Azure AD Premium, on-premises ADFS and signout process

I hope I have the right forum here.  Please point me in the right direction if I don't.

First, our configuration.  We are configuring SSO for Workday using Azure AD Premium.  We use Azure AD Connect to sync our accounts without passwords.  We have on-premises 2012 R2 ADFS servers.

Now, the problem.  When logging out of Workday, we get an Azure AD logout page, then a logout page from our on-premises ADFS servers, then a final page from Azure.  I _think_ this makes sense given our configuration, but the project manager would like to get rid of this and have a single logout page displayed.  He has discovered that if he modifies Workday to use our ADFS server logout URL instead of the Azure URL, this results in a single logout page.  I'm worried that there are security implications of this or that it may break something else.  We use Office 365 and I don't know what the consequences of that are.

I hope this makes sense.

Thanks.