Conditional access not prompting users for MFA

Hi,

Hoping someone has seen this and can point me in the right direction.

We have a couple of conditional access policies set up in AAD, one that blocks users that arent on a trusted site and another that allows users access from untrusted locations if MFA is applied. Users are assigned one policy or the other not both. The block policy works fine, but the MFA policy allows the user to connect regardles of location.

The What IF tool shows the users getting the policy correctly based on IP:


Windows10_Allow_Untrusted_MFA

Require multi-factor authentication

And according to the sign in log MFA was required and done, the result says:

• USER
   
  Kathryn Janeway
• USERNAME
   
  kat.janeway@blahblahblah.com
• APPLICATION ID
   
  00000006-0000-0ff1-ce00-000000000000
• APPLICATION
  Microsoft Office 365 Portal
• CLIENT
   
  ;Windows 10;Edge 16.1629;
• LOCATION
   
  Somewhere
• IP ADDRESS
   
  ::Untrusted IP::
• DATE
   
  5/17/2018, 8:44:37 AM
• MFA REQUIRED
   
  Yes
• MFA AUTH METHOD
   
• MFA AUTH DETAIL
   
• MFA RESULT
  MFA requirement satisfied by claim in the token
• SIGN-IN STATUS
   
  Success

I'm obviously missing something but we need the users to be prompted for MFA every time they sign in when not on once of our sites.