ACS + ADFS won't work when outside of domain

Hi, I'm having some trouble setting up the ADFS 2 + ACS to authenticate, it works fine when I'm logged in the domain network but when I am outside of the network I get this error shown bellow. 

We have UAG sat in front of ADFS for external Access, could this be issue?

ACS20001: An error occurred while processing a WS-Federation sign-in response.

ACS50008: SAML token is invalid.

Trade Id: cfeff2e7-4b1f-4ec9-948b-e9fc9afa1773

I'm not sure why it's not working from outside of the domain network since the message is too generic, this is the message I got from ADFS debug

Ignoring Invalid entry 'signoutCleanup;https%3a%2f%2fxxx.accesscontrol.windows.net%2f&FSS0+-+XXX+Azure&https%3a%2f%2fxxx.accesscontrol.windows.net%2fv2%2fwsfederation&https%3a%2f%2fxxx.accesscontrol.windows.net%2fv2%2fwsfederation' in signout cookie

Constructing MSISAppliesTo from custom URI 'microsoft:identityserver:https://xxx.accesscontrol.windows.net/' if applicable

Constructing MSISAppliesTo from custom URI 'https://xxx.accesscontrol.windows.net/' if applicable

Initiating search for scope https://xxx.accesscontrol.windows.net/

Successfully retrieved scope details for scope https://xxx.accesscontrol.windows.net/

Began processing policy for target: https://xxx.accesscontrol.windows.net/
Request AppliesTo: https://xxx.accesscontrol.windows.net/
Using ActAs: false
Using OnBehalfOf: true

LDAPAttributeStoreReader: Successfully retrieved 1 attribute(s) for user sAMAccountName=k000000

Finished processing policy

Constructing MSISAppliesTo from custom URI 'https://xxx.accesscontrol.windows.net/' if applicable

Constructing MSISAppliesTo from custom URI 'https://xxx.accesscontrol.windows.net/' if applicable

Thanks,

Lucas