AD app for external access throws error on oauth token issue

I am trying to build an AzureAD access app. I have to enable this app for external tenants. I have enabled external access on this app and it is configured to use the Graph API also along with the normal SSO and read-directory rights. I have tested the app in the tenant in which it was created and everything(auth+ graph api access) works. Now here is the flow for the external tenant

- I take the external tenant user to the grant consent URL of my app, the user(who is an admin of the external tenant) grants the access and i get the correct response
- Now i want to get the authorization token for this user, so i take the user through the normal Oauth process via 


        https://login.windows.net/common/oauth2/authorize

followed by

        https://login.windows.net/common/oauth2/token/


At this point Azure throws the following error

    {"error":"invalid_grant","error_description":"AADSTS50000: There was an error issuing a 
    token. AADSTS65005: No permission to access \u0027https://graph.windows.net\u0027 
    resource is configured for \u0027d2037ff7-24e4-4cac-8e5e-16e370b36238\u0027 application, 
    or it is expired or revoked.\r\nTrace ID: 472aa92f-35a2-4ed9-ab07-
    12488cc9e6f5\r\nCorrelation ID: b163dde5-eac5-4c82-99ad-0e1100487cb9\r\nTimestamp: 2013-
    09-23 05:28:41Z","error_codes":[50000,65005],"timestamp":"2013-09-23 
    05:28:41Z","trace_id":"472aa92f-35a2-4ed9-ab07-12488cc9e6f5","correlation_id":"b163dde5- 
    eac5-4c82-99ad-0e1100487cb9"}


Why this error even when the app has been granted access. I tried to lookup the `STS` errors 
but found no explanation. Any ideas?

SAML process continues to work for the external tenant however i.e I can use the app for SAML(SSO) login for this external tenant. The problem only seems to be coming for getting access to the graph API.