Managing SaaS Users & Groups via Azure AD SCIM

Using Azure AD Premium, Enterprise App & SCIM 2.0 Provisioning Scope - Only assigned Users & Groups

I'm trying to work through the use case below:

1. SCIM provisioning of users that are assigned to a given AD Group
2. When a user is added it correctly fires off a POST /Users to Create the User
3. When a user is removed it skips the user a reports - "Details : User details: Skip reason = NotEffectivelyEntitled, Active = True, Assigned = False, Passed scope filter: True;" But does not send a PATCH or a DELETE to inform the saas app that the user is no longer valid.

So question what is the correct mechanism for using SCIM provisioning to manage only a subset of users in the AD as active users of the system.

e.g. only 1 department in company uses saas app so users list for assigning tickets etc should only be those, and if a user changes departments and no longer has access to the saas app they shouldn't be seen as a valid user of the saas app directory. The saas licensing will count all registered users so syncing 20,000 users for no reason is not an option.

Seems like SCIM supports this use case with PATCH & DELETE, but Azure AD isn't propagating changes from the users & groups in the enterprise app as expected.

Any suggestions appreciated.

Thanks