What is the meaning of the following error - Principal does not have access to API/Operation

1. I registered an Azure Application (my-cog-app") and created a service principal. Within my cognitive service resource (my-cog-svc), I granted access role as "Cognitive Services user" to the registered azure app.
2. Within the "my-cog-app" registered app, I added Microsoft Cognitive services "user_impersonation" permission.
3. I got the bearer access token by authorizing request using my my-cog-app client id, secret and scope as https://www.cognitiveservices.azure.com/.default
4. Using the access token, I called the text analytics service within cognitive service and I get the following response
   {
   "error": {
   "code": "PermissionDenied",
   "message": "Principal does not have access to API/Operation."
   }
   }
5. I noticed when the azure account I signed in is added to the cognitive services API permissions as "Cognitive Services user", I get the cognitive service to return a valid response. However when I remove the signed-in user from the API permissions, I get the Principal does not have access to API/Operation.

I am confused because I am thinking the the registered app "my-cog-app" that is also added as a "cognitive services user" in the API permissions of the cognitive service is supposed to allow access to the service on behalf of signed-in users. Am I wrong to assume that?