Hi SK,
I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to "Accept " the answer.
Issue:
Cluster in state failed with QuotaExceeded after upgrade.
Solution:
By Running this command az aks upgrade --resource-group <resource_group_name> --name <cluster_name> --yes --debug
without specifying a Kubernetes version successfully upgraded our AKS cluster to the latest version, resolving the issue and changing the state from "failed" to "succeeded.
If you have any other questions or are still running into more issues, please let me know. Thank you again for your time and patience throughout this issue.
Please remember to "Accept Answer" if any answer/reply helped, so that others in the community facing similar issues can easily find the solution.
Thank you.