FIPS error with Cosmos Postgresql DB while running stored procedure

Vipal Mehta 40

Hi Team,

I am facing a strange error while executing stored proc which executes a query on another DB.

The same proc works fine in our staging DB hosted on Cosmos Postgresql.

We have similar configuration and settings for both the DB and we checked all the parameters, however there are no mismatch.

Please see below error we are facing with our CosmosDB Production instance

ERROR: could not establish connection DETAIL: connection to server at "db.amazonaws.com" , port 5432 failed: could not encrypt password: disabled for FIPS fe_sendauth: error sending password authentication.

Please note that there are no changes made on DB cluster on either side. Interesting thing is it is working on other CosmosDB which we have running

@Oury Ba-MSFT

Mahesh Kurva 1,625 Reputation points Microsoft Vendor

2024-12-20T02:28:18.62+00:00
Hi @Vipal Mehta,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

Apologies for causing the issue.

As I understand it, you're encountering a FIPS-related issue with password encryption on your CosmosDB production instance.

Here are a few steps you can take to troubleshoot and resolve this issue:

Ensure that FIPS mode is not enabled on your production instance if it’s not required. FIPS (Federal Information Processing Standards) mode enforces stricter security protocols, which might be causing the password encryption issue.

Verify the PostgreSQL configuration settings on your production instance. Specifically, check the pg_hba.conf file to ensure that the authentication method is compatible with FIPS mode.

Confirm that the network and security settings, such as security groups and firewall rules, are correctly configured to allow connections between your databases. Ensure that the necessary ports (e.g., 5432) are open and accessible.

Make sure that the database user has the necessary permissions and that the credentials are correctly configured in both the staging and production environments.

If SSL/TLS is enabled, verify that the SSL certificates are correctly configured and that the encryption settings are compatible with FIPS mode.

Check the PostgreSQL logs for any additional error messages or details that might provide more insight into the issue. This can help identify if there are any other underlying problems.

For more information, please refer the documents:

https://zcusa.951200.xyz/en-us/compliance/regulatory/offering-FIPS-140-2

https://www.jan0sch.de/post/postgresql-authentication-migrate-from-md5-to-scram-sha-256/

https://zcusa.951200.xyz/en-us/azure/cosmos-db/postgresql/concepts-security-overview?source=recommendations

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/PostgreSQL_Password_Encryption_configuration.html#:~:text=We%20recommend%20that%20you%20use%20SCRAM%20rather%20than,the%20scram-sha-256%20algorithm%20for%20password%20authentication%20and%20encryption.

https://zcusa.951200.xyz/en-us/azure/postgresql/flexible-server/troubleshoot-password-authentication-failed-for-user?source=recommendations

I hope this information helps. Please do let us know if you have any further queries.
Vipal Mehta 40 Reputation points

2024-12-20T05:16:41.97+00:00

Hi Team,

As mentioned, there have been no changes to the DB instance. The same thing is working fine on Staging DB.

Strangely I am not even able to restore the DB to a previous date before 19th December, which is not expected.

There is definitely something wrong with the host which is running the DB. Something has changed on the Azure side relating to the encryption or connection.

Please note that I am able to connect to the DB, however there is a stored procedure which fetches data from other DB where we are getting this issue.
Mahesh Kurva 1,625 Reputation points Microsoft Vendor

2024-12-20T19:24:41.93+00:00
Hi @Vipal Mehta,

Thanks for getting back to me.

Here are a few steps and considerations that might help you troubleshoot and resolve the problem:

Sometimes, Azure applies updates that can affect your services. Review the Azure Service Health dashboard to see if there have been any recent updates or incidents that might be impacting your CosmosDB instance.

Since the error mentions encryption issues, ensure that the encryption settings on your CosmosDB instance haven’t changed. Verify that the encryption protocols and settings are consistent with those on your staging environment.

If you’re unable to restore your database to a date before December 19th, it might be due to limitations or issues with the point-in-time restore feature. Ensure that you have the necessary permissions and that the backup retention settings are correctly configured.

For more information, please refer the documents:

https://zcusa.951200.xyz/en-us/azure/cosmos-db/restore-account-continuous-backup

https://zcusa.951200.xyz/en-us/azure/cosmos-db/how-to-restore-in-account-continuous-backup?tabs=azure-portal&pivots=api-nosql

If the above steps do not resolve the issue, it might be best to create a support ticket. If you have a support plan, could you please file a support ticket for deeper investigation and do share the SR# with us? In case if you don't have a support plan, please let us know here.

I hope this information helps. Please do let us know if you have any further queries.
Mahesh Kurva 1,625 Reputation points Microsoft Vendor

2024-12-24T16:07:52.9533333+00:00

Hi @Vipal Mehta,

Apologies for causing the issue.

The connection detail db.amazonaws.com suggests that your stored procedure might be connecting to an external AWS database.

There seems some config issue to target db for accepting connection.

If this issue is related to AWS, you may want to consider asking for assistance on the AWS forum.
Vipal Mehta 40 Reputation points

2024-12-24T16:13:24.4+00:00

@Mahesh Kurva

Thanks for the suggestion!

There have been no changes on RDS side. The connection is not going out from Azure Cosmos DB. There are no logs getting generated on AWS DB regarding connection from our DB.

There have been changes from Azure platform, which broke our feature.
Mahesh Kurva 1,625 Reputation points Microsoft Vendor

2024-12-24T20:29:02.72+00:00

Hi @Vipal Mehta,

As you mentioned that you do not have a support plan, we can raise a one-time free support case for you.
Vipal Mehta 40 Reputation points

2024-12-26T05:35:35.0266667+00:00

Hello @Mahesh Kurva
I had raised it with support team on a different chain. They have reverted some changes and disabled FIPS, it has started working now.
Thanks for the assistance and all the help on this.
Mahesh Kurva 1,625 Reputation points Microsoft Vendor

2024-12-26T05:53:51.5033333+00:00

Hi @Vipal Mehta,

Glad to know your issue has been resolved. Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "I'll repost your solution in case you'd like to accept the answer.

Accepted answer

Mahesh Kurva 1,625 Reputation points Microsoft Vendor

2024-12-26T05:53:06.8+00:00

Hi @Vipal Mehta,

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer.

Issue:

Hi Team,

I am facing a strange error while executing stored proc which executes a query on another DB.

The same proc works fine in our staging DB hosted on Cosmos Postgresql.

We have similar configuration and settings for both the DB and we checked all the parameters, however there are no mismatch.

Please see below error we are facing with our CosmosDB Production instance

ERROR: could not establish connection DETAIL: connection to server at "db.amazonaws.com" , port 5432 failed: could not encrypt password: disabled for FIPS fe_sendauth: error sending password authentication.

Please note that there are no changes made on DB cluster on either side. Interesting thing is it is working on other CosmosDB which we have running

Solution:

I had raised it with support team on a different chain. They have reverted some changes and disabled FIPS, it has started working now.

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

Hope this helps. Do let us know if you have any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Please sign in to rate this answer.

1 person found this answer helpful.
Vipal Mehta 40 Reputation points

2024-12-26T06:00:41.79+00:00

Hi @Mahesh Kurva
Thanks for this. I would accept this as a solution for the issue
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

1 additional answer

Sina Salam 14,616 Reputation points

2024-12-22T16:50:11.44+00:00
Hello Vipal Mehta,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you are having FIPS error with Cosmos Postgresql DB while running stored procedure.

If your production Cosmos PostgreSQL instance has continuous backups enabled and that the retention period includes December 19th, there are four areas I will like you to check:

Firstly, you will need to diagnose and confirm Azure-Side changes: Use the Azure Service Health dashboard to identify if any updates were applied to CosmosDB or the hosting infrastructure, particularly related to FIPS enforcement. - https://portal.azure.com/#blade/Microsoft_Azure_Health/AzureHealthBrowseBlade/serviceIssues

Secondly, check if FIPS compliance has been enabled in your production environment unexpectedly:

Run SHOW config_file; in PostgreSQL to locate the configuration file.

Look for parameters like password_encryption and confirm if they are set to SCRAM-SHA-256 or similar.

On the host or application level, ensure that FIPS mode is disabled if it is causing issues. This might involve updating operating system policies or PostgreSQL configurations.

Ensure both staging and production instances have the same authentication methods in pg_hba.conf (e.g., scram-sha-256 or md5), and if using SSL/TLS, confirm that certificates are valid and correctly configured in both environments.

Lastly, this is an immediate action, run Diagnostic Queries using SQL to SHOW assword_encryption; and SHOW ssl; Compare the output between staging and production. Also, use tools like telnet or nc to test connectivity to the target database (db.amazonaws.com) on port 5432 from the CosmosDB production instance. If the issue persists, escalate to Microsoft Azure Support. Hopefully, @Oury Ba-MSFT will respond if she is not on vacation.

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.
Please sign in to rate this answer.
Vipal Mehta 40 Reputation points

2024-12-23T06:54:54.0366667+00:00

Hello @Sina Salam

Thanks for the response and sharing the steps.
We have validated both stage and prod settings and all the settings are matching.

We checked all the steps as mentioned.

Upon checking the Azure Health Dashboard, it seems there were changes on the Azure side and I can see that there were changes applied from Azure platform as attached below -

This has caused this issue I believe and it is breaking our system. Can you help me to raise a support ticket and investigate this on Azure end, since this has been caused by changes done from the Azure team?

Sina Salam 14,616 Reputation points

2024-12-23T08:40:41.6066667+00:00

Hello Vipal Mehta,

Thank you for your feedback and I'm happy you're able to find the answer helpful.

I will contact Microsoft Internal and I'm sure @Oury Ba-MSFT will respond soon.

Meanwhile, you can also raise ticket via your Azure Portal.

Don't forget to close up the thread here by upvoting and accept it as an answer,

Thank you.

Vipal Mehta 40 Reputation points

2024-12-23T08:43:56.12+00:00

Hi @Sina Salam
We don't have a Support plan, could you help me to raise a support ticket for this to investigate from Azure side, as to what are the changes made and how this can be resolved?

My issue is still persistent and I don't have a solution, I could only see that updates were applied from Azure team, but this doesn't solve the error I am facing.

Sina Salam 14,616 Reputation points

2024-12-23T08:53:06.8133333+00:00

Hello Vipal Mehta,

Alright, as I have said I will.

Regarding this:

My issue is still persistent and I don't have a solution, I could only see that updates were applied from Azure team, but this doesn't solve the error I am facing.

Since the answer points you to the right direction, it means a lot. However, Microsoft Internal update will affect most of users not only you.

It will be resolved.

Vipal Mehta 40 Reputation points

2024-12-23T08:58:20.15+00:00

Hello @Sina Salam

I am aware that Microsoft updates affect all who are using the service, however,
Microsoft updates should not break our applications ideally.

This breaking change should have been informed prior before applying to my DBs. There was no prior intimation that Azure is rolling out this changes.

This change has also led my DB not able to restore from the point it went for maintenance, hence I am not able to revert as well and now I have missing restore points.

Since this is affecting our Prod DB and now it has been applied to Stage and Dev as well, our functionality has broken.

Could you share an ETA on when this would get resolved? We have certain commitments and deliverables to be achieved and this has blocked us now.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

FIPS error with Cosmos Postgresql DB while running stored procedure

1 additional answer

Your answer