Docker and SSL connections

cajund · September 15, 2023 12:45PM

Forgive me for any perceived frustration, but I honestly think this should be easier.

I've used Flyway for many years, and I am currently in a situation where I have an RDS Aurora server in a locked down VPC, so any migration needed must take place using a CodeBuild project in the VPC. Also, that RDS will only accept TLS connections due to compliance requirements. I have a separate and similar configuration that doesn't need TLS, and I have Flyway running fine using the off-the-shelf Docker container.

However, getting a secure connection to the database for migration using the standard Docker container is not possible. In order to do so, it is necessary to build a NEW container to insert certificates into the Java keystore. I have run across several folks trying to explain how to do this (including the one recommended by Redgate (https://www.joaorosa.io/2019/01/13/using-flyway-and-gitlab-to-deploy-a-mysql-database-to-aws-rds-securely/), and have also followed the description here: (https://documentation.red-gate.com/fd/ssl-support-184127478.html) and nothing seems to be working. What's more frustrating in my case is that I don't really know Java that well, especially managing certs in their native system. I would expect that with a Docker container, I wouldn't have to.

I've narrowed this down to two different conditions - using the default keystore (of which I don't have the password), and a new keystore that gives a fairly common error for which I have yet to find the magic command. These are from my Dockerfile:

Default keystore (I don't know the password)

<div>RUN $JAVA_HOME/bin/keytool -keystore $JAVA_HOME/lib/security/cacerts -storepass popcorn -noprompt -trustcacerts -alias "rds-ca-bundle" -import -file ./rds-ca-bundle.pem<br></div><div></div>

Error:

Warning: use -cacerts option to access cacerts keystore<br>keytool error: java.io.IOException: Keystore was tampered with, or password was incorrect

Custom keystore:

RUN $JAVA_HOME/bin/keytool -keystore $JAVA_HOME/rds_ca_keystore -storepass popcorn -noprompt -trustcacerts -alias "rds-ca-bundle" -import -file ./rds-ca-bundle.pem<br>ENV JAVA_ARGS=-Djavax.net.ssl.trustStore=$JAVA_HOME/rds_ca_keystore -Djavax.net.ssl.trustStorePassword=popcorn<br>

Error:

Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

Thanks for any help you can provide. I would be nice to include via a volume the .pem that AWS provides. I see that there was a PR created last year that attempts to fix this. but it appears to be dormant. I would also state that I can see my case being unusual. Aren't cloud based DB's and TLS connections on the rise?

Thanks.

Peter_Laws · September 18, 2023 7:46AM

Not at all cajund, I can empathise, modifying pre-existing Docker isn't as straight forward as one would hope!

Our doc on SSL support is admittedly must more useful in a native environment rather than a prepackaged one. I'm not familiar with the joaorosa article, but having read it showing a modified Docker file, I'd conclude that at minimum that required recomposing the Docker image, which isn't the sort of minor edit I anticipate most people would be looking for.

I feel like it should be possible to introduce a Docker entrypoint to sideload the required cert, but I don't know for sure, I'll investigate and let you know what I find.

cajund · September 18, 2023 10:30PM

Thanks for your help. This is from last year:

https://github.com/flyway/flyway-docker/pull/67

I think it's headed in the right direction, but doesn't work as is. My attempt was even simpler this this one. No dice.

Peter_Laws · September 19, 2023 9:26AM

Ah thank you, I can see from that issues history that some of the devs who would likely perform this work have also acted on it so I'll discuss this with them and then come back to you.

cajund · March 20, 2024 7:31PM

Hi Folks,

Finally able to come back to this. I have decided to take the "entrypoint.sh" approach as opposed to rebuilding the docker container as some instructions suggest.

Here are the details:

entrypoint.sh

#!/bin/bash
set -euo pipefail

echo "Adding RDS Cert"

export JAVA_ARGS='-Djavax.net.ssl.trustStore="/flyway/keystore" -Djavax.net.ssl.trustStorePassword="popcorn"'
keytool -keystore /flyway/keystore -alias "AWS RDS Aurora" -noprompt -trustcacerts -storepass "popcorn" -importcert -file rds-combined-ca-bundle.pem
keytool -list  -keystore /flyway/keystore -storepass "popcorn"

And my docker command:

docker run --rm \
    -e FLYWAY_USER=$DB_USER \
    -e FLYWAY_PASSWORD=$DB_PASS \
    -v $(pwd)/rds-combined-ca-bundle.pem:/flyway/rds-combined-ca-bundle.pem \
    -v $(pwd)/build/entrypoint.sh:/flyway/entrypoint.sh \
    -v $(pwd)/db:/flyway/sql \
    --entrypoint=/flyway/entrypoint.sh \
    redgate/flyway:latest migrate \
    -url=$DB_URL \
    -locations=$LOCATIONS \
    -baselineOnMigrate="true"

Some notes:

These commands are pulled from your instructions at: https://documentation.red-gate.com/fd/ssl-support-224003085.html
The output from the entry point looks like this:

Adding RDS Cert
Certificate was added to keystore
Keystore type: PKCS12
Keystore provider: SUN
Your keystore contains 1 entry
aws rds aurora, Mar 19, 2024, trustedCertEntry, 
Certificate fingerprint (SHA-256): EB:BD:7E:AC:8B:02:17:12:95:35:ED:C5:2F:D6:D9:56:7D:42:4D:7E:B4:32:41:D8:35:26:FD:9C:46:6D:3F:40

The error is:

SQL State  : 08000
Error Code : -1
Message    : Could not connect to address=(host=flare-cluster-dev.cluster-************.us-west-2.rds.amazonaws.com)(port=3306)(type=master) : Could not connect to flare-cluster-dev.cluster-************.us-west-2.rds.amazonaws.com:3306 : No X509TrustManager implementation available<br>
Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to address=(host=flare-cluster-dev.cluster-************.us-west-2.rds.amazonaws.com)(port=3306)(type=master) : Could not connect to flare-cluster-dev.cluster-************.us-west-2.rds.amazonaws.com:3306 : No X509TrustManager implementation available<br>Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to flare-cluster-dev.cluster-************.us-west-2.rds.amazonaws.com:3306 : No X509TrustManager implementation available<br>Caused by: javax.net.ssl.SSLHandshakeException: No X509TrustManager implementation available
Caused by: java.security.cert.CertificateException: No X509TrustManager implementation available

I suspect that there is some issue with setting up the JAVA_ARGS. As these were pulled directly from your documentation (with some adjustment, as your docs aren't setting a password), I was hoping that you can tell me what is missing.

Thanks for your help.

cajund · April 10, 2024 1:36PM

Anything on this folks? This is marked as answered, it's not...

Thanks.

Peter_Laws · April 15, 2024 4:09PM

Hi cajund,

Which post shows as answered please?
They all appear as rejected for me, even the ones that aren't posting a solution and your own posts too.

Thanks for the remainder, I was on leave when you posted, I'll check with the devs regarding the cert implementation options, apologies for not having an answer off-hand, this isn't a typical procedure.

cajund · April 15, 2024 4:22PM

On the main page showing the list of the topics, this question shows as answered. I figured you wouldn't delve into it as a result.

But I appreciate that you have. Thanks for the further research. I hope that it results in updating your documentation.

Peter_Laws · April 16, 2024 11:02AM

Oh I see what you mean, after a little bit of tinkering I've managed to get the status corrected, thanks for highlighting.

Given where this exception is now occurring please can you clarify what kind of Aurora DB are you targetting as it may alter the expected cert parameters, so I'll need to cross-reference the respective JDBC drivers.

cajund · April 16, 2024 1:27PM

Aurora MySQL. I am using the latest version of the Flyway container.

Thanks.

cajund · April 30, 2024 9:24PM

Hi Folks, following up here again. Anything more I can provide?

Thanks,

Peter_Laws · May 3, 2024 10:29AM

I'm afraid we don't have much to offer cajund, the definition of what's needed is governed by Amazon, so we're not in a great position to advise.

There seems to be a noteworthy number of instances of this issue when I was researching it, to be transparent at this juncture the flyway side of things looks good and otherwise I'd largely be recycling advice from threads like https://stackoverflow.com/questions/39650898/no-x509trustmanager-implementation-available. The db host will be much better equipped than we to access your connectivity approach.

I'm sorry we can be more help, the areas we're responsible for look to have been correctly configured from what you've shared.

cajund · May 3, 2024 1:12PM

Peter,

Thanks for this reply.

My issue is not connecting to RDS securely, it's connecting to RDS using Flyway in the Docker container. This process is opaque, and my familiarity with Java is not very high. So discerning what is incorrect is not very easy for me in this context, which is why I turned to your forums.

I have posted all of my specifics in my March 11th post. I have used your documentation as best as I can to get this to work. The link to the Docker implementation in those docs is vastly different than your approach (he's building a new container!), and is specific to GitLab and their CI/CD system. If what I have provided looks correct, then perhaps it would be a good idea for one of your programmers to get involved and troubleshoot it. It's clear to me that Flyway is not picking up the Keystore when connecting.

My overal point is that connecting via TLS to RDS Aurora/MySQL is a common thing. If you want to support your users and further your product in the market, it's imperitive that your users be successful. At some point in the near future when we move to production, we will need to move off the community version. But this becomes imposible if I can't get it to work.

Thanks again.

Docker and SSL connections

Answers

Leave a Comment

Product Learning

Community Forums

Events & Friends

Simple Talk