Docker and SSL connections

cajund

Forgive me for any perceived frustration, but I honestly think this should be easier.

I've used Flyway for many years, and I am currently in a situation where I have an RDS Aurora server in a locked down VPC, so any migration needed must take place using a CodeBuild project in the VPC. Also, that RDS will only accept TLS connections due to compliance requirements. I have a separate and similar configuration that doesn't need TLS, and I have Flyway running fine using the off-the-shelf Docker container.

However, getting a secure connection to the database for migration using the standard Docker container is not possible. In order to do so, it is necessary to build a NEW container to insert certificates into the Java keystore. I have run across several folks trying to explain how to do this (including the one recommended by Redgate (https://www.joaorosa.io/2019/01/13/using-flyway-and-gitlab-to-deploy-a-mysql-database-to-aws-rds-securely/), and have also followed the description here: (https://documentation.red-gate.com/fd/ssl-support-184127478.html) and nothing seems to be working. What's more frustrating in my case is that I don't really know Java that well, especially managing certs in their native system. I would expect that with a Docker container, I wouldn't have to.

I've narrowed this down to two different conditions - using the default keystore (of which I don't have the password), and a new keystore that gives a fairly common error for which I have yet to find the magic command. These are from my Dockerfile:

Default keystore (I don't know the password)

<div>RUN $JAVA_HOME/bin/keytool -keystore $JAVA_HOME/lib/security/cacerts -storepass popcorn -noprompt -trustcacerts -alias "rds-ca-bundle" -import -file ./rds-ca-bundle.pem<br></div><div></div>

Error:

Warning: use -cacerts option to access cacerts keystore<br>keytool error: java.io.IOException: Keystore was tampered with, or password was incorrect

Custom keystore:

RUN $JAVA_HOME/bin/keytool -keystore $JAVA_HOME/rds_ca_keystore -storepass popcorn -noprompt -trustcacerts -alias "rds-ca-bundle" -import -file ./rds-ca-bundle.pem<br>ENV JAVA_ARGS=-Djavax.net.ssl.trustStore=$JAVA_HOME/rds_ca_keystore -Djavax.net.ssl.trustStorePassword=popcorn<br>

Error:

Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

Thanks for any help you can provide. I would be nice to include via a volume the .pem that AWS provides. I see that there was a PR created last year that attempts to fix this. but it appears to be dormant. I would also state that I can see my case being unusual. Aren't cloud based DB's and TLS connections on the rise?

Thanks.

Peter_Laws

Not at all cajund, I can empathise, modifying pre-existing Docker isn't as straight forward as one would hope!

Our doc on SSL support is admittedly must more useful in a native environment rather than a prepackaged one. I'm not familiar with the joaorosa article, but having read it showing a modified Docker file, I'd conclude that at minimum that required recomposing the Docker image, which isn't the sort of minor edit I anticipate most people would be looking for.

I feel like it should be possible to introduce a Docker entrypoint to sideload the required cert, but I don't know for sure, I'll investigate and let you know what I find.

cajund

Thanks for your help. This is from last year:

https://github.com/flyway/flyway-docker/pull/67

I think it's headed in the right direction, but doesn't work as is. My attempt was even simpler this this one. No dice.

Peter_Laws

Ah thank you, I can see from that issues history that some of the devs who would likely perform this work have also acted on it so I'll discuss this with them and then come back to you.

cajund

Hi Folks,

Finally able to come back to this. I have decided to take the "entrypoint.sh" approach as opposed to rebuilding the docker container as some instructions suggest.

Here are the details:

entrypoint.sh

#!/bin/bash
set -euo pipefail

echo "Adding RDS Cert"

export JAVA_ARGS='-Djavax.net.ssl.trustStore="/flyway/keystore" -Djavax.net.ssl.trustStorePassword="popcorn"'
keytool -keystore /flyway/keystore -alias "AWS RDS Aurora" -noprompt -trustcacerts -storepass "popcorn" -importcert -file rds-combined-ca-bundle.pem
keytool -list  -keystore /flyway/keystore -storepass "popcorn"

And my docker command:

docker run --rm \
    -e FLYWAY_USER=$DB_USER \
    -e FLYWAY_PASSWORD=$DB_PASS \
    -v $(pwd)/rds-combined-ca-bundle.pem:/flyway/rds-combined-ca-bundle.pem \
    -v $(pwd)/build/entrypoint.sh:/flyway/entrypoint.sh \
    -v $(pwd)/db:/flyway/sql \
    --entrypoint=/flyway/entrypoint.sh \
    redgate/flyway:latest migrate \
    -url=$DB_URL \
    -locations=$LOCATIONS \
    -baselineOnMigrate="true"

Some notes:

• These commands are pulled from your instructions at: https://documentation.red-gate.com/fd/ssl-support-224003085.html
• The output from the entry point looks like this:

Adding RDS Cert
Certificate was added to keystore
Keystore type: PKCS12
Keystore provider: SUN
Your keystore contains 1 entry
aws rds aurora, Mar 19, 2024, trustedCertEntry, 
Certificate fingerprint (SHA-256): EB:BD:7E:AC:8B:02:17:12:95:35:ED:C5:2F:D6:D9:56:7D:42:4D:7E:B4:32:41:D8:35:26:FD:9C:46:6D:3F:40

• The error is:

SQL State  : 08000
Error Code : -1
Message    : Could not connect to address=(host=flare-cluster-dev.cluster-************.us-west-2.rds.amazonaws.com)(port=3306)(type=master) : Could not connect to flare-cluster-dev.cluster-************.us-west-2.rds.amazonaws.com:3306 : No X509TrustManager implementation available<br>
Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to address=(host=flare-cluster-dev.cluster-************.us-west-2.rds.amazonaws.com)(port=3306)(type=master) : Could not connect to flare-cluster-dev.cluster-************.us-west-2.rds.amazonaws.com:3306 : No X509TrustManager implementation available<br>Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to flare-cluster-dev.cluster-************.us-west-2.rds.amazonaws.com:3306 : No X509TrustManager implementation available<br>Caused by: javax.net.ssl.SSLHandshakeException: No X509TrustManager implementation available
Caused by: java.security.cert.CertificateException: No X509TrustManager implementation available

I suspect that there is some issue with setting up the JAVA_ARGS. As these were pulled directly from your documentation (with some adjustment, as your docs aren't setting a password), I was hoping that you can tell me what is missing.

Thanks for your help.

cajund

Anything on this folks? This is marked as answered, it's not...

Thanks.

Peter_Laws

Hi cajund,

Which post shows as answered please?
They all appear as rejected for me, even the ones that aren't posting a solution and your own posts too.

Thanks for the remainder, I was on leave when you posted, I'll check with the devs regarding the cert implementation options, apologies for not having an answer off-hand, this isn't a typical procedure.

cajund

On the main page showing the list of the topics, this question shows as answered. I figured you wouldn't delve into it as a result.

But I appreciate that you have. Thanks for the further research. I hope that it results in updating your documentation.

Peter_Laws

Oh I see what you mean, after a little bit of tinkering I've managed to get the status corrected, thanks for highlighting.

Given where this exception is now occurring please can you clarify what kind of Aurora DB are you targetting as it may alter the expected cert parameters, so I'll need to cross-reference the respective JDBC drivers.

cajund

Aurora MySQL. I am using the latest version of the Flyway container.

Thanks.

cajund

Hi Folks, following up here again. Anything more I can provide?

Thanks,

Peter_Laws

I'm afraid we don't have much to offer cajund, the definition of what's needed is governed by Amazon, so we're not in a great position to advise.

There seems to be a noteworthy number of instances of this issue when I was researching it, to be transparent at this juncture the flyway side of things looks good and otherwise I'd largely be recycling advice from threads like https://stackoverflow.com/questions/39650898/no-x509trustmanager-implementation-available. The db host will be much better equipped than we to access your connectivity approach.

I'm sorry we can be more help, the areas we're responsible for look to have been correctly configured from what you've shared.

cajund

Peter,

Thanks for this reply.

My issue is not connecting to RDS securely, it's connecting to RDS using Flyway in the Docker container. This process is opaque, and my familiarity with Java is not very high. So discerning what is incorrect is not very easy for me in this context, which is why I turned to your forums.

I have posted all of my specifics in my March 11th post. I have used your documentation as best as I can to get this to work. The link to the Docker implementation in those docs is vastly different than your approach (he's building a new container!), and is specific to GitLab and their CI/CD system. If what I have provided looks correct, then perhaps it would be a good idea for one of your programmers to get involved and troubleshoot it. It's clear to me that Flyway is not picking up the Keystore when connecting.

My overal point is that connecting via TLS to RDS Aurora/MySQL is a common thing. If you want to support your users and further your product in the market, it's imperitive that your users be successful. At some point in the near future when we move to production, we will need to move off the community version. But this becomes imposible if I can't get it to work.

Thanks again.

Peter_Laws

You are of course correct cajund, please don't think it's unwillingness!

Maybe you could help me test something, I was concluding that the 'TrustManager implementation available' exception was the JRE complaining about your keystore implementation not meeting the expected format, an alternative possibility exists. It could be that as you're not explicitly stating to use SSL, it's not even attempting to use the keystore and then the connection is correctly rejected.

To validate this, please could you postpend your connection string with 

?useSsl=true

and let us know the results?

cajund

Yes, I am using it.

cajund

Peter,

I've tried a few other things and this simply doesn't work. The flyway executable isn't seeing the `JAVA_ARGS` variable, even if I send it in via the `docker run` command. Unless your tech folks want to get involved with this, I have to cut my losses and look for another solution that solves this problem.

Thanks again for your time.