Docker and SSL connections

cajund

Forgive me for any perceived frustration, but I honestly think this should be easier.

I've used Flyway for many years, and I am currently in a situation where I have an RDS Aurora server in a locked down VPC, so any migration needed must take place using a CodeBuild project in the VPC. Also, that RDS will only accept TLS connections due to compliance requirements. I have a separate and similar configuration that doesn't need TLS, and I have Flyway running fine using the off-the-shelf Docker container.

However, getting a secure connection to the database for migration using the standard Docker container is not possible. In order to do so, it is necessary to build a NEW container to insert certificates into the Java keystore. I have run across several folks trying to explain how to do this (including the one recommended by Redgate (https://www.joaorosa.io/2019/01/13/using-flyway-and-gitlab-to-deploy-a-mysql-database-to-aws-rds-securely/), and have also followed the description here: (https://documentation.red-gate.com/fd/ssl-support-184127478.html) and nothing seems to be working. What's more frustrating in my case is that I don't really know Java that well, especially managing certs in their native system. I would expect that with a Docker container, I wouldn't have to.

I've narrowed this down to two different conditions - using the default keystore (of which I don't have the password), and a new keystore that gives a fairly common error for which I have yet to find the magic command. These are from my Dockerfile:

Default keystore (I don't know the password)

<div>RUN $JAVA_HOME/bin/keytool -keystore $JAVA_HOME/lib/security/cacerts -storepass popcorn -noprompt -trustcacerts -alias "rds-ca-bundle" -import -file ./rds-ca-bundle.pem<br></div><div></div>

Error:

Warning: use -cacerts option to access cacerts keystore<br>keytool error: java.io.IOException: Keystore was tampered with, or password was incorrect

Custom keystore:

RUN $JAVA_HOME/bin/keytool -keystore $JAVA_HOME/rds_ca_keystore -storepass popcorn -noprompt -trustcacerts -alias "rds-ca-bundle" -import -file ./rds-ca-bundle.pem<br>ENV JAVA_ARGS=-Djavax.net.ssl.trustStore=$JAVA_HOME/rds_ca_keystore -Djavax.net.ssl.trustStorePassword=popcorn<br>

Error:

Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

Thanks for any help you can provide. I would be nice to include via a volume the .pem that AWS provides. I see that there was a PR created last year that attempts to fix this. but it appears to be dormant. I would also state that I can see my case being unusual. Aren't cloud based DB's and TLS connections on the rise?

Thanks.

Peter_Laws

Not at all cajund, I can empathise, modifying pre-existing Docker isn't as straight forward as one would hope!

Our doc on SSL support is admittedly must more useful in a native environment rather than a prepackaged one. I'm not familiar with the joaorosa article, but having read it showing a modified Docker file, I'd conclude that at minimum that required recomposing the Docker image, which isn't the sort of minor edit I anticipate most people would be looking for.

I feel like it should be possible to introduce a Docker entrypoint to sideload the required cert, but I don't know for sure, I'll investigate and let you know what I find.

cajund

Thanks for your help. This is from last year:

https://github.com/flyway/flyway-docker/pull/67

I think it's headed in the right direction, but doesn't work as is. My attempt was even simpler this this one. No dice.

Peter_Laws

Ah thank you, I can see from that issues history that some of the devs who would likely perform this work have also acted on it so I'll discuss this with them and then come back to you.

cajund

Hi Folks,

Finally able to come back to this. I have decided to take the "entrypoint.sh" approach as opposed to rebuilding the docker container as some instructions suggest.

Here are the details:

entrypoint.sh

#!/bin/bash
set -euo pipefail

echo "Adding RDS Cert"

export JAVA_ARGS='-Djavax.net.ssl.trustStore="/flyway/keystore" -Djavax.net.ssl.trustStorePassword="popcorn"'
keytool -keystore /flyway/keystore -alias "AWS RDS Aurora" -noprompt -trustcacerts -storepass "popcorn" -importcert -file rds-combined-ca-bundle.pem
keytool -list  -keystore /flyway/keystore -storepass "popcorn"

And my docker command:

docker run --rm \
    -e FLYWAY_USER=$DB_USER \
    -e FLYWAY_PASSWORD=$DB_PASS \
    -v $(pwd)/rds-combined-ca-bundle.pem:/flyway/rds-combined-ca-bundle.pem \
    -v $(pwd)/build/entrypoint.sh:/flyway/entrypoint.sh \
    -v $(pwd)/db:/flyway/sql \
    --entrypoint=/flyway/entrypoint.sh \
    redgate/flyway:latest migrate \
    -url=$DB_URL \
    -locations=$LOCATIONS \
    -baselineOnMigrate="true"

Some notes:

• These commands are pulled from your instructions at: https://documentation.red-gate.com/fd/ssl-support-224003085.html
• The output from the entry point looks like this:

Adding RDS Cert
Certificate was added to keystore
Keystore type: PKCS12
Keystore provider: SUN
Your keystore contains 1 entry
aws rds aurora, Mar 19, 2024, trustedCertEntry, 
Certificate fingerprint (SHA-256): EB:BD:7E:AC:8B:02:17:12:95:35:ED:C5:2F:D6:D9:56:7D:42:4D:7E:B4:32:41:D8:35:26:FD:9C:46:6D:3F:40

• The error is:

SQL State  : 08000
Error Code : -1
Message    : Could not connect to address=(host=flare-cluster-dev.cluster-************.us-west-2.rds.amazonaws.com)(port=3306)(type=master) : Could not connect to flare-cluster-dev.cluster-************.us-west-2.rds.amazonaws.com:3306 : No X509TrustManager implementation available<br>
Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to address=(host=flare-cluster-dev.cluster-************.us-west-2.rds.amazonaws.com)(port=3306)(type=master) : Could not connect to flare-cluster-dev.cluster-************.us-west-2.rds.amazonaws.com:3306 : No X509TrustManager implementation available<br>Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to flare-cluster-dev.cluster-************.us-west-2.rds.amazonaws.com:3306 : No X509TrustManager implementation available<br>Caused by: javax.net.ssl.SSLHandshakeException: No X509TrustManager implementation available
Caused by: java.security.cert.CertificateException: No X509TrustManager implementation available

I suspect that there is some issue with setting up the JAVA_ARGS. As these were pulled directly from your documentation (with some adjustment, as your docs aren't setting a password), I was hoping that you can tell me what is missing.

Thanks for your help.

cajund

Anything on this folks? This is marked as answered, it's not...

Thanks.

Peter_Laws

Hi cajund,

Which post shows as answered please?
They all appear as rejected for me, even the ones that aren't posting a solution and your own posts too.

Thanks for the remainder, I was on leave when you posted, I'll check with the devs regarding the cert implementation options, apologies for not having an answer off-hand, this isn't a typical procedure.

cajund

On the main page showing the list of the topics, this question shows as answered. I figured you wouldn't delve into it as a result.

But I appreciate that you have. Thanks for the further research. I hope that it results in updating your documentation.

Peter_Laws

Oh I see what you mean, after a little bit of tinkering I've managed to get the status corrected, thanks for highlighting.

Given where this exception is now occurring please can you clarify what kind of Aurora DB are you targetting as it may alter the expected cert parameters, so I'll need to cross-reference the respective JDBC drivers.

cajund

Aurora MySQL. I am using the latest version of the Flyway container.

Thanks.