What are the challenges you face when working across database platforms? Take the survey
Options

Docker and SSL connections

Forgive me for any perceived frustration, but I honestly think this should be easier.

I've used Flyway for many years, and I am currently in a situation where I have an RDS Aurora server in a locked down VPC, so any migration needed must take place using a CodeBuild project in the VPC. Also, that RDS will only accept TLS connections due to compliance requirements. I have a separate and similar configuration that doesn't need TLS, and I have Flyway running fine using the off-the-shelf Docker container.

However, getting a secure connection to the database for migration using the standard Docker container is not possible. In order to do so, it is necessary to build a NEW container to insert certificates into the Java keystore. I have run across several folks trying to explain how to do this (including the one recommended by Redgate (https://www.joaorosa.io/2019/01/13/using-flyway-and-gitlab-to-deploy-a-mysql-database-to-aws-rds-securely/), and have also followed the description here: (https://documentation.red-gate.com/fd/ssl-support-184127478.html) and nothing seems to be working. What's more frustrating in my case is that I don't really know Java that well, especially managing certs in their native system. I would expect that with a Docker container, I wouldn't have to.

I've narrowed this down to two different conditions - using the default keystore (of which I don't have the password), and a new keystore that gives a fairly common error for which I have yet to find the magic command. These are from my Dockerfile:

Default keystore (I don't know the password)
<div>RUN $JAVA_HOME/bin/keytool -keystore $JAVA_HOME/lib/security/cacerts -storepass popcorn -noprompt -trustcacerts -alias "rds-ca-bundle" -import -file ./rds-ca-bundle.pem<br></div><div></div>
Error:
Warning: use -cacerts option to access cacerts keystore<br>keytool error: java.io.IOException: Keystore was tampered with, or password was incorrect

Custom keystore:
RUN $JAVA_HOME/bin/keytool -keystore $JAVA_HOME/rds_ca_keystore -storepass popcorn -noprompt -trustcacerts -alias "rds-ca-bundle" -import -file ./rds-ca-bundle.pem<br>ENV JAVA_ARGS=-Djavax.net.ssl.trustStore=$JAVA_HOME/rds_ca_keystore -Djavax.net.ssl.trustStorePassword=popcorn<br>
Error:
Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

Thanks for any help you can provide. I would be nice to include via a volume the .pem that AWS provides. I see that there was a PR created last year that attempts to fix this. but it appears to be dormant. I would also state that I can see my case being unusual. Aren't cloud based DB's and TLS connections on the rise?

Thanks.



Tagged:

Answers

  • Options
    Not at all cajund, I can empathise, modifying pre-existing Docker isn't as straight forward as one would hope!

    Our doc on SSL support is admittedly must more useful in a native environment rather than a prepackaged one. I'm not familiar with the joaorosa article, but having read it showing a modified Docker file, I'd conclude that at minimum that required recomposing the Docker image, which isn't the sort of minor edit I anticipate most people would be looking for.

    I feel like it should be possible to introduce a Docker entrypoint to sideload the required cert, but I don't know for sure, I'll investigate and let you know what I find.
    Kind regards
    Peter Laws | Redgate Software
    Have you visited our Help Center?
  • Options
    cajundcajund Posts: 14 New member
    Thanks for your help. This is from last year:


    I think it's headed in the right direction, but doesn't work as is. My attempt was even simpler this this one. No dice.

  • Options
    Ah thank you, I can see from that issues history that some of the devs who would likely perform this work have also acted on it so I'll discuss this with them and then come back to you.
    Kind regards
    Peter Laws | Redgate Software
    Have you visited our Help Center?
  • Options
    cajundcajund Posts: 14 New member
    edited March 20, 2024 7:37PM
    Hi Folks,

    Finally able to come back to this. I have decided to take the "entrypoint.sh" approach as opposed to rebuilding the docker container as some instructions suggest.

    Here are the details:

    entrypoint.sh
    #!/bin/bash
    set -euo pipefail

    echo "Adding RDS Cert"

    export JAVA_ARGS='-Djavax.net.ssl.trustStore="/flyway/keystore" -Djavax.net.ssl.trustStorePassword="popcorn"'
    keytool -keystore /flyway/keystore -alias "AWS RDS Aurora" -noprompt -trustcacerts -storepass "popcorn" -importcert -file rds-combined-ca-bundle.pem
    keytool -list -keystore /flyway/keystore -storepass "popcorn"



    And my docker command:
    docker run --rm \
        -e FLYWAY_USER=$DB_USER \
        -e FLYWAY_PASSWORD=$DB_PASS \
        -v $(pwd)/rds-combined-ca-bundle.pem:/flyway/rds-combined-ca-bundle.pem \
        -v $(pwd)/build/entrypoint.sh:/flyway/entrypoint.sh \
        -v $(pwd)/db:/flyway/sql \
        --entrypoint=/flyway/entrypoint.sh \
        redgate/flyway:latest migrate \
        -url=$DB_URL \
        -locations=$LOCATIONS \
        -baselineOnMigrate="true"
    


    Some notes:
    Adding RDS Cert
    Certificate was added to keystore
    Keystore type: PKCS12
    Keystore provider: SUN
    Your keystore contains 1 entry
    aws rds aurora, Mar 19, 2024, trustedCertEntry, 
    Certificate fingerprint (SHA-256): EB:BD:7E:AC:8B:02:17:12:95:35:ED:C5:2F:D6:D9:56:7D:42:4D:7E:B4:32:41:D8:35:26:FD:9C:46:6D:3F:40
    • The error is:
    SQL State  : 08000
    Error Code : -1
    Message    : Could not connect to address=(host=flare-cluster-dev.cluster-************.us-west-2.rds.amazonaws.com)(port=3306)(type=master) : Could not connect to flare-cluster-dev.cluster-************.us-west-2.rds.amazonaws.com:3306 : No X509TrustManager implementation available<br>
    Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to address=(host=flare-cluster-dev.cluster-************.us-west-2.rds.amazonaws.com)(port=3306)(type=master) : Could not connect to flare-cluster-dev.cluster-************.us-west-2.rds.amazonaws.com:3306 : No X509TrustManager implementation available<br>Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to flare-cluster-dev.cluster-************.us-west-2.rds.amazonaws.com:3306 : No X509TrustManager implementation available<br>Caused by: javax.net.ssl.SSLHandshakeException: No X509TrustManager implementation available
    Caused by: java.security.cert.CertificateException: No X509TrustManager implementation available

    I suspect that there is some issue with setting up the JAVA_ARGS. As these were pulled directly from your documentation (with some adjustment, as your docs aren't setting a password), I was hoping that you can tell me what is missing.

    Thanks for your help.
  • Options
    cajundcajund Posts: 14 New member
    edited April 10, 2024 1:39PM
    Anything on this folks? This is marked as answered, it's not...

    Thanks.
  • Options
    Hi cajund,

    Which post shows as answered please?
    They all appear as rejected for me, even the ones that aren't posting a solution and your own posts too.

    Thanks for the remainder, I was on leave when you posted, I'll check with the devs regarding the cert implementation options, apologies for not having an answer off-hand, this isn't a typical procedure.
    Kind regards
    Peter Laws | Redgate Software
    Have you visited our Help Center?
  • Options
    cajundcajund Posts: 14 New member
    On the main page showing the list of the topics, this question shows as answered. I figured you wouldn't delve into it as a result.

    But I appreciate that you have. Thanks for the further research. I hope that it results in updating your documentation.

  • Options
    Oh I see what you mean, after a little bit of tinkering I've managed to get the status corrected, thanks for highlighting.

    Given where this exception is now occurring please can you clarify what kind of Aurora DB are you targetting as it may alter the expected cert parameters, so I'll need to cross-reference the respective JDBC drivers.
    Kind regards
    Peter Laws | Redgate Software
    Have you visited our Help Center?
  • Options
    cajundcajund Posts: 14 New member
    edited April 16, 2024 1:38PM
    Aurora MySQL. I am using the latest version of the Flyway container.

    Thanks.
  • Options
    cajundcajund Posts: 14 New member
    Hi Folks, following up here again. Anything more I can provide?

    Thanks,
  • Options
    I'm afraid we don't have much to offer cajund, the definition of what's needed is governed by Amazon, so we're not in a great position to advise.

    There seems to be a noteworthy number of instances of this issue when I was researching it, to be transparent at this juncture the flyway side of things looks good and otherwise I'd largely be recycling advice from threads like https://stackoverflow.com/questions/39650898/no-x509trustmanager-implementation-available. The db host will be much better equipped than we to access your connectivity approach.

    I'm sorry we can be more help, the areas we're responsible for look to have been correctly configured from what you've shared.
    Kind regards
    Peter Laws | Redgate Software
    Have you visited our Help Center?
  • Options
    cajundcajund Posts: 14 New member
    edited May 3, 2024 1:13PM
    Peter,

    Thanks for this reply.

    My issue is not connecting to RDS securely, it's connecting to RDS using Flyway in the Docker container. This process is opaque, and my familiarity with Java is not very high. So discerning what is incorrect is not very easy for me in this context, which is why I turned to your forums.

    I have posted all of my specifics in my March 11th post. I have used your documentation as best as I can to get this to work. The link to the Docker implementation in those docs is vastly different than your approach (he's building a new container!), and is specific to GitLab and their CI/CD system. If what I have provided looks correct, then perhaps it would be a good idea for one of your programmers to get involved and troubleshoot it. It's clear to me that Flyway is not picking up the Keystore when connecting.

    My overal point is that connecting via TLS to RDS Aurora/MySQL is a common thing. If you want to support your users and further your product in the market, it's imperitive that your users be successful. At some point in the near future when we move to production, we will need to move off the community version. But this becomes imposible if I can't get it to work.

    Thanks again.

  • Options
    You are of course correct cajund, please don't think it's unwillingness!

    Maybe you could help me test something, I was concluding that the 'TrustManager implementation available' exception was the JRE complaining about your keystore implementation not meeting the expected format, an alternative possibility exists. It could be that as you're not explicitly stating to use SSL, it's not even attempting to use the keystore and then the connection is correctly rejected.

    To validate this, please could you postpend your connection string with 
    ?useSsl=true
    and let us know the results?
    Kind regards
    Peter Laws | Redgate Software
    Have you visited our Help Center?
  • Options
    cajundcajund Posts: 14 New member
    Yes, I am using it.
  • Options
    cajundcajund Posts: 14 New member
    Peter,

    I've tried a few other things and this simply doesn't work. The flyway executable isn't seeing the `JAVA_ARGS` variable, even if I send it in via the `docker run` command. Unless your tech folks want to get involved with this, I have to cut my losses and look for another solution that solves this problem.

    Thanks again for your time.

Leave a Comment

BoldItalicStrikethroughOrdered listUnordered list
Emoji
Image
Align leftAlign centerAlign rightToggle HTML viewToggle full pageToggle lights
Drop image/file