MySQL & Aurora 5.7 to 8.0 upgrade

I run into all kinds of challenges with consulting services. At this occassion I was involved into a performance issue with a rapidly growing Church app called DonkeyMobile. Since upgrading their Aurora RDS MySQL instances from 5.7 to 8.0 they were hitting a CPU bottleneck on one of the reader nodes.

This is the kind of problem where you need to check some boxes. Did they use the right storage types, is the sizing of the instances not okay, are they using the optimal way of connecting to the instances, are they using RDS Proxy ?

You should always search the internet for other people encountering the same kind of problems. That way I found this article on Medium about “iCreditWorks’ Practical Insights on Upgrading Aurora MySQL 5.7.x to 8.0.x using Amazon RDS Blue/Green Deployment“.

Address MySQL 8.0 Specific Issues — Once our pre-prod database was upgraded, we also observed a pesky MySQL 8.0 specific issue — Aurora monitoring and performance insight tools reported significantly higher database CPU usage when we ran our application performance load test on the upgraded MySQL 8.0 database. This was by far the most complicated issue we faced and it took us a few weeks to get to the bottom of the issue. After much hard work and research we identified that we were hitting this MySQL JDBC driver related bug:

https://bugs.mysql.com/bug.php?id=104247

As noted in this bug report, you should add cacheCallableStmts=true&callableStmtCacheSize=100 parameters to the JDBC connect string of all our application modules and this brought the CPU level down to the same scale as it was in the MySQL 5.7 database! Below is a graph depicting CPU utilization before/after Callable Statement Caching.