Daniel Green's Homepage

RDS Gotcha - Replication

When we migrated from our datacenter rack to AWS, most of our stress revolved around our 2TB MySQL database.  With a database this size, one does not simply mysqldump your way into Mordor.  Our final migration plan required us to setup a MySQL replica in RDS which mirrored the primary in our rack.  This minimized our day-of downtime, and gave us a hot-n-ready database (something restoring from a RDS snapshot does not, at least immediately.)

What I did not realize was that CALL mysql.rds_stop_replication; stops replication but does not disable it entirely; the replication settings remain, and will be reactivated during reboot.  We didn't need to reboot our RDS instance since our iitial migration, so the first time we deployed changes that required a reboot, the deploy hung due to replication being unable to connect to our datacenter.

The solution here was to execute CALL mysql.rds_set_external_master; which clears the replication settings.  We then rebooted once more to ensure all errors had been cleared.  Phew, deployments continue.