• AWS RDS MySQL restore stalling
When restoring an RDS instance from a snapshot, AWS unavoidably lazy loads the entire database on-demand. This leads to a huge performance penalty until the entire database has been accessed at least once.
This is fine if your database is small and you can run mysqldump against it, or SELECT * FROM <all tables>. In our unfortunate and admittedly sub-optimal case, we have a nearly 2TB database. We cannot simply dump or select all of it. And production usage may not touch it all very quickly, leading to slow performance for every first touch query.
How can this be avoided? If you are migrating to AWS, you can "Restore from S3" instead. This lets you load an xtrabackup-based backup into RDS, which does no lazy loading.
Unfortunately, there is no way to avoid lazy loading if you are creating a replica, or restoring, from a snapshot. It seems to be a core part of how RDS snapshots work, and cannot be disabled.