Daniel Green's Homepage

About Me

My name is Daniel Green.  I am a programmer from the western USA.  In 2008, I created camelcamelcamel.com, the Internet's most camel-themed Amazon price tracker, and here in the far-distant future I am very gratefully still running the site.... continue reading

 

DynamoDB Storage Overhead and Billing

Today we learned that DynamoDB has a per-item storage overhead of at least 100 bytes (https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/CapacityUnitCalculations.html).  Depending on your item size, this can significantly increase your storage costs.  And, conveniently, AWS... continue reading

AWS DynamoDB Billing Storage Overhead  

Rails, Sidekiq, and the Database Connection Pool

Update: The creator of Sidekiq, Mike Perham, quickly and correctly pointed out that standard ActiveRecord use inside Sidekiq should include automatic connection pool checkins when jobs are complete. I failed to realize that using ActiveRecord::Base.connection checks out a... continue reading

MySQL Ruby on Rails Sidekiq  

Amazon's URL Shortener Domains

Amazon recently updated their mobile app's "share" feature to use shortened URLs.  Here are the domain names I'm aware of. * a.co * amzn.to * amzn.eu * amzn.asia Probably more to come eventually!... continue reading

Amazon  

RDS Gotcha - Backup Retention

If you have your RDS backup retention period set to a non-zero value, then change it to zero, all of your automatic snapshots get deleted...at least when deploying via CDK. Easy to imagine that causing trouble!... continue reading

AWS RDS  

RDS Gotcha - Replication

When we migrated from our datacenter rack to AWS, most of our stress revolved around our 2TB MySQL database.  With a database this size, one does not simply mysqldump your way into Mordor.  Our final migration plan required us to... continue reading

AWS RDS  

The dangers of restoring RDS instances from snapshots

When restoring an RDS instance from a snapshot, AWS unavoidably lazy loads the entire database on-demand. This leads to a huge performance penalty until the entire database has been accessed at least once. This is fine if your database is... continue reading

AWS RDS MySQL  

Avoiding "Storage-optimization" state when restoring RDS instance from snapshot

This seems somewhat obvious in hindsight, but the always lovely AWS support taught me that the "Storage-optimization" state occurs – during restore, anyway – when the snapshot's IO provisioning doesn't match that of the newly restored instance.  That is, the IO configuration... continue reading

AWS RDS  

AWS RDS MySQL restore stalling

When working with RDS, I would occasionally find my web console-initiated restores-from-s3 stall out indefinitely with no errors of any kind.  Confused, I ran to AWS Support, who told me that I should not specify "root" as the admin user... continue reading

AWS RDS MySQL  

Restoring a MySQL backup to AWS RDS without the automated post-restore snapshot

While getting our CDK-based AWS infrastructure going, I often found myself restoring a MySQL backup to RDS.  This went fairly smoothly, except that I could not avoid the end-of-restore snapshot that the AWS web console requires.  This adds several hours... continue reading

AWS MySQL RDS  

Speeding up and compressing MySQL backups using xtrabackup

I love Percona and its MySQL tools.  They have really enabled me to pretend I'm a DBA for much longer than I probably should have. One of the main Percona tools we use is called "xtrabackup", an online / hot backup... continue reading

MySQL Percona xtrabackup