Back in November, I noticed that I was getting a lot of MySQL crashes, with MySQL’s logs saying that it had a log sequence error. It was actually MariaDB, the community-managed version of MySQL (as opposed to Oracle’s “MySQL proper“), but for all intents and purposes, it was MySQL.
I tried everything I could think of: delete the logs, restore from binary backup, restore from text backup, everything. After switching away from MariaDB and to the “actual” MySQL, it seemed to be more stable at last…
But last week my wife reported that her site was having some of the same problems. I took a look at the logs again, and .. the log sequence errors were back. What’s more, while watching the logs (via tail -f mysql.log I could see new failures happen, through routine usage: MySQL crashed, then restarted. Then crashed, and restarted.
Back to the drawing board: first, I repeated some of the same steps I’d done before, making backups just in case. I cleared out the database, and recreated it from scratch; same problems.
By this time I was really frustrated, but then I noticed someone mentioning… memory.
I immediately cranked up top (actually, htop), and noticed mysqld was using a ton of RAM, and that my server was swapping quite a bit.
I’d not tuned MySQL (as the RamNode wiki actually suggests you do!) but my first thought was that I wanted this server to actually be used for more things – so I bumped up my VPS to give it three gigabytes of RAM. It’s a new service plan, but the nice thing about it was that as soon as I ordered it, the instance was allocated more RAM – I literally watched it happen via htop.
And all my MySQL crashes went away. It hasn’t had a crash since then.
My other server – oddly enough, the one that runs this site – is still on a more constrained instance. I may move it at some point, because I don’t know that I need or want two servers; they’re just cheap enough that it wasn’t a big deal.