“Table marked as crashed”

I encountered a pretty scary error on the WordPress Multisite network I manage yesterday, which turned out to be something relatively minor. I thought I’d blog it in case someone else found themselves in a similar situation and began frantically Googling… which is what I was doing last night as my wife (thankfully) handled getting the kids to bed.

We think we’ve been having some brute force attacks on our server over the past few months, and while our server mostly weathers them, we had one yesterday morning that required a manual restart of the virtual machine where our system is housed. We were back online in no time and all looked good. I then had a pretty busy day, and wasn’t spending much time on the system, but got no notices that there were problems (usually I’ll hear from users if something’s amiss).

Towards the end of the day, on either side of my commute, I saw the following symptoms emerge:

  • I noticed that it was taking a long time to access my dashboard even though the site load was normal. I was though still able to eventually get in.
  • On my way out of the office, I noticed that the BuddyPress admin bar was visible even though I had set it to be hidden for logged out users. I noted to take a look at this when I got home. When I did, I found that I couldn’t change that or any other BP setting… when I clicked “save,” the setting reverted to “not hidden.”
  • While going back and forth between the front and back ends, I suddenly lost the super admin capability on my personal account, though I was still able to access the super admin menu with the “admin” user.
  • I then noticed several plugins — including BuddyPress, WP-Super Cache — had been suddenly deactivated, and I couldn’t reactivate them.
  • Then, I noticed that no network themes were available in the Appearance>Themes menu, and that the “enabled” setting on all themes on the wp-admin/ms-themes.php page had been toggled to “no”
  • The home site went white, probably because BuddyPress was no longer enabled and that site runs a BP theme
  • All subdirectory sites seemed fine and were accepting posts.

I’ve never seen this collection of problems before, and began to worry that perhaps the system got hacked…although the pattern of symptoms suggested something was interfering with the database. I first deleted all plugins (except the shar-db plugin), but doing so changed nothing.

Then, I thought, if this is really bad, I need to make sure I have a current MYSQL dump. When running the dump, I got this error:

mysqldump: Got error: 145: Table ‘./wpmu_global/wp_sitemeta’ is marked as crashed and should be repaired when using LOCK TABLES

Ah… crashed database table! I repaired that table via PhpMyAdmin, and, voila, problem resolved.

It seems as though the hard reset that the system required in the morning corrupted the database, and led to the gradual emergence of these symptoms.

Live, learn, and make sure you check your database.

2 thoughts on ““Table marked as crashed””

  1. Wow, scary indeed, but very glad you blogged about this one, Luke! (and glad the fix was not so terrible).

    I’m sure you’ve saved many of us a couple of hours (at least) of major trauma some day in the future.

Comments are closed.