WordPress Plugin update drops w_options database table – site down

It appears that recently WordPress via GoDaddy is suffering of a serious bug recently, which to my understanding is likely to caused by plugins auto update. So aside of screaming “help!” let’s see what can we do.

What’s the symptom?

The entire site goes down with message: “could not connect to database”, this site is currently experiencing issues, please try again visiting at later time Error 0.

Looking close at the php-errors.log when this happened I could find that a few critical database tables become missing, dropped by an unknown ghost:

WordPress database error Table 'wp w_options' doesn't exist for query SELECT option_value FROM wp_options WHERE ...

WordPress database error Table 'wp_postmeta' doesn't exist for query SELECT _posts ...

You can use below code to count how many tables are in the database via phpMyAdmin:

SELECT COUNT(*) AS table_count FROM information_schema.tables WHERE table_schema = 'your_wordpress_database_name';

Godaddy Admin Scripts, coincidence?

When the issue happened the first time it happened after I got from WordFence an alert as per below

A user with username "admin" who has administrator access signed in to your WordPress site.

User IP: 54.191.137.17

User hostname:
ec2-54-191-137-17.us-west-2.compute.amazonaws.com

User location: Boardman, Oregon, United States

This seems to be a WPAdmin script executed by the whitelisted Godaddy Pro IPs that is used for Cloner.php and other Managed WP features, after opening a case to GoDaddy Support they told me this is fine and should not be the cause, they pointed fingers at the WordFence Security Plugin even when the site is so corrupted that not even the file manager can’t be accessed.

What’s the solution?

So far I have not been able to find for certain the root cause and the solution as I am still unable to reproduce the issue on command.

What I can recommend so far for who may be encountering a similar issue is:

1. Backup your website and database frequently
2. Disable plugins auto update on production
3. Use the staging site to troubleshoot and experiment
4. Once a week take a backup of your database and then try to update the plugins, keep an eye on the number of tables in your Pgsql or Mysql database before and after upgrading the plugins.
5. Execute the plugin update one at time
6. if something goes wrong, restore the database and check the table count staid the same, all critical tables as wp_options and wp_posts are back in place

If you experience similar issues don’t hesitate to contact me so we may try to troubleshoot it together or if you have found a solution already please let me know!