Outage Response Statement

Members of MxM,

Due to a catastrophic system failure, our entire server was knocked offline on Monday, May 24th 2021 at 4:00PM Central. Our own internal team, along with a friend of MxM with defense/cybersecurity background, the interior console's vendor, and two DBAs (Database Administrators) we commissioned helped us successfully recover MxM Network as of Wednesday, May 26th 2021 at 6:00PM Central.

The remainder of that time was spent stabilizing the server, and making sure you had a complete summary of what happened.

We wouldn’t have come back without all of these people. It was a razor-close shave with our future, and we learned from it.

What happened exactly?

A routine kernel update went horribly wrong. Both the interior console (controls our entire Dedicated Server) and the database management system (MySQL-based, controls all of MxM Network) corrupted so badly it took the server offline and we couldn’t restart it.

Impact of the Outage

To put it simply, our database went “back in time” to Wednesday, May 19th 2021. This means nothing that occurred on MxM Network between May 19th and last Monday (5/24/21) at 4PM Central made it through site recovery and therefore won’t show on the site.

This includes:

• Member Registrations and Account Removals

• Any posts (photos, video, text, audio, etc.) made during that time range

• Any messages exchanged during that time range

• Any groups or pages created

• Likes and comments made on posts

• Any forum threads opened or replied to

• Albums created

• Profile changed

What about backups?

Funny story, actually. We not only had daily backups, but we also had a comprehensive snapshot backup that captured our entire server weekly. Basically, a primary and secondary. Both failed to recover the system (to our total shock) because of the extent of the database failure and where it happened.

That being said, our backup protocols saved MxM Network exactly as it was intended to in a situation like this. And needless to say, we’ve massively upgraded our outage response based on what happened.

What will MxM do to keep this from happening again?

Beat you to that question several times over. We have:

• Moved our backup process outside of the server to a far more capable vendor

• Cloned our entire console in case a database corruption / system failure ever happens again

• Started the process of moving our database outside of the server for greater redundancy (safety in case something goes wrong again)

What do I need to do?

Forgive us. 😛 No one could have seen that coming. And if you notice anything out of the ordinary, contact us.

Let’s keep making memories and sharing stuff together.

~ Logan