Thursday, March 26, 2009

Backup is important, Protecting the Backup is equally important

It is a sad day when hacker(s) deliberately attached WebHostingTalk and this is just another incident recently that relates to database backups (after Carbonite, Ma.gnolia, JournalSpace)

As a DBA, it is hard to not imagine the worst for the databases we manage, and I will consult with my colleagues to ensure that our tape backup is safe from attack (both on-site AND off-site)


Hello fellow WHTers!

It's been pretty hectic around here, but I wanted to make sure as many members as possible know what's going on. At approximately 8:30 pm EST on Saturday, March 21 The malicious attacker deleted all backups from the backup servers within the infrastructure before deleting tables from our db server. We were alerted of the db exploitation and quickly shut down the site to prevent further damage.

We've tried to answer any questions or concerns in the following thread posted at
Be sure to subscribe if you want to stay informed.

Remember, you can follow us on Twitter @WebHostingTalk.

WHT Data - Q&A Information

What do we know about the damage done?
This attack was very deliberate, sophisticated and calculated. The attacker was able to circumvent our security measures and access via an arcane backdoor protected by additional firewall. We are still investigating the situation, but we know the attacker infiltrated and deleted the backups first and then deleted three databases: user/post/thread. We have no record or evidence that private message data was accessed. Absolutely no credit card or PayPal data was exposed.

Do we know the motivation behind the attack?
We don’t know enough at this time, so any insight would be purely speculative in nature. WHT is a platform where positive and negative information is shared and exposed about business and individuals. Under TOS policy, we cannot edit or remove user-generated content at the request of an unsatisfied third party. Therefore, WHT tends to become the target for disgruntled individuals and businesses.

Have we been able to restore more recent back-ups?
The offsite backup, the onsite backup and the operational data were destroyed by the attacker, so we’ve resorted to a physical back-up of last resort. Unfortunately, we are experiencing difficulty restoring from our most recent physical backup. At this point, October is the most recent backup that we were able to restore. We continue to work to extract data from a more recent set of DVDs. What is WHT focused on doing now?

The first priority, which kicked in immediately upon discovering the hack while in process, was locking down the infrastructure to avoid further damage and restoring the site. We also had to block the potential for a repeat attack. Now we are working on investigating how much prior data is restorable, reinstating premium memberships, contacting business partners, and communicating with the community members. We are also doing everything possible to identify the attacker and bring them to justice. Disappointments happen – we are working hard to restore trust among community members and to bring things back to normal.

Is WHT doing anything different due to this attack?
WHT has been targeted before and our infrastructure has withstood previous attacks. However, following this well-planned and targeted attack, we will be altering aspects of our architecture to ensure that this type of attack does not happen again. Needless to say, we have learned from this situation and will address any discrepancies accordingly.

We had three, protected data back-up units with one offsite behind a firewall and a fourth physical data back-up layer. We evaluated our disaster recovery plan as recent as late-2008, and carefully reviewed how to recover from a disaster situation. The attacker appeared to have deliberately targeted our data back-up systems, a scenario that our disaster recovery plan did not fully anticipate. We have implemented changes to our data backup and disaster recovery plans to address this weakness. And we advise others to consider a scenario of deliberate, malicious data destruction in their backup and recovery plans.

What should members do now?
The password encryption technology we use is strong for securing non-financial data. However, we suggest that members change their passwords frequently and do not use the same user name and password for the forum as they may use for more sensitive services like online banking. If a member feels more comfortable changing their password, then we recommend that they do what makes them feel more secure.

A concern is that members may receive more spam because the attacker posted stolen email addresses on file sharing sites. I haven’t personally seen an increase in the amount of spam I usually receive to my email address, but it is a risk that we cannot easily alleviate. As we become aware of specific file sharing sites with these email addresses, we are requesting that the emails be removed promptly. So far, most have been quick to comply.

What if I can’t use my WHT account?
We are temporarily using a version of the database from October 2008. This means that if you joined WHT after October 2008, you’ll need to register again to post now. We may still be able to recover your account, but we don’t know yet. Please register with the same username you used before.

If you joined WHT before October 2008 and get a password error, the system is probably asking for the password you were using in October 2008. If you don’t remember your previous password and have access to the email address for your WHT account in October 2008, please use the password recovery tool.

Get updates on this topic here.

For help accessing your account, please open a helpdesk ticket.

If you’ve subscribed to a Premium or Corporate membership prior to October 2008, someone from iNET has contacted you by now. If you’ve subscribed (or re-subscribed) since October 2008 and haven’t heard from iNET, please contact us on the helpdesk.

Moving forward ...
We take the protection of user-contributed data very seriously, and we strongly regret what happened. iNET has a sophisticated infrastructure with advanced security. Yet even institutions that spend millions of dollars a year on Internet security are exploited. Anyone recall NASA being hacked some years back?

It’s not what you’ve done, it’s what you do. And from this day forward, we continue.

We’ve been overwhelmed by all the offers of help and support we’ve received from our members. What can I say about that beyond my heartfelt thanks? I love this community!

No comments:

Post a Comment