ezBoard Attack Updates
Just posting this for reference.
__________________________________
Update: 5/31/05 20:30
Dear ezboard Family,
On Tuesday May 31 ezboard's systems were the victim of a very precise and malicious internet attack. This attack resulted in the loss of a significant amount of current and historical board postings and interrupted services across all systems. I deeply apologize for the inconvenience that this has caused and am here to give you the first of several in-depth updates.
At this time most board management services have been restored and our goal is to have post/reply repaired on Wednesday . We are working to restore board data from our back-up systems. This will take some time, and complete data restoration may not be possible, but it is what we are working towards. What we know now is that the attacker deleted current board and back-up data. We are taking the fastest route to restore data, but it may take up to 48 hours to get data back to approximately 9,000 boards. A more manual method requiring upwards of seven to ten days is required to restore data to the other boards that lost data. Regrettably we may not be able to restore data to all boards that lost data.
Second to restoring board data we are gathering information on how the attacker went about his or her business and what evidence was left behind. We are using this information to reduce the opportunity for future attackers as we build our new release. Should you hear or read anything about this attack we would love to know about it at legal@ezboard.com.
You should be aware that there is no evidence that personal information was touched. There was also no opportunity to see or download credit card information as ezboard does not keep credit card numbers on file.
Again, I apologize to you for any inconvenience caused as we value your participation on ezboard today and as we move towards an exciting - and more robust - version of the experience. We are working to restore your systems as fast as possible. Gold Board managers will be receiving information on GB subscription credits in the next few days.
We we keep you updated as the restoration progresses.
Sincerely,
The following is a summary of the known issues resulting from yesterday's attack:
1) Certain boards and forums are missing topics.
2) Various request errors are hindering boards (when opening certain topics, when replying, etc.).
If your board gives a request error on the main page, i.e. you cannot view the board at all, please post here.
3) Some boards have reported losses of funds from their Community Chests.
4) Various boards' user member lists have lost some or all members.
5) Certain boards' security logs have gone blank.
6) The ezboard network is slow. update (5/31, 5:20 pm):: The network's performance has improved a good deal, but there may still be sporadic slowness.
Along with our technical operations department, we are continuing to work on potential solutions to each of these problems. Until we are able to come up with a detailed plan of attack for each issue, please continue to sit tight. We will keep updating this topic as frequently as possible.
What you should do now
At the moment, all problems listed above are being worked on by the technical staff. We would ask that all admins wait for further information and updates on these issues. Once these are forthcoming you may be asked to submit your board url and describe which problems affect you, but at this point in time it is better if you do not post or submit tickets, as the staff are unable to take action yet.
You can continue using your boards. If certain forums at your board are giving request errors (e.g., when replies are submitted), an administrator may want to create NEW, temporary forums. Then, when the "broken" forums are fixed, the topics from the new forums can be moved into the appropriate, old one.
The face of nature reflects all of life's ups and downs.
By now you I am sure that you are quite aware that ezboard's systems were severely attacked early on Tuesday morning. Since that time, we have been focused on getting boards running properly and restoring data. We have restored a significant number of boards and we are continuing to restore boards and system functionality as quickly as possible. I want to bring you up to speed on what we know, what we are doing and what happens next:
Here is what we know:
1/ The attacker erased all historical post data on all boards and a significant amount of back-up data - the reason we have been able to restore some boards at this time is that we have back up data from the May 9th incremental back-up (we perform back-ups every day, but the attacker deleted many of them). As a result, some board data will be restored, with entries between May 9th and the attack missing. We *may* be able to restore this more recent data from data that we can retrieve from RAM on the servers. The probability of this being successful is not known right now.
2/ Personal and credit card information was not compromised - ezboard does not have CC information on file,
3/ User data, ezSupporter and board configuration data was not compromised,
4/ We have restored data to many boards at this point,
5/ We have additional back-up data for approximately 8,000 boards and will be able to restore over half of all Gold Boards,
6/ We have information on the attacker and are building a case,
7/ New security measures have been put into place to reduce the potential for future attacks,
8/ Your Community Chest and board expiry data was in separate systems and will be restored soon,
9/ Trial boards that were moved off of p097 before the attack will likely be OK. Boards that were created on p097 may not be available.
What we are doing:
1/ Repairing systems to allow posting on all boards,
2/ Restoring data where possible,
3/ Actions have been taken to close the method the attacker used to gain access to the systems.
Next Steps:
We want to be sure that your concerns are addressed as directly as possible. In addition to the above information, we ask you to read the Important Announcements and New Server Status forums frequently for updates. Please read these announcements prior to posting your questions in a forum. Because of the scope of the communities were affected by this attack, it will not be possible to give individualized responses to every user, every poster, or every help ticket. Keep in mind that we do read your questions and will certainly combine frequent questions and themes into broadcast messages like this one.
I would like to thank the ezboard operations staff who have been on the job since Tuesday at 5AM and continue to work hard restoring boards and modifying systems to reduce the threat of future attacks. While you may not be aware of their commitment, please believe me that they are working non-stop for you.
In addition to the hard work that these folks are doing, other members of our team are working hard on the next version of your community experience. An experience that will be vastly improved using the knowledge gained from the last five years of ezboard community and the exciting new developments in online interaction happening now.
Thank you for your patience. We will be keeping you updated every until this is resolved. At this point you should expect to see an update at Noon Pacific time each day.
Sincerely,
Rob
Robert Labatt
CEO
ezboard, Inc.
robertlabatt@ezboard.com
The face of nature reflects all of life's ups and downs.
ezBoard Attack Updates
Posted: 6/1/05 19:35
Re: recent ezboard problems (6/1/05 update)
--------------------------------------------------------------------------------
Regarding the bug replying to topics: we have released a patch to all servers to fix this bug. You just have to make one post in each forum of your board, and after that you should be able to reply to topics without the request error.
This does not restore missing topics, and it does not fix the problem of not being able to move topics. It just makes it so that you can reply.
The face of nature reflects all of life's ups and downs.
We are continuing to restore posts and are working on additional means to restore further data. At this time we do not have an ETA for the completion of the restore process. Indications are that the restoration will be completed in the next 24 to 48 hours. We appreciate your patience and recommend that you look for updates in the Help forums. Please help us to get you answers more quickly by reading the sticky topics in the Help forums before posting your question. Many times your questions will have already been answered.
The face of nature reflects all of life's ups and downs.
Good morning and thank you for your patience as we continue to restore your data and return our systems to normal operation.
This will be a long update, as I want to be as thorough as possible and incorporate threads from the many phone calls and e-mail discussions I have had with ezOps, admins and users in the last few days. I hope that by being open and explicit we can give you a better understanding of what has happened, where we are in the restoration process, and when we can all expect to get back to normal.
I've broken all this information into topic areas to make it easier for you to find the topic that you personally care about the most.
When will my board get its data back?
The most difficult and painful thing for me to report is that it will be impossible to restore full data on all boards. We will send out a Global Announcement once all possible restoration procedures have been completed. During this process, you can expect to see some slowness on the boards, but we’ll return to normal speeds as soon as the restoration is complete.
I realize this creates great uncertainty for many of you still waiting for data restoration, and so I want to give you the following, much more detailed description of the data restoration process we are undertaking in our efforts to get the most back we can, on the fastest time frame possible:
Data restoration began the day of the attack with the restoration of eight servers that we could salvage immediately. Sadly, no other intervention was successful at that time.
Since that time, we have been assembling data from three sources: the remaining historical backups of some 8,000 Gold and trial boards from early May, the current data recently written to hard drives since the attack, and data in RAM memory since the attack.
The actual restorations using this assembled data have been underway since Thursday afternoon, and are being done on a server-by-server basis. The process is automated and each server takes approximately 6 hours to restore. We can restore between 8 and 10 servers in parallel. To some of you this may seem like a long time to restore data on a single server. And it is. However, we are using an innovative process to maximize the data recovery and testing it thoroughly to make sure that (a) we don't erase the posts that have been made since the attack and (b) the data is being properly restored.
When this server-by-server process is complete, we have one final source of data that we may be able to use for restoration. With quick thinking and fast action on the part of your technical team we were able to do an emergency backup of active data. To be clear, this is by no means all the board post data, only the active data at that time. We will be restoring this data as a last step in the process.
We are working as fast as possible on the restoration process - believe me, the faster this is completed the happier we all will be.
What happened in the attack?
What we know is that late in the evening on the holiday Monday, May 30 here in the US, a person with significant and specific information attacked our systems. The attacker was successful in erasing historical board posts and the backup systems, which were password protected and on a separate network. They did not erase the posts of boards that were active on the system at the time.
What more can you tell me about the back-ups?
The issue was not the back-up systems themselves. Our back-up systems have been working well and are separate from our production environment. They are not accessible without inside technical information and passwords. The issue was that either through luck or (more likely) specific information, the attacker was able to delete both the historical files and the back-ups.
What are you doing so this doesn’t happen again?
We have put additional safeguards in place to reduce the opportunity for this to happen in the future. We will also be undertaking a highly detailed security and back-up audit in the coming weeks.
You should also know that we have been Alpha testing the new community software for three weeks and in addition to increased security, redundancy and back-up systems, it will have the ability to allow ezOps to back-up their communities (in addition to daily and weekly incremental backups) to their home computers in an automated fashion. For some ezOps this should provide an additional level of safety and peace of mind.
Can't you give us more frequent updates?
We have been updating you as soon as there is new information. Since the attack we have put out five global messages and have had two conference calls with ezOps and admins. In addition, the Customer Service help desk and Support Forums have been very active. On a more personal basis, Steve DeMello and I have been responding to hundreds of e-mails from users, ezOps and admins. As of 7AM PST on Friday we have responded to every e-mail that has been sent directly to the two of us.
If you are looking for ways to get up to date information fast I recommend reading p079.ezboard.com/fezboard...ouncements first. Please read the materials already available before contacting CS -- the response queues are pretty full, as you might imagine.
What can you tell us about the attacker?
I can tell you that we have suspects, we have evidence and we are working with the FBI. I can also tell you that we will be working to have the attacker punished to the full extent of the law.
While we do have evidence, we can always use more, good, factual evidence. We are offering a $5,000 reward to the person or persons that provide us with factual information that leads to the direct arrest and conviction of the attacker. We are not interested in conspiracy theories, finger pointing or childish pranks. We are interested in information such as community postings or blog entries where the attacker is bragging about the attack and other factual, real information. Please e-mail information to legal@ezboard.com. When you e-mail please provide a link to the original material AND a screen capture of the original material or at the very least, copy and paste the original material into your e-mail. Please be smart, ethical, law abiding and honest about the way you find the information your provide to us.
Now the fine print: A reward will be paid to the person or persons that deliver to legal@ezboard.com information that leads to the direct arrest and conviction of the person or persons that attacked ezboard.com earlier this week. Should more than one person provide identical or substantially similar information that is used for the arrest and conviction of the attacker, the reward will be split equitably amongst the providers of that information.
Will you be providing Credits for Gold Boards?
Yes we plan to provide credits. We need to understand the scope of the data restoration before we decide on the amount of the credit. I would anticipate further news on credits to be available in the next week.
What happened to my Community Chest Balance?
The Community Chest balances were deleted as well, but we do have a back up of that data from a few days before the attack. We need to recalculate the actual Community Chest balances by taking the balance as of the date of the backup, adding Community Chest contributions and subtracting payments from Community Chest. We hope you agree that restoring Community Chest data is a second priority to getting the board posts restored
Please don't worry about your Community Chest balance, it will be restored and updated.
So, where do we go from here?
First let me say that we at ezboard are deeply sorry about the attack this week. The person that did this to us probably had a great chuckle when they pressed the send key to initiate the attack. Little did they know that they would affect so many lives in so many ways. I have heard from cancer patients, historians, mothers and schoolteachers about the way that their boards were damaged and were of vital importance to them. What the attacker did was take a great big piece of many peoples lives including the folks here at ezboard. For that, I hope we catch the attacker and return the favor by taking a bit of their time - behind bars.
We are working to restore the boards as fast as we can. We know your trust in ezboard has been deeply shaken. Now we are trying to rebuild that trust, and are realistic that this will be difficult and take time.
We plan to keep messaging you on our progress. Some of you may have already seen data restored as we started the process on Thursday. Please check p079.ezboard.com/fezboard...ouncements for updates.
In closing, we are working to get systems back to normal and believe that they will be back soon. We are fixing the security and backup processes now and will be performing a security and back-up audit in the coming weeks. Thank you for being an ezboard user, and for reading this, and we hope that you will continue to use ezboard. Let’s put our communities and our network back together, and bring the attacker to justice.
Sincerely,
Rob
Robert Labatt
CEO
ezboard, Inc.
robertlabatt@ezboard.com
The face of nature reflects all of life's ups and downs.
Good evening and thank you for your patience as we continue to restore your data and return our systems to normal operation.
I hope you don’t mind if I continue with the Q&A format for this update as well. Your ezboard team is still at it. We have not taken the weekend off and the restoration process is continuing. You should be aware that in every Global Announcement I am including responses to the common questions that I am getting via e-mail and that we are seeing in the CS queues.
Where are you with the restoration of board data?
The automated board restoration is continuing. We were able to increase the number of server restorations to more than ten at a time. As has been discussed in the help forums and elsewhere, we started from the low numbered p servers and worked our way up. This was not because of where certain boards are housed. We had to pick a place to start and we could not restore all servers at the same time.
I have had many e-mails asking when the restoration process will be finished. We are restoring data from five sources since the main back-up system was destroyed in the attack. Depending on which sources we are using to restore your board/server the restoration process will take more or less time.
We deeply care about getting ezboard back to normal and are working as fast as possible to get there. Please let us focus on that. Talking time to generate timelines and debate the nuance of a changing and complex restoral process will only delay the completion of the restores. Given this thinking it is my hope that you agree with our decision to provide you with regular updates and an announcement when the restoration is complete.
What about my Gold Board renewal in a few days?
Don’t panic. We will not expire your board. Starting next week we will be addressing a gold board credit and restoring the Community Chest data. We have your Gold Board Community Chest balance but need to adjust it for deposits and withdrawals.
Why do posts seem to be disappearing on my Gold Board?
If a thread was not replied to in the time between the attack and when the server your board is on was rebooted, then the information in that thread is most likely permanently lost. We released a fix for this on Thursday last week. It should no longer be an issue, but you may just be seeing the effects of this now.
If your board is currently experiencing thread losses and you have already contacted CS – do not re-submit your information - the information you provided has been given to the tech team. Thank you to everyone who submitted your details, which helped our development team find and fix the problem. No further information is needed at this time. As of midnight June 6, you should not be seeing new data losses.
Was my personal or Credit Card data taken?
We don’t have reason to believe that personal or credit card data was taken. Your credit card information is not stored in our systems.
Do attacks like this happen often?
Sadly, yes they do. Though often they are not reported and from my personal experience customer service related to an attack is spotty. We are trying to do the best job we can. The tech and CS teams are working 20 hour days these days. I’d like some sleep as well.
By the way, it seems that ezboard is not the only destination for attacks this week: abcnews.go.com/Technology...?id=817338 . I feel for their technical staff as it seems that significant user data may have been taken.
When can I expect further updates?
I apologize for not getting the noon update out to you today. We made security changes to the site that made it impossible for the Global Announcements to go out for a few hours. During that time I was responding to individual e-mails that I had received.
We are working to restore the boards as fast as we can. We know your trust in ezboard has been deeply shaken. Now we are trying to rebuild that trust, and are realistic that this will be difficult and take time.
We plan to keep messaging you on our progress. Some of you may have already seen data restored as we started the process on Thursday. Please check p079.ezboard.com/fezboard...ouncements for updates.
In closing, we are working to get systems back to normal and believe that they will be back soon. We are fixing the security and back-up processes now and will be performing a security and back-up audit in the coming weeks. Thank you for being an ezboard user, and for reading this, and we hope that you will continue to use ezboard. Let’s put our communities and our network back together, and bring the attacker to justice.
Sincerely,
Rob
Robert Labatt
CEO
ezboard, Inc.
robertlabatt@ezboard.com
The face of nature reflects all of life's ups and downs.
Hello on Sunday and thank you for your patience as we continue to restore your data and return our systems to normal operation.
The ezboard tech and CS teams are still working for you and working hard to get data back into the system. They are not a group of complainers, but I will tell you that on average the team has been working 20+ hour days to get your data back.
Here is where we are at this point:
Where are you with the restoration of board data?
The automated board restoration is continuing. Approximately 1/2 of the boards are restored to the fullest extent possible, 1/4 are being restored now and 1/4 have yet to be restored. We started with the low numbered servers and are working toward the higher numbered servers.
You should know that boards are being restored on a server-by-server basis. We don't actually restore a 'board' but rather we restore a server. Many servers are being restored in parallel and your server will go through several restoration processes, several times with data arriving from five different sources. This is not simple stuff.
We apologize, but we cannot be more clear on what is happening on each server and board at this point. Please be aware that while we want to respond to all of your very important questions as fast as possible, every bit of energy we expend doing that is taking away from energy that could be spent on managing the restoration of the boards. Taking time to generate timelines and debate the nuance of a changing and complex restoral process will only delay the completion of the restores. While we will continue to respond to direct inquiries, the more you can simply refer to this page for updates rather than send direct inquiries, the better.
What about my Gold Board renewal in a few days?
How can I renew when the money in my chest is missing?
Starting next week we will be addressing a Gold Board credit and restoring the Community Chest data. We have your Gold Board Community Chest balance but need to adjust it for deposits and withdrawals.
Why is my board slow?
The restoration process uses computer power and network bandwidth that is shared with serving web pages. As a result of the restore process you will experience slowness. On occasion a server will be taken off-line as part of the restore process. In such cases your browser will time out when trying to load a page from your board. Don’t panic. The server will be brought back on line once the tech team has finished its work.
Why do posts seem to be still disappearing on my board?
If your board is currently experiencing thread losses and you have already contacted CS – do not re-submit your information - the information you provided has been given to the tech team. As we stated yesterday, if a thread was not replied to in the time between the attack and when the server your board is on was rebooted, then the information in that thread is most likely permanently lost. We have discovered that unless a topic is replied to the data will be lost.
Thank you to everyone who submitted your details, which helped our development team find and fix the problem. No further information is needed at this time. As of midnight June 6, you should not be seeing new data losses. If we get new reports after tonight, we may ask you for more information at that time.
Was my personal or Credit Card data taken?
We don’t have any reason to believe that personal or credit card data was taken. Your credit card information is not stored in our systems.
In closing, we are working to restore the boards as fast as we can and believe ongoing systems and posts will be back to normal soon. We are fixing the security and back-up processes now and will be performing a security and back-up audit in the coming weeks. We know your trust in ezboard has been deeply shaken. Now we are trying to rebuild that trust, and are realistic that this will be difficult and take time.
We plan to keep messaging you on our progress. Please check p079.ezboard.com/fezboard...ouncements for updates.
Thank you for being an ezboard user, and for reading this, and we hope that you will continue to use ezboard. Let’s put our communities and our network back together, and bring the attacker to justice.
Sincerely,
Rob
Robert Labatt
CEO
ezboard, Inc.
robertlabatt@ezboard.com
The face of nature reflects all of life's ups and downs.
Hello on Monday and thank you for your patience as we continue to restore your data and return our systems to normal operation.
We are taking stock of all aspects of the attack and restore processes this morning and we have a board of directors meeting this afternoon to discuss the attack, the restoration process and our next steps.
Your ezboard team is working to get data back and to fix the posting/thread issues that some boards have reported.
We will provide a comprehensive update before the end of our business day here in San Francisco.
Thank you again for your patience and understanding.
Sincerely,
Rob
Robert Labatt
CEO
ezboard, Inc.
robertlabatt@ezboard.com
The face of nature reflects all of life's ups and downs.
As promised earlier today and to some of you who I’ve been e-mailing today here is a comprehensive update of what we know at this point and what we don’t know.
I would like to offer some clarification before we get into the update. The information we use for decision-making and reporting to you is constantly changing. I realize that earlier global announcements have contradicted later ones and confusion has resulted from that. We have been giving you as much information as possible as early as possible and at times that information has changed between global announcements. Please accept my apologies for any confusion the different reports have caused and understand that each global announcement stands on its own at the time it is posted.
Now, on to answering the questions you all have on your minds:
How many servers have been restored?
At this point there is no such thing as a completely restored server.
Let me give you some background to put this in perspective. We have gone back into terabytes of non-back-up data looking for any board data we can find. Where possible we’ve put this data in the restore queue. When we learned of the hack we also dumped the contents of RAM from each server onto a hard drive. We have been restoring that data as well. With the main mode of data restoration gone (the regular back-ups) we have had to create new restoration processes from scratch. That means writing software, testing it and then putting it into production - all on no sleep. In some cases working slower IS safer - but more frustrating for you and for us.
At this time we are restoring from five different data sources. Each source requires a separate step or process to restore data. Roughly half the servers have finished the first step of restoration and are in the second step of what are ultimately five steps of restoration. No severs will be fully restored until they have passed through all five steps. We cannot give you specifics of what servers/boards are where in the process and we cannot have your board opt out of the restore processes.
Why am I still losing threads that were on my board after the attack?
We have isolated the problem and fixed three issues that were leading to thread loss. These fixes have been applied to all servers and you should not be seeing additional losses after today.
Thank you to the over 500 people who submitted information on this problem. We do not need additional information at this time, but if we get new reports of new data loss after tonight we may ask you for more information.
Why am I getting request errors on my board?
We have identified twelve variations of this error. We believe it will be resolved as part of the restoration process. Thanks to the hundreds of people who submitted bug reports on this error already. Should you still be experiencing this bug after the restorals are complete, we will ask for more information at that time.
What is happening to Gold Community subscriptions that have expired since the attack, or will expire soon?
We are currently identifying every Gold Community whose subscription has expired on 5/31 or later and will be reinstating or extending subscription services through at least the next two weeks. This will take a few days to complete. In the meantime, you do not need to let us know your board is expiring.
Why is it taking so long?
Believe me, we are not dragging this out for fun. We are working carefully to ensure that we don’t lose more data. That means double and triple checking what we are doing. For example, we don’t want to overwrite new posts in a forum when we restore that forum. So, we need to perform tests to ensure that does not happen. This caution takes time, but reduces the chance that we lose data that can be restored.
How successful are the restorations?
We don’t know yet. The data that is restored will not be evenly distributed across all boards. Some boards will get all their data restored and some will not get any data back. We will not know exactly which boards will get all, some or none of their data back until the five restoration phases are completed on all servers/boards. We would love to send you a list of which boards are getting what data restored, but it is not information that can be generated at this time.
What don’t we know at this point?
We don’t know how long the five phased recovery process will take. Different servers restore at different rates and there is no pattern to the time it takes.
We don’t know if we will find more data that can be restored. But we are looking for it.
What are you doing to make sure this does not happen again?
We are changing your back-up processes and we are improving security. For security reasons what specifically we are doing is not something we will share in a public forum.
Why are board web pages loading slowly?
The restore process is taking computing power from serving web pages. Web page delivery speeds will return to normal when the restoration process is completed.
Can I e-mail you with questions?
Yes, I have been responding to every e-mail I have received. Sometime it takes a few hours, but I am responding as fast as possible. Please understand that the global announcements have the most up to date information. We cannot provide any information or special restores for specific boards or servers.
Thank you again for your patience and understanding.
Sincerely,
Rob
Robert Labatt
CEO
ezboard, Inc.
robertlabatt@ezboard.com
The face of nature reflects all of life's ups and downs.
Let me start today’s update by telling you that the information published yesterday evening remains accurate – for quick reference, I have attached the Q&A from yesterday to this message.
Based on your emails, and questions to staff and in the Help Forums, I’d like to address five additional questions today.
The answer to the first two questions is the same. The questions are:
Will new forums made since the attack, and new topics made in those forums, be affected by the restoral process?
Will new boards created since the attack be affected by either the attack or the restoral process?
Based on everything we know at this time, we believe that these boards, forums and topics will not be affected by the restoral process. However, we continue to test these issues, and research any reports that indicate otherwise. Based on that research, the answer to one or both of these questions may change.
Can we start posting in our old forums again?
Based on everything we know at this point, we believe that it is safe to post in forums created before the attack again. However, we continue to test, and research any reports that indicate otherwise. Based on that research, the answer may change.
What is the status of restoring Community Chest balances and data?
We will be working on the Community Chest restoral once we have freed up staff from the board restoral, security and back-up improvements.
Will non-Gold Communities be restored?
Non-Gold Communities are not routinely backed up. However, we have data from about 4,000 Non-Gold Communities and we *may* be able to restore additional Non-Gold Communities from RAM. No promises, but we are trying everything.
Thank you again for your patience and understanding.
At this point there is no such thing as a completely restored server.
Let me give you some background to put this in perspective. We have gone back into terabytes of non-back-up data looking for any board data we can find. Where possible we’ve put this data in the restore queue. When we learned of the hack we also dumped the contents of RAM from each server onto a hard drive. We have been restoring that data as well. With the main mode of data restoration gone (the regular back-ups) we have had to create new restoration processes from scratch. That means writing software, testing it and then putting it into production - all on no sleep. In some cases working slower IS safer - but more frustrating for you and for us.
At this time we are restoring from five different data sources. Each source requires a separate step or process to restore data. Roughly half the servers have finished the first step of restoration and are in the second step of what are ultimately five steps of restoration. No severs will be fully restored until they have passed through all five steps. We cannot give you specifics of what servers/boards are where in the process and we cannot have your board opt out of the restore processes.
Why am I still losing threads that were on my board after the attack?
We have isolated the problem and fixed three issues that were leading to thread loss. These fixes have been applied to all servers and you should not be seeing additional losses after today.
Thank you to the over 500 people who submitted information on this problem. We do not need additional information at this time, but if we get new reports of new data loss after tonight we may ask you for more information.
Why am I getting request errors on my board?
We have identified twelve variations of this error. We believe it will be resolved as part of the restoration process. Thanks to the hundreds of people who submitted bug reports on this error already. Should you still be experiencing this bug after the restorals are complete, we will ask for more information at that time.
What is happening to Gold Community subscriptions that have expired since the attack, or will expire soon?
We are currently identifying every Gold Community whose subscription has expired on 5/31 or later and will be reinstating or extending subscription services through at least the next two weeks. This will take a few days to complete. In the meantime, you do not need to let us know your board is expiring.
Why is it taking so long?
Believe me, we are not dragging this out for fun. We are working carefully to ensure that we don’t lose more data. That means double and triple checking what we are doing. For example, we don’t want to overwrite new posts in a forum when we restore that forum. So, we need to perform tests to ensure that does not happen. This caution takes time, but reduces the chance that we lose data that can be restored.
How successful are the restorations?
We don’t know yet. The data that is restored will not be evenly distributed across all boards. Some boards will get all their data restored and some will not get any data back. We will not know exactly which boards will get all, some or none of their data back until the five restoration phases are completed on all servers/boards. We would love to send you a list of which boards are getting what data restored, but it is not information that can be generated at this time.
What don’t we know at this point?
We don’t know how long the five phased recovery process will take. Different servers restore at different rates and there is no pattern to the time it takes.
We don’t know if we will find more data that can be restored. But we are looking for it.
What are you doing to make sure this does not happen again?
We are changing your back-up processes and we are improving security. For security reasons what specifically we are doing is not something we will share in a public forum.
Why are board web pages loading slowly?
The restore process is taking computing power from serving web pages. Web page delivery speeds will return to normal when the restoration process is completed.
Can I e-mail you with questions?
Yes, I have been responding to every e-mail I have received. Sometime it takes a few hours, but I am responding as fast as possible. Please understand that the global announcements have the most up to date information. We cannot provide any information or special restores for specific boards or servers.
The face of nature reflects all of life's ups and downs.
Let me start today’s update by telling you that the information published yesterday remains accurate – for quick reference, I have attached yesterday’s update to the bottom of this message.
Based on your emails, and questions to staff and in the Help Forums, I’d like to address three additional questions today.
Should I contact you with my questions on payment, account issues etcetera?
No. Please direct questions on those topics and questions you would normally send to CS to helpdesk.ezboard.com
Please do not e-mail me directly with these issues, as I will forward you on to the CS team. You will not experience a faster response from CS as a result of my being involved.
Can I resume normal activities like posting and reading on my boards?
Yes, you may resume normal posting at the majority of boards. We are aware of the issues (i.e., request errors) that are affecting approximately 1% of boards and are working to repair them. If you are experiencing problems that are new in the last 24 hours and have not been reported or seen before, please post a bug report at p079.ezboard.com/fbugsfrm177
I’ve been told of a way that I can restore my own topics, should I do it?
Yes and No. Many topics have been restored, but do not appear on the topic listing page. It is not harmful to reply to these topics, but we have implemented a way to automate the restoration of topic titles.
If your question is not answered above please continue to carefully read the balance of this update. Should your question still not be answered after reading this update and you don’t think that our customer service team or help forums can answer your question then please feel free to e-mail me at the address below.
Thank you again for your patience and understanding.
Sincerely,
Rob
Robert Labatt
CEO
ezboard, Inc.
robertlabatt@ezboard.com
Q&A FROM YESTERDAY:
The answer to the first two questions is the same. The questions are:
Will new forums made since the attack, and new topics made in those forums, be affected by the restoral process?
Will new boards created since the attack be affected by either the attack or the restoral process?
Based on everything we know at this time, we believe that these boards, forums and topics will not be affected by the restoral process. However, we continue to test these issues, and research any reports that indicate otherwise. Based on that research, the answer to one or both of these questions may change.
Can we start posting in our old forums again?
Based on everything we know at this point, we believe that it is safe to post in forums created before the attack again. However, we continue to test, and research any reports that indicate otherwise. Based on that research, the answer may change.
What is the status of restoring Community Chest balances and data?
We will be working on the Community Chest restoral once we have freed up staff from the board restoral, security and back-up improvements.
Will non-Gold Communities be restored?
Non-Gold Communities are not routinely backed up. However, we have data from about 4,000 Non-Gold Communities and we *may* be able to restore additional Non-Gold Communities from RAM. No promises, but we are trying everything.
How many servers have been restored?
At this point there is no such thing as a completely restored server.
Let me give you some background to put this in perspective. We have gone back into terabytes of non-back-up data looking for any board data we can find. Where possible we’ve put this data in the restore queue. When we learned of the hack we also dumped the contents of RAM from each server onto a hard drive. We have been restoring that data as well. With the main mode of data restoration gone (the regular back-ups) we have had to create new restoration processes from scratch. That means writing software, testing it and then putting it into production - all on no sleep. In some cases working slower IS safer - but more frustrating for you and for us.
At this time we are restoring from five different data sources. Each source requires a separate step or process to restore data. Roughly half the servers have finished the first step of restoration and are in the second step of what are ultimately five steps of restoration. No severs will be fully restored until they have passed through all five steps. We cannot give you specifics of what servers/boards are where in the process and we cannot have your board opt out of the restore processes.
Why am I still losing threads that were on my board after the attack?
We have isolated the problem and fixed three issues that were leading to thread loss. These fixes have been applied to all servers and you should not be seeing additional losses after today.
Thank you to the over 500 people who submitted information on this problem. We do not need additional information at this time, but if we get new reports of new data loss after tonight we may ask you for more information.
Why am I getting request errors on my board?
We have identified twelve variations of this error. We believe it will be resolved as part of the restoration process. Thanks to the hundreds of people who submitted bug reports on this error already. Should you still be experiencing this bug after the restorals are complete, we will ask for more information at that time.
What is happening to Gold Community subscriptions that have expired since the attack, or will expire soon?
We are currently identifying every Gold Community whose subscription has expired on 5/31 or later and will be reinstating or extending subscription services through at least the next two weeks. This will take a few days to complete. In the meantime, you do not need to let us know your board is expiring.
Why is it taking so long?
Believe me, we are not dragging this out for fun. We are working carefully to ensure that we don’t lose more data. That means double and triple checking what we are doing. For example, we don’t want to overwrite new posts in a forum when we restore that forum. So, we need to perform tests to ensure that does not happen. This caution takes time, but reduces the chance that we lose data that can be restored.
How successful are the restorations?
We don’t know yet. The data that is restored will not be evenly distributed across all boards. Some boards will get all their data restored and some will not get any data back. We will not know exactly which boards will get all, some or none of their data back until the five restoration phases are completed on all servers/boards. We would love to send you a list of which boards are getting what data restored, but it is not information that can be generated at this time.
What don’t we know at this point?
We don’t know how long the five phased recovery process will take. Different servers restore at different rates and there is no pattern to the time it takes.
We don’t know if we will find more data that can be restored. But we are looking for it.
What are you doing to make sure this does not happen again?
We are changing your back-up processes and we are improving security. For security reasons what specifically we are doing is not something we will share in a public forum.
Why are board web pages loading slowly?
The restore process is taking computing power from serving web pages. Web page delivery speeds will return to normal when the restoration process is completed.
Can I e-mail you with questions?
Yes, I have been responding to every e-mail I have received. Sometime it takes a few hours, but I am responding as fast as possible. Please understand that the global announcements have the most up to date information. We cannot provide any information or special restores for specific boards or servers.
The face of nature reflects all of life's ups and downs.
ezBoard Attack Updates
Posted: 6/9/05 13:00
ezboard Attack Update - 06/09/05
--------------------------------------------------------------------------------
Edited to add one additional Q&A
Dear ezboard Family,
Let me start today’s update by telling you that the information published yesterday remains accurate – for quick reference, I have attached the Q&A from previous day’s update to the bottom of this message.
Based on your emails, and questions to staff and in the Help Forums, I’d like to address the two most frequently asked questions:
What is the status and timing of the restoral process?
I’m pleased to announce that almost all servers have completed the first three steps of the restoral process, and we plan to start the final two steps on a server-by-server basis later today. Based on our testing, we expect boards that were active at the time of the attack to see a significant amount of data restored during these final two steps.
We anticipate that the final two restoral steps will take a total of seven to ten days to be completed. Individual servers will be restored throughout that time. This means that you will see your board/server restoral sometime within that period. We are in the process of acquiring additional servers in an effort to accelerate this process.
Please be aware that the final steps are the most computer power intensive, and you will notice at least moderate slowness on your server when it is being restored. We will not respond to requests to have your server restored first, and cannot restore an individual board on a server.
When you said “we expect boards that were active at the time of the attack to see a significant amount of data restored during these final two steps”, what did you mean by “active at the time of the attack”?
“Active at the time of the attack” means that members were actively posting and reading. While we cannot determine how much data will be retrieved for a given level of posting and reading activity, we can tell you that the more active a board was (therefore more of its data was in RAM) the more data will likely be restored. The flip side of this is that an archive board that was not active at that time would not have much or any data in RAM and therefore would likely not be as likely to have data restored.
Gold Community subscriptions are expiring – what are you planning to do about this?
We have completed, and are in final testing of, code that will “freeze” all subscriptions. Assuming a successful final test, we will deploy the code today. We have also established a list of all Gold Communities that have expired since the attack, and are developing a process to restore subscription benefits to those boards.
If your question is not answered above please continue to carefully read the balance of this update. Should your question still not be answered after reading this update and you don’t think that our customer service team or help forums can answer your question then please feel free to e-mail me at the address below.
Thank you again for your patience and understanding.
Sincerely,
Rob
Robert Labatt
CEO
ezboard, Inc.
robertlabatt@ezboard.com
The face of nature reflects all of life's ups and downs.
Let me start today’s update by telling you that the information published yesterday remains accurate – for quick reference, I have attached the Q&A from previous day’s update to the bottom of this message.
Let me briefly update you on progress since the last announcement:
All servers have now completed the first three steps of the restoral process, and have started the final two steps on a server-by-server basis. We continue to anticipate that the final two restoral steps will take a total of seven to ten days to be completed. Individual servers will be restored throughout that time. This means that you will see your board/server restoral sometime within that period. We are installing additional servers to accelerate the restoral process.
In order to return content to the boards as soon as possible, the restoral process will bring topics back out of chronological order. After we have recovered all data from the process, we will then do a final resort of topics to re-establish chronological order.
We have implemented code that has frozen all active Gold Community subscriptions as of yesterday. We have also established a list of all Gold Communities that have expired since the attack, and are finalizing a process to restore subscription benefits to those boards.
If you have a question that is not answered above please continue to carefully read the balance of this update. Should your question still not be answered after reading this update and you don’t think that our customer service team or help forums can answer your question then please feel free to e-mail me at the address below.
Thank you again for your patience and understanding.
Sincerely,
Rob
Robert Labatt
CEO
ezboard, Inc.
robertlabatt@ezboard.com
The face of nature reflects all of life's ups and downs.
Let me start today’s update by telling you that the information published yesterday remains accurate – for quick reference, I have attached the Q&A from previous day’s update to the bottom of this message.
The restoral process continues, with additional servers brought online yesterday to accelerate the process. Please remember that in order to return content to the boards as soon as possible, the restoral process will bring topics back out of chronological order. After we have recovered all data from the process, we will then do a final resort of topics to re-establish chronological order.
If you have a question that is not answered above please continue to carefully read the balance of this update. Should your question still not be answered after reading this update and you don’t think that our customer service team or help forums can answer your question then please feel free to e-mail me at the address below.
Thank you again for your patience and understanding.
Sincerely,
Rob
Robert Labatt
CEO
ezboard, Inc.
robertlabatt@ezboard.com
The face of nature reflects all of life's ups and downs.
Let me start today’s update by telling you that with the exception of the new information on Gold Community subscriptions, the information published yesterday remains accurate – for quick reference, I have attached the Q&A from previous day’s update to the bottom of this message.
Based on your emails, and questions to staff and in the Help Forums, I’d like to address three questions today.
My Gold Community subscription expiration date is not being extended as it was announced it would last week – what’s happening?
We are in the process of taking three steps to resolve the major board subscription issues. Those steps are: re-establishing Gold status for communities whose subscription expired since the attack, correcting subscription terms for communities that renewed since the release of the code (last Thursday), and globally extending all Gold Community subscriptions.
I’m experiencing problems with my server – is this related to the attack?
We have been responding to reports of server failure over the weekend, most notably p087. The problems on p087 were a result of hardware failure, and were in no way related to the attack. Our technical team has brought p087 back online, and is working to restore data.
Why are you cutting and pasting material from update to update?
You should know that, if there are no changes and we are progressing as planned and as indicated the day before, then there is nothing to update other than we are on track. We do provide changed updates when there are changes from the plans that we messaged to you in a previous message. In a way, a cut and paste update is good news.
What is the status of the restoral process?
The final two steps of the restoral process continue across the network. Please remember that in order to return content to the boards as soon as possible, the restoral process will bring topics back out of chronological order. After we have recovered all data from the process, we will then do a final resort of topics to re-establish chronological order.
If you have additional questions please continue to carefully read the balance of this update. Should your question still not be answered after reading this update and you don’t think that our customer service team or help forums can answer your question then please feel free to e-mail me at the address below.
Thank you again for your patience and understanding.
Sincerely,
Rob
Robert Labatt
CEO
ezboard, Inc.
robertlabatt@ezboard.com
The face of nature reflects all of life's ups and downs.
The good news is that we have nothing new to report today – the final two steps of the restoral process are happening across the network, and we expect to be on schedule for completion over the weekend. It is important to understand that at this point there is no such thing as a completely restored server – even the servers farthest along in the process require some additional work. On a related subject, we are continuing to work on recovering p087 data from this weekend’s hard drive problem.
For quick reference, I have attached the Q&A from previous day’s update to the bottom of this message.
If you have additional questions please continue to carefully read the balance of this update. Should your question still not be answered after reading this update and you don’t think that our customer service team or help forums can answer your question then please feel free to e-mail me at the address below.
I remain available for and welcome your emails. However, please do not write to me as a way of circumventing the normal CS process, or to request information specific to the status of your board. I will be referring all normal CS requests to the appropriate forum or queue, and cannot provide information on the status of specific boards and servers.
Thank you again for your patience and understanding.
Sincerely,
Rob
Robert Labatt
CEO
ezboard, Inc.
robertlabatt@ezboard.com
The face of nature reflects all of life's ups and downs.
The restoral process continues on schedule. Four of five steps are nearly complete across the network; the final step will reload all restored data. We still expect all steps to be completed through the weekend. At that point, we will then go to work on restoring all community financial data, addressing Gold Community credits and other financial questions, and sharing our plans for current system improvements and beta testing of the new product.
We are so very sorry for the time the restoral process has taken, and we hope that these updates have helped to reduce your anxiety. We greatly appreciate your patience and support, and want to complete this process as quickly and well as we possibly can.
I have attached the Q&A from previous day’s update to the bottom of this message. If you have additional questions please continue to carefully read the balance of this update. Should your question still not be answered after reading this update and you don’t think that our customer service team or help forums can answer your question then please feel free to e-mail me at the address below.
I remain available for and welcome your emails. However, please do not write to me as a way of circumventing the normal CS process, or to request information specific to the status of your board. I will be referring all normal CS requests to the appropriate forum or queue, and cannot provide information on the status of specific boards and servers.
Thank you again for your patience and understanding.
Sincerely,
Rob
Robert Labatt
CEO
ezboard, Inc.
robertlabatt@ezboard.com
The face of nature reflects all of life's ups and downs.
We can see light at the end of the restoration tunnel. The work we are doing now is delivering great results with many boards getting significant amounts of data back and more to come.
Please don't panic if your particular board has not seen data restored in the last day. Unfortunately it is taking time for the restoration process to propagate throughout to all servers in our systems. Set your expectations to seeing data by Monday. Please hold off on asking CS or me to clarify things about your board until next week. This will allow us to focus on getting your data back faster.
While we are on a positive note, in the last 24 hours we have located an additional pocket of data that we will work to restore. Restoring this data will commence next week after the current restoration processes are completed. You can consider this a sixth restoration process. You may be asking why we just found this data now, and the truth is that we have been running additional tests on old and new equipment and during one of these tests we located additional data. I can't tell you more about this now and please do not ask for more information. We will tell you what we know when we can. Stay tuned here for information.
Finally, we have identified and are testing a back-up and restore process that will be much faster than we had implemented in the past. Remember that the restore processes we are using now are NOT the normal processes. This new procedure means that during the back-up process you will experience much less server slowness and much faster restores. More on this after we have the process nailed down and in production.
Thank you again for your patience and we hope that our daily communications and open dialogue have helped you and all ezboard users to better understand what we are doing to get ezboard back to normal.
Sincerely,
Rob
Robert Labatt
CEO
ezboard, Inc.
The face of nature reflects all of life's ups and downs.
We continue to work through the last steps of the restoral process across the network, and have reports of a great many boards getting significant amounts of data back. There is more to come, as we have also begun restoring the newly located additional pocket of data.
Again, please don't panic if your particular board has not seen data restored in the last day. It is taking time for the restoration process to propagate throughout to all servers in our systems.
We continue to test the back-up and restore process and remain optimistic that you will experience much faster restores. More on this after we have the process nailed down and in production.
Thank you again for your patience and we hope that our daily communications and open dialogue have helped you and all ezboard users to better understand what we are doing to get ezboard back to normal.
Sincerely,
Rob
Robert Labatt
CEO
ezboard, Inc.
The face of nature reflects all of life's ups and downs.
No additional news today – we continue through the last steps of the restoral process across the network, including restoring the newly located additional pocket of data.
The restoration process continues to propagate throughout all servers in our system. Your particular board may not have seen data restored as yet.
We continue to test the new back-up and restore process and remain optimistic that you will experience much faster restores.
Thank you again for your patience and we hope that our daily communications are helping clarify the status of the network.
Sincerely,
Rob
Robert Labatt
CEO
ezboard, Inc.
The face of nature reflects all of life's ups and downs.
Email This To a Friend
Topic Commands
Click to receive email notification of replies
Click to stop receiving email notification of replies