I just noticed the following message on Entrecard’s main page:
There was a fire at our datacenter resulting in the failure of some of our systems, our database server appears to be offline still, we’re working on it and we’re really sorry about the issues at this time. more information arriving shortly.
Update: The outage is pretty serious, you can read more about the datacenter fire at ThePlanet forums. They’ve got teams of people there and they’re working through the night. We have no ETA at this time, very sorry.
I rushed on ThePlanet forums and the matter sounds a bit serious, here’s some quote of the recent news:
Today at approximately 5:45 p.m., a transformer in our H1 data center in Houston caught fire, thus requiring us to take down all generators as instructed by the fire department. All servers are down.
We are working with the fire department, with our facilities staff on site, to assess the situation.
To keep you updated, we will send messages every 15 minutes in Orbit and here in our forum.
From Doug Erwin:
This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.
We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.
This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.
We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well.
There is no impact in any of our other five data centers.
I am sorry that this accident has occurred and I apologize for the impact.
More teams from The Planet are coming along with more contractors from key vendors for electrical and facilities to help get H1 online. At this time, DNS infrastructure continues to propagate. ServerCommand servers are installed, but the teams are making sure all networking is intact and ready. In the meantime, please call our support lines for any issues. Additional support techs are available.
At this time, we do not have an updated Estimated Time to repair. Please continue following this thread for updates.
Director, Product Management
For more news, visit the ThePlanet forums
Let’s hope everything gets back to normal soon and Entrecard comes back.
UPDATED: Entrecard is now back :)