Cenlar Outage – August 27, 2018
Shortly after 12:00 am on Monday morning, August 27, we lost power inside a portion of our production computer room in Ewing, NJ. We dispatched electricians to the site and they worked with our IT staff to redistribute power across the computer room. While the power situation was being assessed and repaired, we routed inbound phone traffic to our Scotch Road facility and leveraged the redundant IVR in our out of region data center to ensure call center continuity. Calls were being answered by 8:00 am and full power was restored in the Ewing computer room by 8:45 am.
As we began to bring systems back online, we determined that a few key network infrastructure servers had been corrupted due to the hard shutdown caused by the power issue. This delayed our recovery time. The IT team began the process of restoring these servers at 9:00 am. The first set of impacted services was fully restored at 1:50 pm, with more services added throughout the day until the last business critical system was restored at 9:30 pm. System testing confirmed that the power issues did not result in any data loss.
The root cause of the power issue was a failed circuit breaker that was not adequately handled by our redundant power infrastructure. This resulted in the partial power failure that shut down a portion of our production server environment. The sudden server shutdown caused the corruption of the critical servers.
In addition to the power restoration that was completed on the 27th, we are working with our electrical contractor to improve power redundancy for the computer room. This effort will be completed by mid-October.