On January 22, 2019, Cenlar’s borrower and client facing systems experienced extreme latency and outages throughout the day. At approximately 2 pm EST, the issues were sufficient to warrant taking Global Teller and CenNet SSO systems off-line. This decision was necessary to troubleshoot and fix the problem rather than allowing the applications to function in a degraded state with limited features. During our troubleshooting activity, we discovered the issue to be related to several smaller outages over the past two weeks.
We were able to identify and remediate the issue in the early evening, improving stability in the systems that had remained operational. At 9:55 pm EST, CenNet SSO was put back into service. Global Teller was restored to service before business hours on Wednesday, January 23, 2019.
The root cause of the issue was a malfunctioning API call to one of our document vendors. The latest version of this API, installed in mid-December, performed as expected in the production environment for over three weeks. The API’s performance began to degrade when it was invoked across multiple platforms during year-end related peak usage.
To remediate the issue, we reverted the API to the version in place prior to mid-December. Additionally, we took the opportunity to allocate additional hardware and memory resources to the environment before restoring it to service.
Note: During this outage, the Call Center and IVR were fully operational. Borrowers could make payments and conduct business even though CenNet experienced a service interruption.