Aug

Important Technology Information from Cenlar

This advisory includes important information on three technology issues: a planned maintenance window for our Document Center this weekend, a root cause analysis of the Aug. 15 connectivity issue in our call center and an update on our pandemic dashboard reporting.

8/15 Call Center Outage Root Cause Analysis

As you may recall, on Monday, Aug. 15, 2022, Cenlar experienced a connectivity issue in our call center that prevented our representatives from receiving in-bound calls from homeowners.

Root Cause
The incident was attributed to a Citrix NetScaler firmware upgrade performed during a scheduled maintenance window on Aug. 14. Although the systems were validated post upgrade, the configuration change caused an issue the next morning, impacting disk space management on one of the two NetScalers.

Each Citrix environment — Production and Disaster Recovery (DR) — has two NetScalers that are in a high availability/load balancing configuration. Due to the nature of the incident, the failover capability did not work as designed, and the primary NetScaler of the Production Citrix environment had to be brought offline completely in order to resolve.

The previous week, the same firmware upgrade was applied on the Citrix NetScalers for the DR environment with no impact. Citrix subsequently notified Cenlar that the NetScaler firmware upgrade package inappropriately changed the NetScaler storage configuration parameters. This flaw resulted in the log files consuming all NetScaler storage capacity. Once the configuration file was fixed and unnecessary files were removed, service was restored to normal operations.

Key Actions Taken to Resolve Incident

Activate Citrix Disaster Recovery Environment, and migrate users to DR
Remove unnecessary files from the Secondary Production NetScaler
Fail over to the Secondary Production NetScaler
Remove the failed Primary Production NetScaler from the load balancer pool

Corrective Action/Next Steps

Correct the NetScaler configuration file for circular logging (Completed)
Modify update procedures to remove previous installation packages and to check NetScaler configuration for circular logging (Completed)
Add Primary Production Netscaler back into Production Citrix load balancing pool
Promote the Primary Production Netscaler as the primary member of the Production Citrix load balancing pool
Add monitoring and alert capability to the NetScalers storage
Scale down the DR Citrix environment

Document Center Maintenance Window This Weekend

We’ve extended our regular maintenance window this weekend from Saturday, Aug. 27 at 11 p.m. through Sunday, Aug. 28 at 3 a.m. to accommodate upgrades to the Document Center. During this period, the Document Center will feature a message alerting users of the maintenance window and directing them to visit again soon.

Pandemic Dashboard Reporting

Pandemic Dashboard Reporting has resumed functioning properly. We thank you for your patience as we fixed the issue.