TIMELINE AND IMPACT
Date: Thursday, November 13th, 2025
Timing: 11 a.m. - 1 p.m. ET (approx. 2 hours)
Impacted: U.S. (AWS) datacenter, webservers, client import/export, and any scheduled mail jobs
SUMMARY
On Thursday, November 13th, 2025, at approximately 11:00 a.m. ET, we began to receive internal alerts indicating degradation of our platform. Clients were seeing slowness in page load times and in their ability to access the platform, as well as delays in scheduled email jobs. We quickly determined that the incident was limited to clients hosted in the U.S. datacenter (AWS), and began working on a fix. By 12 noon ET, the system began gradually recovering, and by 1 p.m. ET, it was fully stabilized.
DETECTION AND MITIGATION
During the week of November 10th, our data team was working on a routine zip code update in the lookup table for the U.S. House and Senate databases. The lookup table supports client advocacy pages that enable supporters to contact U.S. legislators.
After internal testing, the updated table was deployed at about 6:00 a.m. ET on Thursday, November 13th. By 11:00 a.m. ET, platform performance had degraded significantly, and we began to receive reports from clients experiencing issues.
Our operations team initiated an investigation and ultimately traced the issue to the newly added lookup table used by the advocacy page functionality. We immediately reverted to the previous version of the table and restarted all application instances at approximately 12:30 p.m. ET. Performance began to stabilize after the restart. While some minor sluggishness persisted for another 30 minutes, the platform fully recovered by 1:00 p.m. ET.
Our database administrators have collected detailed tracing data and are actively working to reproduce the issue. We have also opened a support case with Oracle for deeper analysis. At this stage, our findings indicate that this was a latent defect triggered by a unique set of conditions that had not occurred previously. Our data team has performed dozens of similar changes in the past without encountering this behavior.
NEXT STEPS
While these are routine updates that we perform regularly without issue, we will not update lookup tables for the remainder of 2025. Moving forward, starting in 2026, we will implement an additional testing phase prior to deploying any lookup tables to client-facing pages. This enhanced process will include creating a separate advocacy database and validating the changes against a defined set of test pages. Only after successful completion of this testing will we deploy updated lookup tables to the live client environment.