Increased error rates on Sigma GCP
Incident
Report for Sigma Computing
Postmortem
Sigma Engineering has completed the postmortem of this service incident. A Root Cause Analysis (RCA) is available on the Sigma Computing Community site: Postmortem for December 2nd, 2022 incident - Sigma GCP
Resolved
Impact Statement: Between 09:30 UTC and 19:00 UTC, Sigma users hosted on GCP may have experienced sporadic errors for some of their queries.
Preliminary Root Cause: Under investigation.
Next Steps: We have seen the rate of these errors decline to zero as a result of our mitigation measures. We will continue to investigate to establish the full root cause and prevent future occurrences. A detailed postmortem will be published.
Preliminary Root Cause: Under investigation.
Next Steps: We have seen the rate of these errors decline to zero as a result of our mitigation measures. We will continue to investigate to establish the full root cause and prevent future occurrences. A detailed postmortem will be published.
Update
We have implemented mitigation measures and seen the rate of these errors decline to zero as a result. We will continue closely monitoring our system's health as our engineers root cause and implement a permanent fix. Scheduled materializations have now been re-enabled and should run as scheduled.
Identified
We have implemented mitigation measures and seen the rate of these errors decline to zero as a result. We will continue closely monitoring our system's health as our engineers root cause and implement a permanent fix. Note that all scheduled materialization runs are still temporarily disabled for the time being.
Update
We are continuing to investigate the root cause for this issue. Users on Sigma GCP will still experience sporadic errors for some of their queries. As part of our mitigation effort, we have temporarily disabled all scheduled materialization runs.
Investigating
Users on Sigma GCP may experience increased error rates affecting the ability to view data. We are currently investigating.
This incident affected: GCP (Sigma - GCP (US)).