Materialization failures impacting GCP cloud-hosted orgs
Incident
Report for Sigma Computing
Postmortem
Sigma Engineering has completed the postmortem of this incident. A Root Cause Analysis (RCA) is available on the Sigma Community site: Post Mortem for Sep 6, 2023: Materialization failures impacting GCP cloud-hosted orgs.
Resolved
Current Status: The mitigation measures have proven effective, and the issue is resolved.
Impacted Cloud Regions & Services: Sigma users hosted on GCP
User Experience: We expect affected materializations to be resolved on the next scheduled run or when manually retriggered for all customers leveraging materialization and hosted on GCP.
Incident Start Time: Approximately 16:30 UTC Sep 6, 2023
Incident End Time: Approximately 19:40 UTC Sep 6, 2023
We will perform a root cause analysis (RCA) and publish our findings in a postmortem note to the Sigma Community site in coming days. You will receive a notification when that occurs.
Impacted Cloud Regions & Services: Sigma users hosted on GCP
User Experience: We expect affected materializations to be resolved on the next scheduled run or when manually retriggered for all customers leveraging materialization and hosted on GCP.
Incident Start Time: Approximately 16:30 UTC Sep 6, 2023
Incident End Time: Approximately 19:40 UTC Sep 6, 2023
We will perform a root cause analysis (RCA) and publish our findings in a postmortem note to the Sigma Community site in coming days. You will receive a notification when that occurs.
Monitoring
Current Status: Our engineers have implemented mitigation measures to circumvent the issue and we are monitoring their effect closely.
Impacted Cloud Regions & Services: Sigma users hosted on GCP
User Experience: We expect the rate of errors for affected materializations to decrease on the next scheduled run or when manually retriggered for all customers leveraging materialization and hosted on GCP.
Incident Start Time: Approximately 17:00 UTC Sep 6, 2023
Incident End Time: Approximately 19:40 UTC Sep 6, 2023
Impacted Cloud Regions & Services: Sigma users hosted on GCP
User Experience: We expect the rate of errors for affected materializations to decrease on the next scheduled run or when manually retriggered for all customers leveraging materialization and hosted on GCP.
Incident Start Time: Approximately 17:00 UTC Sep 6, 2023
Incident End Time: Approximately 19:40 UTC Sep 6, 2023
Investigating
Some customers hosted on Google Cloud Platform (GCP) have observed Materialization and CSV upload failures beginning at 1:00pm EST / 10:00am PST. Our Engineering team is currently conducting an investigation to identify and mitigate the root cause, and Technical Support is actively monitoring the situation.
We are currently investigating an issue with materialization and CSV upload failure for customers on Google Cloud Platform (GCP). This issue is being treated with the highest priority and we will update the status accordingly.
Impacted Cloud Regions & Services: Sigma organizations hosted on GCP utilizing materialization
User Experience: Errors related to materialization and CSV upload failure
Incident Start Time: Approximately 17:00 UTC Sep 6, 2023
We are currently investigating an issue with materialization and CSV upload failure for customers on Google Cloud Platform (GCP). This issue is being treated with the highest priority and we will update the status accordingly.
Impacted Cloud Regions & Services: Sigma organizations hosted on GCP utilizing materialization
User Experience: Errors related to materialization and CSV upload failure
Incident Start Time: Approximately 17:00 UTC Sep 6, 2023
This incident affected: GCP (Sigma - GCP (US)).