Degraded Performance on Sigma AWS - Connection Indexing
Incident
Report for Sigma Computing
Postmortem
Sigma Engineering has completed the postmortem of this incident. A Root Cause Analysis (RCA) is available on the Sigma Community site: Postmortem for May 30, 2023 Incident -Connection Indexing bug - Sigma AWS
Resolved
Current Status: The mitigation measures have proven effective, and the issue is resolved.
Impacted Cloud Regions & Services: Sigma users hosted on AWS US, all connection types
User Experience: As connection indexing has been reenabled, there should no longer be any issue with slow navigation or search of warehouse schema.
Incident Start Time: June 1st 2023, Approximately 13:00 UTC
Incident End Time: June 2nd, 2023, Approximately 06:40 UTC
We will perform a root cause analysis (RCA) and publish our findings in a postmortem note to the Sigma Community site in coming days. You will receive a notification when that occurs.
Impacted Cloud Regions & Services: Sigma users hosted on AWS US, all connection types
User Experience: As connection indexing has been reenabled, there should no longer be any issue with slow navigation or search of warehouse schema.
Incident Start Time: June 1st 2023, Approximately 13:00 UTC
Incident End Time: June 2nd, 2023, Approximately 06:40 UTC
We will perform a root cause analysis (RCA) and publish our findings in a postmortem note to the Sigma Community site in coming days. You will receive a notification when that occurs.
Monitoring
Current Status: Our engineers have implemented mitigation measures to circumvent the issue and we are monitoring their effect closely.
Impacted Cloud Regions & Services: Sigma users hosted on AWS US, all connection types
User Experience: As connection indexing has been reenabled, there should no longer be any issue with slow navigation or search of warehouse schema.
Incident Start Time: June 1st 2023, Approximately 13:00 UTC
Incident End Time: June 2nd, 2023, Approximately 06:40 UTC
Impacted Cloud Regions & Services: Sigma users hosted on AWS US, all connection types
User Experience: As connection indexing has been reenabled, there should no longer be any issue with slow navigation or search of warehouse schema.
Incident Start Time: June 1st 2023, Approximately 13:00 UTC
Incident End Time: June 2nd, 2023, Approximately 06:40 UTC
Update
Current Status: In an effort to resolve the issue, we plan to restart connection indexing services tonight, June 1, between 11PM and midnight PDT. The restart event is expected to complete in under 2 minutes; during this time, workbook loads may fail.
Impacted Cloud Regions & Services: Sigma users hosted on AWS US, all connection types
User Experience: Up to 2 minute disruption in workbook loads during 11PM-midnight PDT maintenance window.
Incident Start Time: June 1st 2023, Approximately 13:00 UTC
Impacted Cloud Regions & Services: Sigma users hosted on AWS US, all connection types
User Experience: Up to 2 minute disruption in workbook loads during 11PM-midnight PDT maintenance window.
Incident Start Time: June 1st 2023, Approximately 13:00 UTC
Update
We are continuing to work on a fix for this issue.
Identified
To mitigate system slowness while we fix the root cause, we temporarily disabled automated connection indexing. Newly-created connections may be slow to navigate and search may not return them. Existing connections may need to have their databases/schemas/tables manually synced in Sigma if changes are made in the warehouse.
Impacted Cloud Regions & Services: Sigma users hosted on AWS US, all connection types
Impacted Cloud Regions & Services: Sigma users hosted on AWS US, all connection types
This incident affected: AWS (Sigma - AWS (US)).