Back to overview
Degraded

Unexpected data contention

Oct 21 at 07:50am GMT-3
Affected services
Stash Platform
Stash Legacy

Resolved
Oct 21 at 02:59pm GMT-3

All data queues are normalized and no abnormal delays are observed or expected.

Updated
Oct 21 at 12:10pm GMT-3

The processing of accumulated events continue to be processed as expected. Estimated time for normalization is around 2.5h

Updated
Oct 21 at 10:36am GMT-3

Our team already identified the issue and addressed the underlying cause. A cluster network issue triggered a bug in the database connection pool of the data ingestion system that prevented connections to be renewed, resulting in events accumulating at the ingestion layer.

Additional capacity has been provisioned to handle the queued data, and events may be inserted out-of-order during this period. It's expected to take around 6h until the queue levels normalize.

Created
Oct 21 at 07:50am GMT-3

Some tenants are experiencing event data contention, that results in stale data being displayed on the dashboards. Our team is already investigating the issue and will update the status as soon as new information is available