Unexpected data contention
Resolved
Oct 21, 2024 at 5:59pm UTC
All data queues are normalized and no abnormal delays are observed or expected.
Affected services
Updated
Oct 21, 2024 at 3:10pm UTC
The processing of accumulated events continue to be processed as expected. Estimated time for normalization is around 2.5h
Affected services
Updated
Oct 21, 2024 at 1:36pm UTC
Our team already identified the issue and addressed the underlying cause. A cluster network issue triggered a bug in the database connection pool of the data ingestion system that prevented connections to be renewed, resulting in events accumulating at the ingestion layer.
Additional capacity has been provisioned to handle the queued data, and events may be inserted out-of-order during this period. It's expected to take around 6h until the queue levels normalize.
Affected services
Created
Oct 21, 2024 at 10:50am UTC
Some tenants are experiencing event data contention, that results in stale data being displayed on the dashboards. Our team is already investigating the issue and will update the status as soon as new information is available
Affected services