All systems are now fully operational and hot storage has been restored. We are however experiencing higher system load than before the upgrade, resulting in somewhat slower ingestion speeds.
We will work on improving performance in an upcoming release.
Please reach out to support if you continue to experience performance issues or bugs.
Posted Sep 12, 2025 - 01:19 CEST
Update
All major issues appear to be resolved. However, some degraded performance on time-series queries can still be expected for the coming hours. Don't hesitate to contact support if you have questions or concerns regarding the incident.
Posted Sep 10, 2025 - 08:17 CEST
Monitoring
Fix has been deployed to production, and we are now seeing recent data written to hot storage again.
All items in hot storage have been set to reload from cold storage to fill missing data. We will keep monitor the situation to confirm that data is getting backfilled correctly.
Posted Sep 09, 2025 - 16:01 CEST
Identified
We have identified an issue with ingestion of "recent data" to our cache and are testing a fix for it. Data written directly to our API, as well as data written via MQTT after 2025-09-09 00:30 is persisted in cold storage, and will become available as soon as we are able to roll our the fix.
Posted Sep 09, 2025 - 15:05 CEST
Update
In addition to the ingestion issue yesterday, we are seeing some issues with our hot/cold storage sync for time-seres data.
This issue is causing data to becoming unavailable in Clarify and the Clarify API. We can see the data reaching cold storage, and have been able to query it from there. We therefore do not believe we have any permanent data loss after 2025-09-09 00:30 based on our stats.
Posted Sep 09, 2025 - 09:19 CEST
Investigating
We have discovered a partial outage of our MQTT ingestion pipeline between 2025-09-08 13:30 - 2025-09-09 00:30. Everything should be back up and running as normal now.
In addition, there could be degreased performance on time-series queries as we are reloading data into hot storage.