Update - Resolved.
Mar 07, 2025 - 12:53 CET
Mar 07, 2025 - 12:53 CET
Update - All data written to Clarify should now be accessible in the API. However, since we have cleared our time-series cache, query performance may be somewhat reduced in the coming hours.
Work that we have done include:
- Deployed a new and highly efficient ingestion pipeline, backed by a new and more scalable persistent storage schema.
- Safely migrated all existing clarify data to the new storage backend.
We do not expect any data loss for data written to Clarify during the incident. However, if you suspect data loss for any data that is critical to your operation, please contact support.
Mar 04, 2025 - 14:36 CET
Work that we have done include:
- Deployed a new and highly efficient ingestion pipeline, backed by a new and more scalable persistent storage schema.
- Safely migrated all existing clarify data to the new storage backend.
We do not expect any data loss for data written to Clarify during the incident. However, if you suspect data loss for any data that is critical to your operation, please contact support.
Mar 04, 2025 - 14:36 CET
Update - We where able to successfully deploy the new ingestion pipeline yesterday. This means that queued up data is getting written to persistent storage. The new solution have proved significantly more performant than the old one, and it does not run into any quota issues with our cloud provider.
However, there are still some more work to do our end before historical data will become visible in Clarify. We will keep you posted of any significant development.
Feb 28, 2025 - 08:10 CET
However, there are still some more work to do our end before historical data will become visible in Clarify. We will keep you posted of any significant development.
Feb 28, 2025 - 08:10 CET
Update - We have completed testing of the new ingestion system, and are ready to start rolling it into production tomorrow. It is expected to take some time for us to roll the new solution into production safely. In the meantime, historic data access may be limited.
Feb 26, 2025 - 19:28 CET
Feb 26, 2025 - 19:28 CET
Update - We have had to shut our ingestion system down again due to Cloud provider issues.
Feb 22, 2025 - 01:03 CET
Feb 22, 2025 - 01:03 CET
Update - Our quota have been restored, and the existing ingestion solution has been started up to keep the backlog from growing.
We will continue to work with our Cloud provider to get a new ingestion solution up next week.
Feb 21, 2025 - 16:08 CET
We will continue to work with our Cloud provider to get a new ingestion solution up next week.
Feb 21, 2025 - 16:08 CET
Update - A quota on our cloud storage provider has been reset again so we have had to pause our ingestion services. We are working with our cloud storage provider to get the quota restored so that we can continue processing the backlog, as well as to progress faster on putting our new ingestion system into production.
We are very sorry for the disruption this causes our customers, and we want you to know that we take the issue very seriously on our end.
Feb 21, 2025 - 10:44 CET
We are very sorry for the disruption this causes our customers, and we want you to know that we take the issue very seriously on our end.
Feb 21, 2025 - 10:44 CET
Update - We are continuing to monitor for any further issues.
Feb 20, 2025 - 13:40 CET
Feb 20, 2025 - 13:40 CET
Update - I have upgraded the severity of the incident to reflect the negative effects we feel this may have on our customers. We apologize for the inconvenience and disruptions this may have to your processes.
The backlog queue of messages is still being processed, although very slowly. We estimate that it will still take several days to completely drain the backlog. We can not make it run any faster with our current solution due to hard quota limitations at our Cloud provider's side. I want to emphasize that we don't currently believe any data to be permanently lost due to this incident.
If it's to any comfort, we are currently in the final testing stages of a new ingestion solution which will be able to handle significantly higher load, and protect against future down-time.
Please subscribe to our status-page if you want to keep yourself up-to-date with any development.
Feb 20, 2025 - 13:37 CET
The backlog queue of messages is still being processed, although very slowly. We estimate that it will still take several days to completely drain the backlog. We can not make it run any faster with our current solution due to hard quota limitations at our Cloud provider's side. I want to emphasize that we don't currently believe any data to be permanently lost due to this incident.
If it's to any comfort, we are currently in the final testing stages of a new ingestion solution which will be able to handle significantly higher load, and protect against future down-time.
Please subscribe to our status-page if you want to keep yourself up-to-date with any development.
Feb 20, 2025 - 13:37 CET
Monitoring - The issue on the Cloud provide has been resolved today, and data is ingesting to persisted storage as normal. It will take some time for the processes the backlog of messages.
Data that was loaded between Feb 13 and today, may still appear to have holes in the data in Clarify. Once the current backlog is handled, we will reset the cache to deal with this situation.
In our last update, we stated that a small portion of data written before Feb 18 at 10:58 could be lost. After further inspection of our metrics, we believe this data to have been recovered. In other words, there appears to be no permanent data loss caused by this instance.
Feb 19, 2025 - 23:46 CET
Data that was loaded between Feb 13 and today, may still appear to have holes in the data in Clarify. Once the current backlog is handled, we will reset the cache to deal with this situation.
In our last update, we stated that a small portion of data written before Feb 18 at 10:58 could be lost. After further inspection of our metrics, we believe this data to have been recovered. In other words, there appears to be no permanent data loss caused by this instance.
Feb 19, 2025 - 23:46 CET
Update - We are continuing to work on a fix for this issue.
Feb 18, 2025 - 15:51 CET
Feb 18, 2025 - 15:51 CET
Identified - We have identified an issue with our Cloud provide between now and Feb 13 2024. The effect on customers is that items that becomes active now (accessed for first time in the last 14 days), may contain holes in the data between Feb 13 and the time the item was first loaded.
The vast majority of this data is retained in our queue, and will become available once our ingestion pipeline comes back online. However, a small portion of the data written before Feb 18 at 10:58 may have been lost.
Feb 18, 2025 - 15:46 CET
The vast majority of this data is retained in our queue, and will become available once our ingestion pipeline comes back online. However, a small portion of the data written before Feb 18 at 10:58 may have been lost.
Feb 18, 2025 - 15:46 CET
Clarify
?
Operational
Auth0 User Authentication
Operational
API
?
Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.