Background

We’re sorry for yesterday’s incident that affected the Serialized Dashboard and the Projections API. We noticed higher error rates in the projection API in the morning and later during the day dashboard also was affected.

While some of our systems were affected, your aggregates and events were completely unaffected by this incident.

Problem identification

After some digging in logs, we discovered that a retry-mechanism of the projection processing did not work as expected. A couple of weeks ago, a new batch processing mechanism was introduced to increase the processing speed of projections. Unfortunately, this retry was not compatible with the new batch handling, which caused partial updates to projection data which made the Dashboard unavailable (since it is also dependent on projections).

We discovered that several projects were affected by the problem and we decided to rebuild the affected projections to avoid any data inconsistencies.

Technical measures

• Error handling in the retry mechanism was improved
• Affected projections were rebuilt and projection data was verified

Conclusion

We know how much you rely on Serialized for your projects and businesses to succeed. We’re passionate about the availability of our services and the correctness of your data. We will continue to analyze this event and continuously improve to serve you better and earn the trust you place in us.