Resolved -
Omnivore API and webhooks have been restored to full functionality.
Sep 11, 12:21 PDT
Monitoring -
A fix has been implemented, with all edge nodes now back in rotation and reporting healthy. We will continue to monitor for any further impacts.
Sep 11, 11:53 PDT
Update -
We are seeing traffic approaching baseline levels. Webhooks have begun firing again. We are still working to improve stability.
Sep 11, 11:45 PDT
Update -
We have partially restored some traffic to Omnivore API. We are still working on full resolution and stabilization.
Sep 11, 10:45 PDT
Update -
We are continuing to work on a fix for this issue.
Sep 11, 10:14 PDT
Update -
We are continuing to work on a fix for this issue.
Sep 11, 10:13 PDT
Identified -
We have identified the trigger to be a bad patch that was released. This patch caused all of our edge servers to become unhealthy. We are currently working to rollback the edge servers to a known good state.
Sep 11, 10:12 PDT
Investigating -
Around 16:47 UTC, the edge nodes serving the Omnivore API went offline. We are currently investigating and hope to provide a better update once we have more information.
Sep 11, 10:05 PDT