500 Service Unavailable Errors Affecting Production Services

Incident Report for Zero Hash

Resolved

This incident has been resolved.
Posted Jan 21, 2025 - 20:59 UTC

Monitoring

A fix has been implemented and we have seen an extended period of successful throughput. We are still actively monitoring the situation.
Posted Jan 21, 2025 - 20:41 UTC

Update

Our team is still working on the fix to bring our services back to a healthy state. We are still all-hands on this issue.
Posted Jan 21, 2025 - 19:11 UTC

Identified

we are experiencing a degradation of services after the initial fix. We are continuing to work on getting systems back to a stable state.
Posted Jan 21, 2025 - 16:57 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Jan 21, 2025 - 16:02 UTC

Update

We are continuing to work on a fix for this issue.
Posted Jan 21, 2025 - 14:46 UTC

Update

We are continuing to work on the resolution for this incident. The issue affects credit check for all RFQ trade executions.
There is a full outage on the below endpoints:
POST /liquidity/execute
POST /convert_withdraw/execute

There is a partial outage on the following endpoint:
POST /fund/rfq deposits are being accepted and queued. The funds will become available once the issue has been resolved.
Posted Jan 21, 2025 - 14:38 UTC

Update

We are continuing to work on a fix for this issue with urgent priority. While we do not have a precise ETA at this time we are hopeful that this will be resolved in the next few hours. We will continue to provide updates as soon as they are available.
Posted Jan 21, 2025 - 11:07 UTC

Identified

We have identified the root cause, and are working towards a fix for the issue.
Posted Jan 21, 2025 - 07:55 UTC

Investigating

We are experiencing an outage across multiple endpoints. We are currently investigating the issue.
Posted Jan 21, 2025 - 02:47 UTC
This incident affected: PROD - REST API (PROD - /liquidity, PROD - /convert_withdraw).