Database Snafu Caused 6-Hour Outage on Bitcoin Exchange
Shared by John Foley from Cloud Database Report · October 8, 2026
Read the original on clouddb.substack.com
View this post on the web at https://clouddb.substack.com/p/database-snafu-caused-6-hour-outage
Welcome to the Cloud Database Report. I’m John Foley, a long-time tech journalist who also worked in strategic comms at Oracle, IBM, and MongoDB. Connect with me on LinkedIn [ https://substack.com/redirect/77100051-fbd8-45dd-9d64-bbdd40c02a85?j=eyJ1IjoiMTR3eXB3In0.f-xldB9G03IXstk7BAMCqX62DEM2cFHI5JhVsvW8cAA ]. Database reliability is off to an inauspicious start in 2026. Just a few weeks into the new year and already a database glitch has resulted in a system failure that might have caused your heart to stop if you were an affected user. In the early hours of January 19, the price of Bitcoin dropped to $0 on Paradex [ https://substack.com/redirect/f29d74b9-0afe-449d-ad8d-66e100581da1?j=eyJ1IjoiMTR3eXB3In0.f-xldB9G03IXstk7BAMCqX62DEM2cFHI5JhVsvW8cAA ], the crypto exchange that encountered the issue. The problem was blamed on database maintenance gone awry, according to reports [ https://substack.com/redirect/cc605719-ad84-4fe3-9e08-0a7c27160818?j=eyJ1IjoiMTR3eXB3In0.f-xldB9G03IXstk7BAMCqX62DEM2cFHI5JhVsvW8cAA ]. The platform was offline for six hours. I hate to say I told you so but…just a few weeks ago I implored the database community to turn its attention to the challenges of database reliability with greater urgency. “We must do better,” I wrote then, and I repeat it now. As I outlined in that earlier post, the past year was pockmarked by database mishaps, from issues affecting data services on Microsoft Azure in January 2025 to database-related outages at Snowflake in December 2025. And there were other bugs, blips, and breakdowns in the 10 months between those ignominious bookends. I’m reposting the article below in case you missed it. It should be required reading for anyone who develops, markets, manages, or invests in database technology, which is most anyone who subscribes to this newsletter. The obvious question this time is what went wrong at Paradex? Just yesterday (Jan. 23), Paradex provided a post-mortem on X. Here’s its explanation: “On Jan 19, a planned 30-minute maintenance window to upgrade our database (to support growing demand) encountered unexpected issues during the scale-up process. A race condition during a service restart, while critical data operations were in progress, caused a corrupted state to be persisted in the cloud and published to Paradex Chain. As a result, some markets reset the funding index to 0, creating abnormal funding PnL that triggered liquidations across multiple markets.” You can read Paradex’s full incident report on X [ https://substack.com/redirect/a8192bca-2b77-420f-ba90-ab28ee33551d?j=eyJ1IjoiMTR3eXB3In0.f-xldB9G03IXstk7BAMCqX62DEM2cFHI5JhVsvW8cAA ]. And here’s some additional info [ https://substack.com/redirect/7f59837e-e2b1-433f-9f5d-0f147d1e487f?j=eyJ1IjoiMTR3eXB3In0.f-xldB9G03IXstk7BAMCqX62DEM2cFHI5JhVsvW8cAA ] it put out. Paradex says user balances were not affected and there was no loss of funds. Complexity has become ‘the dominant failure mode’ Stepping back from the Paradex incident and taking a broader perspective, the ongoing issues with database reliability can’t be blamed simply on software misconfigurations, disk crashes, network outages, or the dozen other factors that sometimes go wrong. It’s that any of those possibilities can happen at any time. I asked Spencer Kimball, CEO of Cockroach Labs [ https://substack.com/redirect/1e276cb0-87d5-48cb-ba34-2a088a993c9a?j=eyJ1IjoiMTR3eXB3In0.f-xldB9G03IXstk7BAMCqX62DEM2cFHI5JhVsvW8cAA ], which strives for “zero downtime,” why these database troubles continue to hamper businesses and everyday life, even after all of the R&D that has gone into resiliency and “self-healing” databases. “Resilience keeps getting harder because modern databases aren’t failing in one place anymore,” Kimball wrote back. “Data volume, distributed systems, human intervention, and infrastructure dependencies all interact in ways that make simple redundancy insufficient.” “It is not for lack of resiliency tools,” Kimball added. “The problem is that complexity itself has become the dominant failure mode.” Yes. And as I pointed out in my earlier post, as tech stack complexity rises so does the need for resiliency. It’s like a dog chasing its tail. At what point does resiliency overcome complexity? Not for the foreseeable future, IMO. But hopefully the emergence of AI-driven automation, along with wider and more rigorous adoption of best practices like distributed architectures, will lessen the frequency and impact of these all-too-frequent fiascos. Additional reading Thanks for reading Cloud Database Report! This post is public so feel free to share it.