The largest cryptocurrency exchange in the United States said its autoscaling was unable to keep up with a huge traffic spike that left many users unable to log in on June 1.
According to a blog posted on June 6 by Coinbase software engineer Michael de Hoog, the exchange experienced a 5x traffic spike over four minutes around 16:05 PDT on June 1, around the time the price of Bitcoin (BTC) was approaching $10,000.
The software engineer said the exchange’s autoscaling was “unable to keep pace with this dramatic increase in traffic”:
“This traffic spike affected a number of our internal services, increasing latency between services. This led to process saturation of the web servers responsible for our API [application programming interface], where the number of incoming requests was greater than the number of listening processes, causing the requests to either be queued and timeout, or fail immediately. Our request error rate spiked to 50%, causing customers to experience errors when interacting with coinbase.com and our mobile apps.”
Traffic on Coinbase. Source: Coinbase
Coinbase said it redeployed the API at 16:20 to increase the number of machines dealing with this spike in traffic. Another two-minute outage followed “due to instances saturating and being marked unhealthy” before the exchange was back online.
History of outages
Cointelegraph reported last week that Coinbase has gone offline four times in the last three months during major Bitcoin price moves, leaving many users unable to access their portfolios.
However, Crypto Twitter later revealed the problems on the exchange went further back than that. CryptoWhale posted a chart to his 18,000+ followers on June 3 showing how Coinbase had gone offline 11 times in its history during major price moves:
After doing more research into exchanges uptimes, I've noticed a substandard pattern from #Coinbase.
— Whale 🐋 (@CryptoWhale) June 3, 2020
Their exchange seems to be programmed to go "offline" anytime theres a $500+ move in #Bitcoin's price. Over the last year, Coinbase has gone offline 11 times during larger moves. pic.twitter.com/3fNOU1QuiZ
Unfortunately, outages during huge downturns or price surges hurt traders wanting to buy and sell the most. Losses can mount when Coinbase users aren’t able to access their accounts to sell their crypto, and potential profits can simply disappear when they can’t buy anything.
Will Coinbase go down again?
The exchange said it was working on improvements in response to the June 1 outage. If Coinbase were to experience another traffic spike to the price of Bitcoin suddenly surging or falling, de Hoog said “pre-scaling and caching” would reduce the impact.
“Longer term we’re planning to improve our deployment process to mitigate some of the autoscaling issues we experienced,” the software engineer said.