“Where Were You When I Needed You?”
– Where Were You When I Needed You, The Grass Roots
On Saturday, the Solana network went down for over 7 hours. Wow. This is on the heels of an even longer outage in September. The price of Solana responded, dropping almost 10%, though it did recover somewhat. And, the post-mortem of the root cause is even uglier, revealing something much more fragile that anyone expected.
If you use the Solana ecosystem, you would have noticed that it went slow, and then basically failed on Saturday. All of the normal services, wallets and infrastructure basically collapsed. The cause was traced to an NFT minting tool known as Candy Machine. This created a huge amount of traffic which overwhelmed bits of the infrastructure. According to the Solana foundation, at its peak, there was 100 gigabits of data every second, a record for the network.
At its core, this sounds like some sort of DDOS attach, where distributed bots launch enormous amounts of data at a network or infrastructure to try to take it down. From a security perspective, this is sort of a Security Class 101 type attack — at least in theory. DDOS attacks have been around since the advent of the internet, and probably before that, on local networks and pre-internet systems.
This of course begs the question, “How could such a theoretically simple attack succeed?” There are a few possibilities, none of them particularly good. First off, it could simply be a bug in the network that was exploited in some way. While this is likely, it doesn’t speak well of a network which is several years old. Other possibilities are also unsatisfying, such as an insider attack of sorts, or simply not enough testing or preparation for an attack of this scale. It looks like something isn’t quite right with Solana, going deeper than just this outage.
In addition, some other information, according to Coindesk (https://www.coindesk.com/tech/2022/05/01/solana-goes-dark-for-7-hours-as-bots-swarm-candy-machine-nft-minting-tool/) emerged that Co-Founder, Anatoly Yakovenko, was traveling during this time, which delayed the response. For a distributed network, this is particularly disturbing. If one person is, and remains super-important in the continued uptime of the network, it doesn’t seem particularly distributed. In addition, if only one person is able to understand and respond…watch out!
The eventual solution also wasn’t particularly confidence inspiring. One of the infrastructure providers raised the price of NFT minting to .01 SOL, effectively creating a BOT-ing penalty. While this worked, it didn’t in anyway solve the underlying problem of a DDOS attack, and simply showed the vulnerability of a low cost, real-time network to an awful lot of transactions, forcing it to, at least temporarily, become a higher cost network.
Solana will eventually figure out what went wrong and solve it. The fear is that it’ll be a one-off fix, and there is something deeper here.
However, while many crypto-haters will look at this as another datapoint that crypto, “isn’t ready for primetime,” it’s important to remember that networks like Solana are still relatively new. In 1999 and 2000, it wasn’t uncommon for the internet to, “have a bad day” caused by things as simple as the I Love You virus script. Eventually, crypto will overcome this too. Unfortunately, it’ll undoubtedly be quite a few more bad days until we get there.