Outages, resilience and the cost of speed

Wonder

1. Why does a chain that produces blocks every 400 milliseconds require coordinated manual intervention to restart after a failure?
2. What specific mechanism allowed a flood of bot transactions in September 2021 to stop the entire network for seventeen hours?
3. How does the arrival of Firedancer change the risk profile of a shared software bug compared to Ethereum’s multi-client ecosystem?

Explore

1. Solana’s architecture prioritizes throughput by executing all transactions on a single, high-speed chain. This monolithic design means that if the client software running on the majority of validators fails, or if load overwhelms validators, block production can halt; since the fixes of 2022-2024, heavy congestion has more often meant dropped transactions than a stopped chain. Unlike modular systems where execution can be distributed across separate layers, Solana concentrates load and risk in one place. A bug in the core client affects every validator simultaneously, leaving no alternative execution path to maintain continuity.
2. The September 2021 outage exemplified this fragility. During a high-profile token launch, a flood of bot transactions overwhelmed the network’s processing capacity. Validators could not keep up with the transaction volume, causing block production to stop for approximately seventeen hours. The recovery required coordinated manual restarts by validator operators. This incident highlighted that while Solana handles normal load efficiently, sudden spikes in activity can trigger cascading failures if the network lacks sufficient headroom or dynamic throttling mechanisms.
3. Subsequent outages continued to test the system’s limits. In May 2022, NFT minting bots caused a seven-hour halt as similar congestion pressures built up. Then, in February 2024, a bug specifically within the program cache halted the chain for about five hours. This incident was not due to external load but an internal software defect. It demonstrated that even without malicious or accidental traffic spikes, a flaw in the client code shared by most validators could bring the network to a standstill until patches were deployed and nodes restarted.
4. Engineers at Solana Labs and other teams shipped several fixes. Switching transaction intake to the QUIC protocol let leaders control and limit incoming traffic, and stake-weighted quality of service shared a leader’s intake capacity among senders in proportion to the stake behind them, so spam could no longer crowd out everyone else. Local fee markets allowed fees to adjust dynamically based on specific account activity rather than global network load, reducing unnecessary congestion for unrelated transactions.
5. Critics argue that the need for coordinated restarts and the historical reliance on a single dominant client create systemic risks. Ethereum, which runs several independent client implementations, has more redundancy: if one client fails, others can continue processing transactions. Solana supporters counter that uptime has improved markedly since February 2024. They emphasize that the network’s performance gains justify the complexity and that the current architecture supports a wider range of applications without the fragmentation inherent in multi-chain designs.
6. The introduction of Firedancer, an independent client developed by Jump Crypto, addresses the client concentration concern. It ran first in a hybrid form from 2024 that still shared much of Agave’s code, and as a fully independent client from December 2025. Only the full client offers real independence: if a bug hit Agave, validators on Firedancer could keep the chain going, provided enough stake runs it. This adds a layer of resilience similar to Ethereum’s multi-client model, reducing the probability that a single software defect will stop block production across the entire network.

Key facts

Exercise nine, six, three, one

Fact check

Dates, figures and claims checked against the published sources listed below.

• Solana Foundation, Network Performance Reports (2021-2024)
• Jump Crypto, Firedancer documentation (2024)
• ethereum.org, 'Client diversity'
• Solana incident reports and status history (2021-2024)