The recent Ethereum outage has left many questioning just how decentralized Ethereum is, and how something similar can be avoided in the future.
Ethereum experienced dramatic network outages and a possible chain split on Wednesday morning, Nov. 11, caused by infrastructure provider Infura’s use of outdated client software and an unexpected code change. Ethereum’s technical woes resulted in knock-on data and transaction issues on several platforms like Binance, MetaMask and Coinbase Wallet while developers scrambled to determine the cause and fix it.
With Ethereum 2.0’s official Dec. 1 launch date soon approaching, the network’s issues did not noticeably impact the price of ETH, but it revived an old discussion over whether the world’s second largest cryptocurrency is truly decentralized, due to the behavior of its developers and what some see as an over-reliance on third parties for services.
What Actually Happened
Infura’s outages, which affected their mainnet API, archive data, filters, logs, websockets and more, were detected around 7AM UTC and soon forced crypto exchanges and wallets like Binance and MetaMask to temporarily suspend Ethereum and ERC20 token withdrawals, as they couldn’t get accurate price data or complete transactions. Meanwhile, distressed investors and projects were demanding to see where their ETH and ERC20 tokens had gone.
Binance and others announced on social media that Ethereum had undergone a network split as a result. CEO Changpeng “CZ” Zhao shared that his exchange was investigating due to conflicting blockchain data shown on block explorers Etherscan and Blockchair.
Developers soon pinpointed the root of the problem to Infura’s outdated client software, which had been affected by Ethereum developers’ stealth deployment of new code that fixed a consensus bug — but Infura hadn’t updated their nodes to this version.
According to Nikita Zhavoronkov, Blockchair’s lead developer, it appeared that Ethereum developers introduced a code change that resulted in a “hard fork” or network split taking place at block 11234873 just after 7AM UTC. This separated third-party applications using outdated Geth versions (like Infura’s clients) from those who ran the latest software, and moved them to a minority chain. Zhavoronkov compared the episode to a similar Bitcoin issue that took place seven years ago.
After Infura updated Geth, services and withdrawals were quickly restored, and the issue was officially resolved by Infura around 2PM UST.
Infura Defends Using Outdated Geth Software
Infura is a major infrastructure provider that operates full nodes used by Ethereum’s ecosystem and is viewed by many as its AWS. It’s also owned by ConsenSys, founded and managed by Ethereum co-founder Joe Lubin.
Infura provides some of the biggest Ethereum and ERC20 projects and businesses with easy access to the blockchain’s applications and services, as well as giving its developers the necessary scale and tools needed to deploy Ethereum’s smart contract-based applications from a test environment onto its live network. Instead of having to build and maintain their own independent nodes, companies can simply use Infura’s, much like tech giants like Netflix and Spotify use Amazon AWS’s cloud services.
According to a Infura post-mortem on its blog, a consensus bug which impacted Geth versions v1.9.9 and v1.9.13 resulted in the stalling of block syncing across several internal subsystems.
The company defended not running the latest Geth version v1.9.23, saying it had decided a while back to take a more “frugal” approach to new client updates from the likes of Geth and Parity after several bug-related issues in the past. Infura now takes its time to ensure the stability of both the update and its network.
Competing Ethereum infrastructure providers like BlockCypher and Alchemy appeared to be unaffected by the Geth issue.
Is Ethereum Decentralized Enough?
Industry leaders were quick to respond to the crisis, citing the ecosystem’s over-reliance on centralized infrastructure such as Infura. Bitfinex CTO Paolo Ardoino (whose exchange runs their own Ethereum nodes) acknowledged the value of the service offered by Infura to the industry, but said that exchanges should run their own nodes to avoid falling in the paradoxical “trap” of depending on centralized entities while operating in a decentralized industry like crypto.
Others took the issue of centralization further, placing the blame on Ethereum developers for altering the network code without properly broadcasting this to its community and allowing them to update their software.
The Ethereum Foundation’s Péter Szilágyi admitted the new code technically caused an unannounced hardfork, from a “bad chain to the good one,” but defended the foundation’s actions as striving to minimize the potential impact:
"Silently fixing a bug dormant for 2+ years has a much lower chance of causing a disruption than raising awareness to it."
While the Ethereum community furiously debates the importance of true decentralization across its growing but occasionally fragile ecosystem, a few recommendations can be made:
- Ethereum projects can help decentralize and stabilize the network by running their own independent nodes;
- Ethereum infrastructure providers should update to the latest client versions where possible;
- Ethereum’s developer team should be more transparent in how they roll out updates and code changes and give more fair warning to projects;
- Ethereum devs should also try to simplify the process of implementing and maintaining an independent node.
As the DeFi sector continues to add even more complexity and security concerns to the Ethereum network, the upcoming Ethereum 2.0 network is seen as a way to provide the scale and ease-of-use needed by its ecosystem.
