In this research report, Xangle explores the history of Ethereum sharding, before delving into Proto-Danksharding and how the economic structure of rollups will change after EIP-4844.
1. Introduction
This report is set up in two parts. The first follows in the footsteps of Ethereum sharding, exploring how the history of changes in sharding and Ethereum’s long-term vision of Danksharding. And in the second part, we'll look at the structure and implications of Proto-Danksharding and how the economic structure of rollups will change after EIP-4844.
2. History and Evolution of Ethereum Sharding
2-1. Early Ethereum sharding efforts: Hypercubes, Hub and Spoke Chains, Super Quadratic Sharding, Quadratic Sharding
*Quadratic sharding: The blockchain is separated into a beacon chain and a chain of 64 shards, and each shard processes transactions in parallel and shoots headers to the beacon block. Each beacon block contains transactions for all 64 shards, and each shard block is validated by 64 committees composed of Ethereum's validators. The committees are randomly assembled through a process called random sampling (see “Sharding: The Future of the Ethereum Blockchain”).
*Crosslink: A signature of a beacon chain validators or committee that has validated a shard chain block, used by the beacon chain to verify the latest state of the shard chain or to interact with the shard chain.
Chain cross-linking, Source: Vitalik Buterin
Ethereum Roadmap, Source: Vitalik Buterin
2-2. The pursuit of simplicity and pragmatism: Full execution sharding → Data sharding
The trajectory of Ethereum's sharding can be characterized as a persistent pursuit of reduction and simplification. From Ethereum's inception, sharding was consistently proposed as a scalability solution. However, given its technical complexity and implementation challenges, the concept of sharding gradually pivoted toward practicality and simplicity. Naturally, it evolved from full execution sharding, where each shard processed all transactions, to data sharding, where shards merely store transaction data executed by the rollups.
3. Danksharding
3-1. Danksharding, a new blockchain architecture for data sharding
In ETH2 sharding, Ethereum validators are randomly sampled to form committees, which take turns simultaneously creating and validating 64 shard blocks. However, the absence of data availability sampling (DAS) necessitates validators to manually store the entire shard data and provide proof of its availability. This system relies on the assumption that all nodes are honest, as there is no way to verify if a particular validator deliberately withholds data. Moreover, the committee must manually tally each node's vote for shard blocks, which can cause time delays and hinder timely inclusion of shard blocks in the beacon block. As such, the earlier sharding methods were quite complex to implement and carried many risks.
*Blob: Short for Binary Large Objects, Blob is a new data type that is stored only in the Beacon Chain. The concept of blobs was first introduced alongside Danksharding and is expected to be used primarily for storing rollup data.
Join us in showcasing the cryptocurrency revolution, one newsletter at a time. Subscribe now to get daily news and market updates right to your inbox, along with our millions of other subscribers (that’s right, millions love us!) — what are you waiting for?
3-2. Minimizing centralization: PBS (Proposer Builder Separation) and crList
MEVs are predominantly leveraged by a small number of powerful nodes that have the HW to update the mempool swiftly and resources to develop advanced MEV algorithms. Basically, PBS revolves around the idea of splitting the role of a block producer into a proposer and a builder, offering a chance for nodes with lower computing power and fewer resources to obtain MEV. Builders are powerful validators capable of developing advanced MEV algorithms and responsible for ordering transactions and creating blocks. Proposers, on the other hand, have the power to decide which of the blocks proposed by builders will be recorded on the chain.
With PBS, even the most computationally powerful nodes will have to share MEV gains with proposers. This stands in contrast to the previous process, whereas previously a single validator had a monopoly on transaction inclusion and block generation. In this process, the proposer receives the bid amount, and the builder receives the MEV, minus transaction fees and the bid amount.
It's important to note that during the bidding process, builders publish only the block header and the desired bid price first, instead of the entire block data. Releasing the full block data upfront could allow other builders to copy the transaction and take the MEV. This is why the full block data is only released after the proposer has selected the block.
Two-slot PBS architecture | Source: Two-slot proposer/builder separation
- A proposer broadcasts a crList containing all valid transactions in their mempool to a builder.
- The builder constructs a block based on the transactions in the crList and submits it to the proposer. In doing so, the builder includes a transaction hash in the block body that proves that it contains all the transactions from the crList.
- The proposer selects the block with the highest bid, constructs the block header, and notifies the nodes.
- The builder submits a block and a proof that it has included all transactions from the crList.
- The block is added to the chain. If the builder does not submit a proof, the block will not be accepted by the fork choice rule.
3-3. Ensuring scalability and trust: DAS, Erasure Coding, and KZG Commitments
In addition to PBS and crList, the secure and efficient implementation of Danksharding requires the preemptive introduction of several other technologies, most notably: 1) Data Availability Sampling (DAS), 2) Erasure Coding, and 3) KZG Commitment.
Erasure coding mechanism | Source: An explanation of the sharding + DAS proposal
4. EIP-4844: Proto-Danksharding
Though the exact specification of Danksharding has yet to be discussed further, and its actual deployment is expected to be years away, the Ethereum community has made the proactive decision to incorporate some of the parameters of Danksharding, such as EIP-4844, into the protocol during the upcoming Dencun upgrade.
4-1. EIP-4844, the cornerstone of Danksharding
Despite its name, Proto-Danksharding, EIP-4844 does not actually shard the Ethereum database. Instead, EIP-4844 aims to 1) pre-introduce some of the logic required for a smooth transition of the protocol architecture to Danksharding in the future, and 2) introduce blob carrying transactions.
Blobs, the centerpiece of EIP-4844, stand for Binary Large Objects. Put simply, they are chunks of data attached to a transaction. Unlike regular transactions, blob data is only stored on the Beacon Chain and incurs very low gas fees. The purpose of blobs is to dramatically reduce the cost of DA (L1 publication) of rollups by creating a storage space exclusively dedicated to data availability, independent of blockspace. Currently, all rollups use the calldata space to write their data to Ethereum, which is expected to be replaced with blobs. For more information about Ethereum's data storage and why rollups currently use calldata, see annotation below.
4-2. Simply reducing calldata cost can cause block size issues
The idea of reducing the DA cost of rollups has been tossed around since before EIP-4844. Initially, it aimed to simply reduce the calldata fee (gas fee), but it was quickly dismissed due to block size issues. Suppose we want to reduce calldata gas fees to one-tenth. Currently, the average block size on Ethereum today is 120KB, with a theoretical maximum size of around 1.8MB (assuming all 30M gas is used for calldata). If we were to reduce the cost of calldata by a factor of 10, the average block size would still be manageable, but the maximum block size would be 18MB, far beyond what the network can handle. In other words, simply reducing the cost of calldata gas can seriously bloat the network.
Source: etherscan
A refinement of the above idea is EIP4488, which was proposed in November 2021. EIP4488 aims to reduce the calldata gas cost from 16 gas per byte to 3 gas per byte, while enforcing a maximum block size of 1.4MB.
The main concept behind EIP4488 is to boost rollup usage by lowering calldata costs by 5.3 times, while simultaneously avoiding excessive block size growth through a hard cap on the maximum block size.
EIP-4844, on the other hand, creates a new data type called blobs and introduces a fee market for them, which helps lower the DA (l1 publication) cost of rollups. While the implementation of EIP-4844 may take longer than EIP4488, which has an intuitive and simple logic facilitating a faster deployment process, EIP-4844 provides the advantage of introducing necessary updates for Danksharding in advance. Upon passing EIP-4844, execution clients will be ready for Danksharding, requiring only consensus clients to upgrade over time—which is another reason EIP-4844 was adopted.
4-3. Structure and creation of blob transactions
*In fact, 125KB is in no way a small amount of data. Assuming 1 byte per character and an average of 6 characters per word, 125KB is enough to hold about 21,000 English words. According to ChatGPT, 21,845 English words would fill about 77 A4 sheets (font Arial, size 10, line spacing 2).
Source: Deep in to EIP-4844, hackmd(@Yicheng-Chris)
In the BlobTransactionNetworkWrapper, which holds the data of a blob carrying transaction, there are three main components: 1) SignedBlobTransaction, comprising the message and signature, 2) blob_kzg, holding the blob commitment, and 3) blob, containing the blob data.
If you look at the message field in the Signedblobtransaction, you will find that a field called blob_versioned_hashes is added to EIP1559 transactions. blob_versioned_hashes is 0x01 (a 1-byte value representing the version) plus the last 31 bytes of a 32-byte SHA256 hash of the KZG commitment of the blob data. The KZG commitment is a summary of the blob data. The reason for hashing the 48-byte KZG commitment once more to 32 bytes is for forward compatibility* with the EVM.
*The EVM uses a stack-based architecture, where the word size, representing the size of a data element on the stack, is 32 bytes or 256 bits.
Source: miror.xyz(@yicheng.eth)
So, why are blob_versioned_hashes included in the transaction in the first place? This can be answered by looking at the creation process of a blob transaction, which will also explain why blob_kzg and blob are separate from Signedblobtransaction (see figure below).
Here’s how a blob transaction is created. First off, rollup users submit transactions and a sequencer collects them and sends them to the L1 transaction mempool. Subsequently, the Ethereum proposer propagates the blob transaction in the mempool in two parts to the nodes: the Signedblobtransaction as the Execution Payload and the blob_kzg and blob as the Sidecar. The Execution Payload, which contains the actual transaction information, can be downloaded from the execution client (L1 execution engine), and the Sidecar from the consensus client. Notably, the blob data is only stored in the Beacon Chain—meaning that the EVM cannot access the sidecar containing the blob data. This explains why blob_versioned_hashes are included in transactions and why a KZG commitment is necessary.
It is also noteworthy that Sidecar information containing blob data is deleted from the Beacon Chain after 30 days. This means that a blob’s DA is only guaranteed for a month, which Buterin explained is because the role of the consensus protocol is to provide "a highly secure real-time bulletin board," not to store data permanently. Storing blob data permanently on the chain would result in accumulating more than 2.5TB of data every year, potentially overwhelming the Ethereum network. On the contrary, the amount of data, about $20 per 1TB on an HDD, is manageable for individuals. Therefore, it would be ideal to split the responsibility of permanently storing blob data across various protocols that are more suitable for handling large data volumes, including rollups, API providers, decentralized storage systems (e.g. BitTorrent), explorers (Etherscan), and indexing protocols (e.g. The Graph). Moreover, since the execution payload data containing the blob_versioned_hashes data is still stored on the blockchain, it should be possible to prove the validity of the data that is over 30 days old against the original KZG commitment.
5. Impact of EIP-4844 on Rollup Costs
5-1. DA (L1 publication) costs currently account for over 90% of total rollup costs
Before examining the impact of EIP-4844 on the rollup economic model, it is essential to understand the revenue structure of rollups. The revenue generated by rollups is divided into two main components: network fees and MEVs. When we refer to revenue, we are specifically referring to operator/sequencer revenue, which is currently monopolized by the project's operating entity, given that all Rollups run on a single operator system.
Network fee revenue consists of the gas fees that rollup users pay to the operator when making transactions, covering the cost of running the rollup plus the additional fee. Charging users the fee serves two purposes: to generate profit similar to any other business and to mitigate potential losses in the event of a sudden surge in gas costs at the base layer (operators collect gas fees from users before submitting batches to the base layer). As of July 13, Optimism's 30-day average daily revenue and profit stood at 41.4 ETH and 11.5 ETH, respectively, resulting in a 27% profit margin.
Rollup costs can be categorized into fixed and variable costs. The fixed costs encompass three components: 1) state write fee, incurred when submitting the state root to the rollup smart contract, 2) validity proofs (applicable for ZK rollups), and 3) Ethereum base transaction fee (21,000gas). On the other hand, the variable costs are divided into 1) L2 gas fee for transaction operation and 2) L1 publication fee for storing batch data in Ethereum blocks.
Currently, the second part of the variable cost is the dominant factor in terms of cost. Rollups use calldata to store batch data in Ethereum blocks, and the calldata fee is 4 gas/byte for zero bytes (0x00) and 16 gas/byte for non-zero bytes (we provided an explanation for rollup’s use for calldata in the annotation). While 16 gas/byte is considerably economical compared to using storage, it's important to consider that batch data is inherently substantial. As of February/March 2023, when demand for Optimistic rollups surged, Arbitrum and Optimistic were storing 14.6KB and 13KB of data per block, respectively. Consequently, the cost of calldata currently constitutes approximately 99% of the total rollup cost.
5-2. With the implementation of EIP4844, the DA cost of rollups is expected to be almost free, and a significant increase in the blob fee would require the demand for rollups to grow by more than 10 times
As per EIP-4844 parameters, blob fees (gas fees) will be dynamically determined by the supply/demand of blobs within its own data gas fee market, which is not affected by blockspace demand. Consequently, after the implementation of EIP-4844, Ethereum will operate with a two-dimensional EIP-1559 fee market consisting of 1) an EIP1559-based fee market for regular transactions and 2) a blob fee market where fees are determined solely by the supply and demand of blobs.
The functioning of the data gas fee market outlined in EIP-4844 is similar to the EIP-1559 mechanism. Blob transactions encompass all the necessary field values required by the EIP-1559 mechanism, including max_fee, max_priority_fee, and gas_used, with the addition of the max_fee_per_data_gas field indicating the user's priority fee. The data gas structure is also divided into base fee and priority fee, similar to EIP-1559. However, there is still an ongoing discussion about whether the base fee should be burned.
The functioning of the data gas fee market outlined in EIP-4844 is similar to the EIP-1559 mechanism. Blob transactions encompass all the necessary field values required by the EIP-1559 mechanism, including max_fee, max_priority_fee, and gas_used, with the addition of the max_fee_per_data_gas field indicating the user's priority fee. The data gas structure is also divided into base fee and priority fee, similar to EIP-1559. However, there is still an ongoing discussion about whether the base fee should be burned.
If the gas target per block of EIP-1559 is set at 15M gas and the max gas is 30M gas, the TARGET_DATA_GAS_PER_BLOCK of the blob would be 375kb and the MAX_DATA_GAS_PER_BLOCK would be 750kb. By default, the blob data is assigned 1 data gas per byte, and the minimum data gas price is set at 1 wei. Similar to EIP-1559, the base fee can vary up to -12.5-12.5% depending on how much data gas is actually used compared to the targeted data gas (TARGET_DATA_GAS_PER_BLOCK). In the initial stages of EIP4884 deployment, the usage of blobs is expected to be relatively low, even with the high network activity. As a result, the cost of the blob itself is likely to be low during this period.
The EIP-4844 gas fee mechanism is highly similar to EIP-1559 | Source: EIP-4844, Vitalik Buterin
Conclusion 1. The current demand for blobs is about 10x lower than the target level (based on 250KB per block) and is estimated to take about 1-2 years to reach the desired level
- As of January-February 2023, Arbitrum generated 1,055 batches of ~99 KB in size every 6.78 blocks (equivalent to 100 MB per day, or 14.6 KB per block). During the same period, Optimism produced 2,981 batches of ~31 KB in size every 2.37 blocks (equivalent to 93 MB per day, or 13 KB per block). When combining the data generated by both rollups, the average size comes to 27.6 KB per block. Consequently, the current demand for the rollups represents only one-tenth of the targeted blob amount of 250 KB (based on having two blobs per block).
- In cases when the latent demand is higher than the blob target, EIP-4884’s price discovery mechanism leads to an escalation in the price of data gas price, while blob transactions with the lowest willingness to pay are removed to maintain price equilibrium. From January 2022 to December 2022, the total data demand for Ethereum rollups increased by approximately 4.4x. At this pace, which means that it would take a substantial 1.5 years for the price discovery mechanism to initiate and for the data price to exceed 1.
Projected evolution of data gas prices based on rollup demand | Source: EIP-4844 Fee Market Analysis
Conclusion 2. Fees will remain nearly zero until blob demand approaches the target level
- It takes about 1.89M gas to record one batch of Arbitrum to Ethereum. Assuming an average gas price of 29gwei and $ETH price of $1500, the DA cost per batch is about $85 and the calldata cost is about $70. When converting this to blobs, the cost to precompile is about $2.2 (50k gas) and the DA cost is $2e-10 (125k data gas), effectively making it virtually free. However, such a low cost could potentially lead to issues such as spamming or Ddos attacks.
Conclusion 3. In the event that the demand for blobs surpasses the target level, the cost of data gas will experience exponential growth, potentially increasing by more than 10x in a matter of hours
- Once the blob demand reaches the target price, the price of DATA GAS increases exponentially every 12 seconds until less willing transactions are dropped and demand decreases. As the demand for rollups is structurally inelastic and remains steady, this mechanism implies that costs can escalate by a factor of 10 or more in a short span of time. Consequently, blob consumers are expected to face significant challenges in dealing with soaring prices.
5-3. Changing the cost structure of blobs is being actively discussed within the Ethereum community
6. Closing Thoughts
EIP-4844 aims to reduce the DA cost of rollups using blobs and introduce some of the upgrades required to implement full Danksharding in advance. Once upgrades are completed at the consensus layer, such as PBS or DAS, following EIP-4884, full Danksharding can be implemented. However, it is important to note that Danksharding is still in the early stages of research and the Ethereum community has yet to establish a concrete specification. As a result, it is likely to be several years before we witness its actual implementation.
Blob fees are expected to be significantly lower than calldata fees, and based on the current EIP-4844 specification and rollup demand, we anticipate that blob fees will be basically free. Given the trend in rollup demand over the past year, we estimate that it might take about one to two years for blob prices to reach price discovery. In the short to medium term after the introduction of EIP-4844, rollups could potentially offer the lowest transaction fees among all existing blockchains. However, some express concerns about potential side effects and propose either 1) higher blob fees or 2) a lower target number of blobs to expedite the price discovery process. Nevertheless, what remains certain is that EIP4844 will significantly contribute to the widespread adoption of rollups. As the crypto industry gains momentum with recent events such as the XRP ruling and major financial institutions like BlackRock and Fidelity filing applications for a spot Bitcoin ETF, the crypto industry is witnessing remarkable growth. It will be intriguing to see whether EIP-4844 can play a role in ushering in a “rollup summer.”
Annotation) Types of Ethereum data storage
A smart contract is a computer program that runs on a blockchain and consists of functions and data called variables or parameters. The data used by these functions is stored in the computer's (EVM) memory, and the EVM's programming language, Solidity, has four main types of memory: storage, memory, stack, and calldata.
A1. The main storage spaces in EVM are categorized as Storage, Memory, Stack, and Calldata
First, storage is a space used to permanently store data in EVM, like a computer's file system (HDD/SDD). Unlike memory, which exists only while a transaction is executed, variables stored here can be read/write continuously. Each contract has its own storage space, and it can only read and write within its own storage. Storing data in storage is relatively expensive because it is permanently stored on the blockchain. Consequently, storage is primarily used to store the state of a contract, such as account balance, contract owner, or other information needed to execute the contract. Storage is organized into 2²⁵⁶ slots of 32 bytes, and all slots start with a value of zero. EVM provides two opcodes to interact with storage: SLOAD (to load a word from storage onto the stack) and SSTORE (to store a word into storage). Opcodes prefixed with an S indicate storage, while M indicates memory.
Memory is a temporary storage space that exists only for the duration of a function call, similar to RAM on a computer. Memory consists of readable and writable byte-array data and is volatile, meaning that message calls always begin with an empty memory. As memory is not permanently written to the blockchain, it incurs significantly lower gas costs compared to storage. This cost-effectiveness makes memory the preferred option for storing variables that are only required temporarily, such as function arguments, local variables, or arrays that are dynamically created during function execution. The EVM provides three opcodes to interact with memory: MLOAD (to load a word from memory to the stack), MSTORE (to store a word into memory), and MSTORE8 (to store a byte into memory).
The stack serves as a very short-term storage space used for immediate computation, storing small local variables. It is designed to have the lowest gas cost, but it also comes with a size limitation (with a maximum capacity of 1,024 * 256 bits, which is 262,144 bits). The stack is used to store small amounts of data during computation, including parameters for instructions and the results of operations. As a developer, you need not overly concern yourself with the stack because the Solidity compiler takes care of it unless you are using an EVM assembly. Nevertheless, the EVM does offer optional opcodes to modify the stack directly, such as POP, PUSH, DUP, and SWAP.
Note that in most cases, Solidity automatically manages the storage location for variables. Developers typically do not need to manage their own storage except for complex data types like structs or dynamic arrays.
A2. The space used by rollups is called Calldata
Finally, the most critical EVM data storage space to understand in the context of EIP-4844 is calldata. Originally, calldata is a space for storing function arguments, particularly for functions that do not necessitate state changes such as view/pure functions. Calldata is read-only and typically constitutes a byte array containing the function selector and encoded parameters essential for a function call. Consequently, calldata is primarily used when making external function calls between contracts.
Similar to memory, calldata is temporary and stored only during the execution of a transaction. However, this does not imply that all data in the calldata is deleted. While the calldata itself is deleted from the EVM, the transaction that contains the calldata is effectively stored permanently because it is included in the Ethereum block. In other words, while the rollup data is inaccessible to the EVM after the transaction is executed, it remains recorded on the Ethereum blockchain and can be accessed by anyone at any time.
Source: Ethereum Yellow Paper