Sharding, Explained

What is sharding?

Sharding is a form of database partitioning, also known as horizontal partitioning.

The process involves breaking up a very large database into smaller, more manageable segments, with the idea of improving performance and reducing the query response time.

Sharding is not a new concept and has been around in traditional, centralized database management since at least the late 1990s. The term was actually popularized by one of the first massively multiplayer online role-playing games (MMORPG), Ultima Online, in which developers split players across different servers (different “worlds” in the game) to cope with the traffic.

In business, a common example of sharding a large database is to break up the customer database into geographic locations. Customers in the same geographic locations are grouped together and placed on unique servers.

What is sharding in terms of blockchain?

It is essentially the same process.

The blockchain network is the database with the nodes representing individual data servers. If we apply sharding to blockchain, this would mean breaking up the blockchain network into individual segments (or shards). Each shard would hold a unique set of smart contracts and account balances.

Nodes would then be assigned to individual shards to verify transactions and operations, instead of each being responsible for verifying every transaction on the entire network.

The idea is that, by breaking up the blockchain into more manageable segments, it should lead to increased throughput of transactions and therefore overcome the scalability issues faced by most of the major blockchains today. We’ll look at this in more detail a bit further down.

How does sharding work?

To explain sharding, let’s use the Ethereum blockchain as an example.

The Ethereum blockchain is made up of thousands of computers, aka nodes — currently 8,622, according to ethernodes.org — each lending a certain amount of hash power to the network. It is this hash power that allows the Ethereum Virtual Machine (EVM) to function — i.e., to execute smart contracts and run decentralized applications (DApps).

At the moment, Ethereum works on a sequential execution basis in which every single one of these nodes has to calculate every single operation and process every single transaction. A transaction passing through this verification process can, therefore, take a long time. Currently, Ethereum is hovering at around 10 transactions per second (TPS) — Visa, for comparison, does 24,000 TPS on average. Adding computers to the network will not necessarily improve efficiency, as the whole ledger is kept on every single computer and the chain of verification will just become longer.

With sharding, the idea is to move from a linear execution model, in which every node has to compute every operation, to a parallel execution model, in which nodes are assigned to process only certain computations. This will allow for multiple, parallel transaction processing at the same time.

The blockchain will be divided into separate shards (subdomains, or “buckets”). Nodes will only have to run the part of the ledger that they are assigned in order to execute processes and validate transactions, instead of maintaining the whole ledger all of the time.

What problem will sharding potentially solve?

Sharding is a potential solution to blockchain’s ongoing scalability issues.

As mentioned above, one of the biggest problems facing blockchain networks is the issue of scalability. The more popular a blockchain network becomes, the more users are initiating transactions, decentralized applications and other processes on the network.

Increased transaction activity places increased demand on nodes to verify transactions, and there’s a real threat that these blockchains could become clogged up (as seen on Ethereum during the CryptoKitties craze, when the game accounted for 11% of transactions on the network). If this happens, transaction speeds become painfully slow, which is not an ideal situation for long-term, sustainable blockchain adoption.

As explained above, if the blockchain is broken up into smaller segments, with teams of nodes assigned responsibility to individual segments, every node won’t have to maintain the entire ledger to execute every operation. Transaction validation can, therefore, happen in parallel rather than in a linear fashion, increasing the speed of the entire network. It provides a solution to the scalability issues surrounding blockchain networks and therefore makes it more sustainable in the long term.

Are there any drawbacks to sharding?

Currently, the main challenges of sharding relate to communication and security.

If you split a blockchain into isolated segments, each shard will appear as a separate blockchain network. Users and applications of one subdomain will not be able to communicate with users and applications of another subdomain, without the implementation of a special, intershard communication mechanism. This adds an additional layer of complication for developers to think about.

In a segmented blockchain, security also becomes a concern, as it is easier for hackers to take over a single shard due to the reduced hash power required to control individual segments, also known as a single-shard takeover attack or a 1% attack. Once a segment has been hacked, the attackers can potentially submit invalid transactions to the main network, or it is possible for information in that specific segment to be invalidated and lost permanently. Ethereum’s proposed answer to this security risk is random sampling, where shard notaries are randomly appointed to different sections to verify block authentication.

Are there alternative solutions to sharding?

Two additional proposals have been suggested by developers to improve performance and transaction speed on blockchains.

The first is to increase the block size, with the basic thinking being that the bigger the block size, the more transactions you can fit into a block — and therefore, the higher the number of transactions per second will be. Although this is true, it also means that the bigger the block size is, the more computing power is needed to verify the block.

If block sizes were to be increased indefinitely, only the most specialized, highly powered computer equipment would be able to handle the required processing power needed to act as a node. The increased cost of this type of equipment would mean node pools would necessarily become smaller and more centralized, increasing the risk of a 51% attack. Increasing the block size would also require a hard fork, which risks splitting the community. If not everyone upgrades to the new blockchain, two separate chains will exist, using two separate coins. Because of these issues, increasing the block size is only a short-term solution.

The second proposal is to use altcoins so that different functions and different applications would run on their own chain with their own coin. This would increase performance because you don’t overload a single blockchain, but it will also increase security risks because all of the hashing power is now split over several blockchains. Again, this makes it much easier to hack the network, as the amount of hashing power needed to execute a successful 51% attack is much smaller. Therefore, it is not a viable solution.

Who uses sharding?

Some blockchains have already implemented a sharding mechanism, while in others, it is still under development.

Zilliqa is the first public blockchain platform to have implemented sharding. It was able to achieve 2,828 TPS in its testnet.

The blockchain ecosystem Near allows developers to easily build and deploy decentralized applications. It also call itself “a sharded, developer-friendly, proof-of-stake blockchain” and state that its sharding technology allows nodes to stay small enough to run on simple cloud-hosted instances — potentially even mobile devices in the future.

Similar to Near, Ethereum provides a blockchain ecosystem for the implementation of smart contract-based, DApps. The Ethereum Foundation is planning to introduce sharding as part of its Ethereum 2.0 update, set for launch in January 2020, as confirmed on an Ethereum Foundation call on June 13, 2019.

Other blockchain projects looking at sharding as a solution to scalability issues include Cardano, QuarkChain and PChain.

What is the future of blockchain sharding?

Sharding gained more attention recently because of Facebook’s Libra coin.

Facebook recently released more details on the Libra coin in its white paper, which is planned for launch in the first half of 2020. It also emerged that Facebook acquired Chainspace, whose developer team was predominantly focused on blockchain sharding. This would suggest that the Libra blockchain will implement some form of blockchain partitioning.

In a more general sense, though, sharding could be the solution that solves blockchain’s trilemma.

In the blockchain trilemma, as described by Vitalik Buterin, co-founder of Ethereum, you can only ever maintain two out of the three blockchain core characteristics at the same time — namely, security, decentralization and scalability. That is because you will always have to compromise on at least one of these elements in order to have the other two.

As Beniamin Mincu — the CEO of the Elrond Network, a blockchain protocol focusing on scalability and interoperability — hinted at in a recent Forbes article, if the sharding challenges can be overcome, it will bring scalability to blockchains without compromising on decentralization or security. This, in turn, could bring blockchain one step closer to sustainable mainstream adoption. Mincu said:

“Sharding is complicated, but it provides the type of throughput capacity improvement that enables public blockchains to rival networks like VISA. Some of its challenges, though, include single-shard takeovers, cross-shard communication and data validity."