Polkadot 2019: Year in review

Well, it’s already the end of 2019; an annual retrospective over what we achieved, where we are now and our plans for the next 12 months.

I started the last year-end round-up with some statistics, and it seems fitting to continue the trend and report some numbers from this year. In total, the Polkadot project, including Substrate, Grandpa, Cumulus and a few bits of support code, has over three hundred thousand lines of code! That’s three times as much as was done last year, and it includes neither our UI nor much of our smart contract codebase. 121 people made these lines of code happen, around half of whom are Parity employees, with the other half being community contributors. The code was added in a little under 3,000 individual contributions, well over double the amount we had at the end of last year. There are now very nearly 1,000 people in the Substrate Technical English-language chatroom and the Substrate community has answered a whole lot of questions relating to Polkadot, Substrate and parachain development that have been answered there.

Teams & Community

The Polkadot ecosystem began with some accomplished teams but this last year has seen some very notable migrations to the ranks. Chronicled, a supply-chain startup operating out of San Francisco announced an upgrade of their Mediledger Network from Parity Ethereum to Substrate, with the eventual goal of using Polkadot for connecting diverse supply-chain networks to each other and allowing for decentralised, integrated supply-chain data-systems.

Around a month ago, Polymath, the comprehensive blockchain-based token securities platform, announced that they had been quietly building out their new platform Polymesh on Substrate for the last nine months. In the announcement they went into detail concerning the various reasons they had come to regard Substrate as the best solution out there, having researched several alternatives. “The decision process involved many variables, involving technology, roadmaps, product fit and community aspects.” Finally, Substrate was selected for a number of factors they found that other frameworks could not deliver, including the modular framework (“Frame”) for composing parcels of business logic, the forkless upgradability, governance and the overall community. On behalf of the community, thanks for the shout ;-)

TheShift Project, a sharded, decentralised cloud hosting service based around IPFS, recently announced that “after conducting weeks of research on the various modular cores currently available”, they would be migrating their implementation over to the Substrate/Polkadot. The three clinching factors included Substrate’s hybrid consensus, the cutting-edge staking system and the off-chain worker framework.

Blockchain.com, the well-known Bitcoin (and now general crypto) wallet/block explorer, announced plans earlier this year to build support for Polkadot’s token and governance right into their toolset. “Blockchain’s over 41M Wallets will not only be able to store, send, and receive DOTs, but also drive the network’s governance by voting on key questions like proposed amendments to the network protocol and council members”. With Polkadot‘s emphasis on solid governance apparatus, integrations like these are invaluable in ensuring good levels of stakeholder-inclusion.

A huge number of workshops, both online and in-person were given around the world on a variety of topics — validation, Polkadot basics, Kusama, addresses, security and Substrate. I witnessed some great work in helping teach Substrate during various community events; I’ll give a particular shout to the Plasm team in Tokyo for running a great seminar on Substrate that I was lucky enough to drop in for. The same team recently announced the release of their final testnet; this continues to be one of the most exciting projects in the Polkadot space and I’m looking forward to playing with it when I get a bit of time.

Our global ambassador programme has now hit 111 people from thirty-seven countries, with 3,700 members in the various Polkadot/Substrate meetup groups. After visiting twenty cities around the world (including three tours of China) and hosting fifteen events with builders, media and influencers in San Francisco alone, 2019 certainly seemed like a major year for Substrate in terms of global outreach. In each city we see renewed vigour by various teams using Substrate and Polkadot to build the next wave of products. From awesome hardware wallets, de-fi products, all the way through to clothing (thanks for my hoodie, Goda :-), I’m always impressed by the entrepreneurial spirit. We will strive to deliver a platform worthy of such an appetite.

We are seeing some really interesting tech coming from team both insides and outside of the Web3 Grant programme, indeed one impressive thing about the Polkadot (and Kusama) community is the energy and productivity shown by its volunteer contributors. Though there are far too many projects in the space these days to be able to give a shout to all of them (I’m talking about you, Centrifuge, Laminar, Acala, KILT, Nodle, MXC, Celer, OAX, Katal, imToken!), I’ll give a quick special mention to Polkascan, whose block explorer is looking ever better — thanks to Emiel for being so attentive to my exploratory needs. Polkastats, a great resource for staking on Kusama, recently released version 2.0 of their site and it’s looking great. Chevdor, a long-time dotter, recently released srtool a means of doing effective deterministic builds of the Polkadot runtime, which is super helpful for verifying runtime upgrades. Subsocial, a general set of Substrate and IPFS-based social-network tools is looking awesome. There are ideas on the table to purpose this for the Kusama, and perhaps Polkadot, governance platform. I think that this is exactly the sort of thing that we’ll be seeing supported increasingly into 2020 through the Polkadot and Kusama treasuries.

Organisations and Sale

Parity has now grown to well over a hundred awesome people and Web3 Foundation, also expanding its ranks, sits at around fifty now. The teams have been expanded with some great hires this year like Ximin (coming to W3F from Dfinity as our senior networking researcher), Yaoqi (coming to Parity from CTO of Ziliqa) and Christine (coming to Parity and W3F from ConsenSys as our CMO). We’re not aiming to grow for the sake of growing, but our doors will continue to be open for great people from both inside and outside of the ecosystem.

Earlier in the year, we completed a number of private sales with a little over five percent of the genesis DOTs sold. There was a broad set of global contributors including enterprises like Innogy (who also expressed an intention to run a validator on the network in their announcement), governance-first VCs like Placeholder Capital and several community teams. The final bunch of DOTs that the Web3 Foundation will be deploying are earmarked for public sale, Kusama stakeholder rewards and long-term ecosystem incentivisation, including for future core developers and community builders who are as-yet uncompensated.

Building it out

Sixty-two projects received grant funding from the Web3 Foundation, across fifty-four teams and twenty countries. Approximately $4.4 million in grants have now been awarded (or are soon expected to be so). This includes two grants awarded for alternative implementations of the Polkadot Runtime Environment (PRE) and one for a full C++ implementation.

The Polkadot Ecosystem Fund was launched alongside Polychain Capital, and we’re excited to see it find its feet among the other great funds in the ecosystem. The Polkadot Incubator was launched with Longhash out of Singapore, with a few teams already in the programme and many more to follow. And, the Web3 Foundation has joined the Proof-of-Stake Alliance (POSA) on behalf of the Polkadot protocol.

Worldwide events

Numerous meetups happened around the world, throughout the year. Polkaworld hackathon saw twenty-two teams compete in Hangzhou, China. Sub0 and Sub0.1 happened in Berlin, seeing over three hundred developers from around the world gather for Substrate seminars, presentations, networking and workshops. We hosted the Web3 Foundation ETHCC track in Paris and Web3 Foundation Ecosystem track at Consensus New York.

The second Web3 Summit took place also in Berlin, seeing more than twelve hundred attendees get together to discuss all matters related to web decentralisation. Edward Snowden was kind enough to address us over video link where he reiterated the dangers of ubiquitous identity provision, increasingly turning society into a permissioned place that is no longer pay-to-play but rather provide-id-to-play.

Polkadot Kusama network launched

The Kusama network (KSM), which we call the “canary” network owing to its status as an early, unaudited (and probably not entirely secure) version of Polkadot, launched successfully as planned. The launch truly demonstrated the advantages of an amorphous network, starting life as a closed proof-of-authority network with validators run by the Web3 Foundation, and progressively opening up staking and governance before becoming a decentralised and permissionless cryptocurrency at the beginning of December. Of the twenty or so upgrades that the network has had so far, the most recent two or three have added KayWhySee, our extensible and federated decentralised identity system as well as new multi-signature, pseudonymity and transaction batching functionality. Many, many more upgrades are coming :-)

As of writing it is maintained by 130 staked validator nodes, fifty or so of them professional validator operations, from around the world with future on-chain developments mulled by an elected council in a public chatroom and ratified through transparent stakeholder referendum. This is open, on-chain governance and it’s happening right now.

Substrate 2.0 launching

Substrate version 2.0 is on the precipice of release with the final issue having been sorted just days ago. This release includes all sorts of awesome features including

• Numerous plug-and-play modules for on-chain governance;
• Run-Ahead, our runtime compiler that means your Substrate nodes run near full speed even when they’re old versions;
• Fast-Forward, our hybrid consensus algorithm which lets us author blocks even in network conditions that would stall other legacy algorithms such as PBFT/Tendermint, and yet lets us finalise tens of thousands of blocks instantly;
• KayWhySee, our extensible, federated identity system; and
• Hand-Off, our technology for effortlessly pairing on-chain and off-chain logic, allowing the chain to hand out arbitrary, non-deterministic work over your network, and even have the workload updated along with on-chain updates.

Compatibility and Bridging

Our four-point plan for ensuring long-term compatibility with Ethereum was published with the feedback being solidly positive. One of the points was duly delivered a few weeks later as promised: Substrate is now compatible with the Ethereum EVM, meaning you can now run Ethereum contracts right inside a Substrate chain! Development on a second point, the Parity PoA Ethereum bridge for Substrate, is well underway and we hope to give a demonstration of that working early in the new year.

In addition to the EVM compatibility our own foray into a smart contract language that I mentioned last year codenamed Fleetwood has morphed and bloomed into the ink! language, now at version 2.0 and looking wonderfully clean and easy to get to grips with. Based in Rust, your smart contracts benefit from Rust’s extensive compile-time correctness guarantees, and its world-class package management, documentation system and test framework. As a core product of Parity and an important piece of Polkadot’s future, our smart contract toolset is something we will be doubling-down on next year, with added documentation, optimisations (like a streaming AOT compiler for native-speed smart contracts) and interoperability infrastructure, particularly for interoperation with Substrate runtimes and the wider Polkadot ecosystem.

The Projects

There has been major progress throughout the key ecosystem projects and components. Parity Signer, the software that lets you turn your unused smartphone into an air-gapped cold-wallet device, is now compatible with Polkadot and Kusama. It supports the extensible HD key derivation, making it really easy to manage all of your accounts. It also works flawlessly with both the Polkadot Web Apps and the Polkadot Chrome and Firefox plugins.

Other hardware wallets are also options; Ledger support for Polkadot is now provided and it will hook right into the Polkadot Web Apps. Completely fresh hardware wallets with native support for Polkadot cryptography are also under development from both Bepal and Wookong.

Speaking of which, the Polkadot Web Apps has progressed at a blinding pace. Providing a beautiful, well-polished experience, it’s an awesome wallet, coming with great key management and key organisation capabilities as well as fully supporting Polkadot’s sophisticated fees and locking mechanisms. It provides great feedback on all on-chain activities, helpfully notifying you when important events like chain upgrades or democratic referenda happen. Polkadot’s staking functionality is fully supported and support for Polkadot’s various other modules like governance and identity is ever increasing.

Polkadot Telemetry has been moved to a high-performance Rust backend, allowing it to support hundreds of nodes across the network all delivering their realtime updates. Substrate Smart Contracts has become increasingly mature, with numerous features implemented and the code polished over the last twelve months.

Our Fast-Forward hybrid consensus system is completed. It’s a combination of the Grandpa finality algorithm, capable of finalising millions of blocks across thousands of validators near-instantly, and the Babe block production algorithm, capable of keeping the chain live even with dramatic network outages. This combination allows us to provide incredible levels of security for each and every one of our Parachain shards while continuously adapting to network conditions. Real-world testing on the Alexander test network and live usage on the Kusama network have confirmed its excellent performance even with high numbers of validators.

In the middle of the year, during a trip to Shanghai, the idea of “parathreads” was born. Parathreads are the technical name for Polkadot’s shards that are not scheduled to be progressed (i.e. have transactions executed) every block. They allow projects an effective “on-ramp” into the Polkadot ecosystem, with the pricing for deployment being fixed and low. Chains are then able to schedule themselves in a pay-as-you-go fashion, processing blocks more frequently when they need to and temporarily suspending their activity when there is less work to do. This offers an alternative, much more agile, economic model to project teams to the heavy-duty parachain slot auctions.

Audits and launch

Key parts of Polkadot have been completed over the course of the year. The initial implementation of the sharding apparatus is complete, supporting both continuously and sporadically scheduled shards (known as parachains and parathreads in the code respectively). Additionally, the Slot Auctions module is finished, as is the Crowdfunding module. These allow teams to permissionlessly procure a shard of Polkadot, possibly with the help of trust-free fixed-term deposits from the public DOT holders. They will both be rolled out to Kusama in the new year.

Two audits of the codebase and design have been completed with broadly positive outcomes. The final slew of external audits has now begun: that’s three of them from three independent, highly respected teams, each covering different parts of the codebase, along with a fourth penetration-testing team just trying to punch holes in our protocol. The results are expected to be returned in the new year.

The final parts of the Polkadot protocol are now well underway in development. The underlying networking, crucial to supporting more than a handful of shards, is now taking a front-seat. Cumulus, which allows Substrate chains to connect into Polkadot as the constituent “Parachain” shards, was successfully demonstrated at the second Substrate developers conference Sub0.1 in Berlin. We expect the first full release of the project early in the new year. Development of XCMP, the cross-chain message-passing system has begun after we released the revised design during November, which takes into account Parathreads.

Wrapping up…

If 2018 was the year of laying the foundations for our new Polkadot house, 2019 was building the bulk of its structure. I think it not a great stretch to see 2020 as the year where we can get the interior decorations done and start moving in. In real terms, this means a Polkadot network launch with a staged rollout of the various pieces of functionality, including governance, parachains, slot auctions, parathreads, XCMP and Spree.

Optimisations will be a focus of the team, aiming to get around Parity-Ethereum levels of TPS by the end of the year on each Substrate-based shard. As well as targeting the CPU and I/O, we’ll also be working on optimising the networking layer away from a “dumb” gossip-based system to something much more sophisticated, which will be needed to get to the numbers of parachains and parathreads that we’d like to see deployed on the mainnet.

On the research end I would hope to see the final designs of Polkadot 2.0 whose key feature is parachain-composability for exponential scalability. Polkadot 2.0 aims to bring support for up to three tiers of parachains, giving a theoretical limit of 1 million parachains, each one operating in a fully parallelised manner. Needless to say, in aggregate, this would enable a mind-bogglingly large amount of trust-free, interoperable transaction capacity.

With that little note, I’ll sign off for now and wish you all happy holidays and a merry new year!