The peer-to-peer network layer is crucial for a decentralized network. Responsible for node communication, the network layer takes care of node discovery, transaction, and block propagation across the network, underpinning all other functions.
At the initial stage of CKB (Common Knowledge Base) development, we have investigated considerably into candidates solutions, but none could meet the CKB needs ideally. We ended up considering setting up a peer-to-peer network layer in-house so that CKB’s needs can be fulfilled accordingly.
Plus, as the Nervos ecosystem keeps growing, we can easily follow up on new requirements and solve problems in time. However, building a new stable and efficient peer-to-peer layer is highly demanding. We nevertheless decided to design a custom peer-to-peer network layer for CKB.
CKB has been operating smoothly for nearly three years. Since the launch of CKB, Tentacle has been running in an open and complex environment. Today, Tentacle has evolved into a reliable and full-featured peer-to-peer network library.
That’s a little bit of Tentacle’s story. As an early member of the developer team, I’m happy to share my experience with you in this article so you can have a basic understanding of Tentacle, why we decided to create it and its working principles. I’ll mainly focus on customized packaging, multiplexing and encrypted communication. After reading, you may be eager to have hands-on practice with Tentacle yourself. No problem! Tentacle is open source under MIT License and can be implemented in Rust. I’ll attach the GitHub link at the end of this post, feel free to try it out in your own project.
Overall Design
Before we start, we need to first understand what CKB aims to achieve.
These goals are:
- Ensuring stable and efficient communication
- Supporting encrypted communication
- Enabling message broadcasting and priority queues
- Assisting parallel multi-version, multi-protocol communication
- Providing fast, lightweight switches between protocols
The first requirement is the fundamental guarantee, while the rest are functional guarantees. I’d like to briefly point out that supporting fast and lightweight switches between protocols are particularly prepared for CBK's hardfork, which is scheduled to complete in the first half of 2022.
Given the requirements above, a few protocol candidates are available: UDP (User Datagram Protocol), QUIC (Quick UDP Internet Connections), and KCP Protocol. We first ruled out the UDP-based solutions. Building a reliable UDP-based streaming protocol is feasible but terribly time-consuming, and requires a comprehensive design of a whole set of control protocols to deliver the same level of performance as TCP, but may not necessarily be more efficient.
Our better candidates are QUIC and KCP Protocol, TCP’s strong competitors. QUIC outperforms TCP when it comes to the Head-of-Line Blocking Problem. QUIC can reduce the handshake overhead during the connection establishment stage and the consumption due to packet loss and retransmission. However, since QUIC was far from perfect in the early stage of CKB development, we would have to implement it all by ourselves based on the RFCs. Plus, most connections in a peer-to-peer network are long connections. Different from an HTTP scenario, a slow handshake is not considered as a problem here.
So, we want to build our peer-to-peer network library based on TCP. Such a library must be well-layered, where each layer functions in a relatively independent manner with minimal complexity, thus providing maximum flexibility for developers while reducing development complexity and enhancing testing convenience. The last two properties are crucial guarantees for high-quality code.
The internal composition of Tentacle is a layered protocol stack illustrated below. From left to right, the protocol layer levels move from high to low.
Figure 1. Processing user data through Tentacle’s layered protocol stack
Let me further explain this process:
- User data is pushed to Tentacle through user-defined protocols from the upper layer applications, e.g., CKB
- Data is processed by Tentacle according to the protocol defined by developers, then transmitted to Yamux (Yamux is a multiplexing library. Don't worry if you don't understand. I'll get to it soon).
- Protocol data is uniformed in a single format by Yamux and transmitted to secio (secure I/O)
- Secio encrypts the data from Yamux and transmits it to the underlying streaming protocols, e.g., TCP
- The remote end receives the data and processes them in reverse. It parses data from the underlying streaming protocol into user data via the 3-layered Tentacle protocol, then pushes the data back to the receiver on the application’s end.
From the perspective of data transmission, the above flow can be illustrated in the diagram below:
Figure 2. Peeling off the protocol layers in user data processing
As you can see, for data sending, Tentacle converts the upper layer data into its own format, then passes the data to the lower layer right next to it. It’s done layer by layer, like peeling an onion; for data receiving, Tentacle follows the same steps in the reverse direction.
Tentacle: Custom Encapsulation
On the top layer, Tentacle has to take care of the following tasks:
- Implement a priority queue of tasks to support the priority delivery of certain messages
- Implement encapsulation of the underlying streaming protocols, to unify the multiple behaviors of streaming protocols
- Unify the abstract protocol behaviors, to allow users to define various protocols with the same socket
Tentacle supports the construction of a complete communication system on TCP, TCP + TLS, WebSocket, and WebAssembly. Because it requires encapsulating the underlying protocols and providing a unified socket, the concept of multi-address is thus introduced into Tentacle. Multi-address is used to unify the address formats of the underlying protocols, and to identify the listening and broadcasting operations users want to perform.
Tentacle regards developer-defined protocols as the first objects, meaning that communication is impossible if a protocol doesn’t exist. CKB’s main communication protocol, Sync, for synchronizing the latest transactions and clocks, or Relayer, for synchronizing the latest transactions and blocks, are two examples of such developer-defined protocols.
After abstracting Yamux into protocols, a handshake between the client and the server is needed to open the protocol. Illustrated as below, this handshake is used to determine both sides’ names and version numbers.
Figure 3. The handshake before opening a protocol
As a whole, Tentacle is a peer-to-peer structure, where any node is both the client and the server. In other words, any node can initiate a request to open a protocol by following the steps below:
- The client initiates the connection by sending a message to the server, providing the name of the protocol to be opened and the numbers of all the versions supported
- After receiving the message, the server confirms that the protocol can be supported. Then it selects the latest version supported by both ends and opens the protocol, then replies with the confirmed protocol and version number.
- Once both ends open the protocol, they can further communicate.
Yamux: Multiplexing
Tentacle relies on a Rust implementation tokio-yamux, which we created based on Yamux spec. Tokio is the common asynchronous runtime in Rust.
The core function of Yamux is multiplexing. Multiplexing is to support the transmission of more than one signal or data stream on a single channel. In other words, multiplexing unifies different encoding rules of the data streams and transmits the unified stream to the remote end. Once a unified message reaches the remote end, it will be separated into different data based on the flags, and passed further to the upper layer users. The transmission of multiple protocols on one channel is thus implemented.
Yamux encapsulates the upper layer data in a custom header|data format and records the source in the header. When the data stream reaches the remote end, Yamux distributes the data according to the header. This is how Tentacle enables the parallel communication of protocols of different versions.
The format of Yamux message is easy to grasp. It contains the two parts below:
- Header indicates the type, stream id, length, flag, and other components of the message
- Body stores message content. It is optional.
Figure 4. Sending and receiving protocol data via Yamux in CKB network
On the Yamux layer, all the CKB protocols (e.g. Sync, Relayer) are unified into a single composite message in the Yamux format. The data in CKB protocols are combined into one signal at the Yamux layer on the sender’s end. After the transmission, it will be decomposed into different CKB protocols at the receiver’s end.
Secio: Encrypted Communication
Secio is a cryptographic communication library we have implemented. Despite the use of mature encryption rules and processes, it is relatively simple in terms of design and implementation.
Secio encrypted communication consists of two stages:
- Handshake negotiates symmetric private key via ECDH (Elliptic Curve Diffie Hellman)
- Communication uses AEAD (Authenticated Encryption with Associated Data) Comparison to encrypt and decrypt messages
The write operations from any message must first pass through a complete encryption process, then be assembled in header|data and sent to the underlying streaming socket. The process of reception goes through the same stages but in reverse.
The following diagram plots the interaction between client and server in the handshake.
Figure 5. Five steps in the secio handshake stage
Suppose the underlying protocol is TCP, three interactions happen during the handshake, the last of which is encrypted communication.
The above diagram illustrates the flow as follows:
- After establishing a TCP connection, both ends send messages to the other, including the encryption methods they support, a random nonce, local public key, hash algorithm, and DH Algorithm.
- On receipt of the first message, a temporary pair of asymmetric encryption public-private key is generated locally.
- Each end generates a hash value composed of three components: the hash of the message previously sent, the message received, and the temporary public key. The newly generated hash needs to be signed with the private key corresponding to the local public key. Then they send back the original data which the other end lacks, i.e., the temporary public key, together with the signature.
- On receipt of the second message, each end reassembles the original data locally and verifies the signature with the public key sent from the remote end, to make sure that the remote end is the owner of the private key corresponding to this public key.
- At this point, each end holds the temporary public key of the remote end and its own temporary private key. They can use DH Algorithm to generate a shared value, which is the symmetric encryption key of the initial state. Due to the difference in AEAD Algorithms, the lengths of key/IV (initialization vector) are consequently different. It requires a unifying algorithm to extend the shared key.
- Next, based on the result of the first interaction, both ends select the algorithm that they support. As two symmetric keys have been generated, we use the result of the hash calculation of the raw pubkey to determine which key is for encryption, while the other is for decryption. The order generated by both ends should be exactly the opposite.
- The nonce sent for the first time is encrypted by the generated key, and resent to the other end to verify the key generated by ECDH is correct.
- When the above steps are gone through, the handshake ends.
The process of encrypted communication is shown as follows:
Figure 6. Data encryption and decryption in communication
The stream structure (“secure stream” in the diagram) is what the user end receives after the handshake. This structure internally encapsulates the secio communication in the following steps:
- Encrypted messages are sent to the underlying protocol.
- Once received, messages are decrypted and returned to the upper layer.
Tentacle is a mature peer-to-peer network protocol library used in CKB network implementations. It has been running stable in an open, real-world environment for years.
Feel free to take a look at the source code and try it out in your project. If you have any questions, please feel free to leave us an issue on GitHub.
✍🏻 Written by Chao Luo
Other articles you might be also interested in: