Enhancing Ergonomics and Extensibility of CKB Contract Development with Lua
Being a cross-platform and embeddable scripting language, Lua is well-known for its simplicity and ease of use. To facilitate contract development on CKB platform, we have recently integrated Lua 5.4 to CKB Virtual Machine (CKB-VM) and created CKB-Lua. This allows CKB developers to either run the standalone Lua interpreter on CKB or embed Lua code into the main CKB contracts, thus greatly lowering the barrier of contract development on CKB.
Introduction
CKB-Lua can be used as a shared library (embedded CKB-Lua) or a standalone interpreter. In the embedded mode, it is integrated into a hosting program and called when needed; in the standalone mode, CKB-Lua itself is the main program and loads Lua scripts from the provided arguments. Standalone CKB-Lua is highly ergonomic thanks to the simplicity and ease of use of the Lua programming language. Even programmers without prior experience in CKB contract development can write their first CKB-Lua contract in just a few hours. With the powerful dynamic loading mechanism of embedded CKB-Lua, developers can extend their main contracts in various ways, be it customizing the rules to issue tokens or writing an all-new NFT trading engine in Lua. To further facilitate the development of CKB contracts with Lua, we created a file system called Simple Lua File System that helps with code reuse. Files within this file system can be made available for Lua scripts to read and execute, e.g., running require('mymodule')
and io.open
('myfile')
. This way, we can share common modules between different contracts.
The easiest way to try out CKB-Lua is by building a Lua project with the Capsule template that supports CKB-Lua. You may create an embedded CKB-Lua project with capsule new --template lua-embedded lua-demo
or a standalone project with capsule new --template lua lua-demo
.
You can obtain the source code of CKB-Lua from the GitHub repository, and build the binaries for both modes by running make all-via-docker
. The shared library for the embedded CKB-Lua is ./build/libckblua.so
, while the standalone CKB-Lua interpreter is./build/lua-loader
.
Choosing between the embedded and standalone modes requires weighing the pros and cons, which are outlined below. It's necessary to consider the trade-offs when making a decision.
Embedded
👍 Time-sensitive operations can be completed in the hosting program
👎 Slightly steeper learning curve
Standalone
👍 Easy to learn
👎 May be not efficient enough for some computation-intensive use cases, such as signature verification
Using CKB-Lua As a Shared Library
In this section, we provide a simple example to show how to use CKB-Lua as a shared library. Since the usage of embedded CKB-Lua is almost a superset of the usage of standalone CKB-Lua, I will primarily focus on illustrating the embedded mode.
A sample program that evaluates Lua code with CKB-Lua shared library is dylib.c. Before we dive into the details, let me showcase how to load and unpack script args by a few lines in Lua below:
local _code_hash, _hash_type, args, err = ckb.load_and_unpack_script()
print(err)
if err == nil then
ckb.dump(args)
end
Once familiar with this mode, you can unleash the full potential of Lua with your creativity. For instance, a more advanced and realistic example of implementing Simple UDT (Simple User Defined Tokens, or sUDT) in Lua can be found here.
Also notice that dylib.c is written in C, but both the main program and the test code can be alternatively written in Rust. Loading a shared library and performing unit tests by mocking transactions can be easily achieved in Rust (Here is an example).
I will now explain what the above sample code does and how to run it in three steps.
Step 1 Building CKB-Lua
In the given sample code, the execution of Lua relies on the libckblua.so
shared library. The hosting program, ./build/dylibexample
can be built from dylib.c with make all-via-docker
.
Step 2 Preparing Shared Library
Packing the Shared Library Into the Dependent Cell Data
To use the CKB-Lua shared library, we need to store it as cell data and specify the cell with shared library data as a dependent cell. The simplest way to do that is to generate a Lua project with Capsule. You can also mock this with the CKB Rust SDK (like this).
Passing the Dependent Cell Information With Script Args
Then we pass the cell information to the args of the main C script by performing some standard operations to obtain the args of the current script, i.e., loading the script with ckb_load_script
and then de-serializing the args with Molecule.
After obtaining the args, we can unpack the cell information contained within the args.
In our example, the args has the following format:
// <reserved args, 2 bytes> <code hash of the shared library, 32 bytes>
// <hash type of shared library, 1 byte>
With the size and offset of the fields, we can easily obtain the code hash and hash type of the dependent cell, and open the shared library by ckb_dlopen2
.
Opening the Shared Library and Calling the Functions
err = ckb_dlopen2(code_hash, hash_type, code_buff, code_buff_size, handle, &consumed_size);
[ckb_dlopen2]
is a ckb-c-stdlib function to load the shared library from dependent cell(s) with specific code_hash
and hash_type
. Once the shared library is found, ckb_dlopen2
will write the shared library handle to the handle
parameter. The handle
parameter can then be used to obtain pointers to our Lua functions.
Step 3 Calling Lua Script
We use the following exported functions to call Lua scripts.
Exposed C Functions in the Shared Library
The following four functions are exposed in the shared library:
void *lua_create_instance(uintptr_t min, uintptr_t max)
int lua_run_code(void *l, const char *code, size_t code_size, char *name)
void close_lua_instance(void *L)
void lua_toggle_exit(void *L, int enabled)
lua_create_instance
With lua_create_instance
, a new Lua instance can be created. In order to avoid the collision between Lua’s malloc and the host program’s malloc function, we need to specify the upper and lower bound of the memory used exclusively by Lua (the host program must not use this memory space when the Lua instance is alive). The host program can allocate this space in the stack or heap. The return value of lua_create_instance
is an opaque pointer that can be used to evaluate Lua code with lua_run_code
. We can reclaim the resources by calling lua_close_instance
.
lua_run_code
The evaluation of Lua code can be initiated by calling lua_run_code
. Both Lua source code and Lua bytecode are acceptable to lua_run_code
. Evaluating binary code can be less resource-hungry in terms of both the cycles needed to run the code and the storage space required to store it. This function takes four arguments:
l
: the opaque pointer returned fromlua_create_instance
code
: a pointer that points to the code buffer, which can be either Lua source code or Lua bytecode.code_size
: the size of thecode
buffername
: the name of the Lua program. This may be helpful in debugging.
Lua script may early return control to the host program by calling ckb.exit_script(code)
, where the return code of this function is the parameter code
. In the normal execution flow (i.e., no ckb.exit_script
is called), the return value of lua_run_code will be 0 if the Lua script is executed successfully, or a negative number if the Lua script exited abnormally. Lua script should pass a non-negative number to ckb.exit_script
in order to distinguish the Lua interpreter error code and deliberately set Lua script return code.
lua_close_instance
Calling lua_close_instance
will release all the resources used by Lua, allowing the memory given to Lua reusable. It takes the opaque pointer returned from lua_create_instance
as a parameter.
lua_toggle_exit
This can be used to toggle to enable/disable the ckb.exit
feature. Since ckb.exit(code)
can stop the execution of the whole VM and return code
to CKB-VM, it may be undesirable in some situations, and we disabled it by default. Users may call lua_toggle_exit(l, 1)
to enable it (the parameter l
is the Lua instance).
Lua Functions
Functions in Lua Standard Library
Most of the functions in Lua standard library have been ported to CKB-Lua. All the platform-independent functions are supported, while some platform-dependent functions (e.g., most functions in the module os
) are not.
CKB Specific Functions
Also, we have added a few ckb-specific helper functions to interact with CKB-VM, including those for issuing syscalls, debugging, and unpacking data structures. All are located in the ckb
Lua module.
See the Appendix at the end of this post for a list of exported constants and functions.
Stitching Things Together
Running Lua code is quite easy. First, we load the functions from the previous returned shared library handle, then create a stack space for Lua to use. And finally, we can call the exported functions:
CreateLuaInstanceFuncType create_func =
must_load_function(handle, "lua_create_instance");
EvaluateLuaCodeFuncType evaluate_func =
must_load_function(handle, "lua_run_code");
CloseLuaInstanceFuncType close_func =
must_load_function(handle, "lua_close_instance");
const size_t mem_size = 1024 * 512;
uint8_t mem[mem_size];
void* l = create_func((uintptr_t)mem, (uintptr_t)(mem + mem_size));
if (l == NULL) {
printf("creating lua instance failed\n");
return;
}
int ret = evaluate_func(l, code, code_size, "test");
if (ret != 0) {
printf("evaluating lua code failed: %d\n", ret);
return;
}
close_func(l);
The Lua code below demonstrates how to load CKB script and unpack it into code_hash
, hash_type
and args
. In case of anything goes wrong during the process, an error would be returned as err
. Since the script args can be arbitrary binary data, it may be hard to inspect. We have also added a dump
function to dump the hex and ASCII representations of the data.
local _code_hash, _hash_type, args, err = ckb.load_and_unpack_script()
print(err)
if err == nil then
ckb.dump(args)
end
Using Standalone CKB-Lua
Now that we’ve seen how to use CKB-Lua as a shared library, let’s move forward to the standalone method. For this, we need to specify the Lua code first, which is the main difference between the two modes of using CKB-Lua. The code can be either Lua source code or byte code.
Standalone CKB-Lua Args Format
The args passed to standalone CKB-Lua should be in the following format:
<lua loader args, 2 bytes> <data, variable length>
where the first two bytes are used by CKB-Lua interpreter internally and the format and meaning of data
depend on lua loader args
in the first two bytes.
Currently, the only accepted value for lua loader args
is 0, and the remaining data
should be of the following format:
<code hash of lua code, 32 bytes> <hash type of lua code, 1 byte>
.
In this case, we load Lua code from the data of the cell with the given code hash and hash type.
Simple Lua File System
We can mount the Simple Lua File System so that CKB-Lua programs can reuse the files within the file system. All the files in the File System are readable, executable, and immutable. Moreover, the normal file system attributes (e.g., owner, readability) are irrelevant in CKB-Lua. We can thus represent the Simple Lua File System as a mapping from the file path to the file content. The file system can be stored in any cell whose data are accessible from Lua scripts, and Lua scripts may mount multiple file systems from multiple cells. We may use the script fs.lua to create a file system from given files, or vice versa, unpacking the file system into files. A file system is represented as a binary file which will be displayed soon below.
Mounting File System Within Lua Scripts
Calling ckb.mount(source, index)
will mount the file system from the data of the cell with the specified source
and index
. For example, if you want to mount the file system within the first output cell, run ckb.mount(ckb.SOURCE_OUTPUT, 0)
. The returned value of ckb.mount
will be nil unless an error occurred. In the latter case, an integer to represent the error will be returned. Afterward, you can read and execute files contained in this file system. By calling ckb.mount
multiple times, you can mount several file systems.
Note that files from later mounts may override those from earlier mounts. For instance, if a file named a.txt
is contained in two file systems, the file a.txt
from a later mount is preferred over that of the earlier mount when reading.
Creating a File System
The binary representation of Simple Lua File System is simple. You may think of it as the C struct SimpleFileSystem
as below:
struct Blob {
uint32_t offset;
uint32_t length;
}
struct Metadata {
struct Blob file_name;
struct Blob file_content;
}
struct SimpleFileSystem {
uint32_t file_count;
struct Metadata metadata[..];
uint8_t blobs[..];
}
It consists of three parts:
file_count
: a number to represent the number of files contained in this file systemmetadata
: an array of metadata used to obtain the filenames and file contentsblobs
(alsoblob
): an array of binary objects to store the actual filenames and file contents.
To pack all Lua files within the current directory into $packed_file
, run find . -name '*lua' -type f | lua "./utils/fs.lua" pack "$packed_file"
. Note that all file paths piped into the Lua script must be in the relative path format. The absolute path of a file in the current system is usually meaningless in the Simple Lua File System. An alternative way to pack files is running lua "./utils/fs.lua" pack "$packed_file" *.lua
. However, the utility of the latter command is limited due to the maximum number of command line arguments in your OS.
Unpacking File System to Files
To unpack the files contained within a file system, you may run lua "./utils/fs.lua" unpack "$packed_file" "$directory"
, where $packed_file
is the file that contains the file system, $directory
is the directory to hold all the unpacked files.
Request for Comments
The development of the CKB-Lua prototype is now in the last stage and we welcome your feedback. If you have any use cases that are not currently addressed by the existing implementation, feel free to share your thoughts by leaving comments on the GitHub repo.
Appendix: Exposed Lua Functions and Constants in the CKB Module
Exposed Functions
Partial Loading
Partial loading refers to the process of loading a portion of a large resource into memory, in order to save memory and improve performance. CKB-Lua provides a partial-loading mechanism where Lua functions can obtain a part of the whole data by passing argument(s) length
and/or offset
.
length | offset | Behavior | Example |
Omitted | Omitted | Load the entire data | Call ckb.load_witness(0, ckb.SOURCE_INPUT) to load the whole witness of input cell of index 0 |
Passed, zero | Omitted | Return the length of the entire data | Call ckb.load_witness(0, ckb.SOURCE_INPUT, 0) to return the length of the witness instead of the witness data |
Passed, non-zero | Omitted | Return the initial data of the arg length bytes | Call ckb.load_witness(0, ckb.SOURCE_INPUT, 10) to return the initial 10 bytes of witness |
Passed, zero | Passed | Return the data length starting from offset | Call ckb.load_witness(0, ckb.SOURCE_INPUT, 0, 10) to return the data length starting from offset 10 |
Passed, non-zero | Passed | Return the data starting from offset of length | Call ckb.load_witness(0, ckb.SOURCE_INPUT, 2, 10) to return 2 bytes of data starting from 10 |
Error Handling
Most args may return an error as the last argument. The error is a non-zero integer that represents the kind of error that happened during the execution of the function. Callers should check errors to be nil before using other arguments.
More Examples
Table of Functions
Note when partial loading support is enabled, the description for arguments length and offset is omitted. You may optionally pass length and offset in that case.
Description | Calling Example | Args | Support Partial Loading | Return Value | Side Effects | See Also | |
ckb.dump | Dump a returned buffer to stdout | ckb.dump(buf) | Buf (a buffer returned from syscalls below) | Not applicable | None | Buf is printed | |
ckb.exit | Exit CKB-VM execution | ckb.exit(code) | Code (exit code) | Not applicable | None | Exit CKB-VM execution | ckb-exit syscall |
ckb.exit_script | Stop Lua script execution, return to the CKB-VM caller | ckb.exit_script(code) | Code (this will be the return code of lua_run_code , should be positive, see lua_run_code above for details) | Not applicable | None | Stop Lua script execution, return to the CKB-VM caller | |
ckb.mount | Load cell data and mount file system within | ckb.mount(source, index) | Source (of the cell to load), index (of the cell to load within all cells with source ) | Not applicable | Err (may be nil object to represent possible error) | Files within the file system will be available to use if no error occurred | File system documentation |
ckb.load_tx_hash | Load transaction hash | buf, err = ckb.load_tx_hash() | None | Yes | Buf (containing the transaction hash), err (may be nil object to represent possible error) | None | ckb_load_tx_hash syscall |
ckb.load_script_hash | Load hash of the current script | buf, error = ckb.load_script_hash() | None | Yes | Buf (containing the script hash), err (may be nil object to represent possible error) | None | ckb_load_script_hash syscall |
ckb.load_script | Load current script | buf, error = ckb.load_script() | None | Yes | Buf (containing script), err (may be nil object to represent possible error) | None | ckb_load_script syscall |
ckb.load_and_unpack_script | Load and unoack current script | code_hash, hash_type, args, error = ckb.load_and_unpack_script() | None | Not applicable | Code_hash (code hash of the current script), hash_type (hash type of the current script), args (args of the current script), err (may be nil object to represent possible error) | None | |
ckb.load_transaction | Load current transaction | buf, error = ckb.load_transaction() | None | Yes | Buf (containing cell), err (may be nil object to represent possible error) | None | ckb_load_transaction syscall |
ckb.load_cell | Load cell | buf, error = ckb.load_cell(index, source) | Index (the index of the cell), source (the source of the cell) | Yes | Buf (containing cell), err (may be nil object to represent possible error) | None | ckb_load_cell syscall |
ckb.load_input | Load input cell | buf, error = ckb.load_input(index, source) | Index (the index of the cell), source (the source of the cell) | Yes | Buf (containing the input cell), err (may be nil object to represent possible error) | None | ckb_load_input syscall |
ckb.load_header | Load cell header | buf, error = ckb.load_header(index, source) | Index (the index of the cell), source (the source of the cell) | Yes | Buf (containing the header), err (may be nil object to represent possible error) | None | ckb_load_header syscall |
ckb.load_witness | Load the witness | buf, error = ckb.load_witness(index, source, length, offset) | Index (the index of the cell), source (the source of the cell) | Yes | Buf (containing the witness), err (may be nil object to represent possible error) | None | ne |
ckb.load_cell_data | Load cell data | buf, error = ckb.load_cell_data(index, source, length, offset) | Index (the index of the cell), source (the source of the cell) | Yes | Buf (containing the cell data), err (may be nil object to represent possible error) | None | ckb_load_cell_data syscall |
ckb.load_by_field | Load cell data field | buf, error = ckb.load_cell_by_field(index, source, field) | Index (the index of the cell), source (the source of the cell), field (the field to load) | Yes | Buf (containing the cell data field), err (may be nil object to represent possible error) | None | ckb_load_cell_by_field syscall |
ckb.load_input_by_field | Load input field | buf, error = ckb.load_input_by_field(index, source, field) | Index (the index of the cell), source (the source of the cell), field (the field to load) | Yes | Buf (containing the cell data field), err (may be nil object to represent possible error) | None | ckb_load_input_by_field syscall |
ckb.load_header_by_field | Load header by field | ckb.load_header_by_field(index, source, field) | Index (the index of the cell), source (the source of the cell), field (the field to load) | Yes | Buf (containing the cell data field), err (may be nil object to represent possible error) | None | ckb_load_header_by_field syscall |
ckb.unpack_script | Unpack the buffer that contains the molecule structure Script | ckb.unpack_script(buf) | Buf (buffer that contains the molecule structure Script ) | Not applicable | Table (containing keys code_hash , hash_type and args ), err (may be nil object to represent possible error) | None | The molecule definition of Script |
ckb.unpack_witnessargs | Unpack the buffer that contains the molecule structure WitnessArgs | ckb.unpack_witnessargs(buf) | Buf (buffer that contains the molecule structure WitnessArgs ) | Not applicable | Table (containing keys lock , input_type and output_type ), err (may be nil object to represent possible error) | None | The molecule definition of WitnessArgs |
ckb.unpack_outpoint | Unpack the buffer that contains the molecule structure OutPoint | ckb.unpack_outpoint(buf) | Buf (buffer that contains the molecule structure OutPoint ) | Not applicable | Table (containing keys tx_hash and index ), err (may be nil object to represent possible error) | None | The molecule definition of OutPoint |
ckb.unpack_cellinput | Unpack the buffer that contains the molecule structure CellInput | ckb.unpack_cellinput(buf) | Buf (buffer that contains the molecule structure CellInput ) | Not applicable | Table (containing keys since and previous_output ), err (may be nil object to represent possible error) | None | The molecule definition of CellInput |
ckb.unpack_celloutput | Unpack the buffer that contains the molecule structure CellOutput | ckb.unpack_celloutput(buf) | Buf (buffer that contains the molecule structure CellOutput ) | Not applicable | Table (containing keys capacity , lock and type ), err (may be nil object to represent possible error) | None | The molecule definition of CellOutput |
ckb.unpack_celldep | Unpack the buffer that contains the molecule structure CellDep | ckb.unpack_celloutput(buf) | Buf (buffer that contains the molecule structure CellDep ) | Not applicable | Table (containing keys out_point and dep_type ), err (may be nil object to represent possible error) | None | The molecule definition of CellDep |
Exported Constants
While most constants here are directly taken from ckb_consts.h, some specific constants, like ckb.LUA_ERROR_INTERNAL
, still have to be defined. These CKB-Lua constants are presented below.
ckb.SUCCESS
ckb.INDEX_OUT_OF_BOUND
ckb.ITEM_MISSING
ckb.LENGTH_NOT_ENOUGH
ckb.INVALID_DATA
ckb.LUA_ERROR_INTERNAL
ckb.LUA_ERROR_OUT_OF_MEMORY
ckb.LUA_ERROR_ENCODING
ckb.LUA_ERROR_SCRIPT_TOO_LONG
ckb.LUA_ERROR_INVALID_ARGUMENT
ckb.SOURCE_INPUT
ckb.SOURCE_OUTPUT
ckb.SOURCE_CELL_DEP
ckb.SOURCE_HEADER_DEP
ckb.SOURCE_GROUP_INPUT
ckb.SOURCE_GROUP_OUTPUT
ckb.CELL_FIELD_CAPACITY
ckb.CELL_FIELD_DATA_HASH
ckb.CELL_FIELD_LOCK
ckb.CELL_FIELD_LOCK_HASH
ckb.CELL_FIELD_TYPE
ckb.CELL_FIELD_TYPE_HASH
ckb.CELL_FIELD_OCCUPIED_CAPACITY
ckb.INPUT_FIELD_OUT_POINT
ckb.INPUT_FIELD_SINCE
ckb.HEADER_FIELD_EPOCH_NUMBER
ckb.HEADER_FIELD_EPOCH_START_BLOCK_NUMBER
ckb.HEADER_FIELD_EPOCH_LENGTH
✍🏻 Witten by Biao YI
You may also be interested in:
By the same author: