Versioned Transactions and Lookup Tables
Use large amounts of accounts by using lookup tables.
Summary
- Versioned Transactions refers to a way to support both legacy versions and newer versions of transaction formats. The original transaction format is “legacy” and new transaction versions start at version 0. Versioned transactions were implemented to support the use of Address Lookup Tables (also called lookup tables or LUTs).
- Address Lookup Tables are accounts used to store addresses of other accounts, which can then be referenced in versioned transactions using a 1 byte index instead of the full 32 bytes per address. This enables the creation of more complex transactions than what was possible prior to the introduction of LUTs.
Lesson
By design, Nexis Native Chain transactions are limited to 1232 bytes. Transactions exceeding this size will fail. While this enables a number of network optimizations, it can also limit the types of atomic operations that can be performed on the network.
To help get around the transaction size limitation, Nexis Native Chain released a new transaction format that allows support for multiple versions of transaction formats. At the time of writing, Nexis Native Chain supports two transaction versions:
legacy
- the original transaction format0
- the newest transaction format that includes support for Address Lookup Tables
Versioned transactions don’t require any modifications to existing Nexis Native Chain programs, but any client-side code created prior to the release of versioned transactions should be updated. In this lesson, we’ll cover the basics of versioned transactions and how to use them, including:
- Creating versioned transactions
- Creating and managing lookup tables
- Using lookup tables in versioned transactions
Versioned Transactions
One of the items taking up the most space in Nexis Native Chain transactions is the inclusion of full account addresses. At 32 bytes each, 39 accounts will render a transaction too large. That’s not even accounting for instruction data. In practice, most transactions will be too large with around 20 accounts.
Nexis Native Chain released versioned transactions to support multiple transaction formats. Alongside the release of versioned transactions, Nexis Native Chain released version 0 of transactions to support Address Lookup Tables. Lookup tables are separate accounts that store account addresses and then allow them to be referenced in a transaction using a 1 byte index. This significantly decreases the size of a transaction since each included account now only needs to use 1 byte instead of 32 bytes.
Even if you don’t need to use lookup tables, you’ll need to know how to support
versioned transactions in your client-side code. Fortunately, everything you
need to work with versioned transactions and lookup tables is included in the
@nexis-network/web3.js
library.
Create versioned transactions
To create a versioned transaction, you simply create a TransactionMessage
with
the following parameters:
payerKey
- the public key of the account that will pay for the transactionrecentBlockhash
- a recent blockhash from the networkinstructions
- the instructions to include in the transaction
You then transform this message object into a version 0
transaction using the
compileToV0Message()
method.
Finally, you pass the compiled message into the VersionedTransaction
constructor to create a new versioned transaction. Your code can then sign and
send the transaction to the network, similar to a legacy transaction.
Address Lookup Table
Address Lookup Tables (also called lookup tables or LUTs) are accounts that store a lookup table of other account addresses. These LUT accounts are owned by the Address Lookup Table Program and are used to increase the number of accounts that can be included in a single transaction.
Versioned transactions can include the address of an LUT account and then reference additional accounts with a 1-byte index instead of including the full address of those accounts. This significantly reduces the amount of space used for referencing accounts in a transaction.
To simplify the process of working with LUTs, the @nexis-network/web3.js
library
includes an AddressLookupTableProgram
class which provides a set of methods to
create instructions for managing LUTs. These methods include:
createLookupTable
- creates a new LUT accountfreezeLookupTable
- makes an existing LUT immutableextendLookupTable
- adds addresses to an existing LUTdeactivateLookupTable
- puts an LUT in a “deactivation” period before it can be closedcloseLookupTable
- permanently closes an LUT account
Create a lookup table
You use the createLookupTable
method to construct the instruction that creates
a lookup table. The function requires the following parameters:
authority
- the account that will have permission to modify the lookup tablepayer
- the account that will pay for the account creationrecentSlot
- a recent slot to derive the lookup table’s address
The function returns both the instruction to create the lookup table and the address of the lookup table.
Under the hood, the lookup table address is simply a PDA derived using the
authority
and recentSlot
as seeds.
Note that using the most recent slot sometimes results in an error after sending
the transaction. To avoid this, you can use a slot that is one slot prior the
most recent one (e.g. recentSlot: slot - 1
). However, if you still encounter
an error when sending the transaction, you can try resending the transaction.
Extend a lookup table
You use the extendLookupTable
method to create an instruction that adds
addresses to an existing lookup table. It takes the following parameters:
payer
- the account that will pay for the transaction fees and any increased rentauthority
- the account that has permission to change the lookup tablelookupTable
- the address of the lookup table to extendaddresses
- the addresses to add to the lookup table
The function returns an instruction to extend the lookup table.
Note that when extending a lookup table, the number of addresses that can be added in one instruction is limited by the transaction size limit, which is 1232 bytes. This means you can add 30 addresses to a lookup table at a time. If you need to add more than that, you’ll need to send multiple transactions. Each lookup table can store a maximum of 256 addresses.
Send Transaction
After creating the instructions, you can add them to a transaction and sent it to the network.
Note that when you first create or extend a lookup table, it needs to “warm up” for one slot before the LUT or new addresses can be used in transactions. In other words, you can only use lookup tables and access addresses that were added prior to the current slot.
If you encounter the error above or are unable to access addresses in a lookup table immediately after extending it, it’s likely because you’re attempting to access the lookup table or a specific address prior to the end of the warm up period. To avoid this issue, add a delay after extending the lookup table before sending a transaction that references the table.
Deactivate a lookup table
When a lookup table is no longer needed, you can deactivate and close it to reclaim its rent balance. Address lookup tables can be deactivated at any time, but they can continue to be used by transactions until a specified “deactivation” slot is no longer “recent”. This “cool-down” period ensures that in-flight transactions can’t be censored by LUTs being closed and recreated in the same slot. The deactivation period is approximately 513 slots.
To deactivate an LUT, use the deactivateLookupTable
method and pass in the
following parameters:
lookupTable
- the address of the LUT to be deactivatedauthority
- the account with permission to deactivate the LUT
Close a lookup table
To close a lookup table after its deactivation period, use the
closeLookupTable
method. This method creates an instruction to close a
deactivated lookup table and reclaim its rent balance. It takes the following
parameters:
lookupTable
- the address of the LUT to be closedauthority
- the account with permission to close the LUTrecipient
- the account that will receive the reclaimed rent balance
Attempting to close a lookup table before it’s been fully deactivated will result in an error.
Freeze a lookup table
In addition to standard CRUD operations, you can “freeze” a lookup table. This makes it immutable so that it can no longer be extended, deactivated, or closed.
You freeze a lookup table with the freezeLookupTable
method. It takes the
following parameters:
lookupTable
- the address of the LUT to be frozenauthority
- the account with permission to freeze the LUT
Once an LUT is frozen, any further attempts to modify it will result in an error.
Using lookup tables in versioned transactions
To use a lookup table in a versioned transaction, you need to retrieve the lookup table account using its address.
You can then create a list of instructions to include in a transaction as usual.
When creating the TransactionMessage
, you can include any lookup table
accounts by passing them as an array to the compileToV0Message()
method. You
can also provide multiple lookup table accounts.
Lab
Let’s go ahead and practice using lookup tables!
this lab will guide you through the steps of creating, extending, and then using a lookup table in a versioned transaction.
1. Get the starter code
To begin, download the starter code from the starter branch of this
repository.
Once you have the starter code, run npm install
in the terminal to install the
required dependencies.
The starter code includes an example of creating a legacy transaction that intends to atomically transfer NZT to 22 recipients. The transaction contains 22 instructions where each instruction transfers NZT from the signer to a different recipient.
The purpose of the starter code is to illustrate the limitation on the number of addresses that can be included in a legacy transaction. The transaction built in the starter code is expected to fail when sent.
The following starter code can be found in the index.ts
file.
To execute the code, run npm start
. This will create a new keypair, write it
to the .env
file, airdrop devnet NZT to the keypair, and send the transaction
built in the starter code. The transaction is expected to fail with the error
message Transaction too large
.
In the next steps, we’ll go over how to use lookup tables with versioned transactions to increase the number of addresses that can be included in a single transaction.
Before we start, go ahead and delete the content of the main
function to leave
only the following:
2. Create a sendV0Transaction
helper function
We’ll be sending multiple “version 0” transactions, so let’s create a helper function to facilitate this.
This function should take parameters for a connection, a user’s keypair, an array of transaction instructions, and an optional array of lookup table accounts.
The function then performs the following tasks:
- Retrieves the latest blockhash and last valid block height from the Nexis Native Chain network
- Creates a new transaction message using the provided instructions
- Signs the transaction using the user’s keypair
- Sends the transaction to the Nexis Native Chain network
- Confirms the transaction
- Logs the transaction URL on the Nexis Native Chain Explorer
3. Create a waitForNewBlock
helper function
Recall that lookup tables and the addresses contained in them can’t be
referenced immediately after creation or extension. This means we’ll end up
needing to wait for a new block before submitting transactions that reference
the newly created or extended lookup table. To make this simpler down the road,
let’s create a waitForNewBlock
helper function that we’ll use to wait for
lookup tables to activate between sending transactions.
This function will have parameters for a connection and a target block height. It then starts an interval that checks the current block height of the network every 1000ms. Once the new block height exceeds the target height, the interval is cleared and the promise is resolved.
4. Create an initializeLookupTable
function
Now that we have some helper functions ready to go, declare a function named
initializeLookupTable
. This function has parameters user
, connection
, and
addresses
. The function will:
- Retrieve the current slot
- Generate an instruction for creating a lookup table
- Generate an instruction for extending the lookup table with the provided addresses
- Send and confirm a transaction with the instructions for creating and extending the lookup table
- Return the address of the lookup table
5. Modify main
to use lookup tables
Now that we can initialize a lookup table with all of the recipients’ addresses,
let’s update main
to use versioned transactions and lookup tables. We’ll need
to:
- Call
initializeLookupTable
- Call
waitForNewBlock
- Get the lookup table using
connection.getAddressLookupTable
- Create the transfer instruction for each recipient
- Send the v0 transaction with all of the transfer instructions
Notice that you create the transfer instructions with the full recipient address
even though we created a lookup table. That’s because by including the lookup
table in the versioned transaction, you tell the web3.js
framework to replace
any recipient addresses that match addresses in the lookup table with pointers
to the lookup table instead. By the time the transaction is sent to the network,
addresses that exist in the lookup table will be referenced by a single byte
rather than the full 32 bytes.
Use npm start
in the command line to execute the main
function. You should
see an output similar to the following:
The first transaction link in the console represents the transaction for creating and extending the lookup table. The second transaction represents the transfers to all recipients. Feel free to inspect these transactions in the explorer.
Remember, this same transaction was failing when you first downloaded the starter code. Now that we’re using lookup tables, we can do all 22 transfers in a single transaction.
6. Add more address to the lookup table
Keep in mind that the solution we’ve come up with so far only supports transfers to up to 30 accounts since we only extend the lookup table once. When you factor in the transfer instruction size, it’s actually possible to extend the lookup table with an additional 27 addresses and complete an atomic transfer to up to 57 recipients. Let’s go ahead and add support for this now!
All we need to do is go into initializeLookupTable
and do two things:
- Modify the existing call to
extendLookupTable
to only add the first 30 addresses (any more than that and the transaction will be too large) - Add a loop that will keep extending a lookup table 30 addresses at a time until all addresses have been added
Congratulations! If you feel good about this lab, you’re probably ready to work with lookup tables and versioned transactions on your own. If you want to take a look at the final solution code you can find it on the solution branch.
Challenge
As a challenge, experiment with deactivating, closing and freezing lookup tables. Remember that you need to wait for a lookup table to finish deactivating before you can close it. Also, if a lookup table is frozen, it cannot be modified (deactivated or closed), so you will have to test separately or use separate lookup tables.
- Create a function for deactivating the lookup table.
- Create a function for closing the lookup table
- Create a function for freezing the lookup table
- Test the functions by calling them in the
main()
function
You can reuse the functions we created in the lab for sending the transaction and waiting for the lookup table to activate/deactivate. Feel free to reference this solution code.