Skip to content
This repository was archived by the owner on Mar 11, 2025. It is now read-only.

Commit 4c0b110

Browse files
anoushk1234ngundotra
anoushk1234
and
ngundotra
authored
docs: documentation for account compression program (#3998)
* docs: added account compression to sidebar * docs: added account compression page * docs: added concepts * docs: added more core concepts + usage docs * fix: remove extra column in table * fix: fixed error in swap example * fix: revert package.json changes for docs * docs: remove digital asset ex from compression docs * Address nit --------- Co-authored-by: Noah Gundotra <[email protected]>
1 parent c7a1e98 commit 4c0b110

File tree

4 files changed

+252
-0
lines changed

4 files changed

+252
-0
lines changed

‎docs/sidebars.js

+10
Original file line numberDiff line numberDiff line change
@@ -52,5 +52,15 @@ module.exports = {
5252
},
5353
],
5454
},
55+
{
56+
type: "category",
57+
label: "Account Compression",
58+
collapsed: true,
59+
items: [
60+
"account-compression",
61+
"account-compression/concepts",
62+
"account-compression/usage",
63+
]
64+
},
5565
],
5666
};

‎docs/src/account-compression.md

+57
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
---
2+
title: Account Compression Program
3+
---
4+
5+
This on-chain program provides an interface for composing smart-contracts to create and use SPL ConcurrentMerkleTrees. The primary application of using SPL ConcurrentMerkleTrees is to make edits to off-chain data with on-chain verification.
6+
7+
## Motivation
8+
9+
- The high throughput of the Solana blockchain has increased the creation of Non fungible assets i.e NFTs due to their custodial ownership and censorship resistance characteristics. However, the practical use cases of these NFTs are limited by network storage costs when these are created at scale. It's rather inexpensive to mint a single non fungible token, however as you increase the quantity the cost of storing the asset's data on-chain becomes uneconomical.
10+
11+
- To fix this we must ensure the cost per token is as close to zero as possible. The solution is to store a compressed hash of the asset data on chain while maintaining the actual data off chain in a database. The program provides a way to verify the off chain data on chain and also make concurrent writes to the data. In-order to do this we introduced a new data structure called a Concurrent Merkle Tree that avoids proof collision while making concurrent writes.
12+
13+
14+
## Background
15+
16+
The account compression program is currently being used for the [Metaplex Bubblegum Program](https://github.com/metaplex-foundation/metaplex-program-library/blob/master/bubblegum/)
17+
18+
To solve the problem of the high on-chain storage cost per unit of these assets, we need to store a compressed fingrprint on-chain that can verify the off-chain asset data. To do this we need
19+
- Concurrent Merkle Trees
20+
- The concurrent merkle tress allows us to compress all the data into a single root hash stored on-chain while allowing concurrent replacements and appends to the data.
21+
- Program indexer
22+
- The indexer is incharge of indexing the latest writes to the tree on chain so you know which nodes have been replaced and which have been appended to so you can avoid proof collision
23+
- Off-chain Database
24+
- The db stores the actual asset data on chain as we are only storing the merkle root on chain and we need to be able to verify the data on chain.
25+
26+
The crux of the this is the concurrent merkle tree and we shall learn about it in the next section.
27+
28+
## Source
29+
30+
The Account Compression Program's source is available on
31+
[github](https://github.com/solana-labs/solana-program-library).
32+
33+
34+
## Interface
35+
The Account Compression Program is written in rust and also has a typescript sdk for interacting with the program.
36+
37+
### Rust Packages
38+
| Name | Description | Program |
39+
| ---------------------------- | ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
40+
| `spl-account-compression` | SDK for interacting with account compression program | [Rust Crate](https://crates.io/crates/spl-account-compression) and [Rust Docs](https://docs.rs/spl-account-compression) |
41+
| `spl-noop` | SDK for interacting with no op program, primarily for circumventing log truncation | [Rust Crate](https://crates.io/crates/spl-noop) and [Rust Docs](https://docs.rs/spl-noop) |
42+
| `spl-concurrent-merkle-tree` | SDK for creating SPL ConcurrentMerkleTrees | [Rust Crate](https://crates.io/crates/spl-concurrent-merkle-tree) and [Rust Docs](https://docs.rs/spl-concurrent-merkle-tree) |
43+
44+
### TypeScript Packages
45+
| Name | Description | Package |
46+
| --------------------------------- | ---------------------------------------------------- | -------------------------------------------------------------------- |
47+
| `@solana/spl-account-compression` | SDK for interacting with account compression program | [NPM](https://www.npmjs.com/package/@solana/spl-account-compression) |
48+
49+
## Testing and Development
50+
51+
Testing contracts locally requires the SDK to be built.
52+
53+
With a built local SDK, the test suite can be ran with:
54+
55+
1. `yarn link @solana/spl-account-compression`
56+
2. `yarn`
57+
3. `yarn test`

‎docs/src/account-compression/concepts.md

+73
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
---
2+
title: Core Concepts
3+
---
4+
5+
## Concurrent Merkle Trees
6+
To understand concurrent merkle trees we must first briefly understand merkle trees.
7+
8+
### Merkle Trees
9+
10+
A merkle tree is a hash based data structure that encodes data into a tree.
11+
The tree has nodes that are hashes of it's children and each leaf node is a hash of the data.
12+
13+
Each node has a 256 bit (32 byte) string represented by X<sub>i</sub> ∈ {0,1}^256 which is hashed using `H: {0, 1}^256 × {0, 1}^256 → {0, 1}^256`, meaning two child nodes with their 256 bit strings are hashed into one parent node with a 256 bit string. You can use can use any hash function that satisfies this property but we use SHA256.
14+
15+
Important properties of merkle trees:
16+
- The tree must be a fully balanced binary tree
17+
- Each Node *X<sub>i</sub> = H(X<sub>2i</sub>, X<sub>2i+1</sub>) for all i < 2^D*
18+
- Each Leaf Node *X<sub>i</sub> for all i <= 2^D*. X<sub>i</sub> is the hash of the data.
19+
20+
Because of these properties we can verify if certain data exists in tree while compressing all the data into a single 256 bit string called the root hash.
21+
22+
Example of a merkle tree of depth 2:
23+
```txt
24+
X1
25+
/ \
26+
X2 X3
27+
/ \ / \
28+
X4 X5 X6 X7
29+
```
30+
You can verify that X5 computes to X1 by doing X1 = H(H(X4,X5),X3)) where {X4,X5,X3} are the proof.
31+
If you change X5 to X5' then you will have to recompute the root hash in the following steps:
32+
- X2' = H(X4,X5')
33+
- X1' = H(X2',X3)
34+
35+
### Concurrent leaf replacement
36+
We know that there can be multiple concurrent requests to write to the same state, however when the root changes while the first write is happenning the second write will generate an invalid root, in other words everytime a root is modified all modifications in progress will be invalid.
37+
```txt
38+
X1' X1''
39+
/ \ / \
40+
X2' X3 X2 X3''
41+
/ \ / \ / \ / \
42+
X4 X5' X6 X7 X4 X5 X6'' X7
43+
```
44+
In the above example let's say we try to modify `X5 -> X5'` and make another request to modify X6 -> X6''. For the first change we get root `X1'` computed using `X1' = H(H(X4,X5'),X3)`. For the second change we get root X1'' computed using `X1'' = H(H(X6'',X7),X2`). However `X1''` is not valid as `X1' != H(H(X6, X7), X2)` because the new root is actually `X1'`.
45+
46+
The reason this happens is because the change in the first trees path actualy changes the proofs required by the second trees change. To circumvent this problem we maintain a changelog of updates that have been made to the tree, so when `X5 -> X5'` the second mutation can actually use X2' instead of X2 which would compute to the correct root.
47+
48+
To swap the nodes when adding a new leaf in the second tree we do the following:
49+
- Take XOR of the leaf indices of the change log path and the new leaf in base 2
50+
- The depth at which you have to make the swap is the number of leading zeroes in the result(we also add one to it because the swap node is one below the intersection node)
51+
- At that depth change the node in the proof to the node in the changelog
52+
53+
Example with the previous trees:
54+
```txt
55+
2 1
56+
Changelog: [X5',X2']
57+
New Leaf: X6'' at leaf index 2
58+
59+
2 1
60+
Old proof for new leaf: [X7,X2]
61+
62+
1 XOR 2 = 001 XOR 010 = 011 (no leading zeroes)
63+
depth to swap at = 0 + 1 = 1
64+
65+
2 1
66+
New proof for new leaf: [X7,X2']
67+
```
68+
**Note:** We use XOR here because changelogs can get large as there can be many concurrent writes so using XOR is more efficient than a simple array search algorithm.
69+
70+
**Note**: Solana imposes a transactions size restriction of 1232 bytes hence the program also provides the ability to cache the upper most part of the concurrent merkle tree called a "canopy" which is stored at the end of the account.
71+
72+
73+

‎docs/src/account-compression/usage.md

+112
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
---
2+
title: Example usage of the TS SDK
3+
---
4+
5+
6+
7+
## Install
8+
9+
```shell
10+
npm install --save @solana/spl-account-compression @solana/web3.js
11+
```
12+
13+
__OR__
14+
15+
```shell
16+
yarn add @solana/spl-account-compression @solana/web3.js
17+
```
18+
19+
### Examples
20+
21+
1. Create a tree
22+
23+
```typescript
24+
// Assume: known `payer` Keypair
25+
// Generate a keypair for the ConcurrentMerkleTree
26+
const cmtKeypair = Keypair.generate();
27+
// Create a system instruction to allocate enough
28+
// space for the tree
29+
const allocAccountIx = await createAllocTreeIx(
30+
connection,
31+
cmtKeypair.publicKey,
32+
payer.publicKey,
33+
{ maxDepth, maxBufferSize },
34+
canopyDepth,
35+
);
36+
// Create an SPL compression instruction to initialize
37+
// the newly created ConcurrentMerkleTree
38+
const initTreeIx = createInitEmptyMerkleTreeIx(
39+
cmtKeypair.publicKey,
40+
payer.publicKey,
41+
{ maxDepth, maxBufferSize }
42+
);
43+
const tx = new Transaction().add(allocAccountIx).add(initTreeIx);
44+
await sendAndConfirmTransaction(connection, tx, [cmtKeypair, payer]);
45+
```
46+
47+
2. Add a leaf to the tree
48+
49+
```typescript
50+
// Create a new leaf
51+
const newLeaf: Buffer = crypto.randomBytes(32);
52+
// Add the new leaf to the existing tree
53+
const appendIx = createAppendIx(cmtKeypair.publicKey, payer.publicKey, newLeaf);
54+
const tx = new Transaction().add(appendIx);
55+
await sendAndConfirmTransaction(connection, tx, [payer]);
56+
```
57+
58+
3. Replace a leaf in the tree, using the provided `MerkleTree` as an indexer
59+
60+
This example assumes that `offChainTree` has been indexing all previous modifying transactions
61+
involving this tree.
62+
It is okay for the indexer to be behind by a maximum of `maxBufferSize` transactions.
63+
64+
65+
```typescript
66+
// Assume: `offChainTree` is a MerkleTree instance
67+
// that has been indexing the `cmtKeypair.publicKey` transactions
68+
// Get a new leaf
69+
const newLeaf: Buffer = crypto.randomBytes(32);
70+
// Query off-chain records for information about the leaf
71+
// you wish to replace by its index in the tree
72+
const leafIndex = 314;
73+
// Replace the leaf at `leafIndex` with `newLeaf`
74+
const replaceIx = createReplaceIx(
75+
cmtKeypair.publicKey,
76+
payer.publicKey,
77+
newLeaf,
78+
offChainTree.getProof(leafIndex)
79+
);
80+
const tx = new Transaction().add(replaceIx);
81+
await sendAndConfirmTransaction(connection, tx, [payer]);
82+
```
83+
84+
4. Replace a leaf in the tree, using a 3rd party indexer
85+
86+
This example assumes that some 3rd party service is indexing the the tree at `cmtKeypair.publicKey` for you, and providing MerkleProofs via some REST endpoint.
87+
The `getProofFromAnIndexer` function is a **placeholder** to exemplify this relationship.
88+
89+
```typescript
90+
// Get a new leaf
91+
const newLeaf: Buffer = crypto.randomBytes(32);
92+
// Query off-chain indexer for a MerkleProof
93+
// possibly by executing GET request against a REST api
94+
const proof = await getProofFromAnIndexer(myOldLeaf);
95+
// Replace `myOldLeaf` with `newLeaf` at the same index in the tree
96+
const replaceIx = createReplaceIx(
97+
cmtKeypair.publicKey,
98+
payer.publicKey,
99+
newLeaf,
100+
proof
101+
);
102+
const tx = new Transaction().add(replaceIx);
103+
await sendAndConfirmTransaction(connection, tx, [payer]);
104+
```
105+
106+
## Reference examples
107+
108+
Here are some examples using account compression in the wild:
109+
110+
* Solana Program Library [tests](https://github.com/solana-labs/solana-program-library/tree/master/account-compression/sdk/tests)
111+
112+
* Metaplex Program Library Compressed NFT [tests](https://github.com/metaplex-foundation/metaplex-program-library/tree/master/bubblegum/js/tests)

0 commit comments

Comments
 (0)