Skip to content

Commit c5ed3ec

Browse files
realChainLiferealChainLife
authored
Update file-systems.md
1 parent 67a1d09 commit c5ed3ec

File tree

1 file changed

+270
-7
lines changed

1 file changed

+270
-7
lines changed

‎docs/concepts/file-systems.md

Lines changed: 270 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -21,18 +21,281 @@ If you're interested in how MFS and UnixFS play into how IPFS works with files i
2121

2222
Because files in IPFS are content-addressed and immutable, they can be complicated to edit. Mutable File System (MFS) is a tool built into IPFS that lets you treat files like you would a normal name-based filesystem — you can add, remove, move, and edit MFS files and have all the work of updating links and hashes taken care of for you.
2323

24-
MFS is accessed through the [`files`](/reference/api/cli/#ipfs-files) commands in the IPFS CLI and API.
24+
### Working with files API
2525

26-
<!-- TODO: add some examples to demonstrate how to use MFS, e.g. creating a directory, adding a file, checking the hash, editing the file, and checking the hash again. -->
26+
MFS is accessed through the files commands in the IPFS CLI and API. The commands on the files API take the format of `ipfs.files.someMethod()`. These methods are used to:
2727

28-
This video also provides a good overview of MFS:
28+
#### Create a directory
2929

30-
@[youtube](FX_AXNDsZ9k)
30+
The MFS method `ipfs.files.mkdir` creates a new directory at a specified path. For example, to add a directory `example` to our root directory (`/`), run:
31+
32+
````
33+
await ipfs.files.mkdir('/example')
34+
````
35+
36+
If you want to create a new directory nested under others that don't yet exist, you need to explicitly set the value of `parents` to `true`, like so:
37+
38+
````
39+
await ipfs.files.mkdir('/my/directory/example', { parents: true })
40+
````
41+
42+
#### Check directory status
43+
44+
The method, `ipfs.files.stat` enables you to check the status of a file or directory on your IPFS node. To check the status of a directory called `example` located within the root directory, you could call the method by running:
45+
46+
````
47+
await ipfs.files.stat('/example')
48+
````
49+
50+
This method returns an object with a `cid`, `size`, `cumulativeSize`, `type`, `blocks`, `withLocality`, `local`, and `sizeLocal`. This returned data is the essential information about the directory.
51+
52+
````
53+
// {
54+
// hash: CID('QmXmJBmnYqXVuicUfn9uDCC8kxCEEzQpsAbeq1iJvLAmVs'),
55+
// size: 60,
56+
// cumulativeSize: 118,
57+
// blocks: 1,
58+
// type: 'directory'
59+
// }
60+
````
61+
62+
If you add, move, copy, or remove a file into a directory the hash of you directory will change with every file modified.
63+
64+
#### Add a file to MFS
65+
66+
To add a file to IPFS, use the MFS `ipfs.files.write` method using the command:
67+
68+
````
69+
await ipfs.files.write(path, content, [options])
70+
````
71+
::: tip
72+
73+
This method can create a brand new file, that accepts file `content` in multiple formats, in a specified `path` within the IPFS instance by providing the boolean option {` create: true` }.
74+
75+
:::
76+
77+
If you, for example, have a file object accessible via a variable `examplePic` that you wanted to add it to the root directory and have it named `example.jpg`, you could run:
78+
79+
````
80+
await ipfs.files.write('/example.jpg', examplePic, { create: true })
81+
````
82+
83+
This method does not provide a return value.
84+
85+
#### View contents of a directory
86+
87+
To check whether the `ipfs.files.write` method has worked as expected, you can use the `ipfs.files.ls` method to inspect directories on the MFS. You can do this by running:
88+
89+
````
90+
await ipfs.files.ls([path], [options])
91+
````
92+
93+
The method will default to listing the contents of your directory (`/`), or you can choose to specify a specific `path` (directory) you'd like to inspect such as `/example`.
94+
95+
This method produces an array of objects for each file or directory with properties such as `name`, `type`, `size`, `cid`, `mode`, and `mtime`. If you wanted to inspect the contents of a `/example` directory, run:
96+
97+
````
98+
await ipfs.files.ls('/example')
99+
````
100+
101+
#### Copy a file or directory
102+
103+
In the Mutable File System, like traditional systems, you can copy a file or directory to a new location while also leaving it intact at its source.
104+
105+
You can do this by using the method:
106+
107+
````
108+
await ipfs.files.cp(...from, to, [options])
109+
````
110+
111+
::: tip
112+
113+
This method offers two formatting options for passing the `from` key:
114+
115+
- an existing MFS path to a file or a directory in your own node (e.g. `/my-example-dir/my-example-file.txt`)
116+
- an IPFS path to a file or directory hosted either by you or by a peer (e.g. `/ipfs/QmWc7U4qGeRAEgtsyVyeW2CRVbkHW31nb24jFyks7eA2mF`)
117+
118+
:::
119+
120+
The `to` key is the destination path in MFS, and there's an option {` create: true` } that can be used to create parent directories that don't already exist.
121+
122+
You can use this method to perform different operations including:
123+
124+
````
125+
// copy a single file into a directory
126+
await ipfs.files.cp('/example-file.txt', '/destination-directory')
127+
await ipfs.files.cp('/ipfs/QmWGeRAEgtsHW3ec7U4qW2CyVy7eA2mFRVbk1nb24jFyks', '/destination-directory')
128+
129+
// copy multiple files into a directory
130+
await ipfs.files.cp('/example-file-1.txt', '/example-file-2.txt', '/destination-directory')
131+
await ipfs.files.cp('/ipfs/QmWGeRAEgtsHW3ec7U4qW2CyVy7eA2mFRVbk1nb24jFyks',
132+
'/ipfs/QmWGeRAEgtsHW3jk7U4qW2CyVy7eA2mFRVbk1nb24jFyre', '/destination-directory')
133+
134+
// copy a directory into another directory
135+
await ipfs.files.cp('/source-directory', '/destination-directory')
136+
await ipfs.files.cp('/ipfs/QmWGeRAEgtsHW3ec7U4qW2CyVy7eA2mFRVbk1nb24jFyks', '/destination-directory')
137+
````
138+
139+
#### Move a file or directory
140+
141+
MFS allows you to move files between directories using the `ipfs.files.mv` which looks like this:
142+
143+
````
144+
await ipfs.files.mv(from, to, [options])
145+
````
146+
`from` is the source path (or paths) of the content you'd like to move while `to` is the destination path.
147+
148+
You can use this method to perform different operations including:
149+
150+
````
151+
// move a single file into a directory
152+
await ipfs.files.mv('/example-file.txt', '/destination-directory')
153+
154+
// move a directory into another directory
155+
await ipfs.files.mv('/source-directory', '/destination-directory')
156+
157+
// overwrite the contents of a destination file with the contents of a source file
158+
await ipfs.files.mv('/source-file.txt', '/destination-file.txt')
159+
160+
// move multiple files into a directory
161+
await ipfs.files.mv('/example-file-1.txt', '/example-file-2.txt', '/example-file-3.txt', '/destination-directory')
162+
````
163+
164+
#### Read the contents of a file
165+
166+
The `ipfs.files.read` method allows you to read and, or display the contents of a file in a buffer. The method takes the format:
167+
168+
````
169+
ipfs.files.read(path, [options])
170+
````
171+
The `path` provided is the path of the file to read, and it must point to a file rather than a directory.
172+
173+
174+
#### Remove a file or directory
175+
176+
MFS allows you to remove files or directories using the method:
177+
178+
````
179+
await ipfs.files.rm(...paths, [options])
180+
````
181+
182+
`paths` are one or more paths to remove.
183+
184+
By default, if you attempt to remove a directory that still has contents, the request will fail. To remove a directory and everything contained in it, you'll need to use the option {` recursive: true `}.
185+
186+
````
187+
// remove a file
188+
await ipfs.files.rm('/my/beautiful/file.txt')
189+
190+
// remove multiple files
191+
await ipfs.files.rm('/my/beautiful/file.txt', '/my/other/file.txt')
192+
193+
// remove a directory and its contents
194+
await ipfs.files.rm('/my/beautiful/directory', { recursive: true })
195+
196+
// remove a directory only if it is empty
197+
await ipfs.files.rm('/my/beautiful/directory')
198+
````
31199

32200
## Unix File System (UnixFS)
33201

34-
A file in IPFS isn’t just content. It might be too big to fit in a single block, so it needs metadata to link all its blocks together. It might be a symlink or a directory, so it needs metadata to link to other files. UnixFS is the data format used to represent files and all their links and metadata in IPFS, and is loosely based on how files work in Unix. When you add a _file_ to IPFS, you are creating a block (or a tree of blocks) in the UnixFS format.
202+
When you add a _file_ to IPFS it might be too big to fit in a single block, so it needs metadata to link all its blocks together. UnixFS is a [protocol-buffers](https://developers.google.com/protocol-buffers/)-based format for describing files, directories, and symlinks in IPFS. This data format is used to represent files and all their links and metadata in IPFS. UnixFS creates a block (or a tree of blocks) of linked objects.
203+
204+
UnixFS currently has [Javascript](https://github.com/ipfs/js-ipfs-unixfs) and [Go](https://github.com/ipfs/go-ipfs/tree/b3faaad1310bcc32dc3dd24e1919e9edf51edba8/unixfs) implementations. These implementations have modules written in to run different functions:
205+
206+
- **Data Formats**: manage the serialization/deserialization of UnixFS objects to protocol buffers
207+
208+
- **Importer**: Build DAGs from files and directories
209+
210+
- **Exporter**: Export DAGs
211+
212+
### Data Formats
213+
214+
On UnixFS-v1 the data format is represented by this protobuf:
215+
216+
````
217+
message Data {
218+
enum DataType {
219+
Raw = 0;
220+
Directory = 1;
221+
File = 2;
222+
Metadata = 3;
223+
Symlink = 4;
224+
HAMTShard = 5;
225+
}
226+
227+
required DataType Type = 1;
228+
optional bytes Data = 2;
229+
optional uint64 filesize = 3;
230+
repeated uint64 blocksizes = 4;
231+
optional uint64 hashType = 5;
232+
optional uint64 fanout = 6;
233+
optional uint32 mode = 7;
234+
optional UnixTime mtime = 8;
235+
}
236+
237+
message Metadata {
238+
optional string MimeType = 1;
239+
}
240+
241+
message UnixTime {
242+
required int64 Seconds = 1;
243+
optional fixed32 FractionalNanoseconds = 2;
244+
}
245+
````
246+
247+
This `Data` object is used for all non-leaf nodes in Unixfs:
248+
249+
- For files that are comprised of more than a single block, the `Type` field will be set to `File`, the `filesize` field will be set to the total number of bytes in the files, and `blocksizes` will contain a list of the filesizes of each child node.
250+
251+
- For files comprised of a single block, the `Type` filed will be set to `File`, `filesize` will be set to the total number of bytes in the file and file data will be stored in the `Data` field.
252+
253+
UnixFS also supports two optional metadata format fields:
254+
255+
- `mode` - used for persisting the file permissions in [numeric notation](https://en.wikipedia.org/wiki/File_system_permissions#Numeric_notation). If unspecified this defaults to `0755` for directories/HAMT shards and `0644` for all the other types where applicable.
256+
257+
- `mtime` - is a two-element structure (`Seconds`, `FractionalNanoseconds`) representing the modification time in seconds relative to the unix eposch `1970-01-01T00:00:00Z`.
258+
259+
### Importer
260+
261+
Importing a file into unixfs is split into two processes. A chunking function, and a layout function. You can test out how all this different features work using the IPFS [DAG builder](https://dag.ipfs.io)
262+
263+
#### Chunking
264+
265+
When an object is added to IPFS, it is chunked up into smaller parts, each part is hashed, and a CID is created for each chunk. This DAG building process has two main parameters, the leaf format and the chunking strategy.
266+
267+
The leaf format takes two format options, Unixfs leaves and raw leaves:
268+
269+
- The Unixfs leaves format adds a data wrapper on newly added objects to produce UnixFS leaves with additional data sizes. This wrapper is used to determine whether newly added objects are files or directories.
270+
271+
- The raw leaves format is the default format on IPFS where nodes output from chunking will be raw data from the file with a CID type of `'raw'`. This is mainly configured for backwards compatibility with formats that used a Unixfs Data object.
272+
273+
The chunking strategy is used to determine the size options available during the chunking process. The strategy currently has two different options, 'fixed size' and 'rabin'.
274+
275+
- Fixed sizing will chunk the input data into pieces of a given size, on IPFS this could be 512 bytes, 1024 bytes, and more. The smaller the byte size, the better the deduplication of data.
276+
277+
- Rabin chunking will chunk the input data using rabin fingerprinting to determine the boundaries between chunks. Rabin also reduces the number of input data chunked nodes.
278+
279+
#### Layout
280+
281+
Layout defines the shape of the tree that gets built from the chunks of the input file.
282+
283+
There are currently two options for layout, balanced, and trickle. Additionally, a 'max width' must be specified. The default max width is 174.
284+
285+
The balanced layout creates a balanced tree of width 'max width'. The tree is formed by taking up to 'max width' chunks from the chunk stream, and creating a unixfs file node that links to all of them. This is repeated until 'max width' unixfs file nodes are created, at which point a unixfs file node is created to hold all of those nodes, recursively. The root node of the resultant tree is returned as the handle to the newly imported file.
286+
287+
If there is only a single chunk, no intermediate unixfs file nodes are created, and the single chunk is returned as the handle to the file.
288+
289+
### Exporter
290+
291+
To export or read the file data out of the unixfs graph, perform an in order traversal, emitting the data contained in each of the leaves.
292+
293+
## Further resources
294+
295+
You can find additional resorces to familiarize with these file systems at:
35296

36-
UnixFS is a [protocol-buffers](https://developers.google.com/protocol-buffers/)-based format. You can find the definitions for it [here](https://github.com/ipfs/go-unixfs/blob/master/pb/unixfs.proto).
297+
- [Protoschool MFS course](https://proto.school/#/mutable-file-system)
298+
- [Understanding how the InterPlanetary File System deals with Files](https://github.com/ipfs/camp/tree/master/CORE_AND_ELECTIVE_COURSES/CORE_COURSE_A), from IPFS Camp 2019
299+
- [Jeromy Coffee Talks - Files API](https://www.youtube.com/watch?v=FX_AXNDsZ9k)
300+
- [UnixFS Specification](https://github.com/ipfs/specs/blob/master/UNIXFS.md)
37301

38-
_Note: we are currently designing an updated version of UnixFS that will be [IPLD](https://ipld.io)-compatible. You can follow along or participate [on GitHub](https://github.com/ipfs/unixfs-v2)._

0 commit comments

Comments
 (0)