How to add a directory with sharding enabled?

[daviddias]

Doesn't seem to be an option today at https://godoc.org/github.com/ipfs/interface-go-ipfs-core#UnixfsAPI

[aschmahmann]

@daviddias I could be wrong here, but from what I can tell it relies on the use of a set global variable:

• The global: https://github.com/ipfs/go-unixfs/blob/25a8acedf8c65251dd15a39f9125dbb8973ab5dc/io/directory.go#L19
• Creating a directory in UnixFS: https://github.com/ipfs/go-unixfs/blob/25a8acedf8c65251dd15a39f9125dbb8973ab5dc/io/directory.go#L85

This variable can be set as an IPFS config option: https://github.com/ipfs/go-ipfs/blob/master/docs/experimental-features.md#directory-sharding--hamt

[daviddias]

Thank you jumping in to help, @aschmahmann!

Does that mean that IPFS operates either in a "every dir is sharded" or "no dir is sharded"? Is there any technical reason why both can't happen at the same time?

[aschmahmann]

It looks like it, but more precisely it's either "every new directory is sharded" or "no new directory is sharded". I think we can still read sharded directories even without passing the global flag.

Not sure why we don't, but the readme on go-ipfs experimental features (https://github.com/ipfs/go-ipfs/blob/master/docs/experimental-features.md#directory-sharding--hamt) has "Make sure that objects that don't have to be sharded aren't" as one of the blockers on the road to being a real feature.

I'm sure @Stebalien or @mikeal would know more details (e.g. is this blocked on unixfs v2, etc.)

[Stebalien]

• UnixFSv2 will switch to an entirely new sharding library that won't have the over-sharding issues we currently have.
• I believe we used a global switch for simplicity. We probably intended to turn it on fairly quickly and didn't want to feed through an option. However, I have absolutely no idea if this is correct.

[hacdias]

I already figured out how to enable sharding with the current configuration. Taking in account that you have a configuration object of the type *config.Config, you can just change Experimental.ShardingEnabled to true, just like on this snippet:

// imports
config "github.com/ipfs/go-ipfs-config"
"github.com/ipfs/go-ipfs/repo/fsrepo"

// code
cfg, _ := config.Init(ioutil.Discard, 2048)
cfg.Experimental.ShardingEnabled = true

// then create the repository with that config
fsrepo.Init(repoPath, cfg)

[hacdias]

@daviddias I think we can close this!

[daviddias]

@hacdias can you make sure this gets documented?

[hacdias]

@daviddias ipfs/kubo#6792