Skip to content
This repository was archived by the owner on Feb 12, 2024. It is now read-only.

Commit dfe15d7

Browse files
achingbrainjacobheun
achingbrain
and
jacobheun
authored
feat: enable DHT by Routing.Type config key (#3153)
Updates use of the kad-dht module in the codebase. New features: 1. Adds [Routing.Type](https://github.com/ipfs/go-ipfs/blob/b86e93045aaa0ac357d939884e1c7996b0fcdbd4/docs/config.md#routing) configuration key with values - `'none'` default, DHT disabled - `'dht'` DHT start-as-client-auto-upgrade-to-server (not implemented in `js-libp2p-kad-dht` yet) - `'dhtserver'` DHT server mode - `'dhtclient'` DHT client mode 2. Updates Core API to be compliant with the docs Co-authored-by: Jacob Heun <[email protected]>
1 parent 281bfe6 commit dfe15d7

File tree

16 files changed

+166
-67
lines changed

16 files changed

+166
-67
lines changed

‎examples/custom-libp2p/package.json

+2-2
Original file line numberDiff line numberDiff line change
@@ -11,9 +11,9 @@
1111
"license": "MIT",
1212
"dependencies": {
1313
"ipfs": "^0.47.0",
14-
"libp2p": "^0.28.0",
14+
"libp2p": "^0.28.5",
1515
"libp2p-bootstrap": "^0.11.0",
16-
"libp2p-kad-dht": "^0.19.5",
16+
"libp2p-kad-dht": "^0.19.9",
1717
"libp2p-mdns": "^0.14.1",
1818
"libp2p-mplex": "^0.9.5",
1919
"libp2p-secio": "^0.12.4",

‎packages/ipfs/.aegir.js

+3-5
Original file line numberDiff line numberDiff line change
@@ -68,11 +68,9 @@ module.exports = {
6868
ipfsHttpModule: require('ipfs-http-client'),
6969
ipfsBin: path.join(__dirname, 'src', 'cli', 'bin.js'),
7070
ipfsOptions: {
71-
config: {
72-
libp2p: {
73-
dialer: {
74-
dialTimeout: 60e3 // increase timeout because travis is slow
75-
}
71+
libp2p: {
72+
dialer: {
73+
dialTimeout: 60e3 // increase timeout because travis is slow
7674
}
7775
}
7876
}

‎packages/ipfs/docs/DELEGATE_ROUTERS.md

+3
Original file line numberDiff line numberDiff line change
@@ -13,8 +13,11 @@ If you need to support Delegated Content and/or Peer Routing, you can enable it
1313
If you are not able to run your own delegate router nodes, we currently have two nodes that support delegated routing. **Important**: As many people may be leveraging these nodes, performance may be affected, which is why we recommend running your own nodes in production.
1414

1515
Available delegate multiaddrs are:
16+
1617
- `/dns4/node0.delegate.ipfs.io/tcp/443/https`
1718
- `/dns4/node1.delegate.ipfs.io/tcp/443/https`
19+
- `/dns4/node2.delegate.ipfs.io/tcp/443/https`
20+
- `/dns4/node3.delegate.ipfs.io/tcp/443/https`
1821

1922
**Note**: If more than 1 delegate multiaddr is specified, the actual delegate will be randomly selected on startup.
2023

‎packages/ipfs/docs/FAQ.md

+23-2
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22

33
- [FAQ](#faq)
44
- [Why isn't there DHT support in js-IPFS?](#why-isnt-there-dht-support-in-js-ipfs)
5+
- [Node.js](#nodejs)
6+
- [Browser](#browser)
57
- [How to enable WebRTC support for js-ipfs in the Browser](#how-to-enable-webrtc-support-for-js-ipfs-in-the-browser)
68
- [Is there WebRTC support for js-ipfs with Node.js?](#is-there-webrtc-support-for-js-ipfs-with-nodejs)
79
- [How can I configure an IPFS node to use a custom `signaling endpoint` for my WebRTC transport?](#how-can-i-configure-an-ipfs-node-to-use-a-custom-signaling-endpoint-for-my-webrtc-transport)
@@ -13,7 +15,26 @@
1315

1416
There is DHT support for js-IPFS in the form of [libp2p/js-libp2p-kad-dht](https://github.com/libp2p/js-libp2p-kad-dht) but it is not finished yet, and may not be the right solution to the problem.
1517

16-
Historically js-IPFS has targeted browser environments. The constraints imposed by browsers do not typically make for good DHT participants - people do not tend to stay on a page for long enough to make or answer DHT queries, and even if they did, most nodes on the network talk TCP - the browser can neither open TCP ports on remote hosts nor accept TCP connections.
18+
### Node.js
19+
20+
To enable DHT support, before starting your daemon run:
21+
22+
```console
23+
$ jsipfs config Routing.Type dht
24+
```
25+
26+
The possible values for `Routing.Type` are:
27+
28+
- `'none'` the default, this means the DHT is turned off any you must manually dial other nodes
29+
- `'dht'` start the node in DHT client mode, if it is discovered to be publicly dialable it will automatically switch to server mode
30+
- `'dhtclient'` A DHT client is able to make DHT queries but will not respond to any
31+
- `'dhtserver'` A DHT server can make and respond to DHT queries. Please only choose this option if your node is dialable from the open Internet.
32+
33+
At the time of writing, only DHT client mode is supported and will be selected if `Routing.Type` is not `'none'`.
34+
35+
### Browser
36+
37+
In the browser there are many constraints that mean the environment does not typically make for good DHT participants - the number of connections required is high, people do not tend to stay on a page for long enough to make or answer DHT queries, and even if they did, most nodes on the network talk TCP - the browser can neither open TCP ports on remote hosts nor accept TCP connections.
1738

1839
A better approach may be to set up [Delegate Routing](./DELEGATE_ROUTERS.md) to use remote go-IPFS to make queries on the browsers' behalf as these do not have the same constraints.
1940

@@ -122,4 +143,4 @@ Still if you run into problems with native modules follow these instructions [he
122143

123144
## Have more questions?
124145

125-
Ask for help in our forum at https://discuss.ipfs.io or in IRC (#ipfs on Freenode).
146+
Ask for help in our forum at https://discuss.ipfs.io or in IRC (#ipfs on Freenode).

‎packages/ipfs/package.json

+2-2
Original file line numberDiff line numberDiff line change
@@ -128,14 +128,14 @@
128128
"iterable-ndjson": "^1.1.0",
129129
"jsondiffpatch": "^0.4.1",
130130
"just-safe-set": "^2.1.0",
131-
"libp2p": "^0.28.0",
131+
"libp2p": "^0.28.5",
132132
"libp2p-bootstrap": "^0.11.0",
133133
"libp2p-crypto": "^0.17.6",
134134
"libp2p-delegated-content-routing": "^0.5.0",
135135
"libp2p-delegated-peer-routing": "^0.5.0",
136136
"libp2p-floodsub": "^0.21.0",
137137
"libp2p-gossipsub": "^0.4.0",
138-
"libp2p-kad-dht": "^0.19.5",
138+
"libp2p-kad-dht": "^0.19.9",
139139
"libp2p-mdns": "^0.14.1",
140140
"libp2p-mplex": "^0.9.3",
141141
"libp2p-noise": "^1.1.1",

‎packages/ipfs/src/cli/commands/daemon.js

-1
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,6 @@ module.exports = {
7676
preload: { enabled: argv.enablePreload },
7777
EXPERIMENTAL: {
7878
ipnsPubsub: argv.enableNamesysPubsub,
79-
dht: argv.enableDhtExperiment,
8079
sharding: argv.enableShardingExperiment
8180
},
8281
init: argv.initProfile ? { profiles: argv.initProfile } : true

‎packages/ipfs/src/core/components/dht.js

+24-14
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,10 @@ module.exports = ({ libp2p, repo }) => {
2121

2222
if (!Buffer.isBuffer(key)) {
2323
try {
24+
key = key.toString().split('/')
25+
.filter(part => part && part !== 'ipfs' && part !== 'ipns')
26+
.shift()
27+
2428
key = (new CID(key)).buffer
2529
} catch (err) {
2630
throw errCode(err, 'ERR_INVALID_CID')
@@ -44,6 +48,10 @@ module.exports = ({ libp2p, repo }) => {
4448
put: withTimeoutOption(async (key, value) => { // eslint-disable-line require-await
4549
if (!Buffer.isBuffer(key)) {
4650
try {
51+
key = key.toString().split('/')
52+
.filter(part => part && part !== 'ipfs' && part !== 'ipns')
53+
.shift()
54+
4755
key = (new CID(key)).buffer
4856
} catch (err) {
4957
throw errCode(err, 'ERR_INVALID_CID')
@@ -89,7 +97,7 @@ module.exports = ({ libp2p, repo }) => {
8997
* Query the DHT for all multiaddresses associated with a `PeerId`.
9098
*
9199
* @param {PeerId} peerId - The id of the peer to search for.
92-
* @returns {Promise<{ id: CID, addrs: Multiaddr[] }>}
100+
* @returns {Promise<{ id: String, addrs: Multiaddr[] }>}
93101
*/
94102
findPeer: withTimeoutOption(async peerId => { // eslint-disable-line require-await
95103
if (typeof peerId === 'string') {
@@ -100,45 +108,47 @@ module.exports = ({ libp2p, repo }) => {
100108

101109
return {
102110
id: peer.id.toB58String(),
103-
addrs: peer.addrs
111+
addrs: peer.multiaddrs
104112
}
105113
}),
106114

107115
/**
108116
* Announce to the network that we are providing given values.
109117
*
110-
* @param {CID|CID[]} keys - The keys that should be announced.
118+
* @param {CID|CID[]} cids - The keys that should be announced.
111119
* @param {Object} [options] - provide options
112120
* @param {bool} [options.recursive=false] - Provide not only the given object but also all objects linked from it.
113121
* @returns {Promise}
114122
*/
115-
provide: withTimeoutOption(async (keys, options) => {
116-
keys = Array.isArray(keys) ? keys : [keys]
123+
provide: withTimeoutOption(async function * (cids, options) {
124+
cids = Array.isArray(cids) ? cids : [cids]
117125
options = options || {}
118126

119-
for (var i in keys) {
120-
if (typeof keys[i] === 'string') {
127+
for (var i in cids) {
128+
if (typeof cids[i] === 'string') {
121129
try {
122-
keys[i] = new CID(keys[i])
130+
cids[i] = new CID(cids[i])
123131
} catch (err) {
124132
throw errCode(err, 'ERR_INVALID_CID')
125133
}
126134
}
127135
}
128136

129137
// ensure blocks are actually local
130-
const hasKeys = await Promise.all(keys.map(k => repo.blocks.has(k)))
131-
const hasAll = hasKeys.every(has => has)
138+
const hasCids = await Promise.all(cids.map(cid => repo.blocks.has(cid)))
139+
const hasAll = hasCids.every(has => has)
132140

133141
if (!hasAll) {
134-
throw errCode('block(s) not found locally, cannot provide', 'ERR_BLOCK_NOT_FOUND')
142+
throw errCode(new Error('block(s) not found locally, cannot provide'), 'ERR_BLOCK_NOT_FOUND')
135143
}
136144

137145
if (options.recursive) {
138146
// TODO: Implement recursive providing
139-
throw errCode('not implemented yet', 'ERR_NOT_IMPLEMENTED_YET')
140-
} else {
141-
await Promise.all(keys.map(k => libp2p._dht.provide(k)))
147+
throw errCode(new Error('not implemented yet'), 'ERR_NOT_IMPLEMENTED_YET')
148+
}
149+
150+
for (const cid of cids) {
151+
yield libp2p._dht.provide(cid)
142152
}
143153
}),
144154

‎packages/ipfs/src/core/components/libp2p.js

+15-4
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,8 @@ module.exports = ({
3434

3535
// Required inline to reduce startup time
3636
const Libp2p = require('libp2p')
37-
return new Libp2p(mergeOptions(libp2pOptions, get(options, 'libp2p', {})))
37+
38+
return new Libp2p(libp2pOptions)
3839
}
3940

4041
function getLibp2pOptions ({ options, config, datastore, keys, keychainConfig, peerId, multiaddrs }) {
@@ -54,7 +55,6 @@ function getLibp2pOptions ({ options, config, datastore, keys, keychainConfig, p
5455
modules: {}
5556
}
5657

57-
const bootstrapList = get(options, 'config.Bootstrap', get(config, 'Bootstrap', []))
5858
const libp2pOptions = {
5959
modules: {
6060
pubsub: getPubsubRouter()
@@ -70,7 +70,7 @@ function getLibp2pOptions ({ options, config, datastore, keys, keychainConfig, p
7070
get(config, 'Discovery.webRTCStar.Enabled', true))
7171
},
7272
bootstrap: {
73-
list: bootstrapList
73+
list: get(options, 'config.Bootstrap', get(config, 'Bootstrap', []))
7474
}
7575
},
7676
relay: {
@@ -84,6 +84,8 @@ function getLibp2pOptions ({ options, config, datastore, keys, keychainConfig, p
8484
}
8585
},
8686
dht: {
87+
enabled: get(config, 'Routing.Type', 'none') !== 'none',
88+
clientMode: get(config, 'Routing.Type', 'dht') !== 'dhtserver',
8789
kBucketSize: get(options, 'dht.kBucketSize', 20)
8890
},
8991
pubsub: {
@@ -110,13 +112,22 @@ function getLibp2pOptions ({ options, config, datastore, keys, keychainConfig, p
110112
// Note: libp2p-nodejs gets replaced by libp2p-browser when webpacked/browserified
111113
const getEnvLibp2pOptions = require('../runtime/libp2p-nodejs')
112114

115+
let constructorOptions = get(options, 'libp2p', {})
116+
117+
if (typeof constructorOptions === 'function') {
118+
constructorOptions = {}
119+
}
120+
113121
// Merge defaults with Node.js/browser/other environments options and configuration
114122
const libp2pConfig = mergeOptions(
115123
libp2pDefaults,
116124
getEnvLibp2pOptions(),
117-
libp2pOptions
125+
libp2pOptions,
126+
constructorOptions
118127
)
119128

129+
const bootstrapList = get(libp2pConfig, 'config.peerDiscovery.bootstrap.list', [])
130+
120131
if (bootstrapList.length > 0) {
121132
libp2pConfig.modules.peerDiscovery.push(require('libp2p-bootstrap'))
122133
}

‎packages/ipfs/src/core/components/start.js

+7-3
Original file line numberDiff line numberDiff line change
@@ -198,13 +198,17 @@ function createApi ({
198198
throw new NotEnabledError('dht not enabled')
199199
}
200200

201+
const dhtNotEnabledIterator = async function * () { // eslint-disable-line require-await,require-yield
202+
throw new NotEnabledError('dht not enabled')
203+
}
204+
201205
const dht = get(libp2p, '_config.dht.enabled', false) ? Components.dht({ libp2p, repo }) : {
202206
get: dhtNotEnabled,
203207
put: dhtNotEnabled,
204-
findProvs: dhtNotEnabled,
208+
findProvs: dhtNotEnabledIterator,
205209
findPeer: dhtNotEnabled,
206-
provide: dhtNotEnabled,
207-
query: dhtNotEnabled
210+
provide: dhtNotEnabledIterator,
211+
query: dhtNotEnabledIterator
208212
}
209213

210214
const dns = Components.dns()

‎packages/ipfs/src/core/runtime/config-browser.js

+3
Original file line numberDiff line numberDiff line change
@@ -37,5 +37,8 @@ module.exports = () => ({
3737
LowWater: 200,
3838
HighWater: 500
3939
}
40+
},
41+
Routing: {
42+
Type: 'none'
4043
}
4144
})

‎packages/ipfs/src/core/runtime/config-nodejs.js

+3
Original file line numberDiff line numberDiff line change
@@ -51,5 +51,8 @@ module.exports = () => ({
5151
LowWater: 200,
5252
HighWater: 500
5353
}
54+
},
55+
Routing: {
56+
Type: 'none'
5457
}
5558
})

‎packages/ipfs/src/core/runtime/libp2p-browser.js

+1
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ module.exports = () => {
4848
dht: {
4949
kBucketSize: 20,
5050
enabled: false,
51+
clientMode: true,
5152
randomWalk: {
5253
enabled: false
5354
},

‎packages/ipfs/src/core/runtime/libp2p-nodejs.js

+1
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,7 @@ module.exports = () => {
5151
dht: {
5252
kBucketSize: 20,
5353
enabled: false,
54+
clientMode: true,
5455
randomWalk: {
5556
enabled: false
5657
},

0 commit comments

Comments
 (0)