Support chrome.sockets.* APIs

[lidel]

About APIs

Chromium OS exposes chrome.sockets.* for its apps:

• https://developer.chrome.com/apps/manifest/sockets#manifest
  • https://developer.chrome.com/apps/sockets_tcp
  • https://developer.chrome.com/apps/sockets_tcpServer
  • https://developer.chrome.com/apps/sockets_udp

We can access those APIs in Brave by adding ipfs-companion (beta+stable) as blessed_extension in brave-core/../_api_features.json (suggested by @bbondy)

See also my notes on bringing IPFS to Brave in 2019: brave/brave-browser#819 (comment)

Potential use

• Exposing HTTP Gateway to embedded js-ipfs node
• Making js-ipfs basically as useful as go-ipfs for regular user
• I imagine we want to detect Brave in js-ipfs/.../config-browser.js#L8
• Local Discovery and UDP/TCP Transports
  (similar to @alanshaw's libdweb experiments on Firefox)

Relevant libraries

Created for and used in webtorrent:

• https://github.com/feross/chrome-net follows Node.js net (TCP) API
• https://github.com/feross/chrome-dgram follows Node.js dgram (UDP) API
• Use with browserify: webtorrent/package.json#L28-L30

MVP: Exposing HTTP Gateway

Quick notes about first steps:

1. Create Brave build that exposes chrome.sockets.* APIs to temporary extensions
   • Check it it works by default. If not, find Brave build instructions here: https://github.com/brave/brave-browser/wiki and whitelist rules are around brave-core/../_api_features.json
2. Detect js-ipfs is running in Brave and enable Gateway by providing a valid multiaddr instead of empty string
   • config-nodejs.js#L10 vs config-browser.js#L8
3. Try building ipfs-companion/js-ipfs with chrome-net and chrome-ngram as replacements for Node APIs in browser contexts
4. Start ipfs-companion in Brave, switch to embedded js-ipfs node
   • did it start and open Gateway on TCP port specified in step 2? is HTTP Gateway working as expected?
     • if yes, then we are lucky and this it! :)
     • if not, then we need to write additional code for exposing Gateway over chrome.sockets.*
5. Continue in Embedded JS-IPFS in Brave (experiment) #716

cc #312

[lidel]

Baby steps :) With some help I was able to confirm APIs are available in ipfs-companion background page after whitelisting extension. Notes below.

How To Develop with Local (Unpacked) Extension (old master)

Click to expand

0. (prerequisite) Created bundle:brave build script that adds sockets section to extension manifest – this is already in this branch

1. Built (npm run dev-build) and started Brave (npm start). First build takes 4h+, but following ones are faster.

2. Enabled "Developer mode" and loaded an unpacked extension with , it got assigned ID paanflafljinkomjoheoobkeigpakegp.

3. Generated ID shortcut in format expected by Brave..

   $ echo -n paanflafljinkomjoheoobkeigpakegp | openssl sha1 | tr '[:lower:]' '[:upper:]'
   (STDIN)= CEBF34BE446B77F39E2C535566F8C27F9A460B1E

4. ..and added it to whitelists for sockets.* APIs in two files in src/brave (brave-core):

--- a/common/extensions/api/_api_features.json
+++ b/common/extensions/api/_api_features.json
@@ -61,17 +61,17 @@
   "sockets.tcp": {
     "dependencies": ["manifest:sockets"],
     "contexts": ["blessed_extension"],
-    "whitelist": ["3D9518A72EB02667A773B69DBA9E72E0F4A37423"]
+    "whitelist": ["3D9518A72EB02667A773B69DBA9E72E0F4A37423","CEBF34BE446B77F39E2C535566F8C27F9A460B1E"]
   },
   "sockets.tcpServer": {
     "dependencies": ["manifest:sockets"],
     "contexts": ["blessed_extension"],
-    "whitelist": ["3D9518A72EB02667A773B69DBA9E72E0F4A37423"]
+    "whitelist": ["3D9518A72EB02667A773B69DBA9E72E0F4A37423","CEBF34BE446B77F39E2C535566F8C27F9A460B1E"]
   },
   "sockets.udp": {
     "dependencies": ["manifest:sockets"],
     "contexts": ["blessed_extension"],
-    "whitelist": ["3D9518A72EB02667A773B69DBA9E72E0F4A37423"]
+    "whitelist": ["3D9518A72EB02667A773B69DBA9E72E0F4A37423","CEBF34BE446B77F39E2C535566F8C27F9A460B1E"]
   },
   "braveRewards": {
     "channel": "stable",
diff --git a/common/extensions/api/_manifest_features.json b/common/extensions/api/_manifest_features.json
index fcacdb73..db024427 100644
--- a/common/extensions/api/_manifest_features.json
+++ b/common/extensions/api/_manifest_features.json
@@ -6,6 +6,6 @@
   "sockets": {
     "channel": "stable",
     "extension_types": ["extension", "platform_app"],
-    "whitelist": ["3D9518A72EB02667A773B69DBA9E72E0F4A37423"]
+    "whitelist": ["3D9518A72EB02667A773B69DBA9E72E0F4A37423","CEBF34BE446B77F39E2C535566F8C27F9A460B1E"]
   }
 }

5. Rebuilt Brave via npm run build
6. Started it via npm start
7. Loaded extension again, it got the same ID (paanflafljinkomjoheoobkeigpakegp)
8. Opened Console for Companion's background page via "Inspect views" on chrome://extensions and confirmed chrome.sockets API is present 🎉

   

How To Develop with Local (Unpacked) Extension (ipfs branch of Brave)

1. Switch extension to feat/brave-build-with-chrome-sockets branch and run npm run build bundle:brave build script to produce extension manifest with sockets section.

2. Switch Brave to ipfs branch, build (npm run build) and start Brave (npm start). First build takes 4h+, but following ones are faster.

3. Enabled "Developer mode" and loaded an unpacked extension.

4. Opened Console for Companion's background page via "Inspect views" on chrome://extensions and confirmed chrome.sockets API is present 🎉

Next

Override js-ipfs config to make it start Gateway and set up ipfs-companion build to see if use of chrome-net and chrome-ngram is enough to get it working.

[lidel]

First checkpoint, http backed by http-node and chrome-net works from ipfs-companion:

let http = require('http')
let server = http.createServer(function (req, res) {                                                                                                                                                                                        
  res.writeHead(200, { 'Content-Type': 'text/plain' })                                                                                                                                                                                      
  res.end('Hello from ipfs-companion exposing HTTP via chrome.sockets in Brave :-)\n')                                                                                                                                                      
})                                                                                                                                                                                                                                          
server.listen(9091, '127.0.0.1')

Next

Getting js-ipfs working may take more than this, as http-node was created from Node 6.x and modern js-ipfs throws ipfs-inactive/js-ipfs-http-client#840.

[lidel]

Second checkpoint: hapi v18 (HTTP server used by js-ipfs) works.

I took js-ipfs 0.35.0-pre.0 for a spin.
It ships with hapi.js v18, HTTP server used for exposing Gateway.

Note: see changes in feat/brave-build-with-chrome-sockets branch

• start raw http server (http.createServer)
• start raw hapi server (Hapi.Server) (screenshot below)
  • after some troubleshooting I managed to start raw Hapi server but got internal error for all responses
• return valid response from hapi server running in browser extension
  • got it to work by replacing url with iso-url during webpack build

Demo: hapi v18 (HTTP server) running in ipfs-companion

TypeError: this._dht.on is not a function comes from libp2p, see Next below

Next

• start embedded js-ipfs with Gateway exposed by embedded Hapi server
  • right now new Ipfs(..) fails due to TypeError: this._dht.on is not a function, which is quite surprisingly not related to HTTP server (which should just work). It seems libp2p does not disable DHT fully when we polyfill net/dgram using chrome.sockets, which results in TypeError around js-libp2p/src/index.js#L404
• start js-ipfs-http-client
  • Does not work for now due to TypeError: res.req.getHeaders is not a function – http API mismatch triggered in js-ipfs-http-client/src/utils/send-request.js#L41

[drbh]

@lidel great progress! How'd you get the Hapi route response to not throw an error? When I try to get a response from :9092/ I get a 500 and the console says TypeError: Uncaught error: Cannot read property '_replied' of null

Aside from that, what are the next steps for handling the roadblocks you noted above?

If we completely disable DHT - should the embedded js-ipfs Gateway run?

[lidel]

@drbh I believe _replied error went away when I switched to new hapi/js-ipfs. Try:

• fetching recent updates from feat/brave-build-with-chrome-sockets branch
• rebulding Companion via npm run dev-build bundle:brave

As for remaining roadblocks:

• this._dht.on is not a function – I tried to disable DHT via config passed to new Ipfs(), but the error remained – not sure why. It is something I will be looking next (need to understand code around js-libp2p/src/index.js#L404 first)
• res.req.getHeaders is not a function – this is a low priority (we want to get js-ipfs online first), but my current guess is this will probably require update of http-node (http api shim which is 3 years old)