Support testing with an in-process ethereum provider

[cgewecke]

🧐 Motivation

solidity-coverage is being rewritten so that it has fewer quirks and problems. One element of the new design is that it listens to the ethereumjs-vm opcode step emitter of an in-process ganache instance. In Truffle terms, it runs on a network which looks like:

const ganache = require("ganache-cli")

modules.exports = {
  networks: {
    ganache: {
      provider: ganache.provider(options),
      network_id: "*"
    }
  }
}

Have been testing it on Zeppelin and the only issue is that some GSN tests rely on an oz-gsn utility run in test.sh which connects to ganache-cli as a stand-alone server.

I'm wondering if anyone has any views are about moving away from test.sh towards an in-process provider approach.

I think the main changes this would involve are:

• Putting the accounts keys and balances into their own file so they can be JS required.
• Writing an oz-gsn deploy relay hub test helper that uses an in-process provider
• Launching the tests with ganache.provider by default.

Pros

• Less shell script
• There are interesting tools written to consume the client this way
  • NomicLabs is actively working on some...
  • 0xProject tooling

Cons/Caveats

• It runs a little more slowly. (ganache-core 481 requests info about this)
• Maintaining compatibility with a solution for Make tests run against Geth or Parity nodes #355?

Happy to help with PRs etc if this is something you're interested in.

[nventuro]

Hello @cgewecke! We of course will want to try out the new solidity-coverage once it comes out, and should modify our setup accordingly. I'm curious though, where does the need to have an in-process ganache come from? Will the new implementation be coupled to truffle, or is there some more generic API that can be used?

[cgewecke]

Hi @nventuro!

where does the need to have an in-process ganache come from?

Good question! An in-process ganache lets us access the VM directly and listen to a "step" event it emits each time an opcode is run. Solidity-coverage instruments contracts with statements which push a bytes32 data hash onto the evm stack. By watching the evm step we can detect the hash and mark it as a line or branch hit in a coverage map.

We need to collect opcode trace data for both calls and sends. At the moment ganache only provides a trace RPC endpoint for sends (debug_traceTransaction).

0xProject's trace tools get around this restriction by replaying calls as transactions so their data is 'available' and then reverting the chain. However they also reported that there's a lot of data-transfer overhead using debug_traceTransaction and there has been a reluctance at ganache to fix that by making the trace method filterable.

Using an in-process provider tries to avoid that complexity... fwiw it should be possible to adapt SC's design to work with a client like Parity which supports both traceTransaction and traceCall, although filtering might be necessary there as well.

Will the new implementation be coupled to truffle, or is there some more generic API that can be used?

It is strongly decoupled from Truffle. When the new version is released it will be available as both a Buidler and Truffle plugin and the API should allow integration with any JS ethereum dev stack or build process.

[nventuro]

(this is mostly me being curious)

Ohh, I see. And I'm guessing it is not possible to listen to step events on ethereumjs-vm when it is running as a separate process? I guess solidity-coverage needs more than just the ganache provider in order to work, is there a gist of the planed API available somewhere?

edit: to clarify, I'm asking because there are instances where we find it valuable to run tests on a standalone ganache instance, and want to see if there is a way to no drop that capability.

[cgewecke]

@nventuro

I'm guessing it is not possible to listen to step events on ethereumjs-vm when it is running as a separate process?

Like a websocket subscription or something? That would be really cool and has been proposed at ganache but not sure what its status is.

there are instances where we find it valuable to run tests on a standalone ganache instance,

I am curious about this - that seems very important. What do you see as the differences between in-process and standalone?

The exposed API will be something like the app class in this file. It's not finalized but it should be something very simple like

coverage.attachToProvider(provider) 
coverage.instrument(files) 
coverage.report()

[nventuro]

Like a websocket subscription or something?

Right, it technically wouldn't need to go over the network since it's running on the same machine, but that would work.

I am curious about this - that seems very important. What do you see as the differences between in-process and standalone?

My main use case is while debugging: I run some tests (a subset of them) on a standalone ganache, and then connect to it (e.g. via truffle console or some other tool) to manually inspect state, call functions, etc., to figure out what's wrong. Because running tests is often so slow, I find this approach lets me arrive at the solution much faster than iterating on the test suite directly.

[nventuro]

It seems to me though that it should be possible to have a sort of 'coverage-node', a stand-alone process that wraps ganache and exposes the instrument and report functions to the network. I imagine all the truffle plugin does is hook those functions to mocha's before and after blocks, which could be replicated this way.

[cgewecke]

@nventuro

I have to think about this a bit... some of the redesign goals are that the tool is :

• usable with any testing framework setup (ex: ethers & jest)
• usable with any ganache version (e.g. user can configure this - no fork dependence)

I imagine all the truffle plugin does is hook those functions to mocha's before and after blocks, which could be replicated this way.

The coverage API is trying to limit its concerns to rewriting solidity files, storing them in a pre-defined temporary folder and attaching to the client to get a trace. How the user compiles and tests, and which client version they use is ideally up to their tooling.

it should be possible to have a sort of 'coverage-node', a stand-alone process that wraps ganache

Just making a note here ... in your model :

• instrument writes instrumented solidity and its coverage mapping to the filesystem.
• there is a node script wrapper around ganache that:
  • consumes any ganache
  • hooks into the ganache vm
  • loads the coverage map (maybe when it receives a message that the tests are about to run)
  • forwards calls & responses between the test runner and the standalone
  • gets a message when the tests are complete and reports the outcome.

Does that seem right?

Very specifically for truffle I think it might be possible to point the console at a network that uses an in-process provider, run the test command at the console, and achieve the same debugging setup.

However you've raised a really important use case - will think about this further...

[nventuro]

Just making a note here ... in your model :

• instrument writes instrumented solidity and its coverage mapping to the filesystem.
• there is a node script wrapper around ganache that:
  • consumes any ganache
  • hooks into the ganache vm
  • loads the coverage map (maybe when it receives a message that the tests are about to run)
  • forwards calls & responses between the test runner and the standalone
  • gets a message when the tests are complete and reports the outcome.

Pretty much, yes. I assume by 'consume a ganache' you mean that it is launched from within node? Also, from my shallow understanding of the matter, we wouldn't have to do anything special to handle the 'call forwarding'.

Sorry to have thwarted your plans! 😅

[cgewecke]

Also, from my shallow understanding of the matter, we wouldn't have to do anything special to handle the 'call forwarding'.

@nventuro Maybe I am misunderstanding but to me this seems a little complicated - a special server wrapper around the provider (which supports http + websockets) and which has to be launched separately for coverage. This is similar to the current approach but I've been thinking it's preferable not to have a different client for every task (e.g normal test, test w/ coverage).

I actually really don't want to do what you're proposing ha ha!! It's simpler to handle as much as possible in the same process & memory as the tests.

However I just remembered that Oraclize has that bridge and is a case similar to your GSN deployer. So in-process is definitely a problem for some very important testing contexts....

[cgewecke]

Also for clarification, the way we need to access the vm trace is a little tortured. It has to be in memory and the attachment step looks something like this...

const self = this;
const provider = ganache.provider(options);

const blockchain = provider.engine.manager.state.blockchain;
const createVM = blockchain.createVMFromStateTrie;

// Attach to VM which ganache has already instantiated and 
// which it uses to execute eth_send
blockchain.vm.on('step', self.collector.step.bind(self.collector));

// Hijack createVM method which ganache uses to run eth_calls
blockchain.createVMFromStateTrie = function(state, activatePrecompiles) {
  const vm = createVM.apply(blockchain, arguments);
  vm.on('step', self.collector.step.bind(self.collector));
  return vm;
}

[nventuro]

I actually really don't want to do what you're proposing ha ha!! It's simpler to handle as much as possible in the same process & memory as the tests.

To be fair, I don't think you need to: from our discussion here and my understanding of the proposed API it seems to me like it shouldn't be hard to get ganache-core and the new solidity-coverage bundled together in the fashion I described above. We could do this ourselves if needed.

By the way, I just realized I never replied to the GSN deployment bit: that'll be a non-issue, our gsn-helpers already support being called from JavaScript directly, so that migration will be simple.

[cgewecke]

from our discussion here and my understanding of the proposed API it seems to me like it shouldn't be hard to get ganache-core and the new solidity-coverage bundled together in the fashion I described above. We could do this ourselves if needed.

Ok well you've persuaded me this is likely necessary anyway. If you'd like to collaborate in some way I'd be very happy to.

[cgewecke]

@nventuro Thanks so much for your advice in this thread - incredibly helpful.

Have now modified solidity-coverage to run ganache as an http server and added async hook options that allow the user to launch whatever external processes they want and connect to the coverage server as required.

Have opened #1923 upgrading solidity-coverage using this approach for GSN. It's not exactly what you suggested but it preserves the stand-alone server launch for regular tests. . .

Closing in favor #1923.