Deploy flags e2e #2755

sukantoraymond · 2025-04-17T22:48:33Z

Our current E2E for tests/e2e/testcases is pretty extensive for local network across all commands. However, it does not test flags input extensively for happy path and error path.

For example, our E2E tests for blockchain deploy in https://github.com/ava-labs/avalanche-cli/blob/main/tests/e2e/testcases/subnet/sov/ currently tests that we are able to deploy to local network using PoA / PoS / Initialize Validator Manager. However, it does not test the multiple ways that we can achieve each objectives (for example we can use deploy an L1 with by setting num-local-machine=2 instead of using default value of 1).

This PR proposes using json files as input that is fed into a general function (TestCommandWithJSONConfig), which can be reused for all blockchain subcommands (e.g. blockchain addValidator, etc) simply by feeding it with different JSON file. Reusing same core testing logic enables us to achieve extensive happy path and error path coverage without huge additional testing code.

sukantoraymond · 2025-04-17T23:22:00Z

tests/e2e/commands_e2e/blockchain/deploy/deploy_tests.json

+  "happyPath": [
+    {
+      "name": "local_deploy",
+      "flags": {}
+    }


we can set globalFlags in each testCases as well, which will override the globalFlags value

sukantoraymond · 2025-04-18T16:20:14Z

tests/e2e/commands_e2e/blockchain/deploy/suite.go

+var _ = ginkgo.Describe("[Blockchain Deploy Flags]", ginkgo.Ordered, func() {
+	_ = ginkgo.BeforeEach(func() {
+		// Create test subnet config
+		commands.CreateSubnetEvmConfigSOV(subnetName, utils.SubnetEvmGenesisPath, ewoqEVMAddress)


We use ewoq here because we use ewoq as validator manager owner when we deploy in local network

Tonix517 · 2025-04-18T18:00:38Z

tests/e2e/commands/blockchain/blockchain_command.go

+}
+
+// TestCommandWithJSONConfig tests a CLI command with flag inputs from a JSON file
+func TestCommandWithJSONConfig(command string, configPath string, testCase *TestCase) (string, error) {


love the idea - any plan to move it to some test common directories to enable it reusable by all other commands?

Thanks! Yes i will move it to external directories

Generalized the function

sukantoraymond · 2025-04-21T16:09:52Z

Example of how this PR reduces code while increasing code coverage:

Currently, when we are testing key transfer:

func KeyTransferSend(keyName string, targetAddr string, amount string) (string, error) {
	// Create config
	args := []string{
		KeyCmd,
		"transfer",
		"--local",
		"--key",
		keyName,
		"--destination-addr",
		targetAddr,
		"--amount",
		amount,
		"--p-chain-sender",
		"--p-chain-receiver",
		"--" + constants.SkipUpdateFlag,
	}
	cmd := exec.Command(CLIBinary, args...)

	out, err := cmd.CombinedOutput()
	return string(out), err
}

We have to write a custom function to call key transfer command (and this function only does p chain to p chain transfer). Additionally, since we will be testing transfer from c to p chain, we have to make several more custom functions -> each function is 20 lines!

Compare that with using TestCommandWithJSONConfig. We no longer need to write custom function. Just create a json file with any scenarios you want (P->C, C->X, etc) and we can test happy path and error path! (e.g. balance not enough for transfer, etc).

Using JSON ensures that every possible flag is tested, compared to doing this right now, where we can't verify that we have tested every scenario:

ginkgo.It("can transfer between keys", func() {
		_ = utils.DeleteKey(keyName)
		_ = utils.DeleteKey(ewoqKeyName)
		output, err := commands.CreateKeyFromPath(ewoqKeyName, utils.EwoqKeyPath)
		if err != nil {
			fmt.Println(output)
			utils.PrintStdErr(err)
		}
		gomega.Expect(err).Should(gomega.BeNil())
		output, err = commands.CreateKey(keyName)
		if err != nil {
			fmt.Println(output)
			utils.PrintStdErr(err)
		}
		gomega.Expect(err).Should(gomega.BeNil())
		commands.StartNetworkWithVersion("")

		amount := 0.2
		amountStr := fmt.Sprintf("%.2f", amount)
		amountNAvax := uint64(amount * float64(units.Avax))

		// send/receive without recovery

		output, err = commands.ListKeys("local", true, true)
		gomega.Expect(err).Should(gomega.BeNil())
		keyAddr, keyBalance1, err := utils.ParseAddrBalanceFromKeyListOutput(output, keyName)
		gomega.Expect(err).Should(gomega.BeNil())
		_, ewoqKeyBalance1, err := utils.ParseAddrBalanceFromKeyListOutput(output, ewoqKeyName)
		gomega.Expect(err).Should(gomega.BeNil())

		output, err = commands.KeyTransferSend(ewoqKeyName, keyAddr, amountStr)
		if err != nil {
			fmt.Println(output)
		}
		gomega.Expect(err).Should(gomega.BeNil())

		feeNAvax, err := utils.GetKeyTransferFee(output)
		gomega.Expect(err).Should(gomega.BeNil())

		output, err = commands.ListKeys("local", true, true)
		gomega.Expect(err).Should(gomega.BeNil())
		_, keyBalance2, err := utils.ParseAddrBalanceFromKeyListOutput(output, keyName)
		gomega.Expect(err).Should(gomega.BeNil())
		_, ewoqKeyBalance2, err := utils.ParseAddrBalanceFromKeyListOutput(output, ewoqKeyName)
		gomega.Expect(err).Should(gomega.BeNil())
		gomega.Expect(feeNAvax + amountNAvax).Should(gomega.Equal(ewoqKeyBalance1 - ewoqKeyBalance2))
		gomega.Expect(keyBalance2 - keyBalance1).Should(gomega.Equal(amountNAvax))

		output, err = commands.ListKeys("local", true, true)
		gomega.Expect(err).Should(gomega.BeNil())
		_, keyBalance3, err := utils.ParseAddrBalanceFromKeyListOutput(output, keyName)
		gomega.Expect(err).Should(gomega.BeNil())
		_, ewoqKeyBalance3, err := utils.ParseAddrBalanceFromKeyListOutput(output, ewoqKeyName)
		gomega.Expect(err).Should(gomega.BeNil())
		gomega.Expect(ewoqKeyBalance1 - ewoqKeyBalance3).Should(gomega.Equal(feeNAvax + amountNAvax))
		gomega.Expect(keyBalance3 - keyBalance1).Should(gomega.Equal(amountNAvax))
	})

felipemadero · 2025-04-22T12:46:39Z

tests/e2e/commands/subnet.go

@@ -85,7 +85,7 @@ func CreateSubnetEvmConfigWithVersionNonSOV(subnetName string, genesisPath strin
 	gomega.Expect(exists).Should(gomega.BeTrue())
 }

-func CreateSubnetEvmConfigWithVersionSOV(subnetName string, genesisPath string, version string) {
+func CreateSubnetEvmConfigGenesisPathWithVersionSOV(subnetName string, genesisPath string, version string, validatorManagerOwner string) {


we need a generic function that just take the command name, the non flag args, and a map for flags, and
stopping doing this

felipemadero · 2025-04-22T12:49:12Z

tests/e2e/commands/subnet.go

@@ -236,9 +236,9 @@ func CreateCustomVMConfigSOV(subnetName string, genesisPath string, vmPath strin
 		genesisPath,
 		"--proof-of-authority",
 		"--validator-manager-owner",


we also need to stop using flag constants here, and take those from cmd/flags

felipemadero · 2025-04-22T12:49:42Z

.github/workflows/e2e-test.yml

@@ -35,6 +35,7 @@ jobs:
            "\\[Node create\\]",
            "\\[Node devnet\\]",
            "\\[Docker\\]",
+            "\\[Blockchain Deploy Flags\\]",


not sure we want to start modifying CI at this stage

felipemadero · 2025-04-22T12:50:35Z

tests/e2e/commandse2e/blockchain/deploy/deploy_tests.json

@@ -0,0 +1,22 @@
+{


I like this cmd tree. probably this can have a default name, like test_table.json

felipemadero · 2025-04-22T12:51:28Z

tests/e2e/commandse2e/command.go

+type CommandGroup string
+
+const (
+	BlockchainCmd CommandGroup = "blockchain"


I would rather add this constant on cmd/blockchaincmd/ and import that here

felipemadero · 2025-04-22T12:55:43Z

tests/e2e/commandse2e/command.go

+// TestCase represents a single test case configuration
+type TestCase struct {
+	Name          string            `json:"name"`
+	Flags         map[string]string `json:"flags"`


You also need non flag args here

felipemadero · 2025-04-22T12:56:19Z

tests/e2e/commandse2e/command.go

+	}
+
+	// Append any additional arguments
+	if len(args) > 0 {


this should belong to the test case

felipemadero · 2025-04-22T13:01:52Z

tests/e2e/commandse2e/blockchain/deploy/suite.go

+			ginkgo.By(fmt.Sprintf("Running test case: %s", testCase.Name))
+			_, err = commandse2e.TestCommandWithJSONConfig(
+				commandse2e.BlockchainCmd,
+				"deploy",


this should be a constant on cmd/blockchaincmd/

felipemadero · 2025-04-22T13:03:02Z

tests/e2e/commandse2e/blockchain/deploy/suite.go

+		// Cleanup test subnet config
+		commands.DeleteSubnetConfig(subnetName)
+	})
+	blockchainCmdArgs := []string{subnetName}


this should belong to the json test case IMO

felipemadero · 2025-04-22T13:03:49Z

tests/e2e/commandse2e/blockchain/deploy/suite.go

+var _ = ginkgo.Describe("[Blockchain Deploy Flags]", ginkgo.Ordered, func() {
+	_ = ginkgo.BeforeEach(func() {
+		// Create test subnet config
+		commands.CreateSubnetEvmConfigSOV(subnetName, ewoqEVMAddress, commands.PoA)


maybe this should be added to the json spec (how to setup required env)

deploy flags e2e

afbba82

sukantoraymond requested a review from a team as a code owner April 17, 2025 22:48

sukantoraymond requested review from learyce, Tonix517, jasonatran, artemanosov, meaghanfitzgerald and cwend-ava April 17, 2025 22:48

github-project-automation bot added this to avalanchego Apr 17, 2025

github-project-automation bot moved this to Backlog 🗄️ in avalanchego Apr 17, 2025

github-actions bot assigned sukantoraymond Apr 17, 2025

deploy flags e2e

6a6fb04

sukantoraymond marked this pull request as draft April 17, 2025 22:55

sukantoraymond commented Apr 17, 2025

View reviewed changes

use evm address

7a7ec4f

sukantoraymond marked this pull request as ready for review April 18, 2025 16:19

sukantoraymond commented Apr 18, 2025

View reviewed changes

fix lint

f22428f

Tonix517 reviewed Apr 18, 2025

View reviewed changes

sukantoraymond added 7 commits April 21, 2025 12:35

rename tests

e491435

remove duplicate

0c7e5df

make test function general

f8f9225

lint

b2f2222

lint

a741643

lint

47f4a46

lint

12c704f

felipemadero reviewed Apr 22, 2025

View reviewed changes

sukantoraymond added 2 commits April 22, 2025 17:39

add output

438fe47

add more testcases

abdc06a

add more test cases

23bb74c

sukantoraymond marked this pull request as draft April 30, 2025 15:36

sukantoraymond closed this May 9, 2025

github-project-automation bot moved this from Backlog 🧊 to Done 🎉 in avalanchego May 9, 2025

Deploy flags e2e #2755

Deploy flags e2e #2755

Uh oh!

Conversation

sukantoraymond commented Apr 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sukantoraymond Apr 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sukantoraymond commented Apr 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sukantoraymond commented Apr 17, 2025 •

edited

Loading

sukantoraymond Apr 21, 2025 •

edited

Loading

sukantoraymond commented Apr 21, 2025 •

edited

Loading