How to write and test changes to standard libraries?

[hb9cwp]

Hello, how do TinyGo developers actually develop and test their changes in standard libraries?

In the release branch (v0.37.0), I have debugged and fixed a problem #4903 in src/machine/machine_rp2_spi.go which on new RP2350 processors proved blocking for example the display driver tinygo.org/x/drivers/st7789. Currently, I try to rebase and verify those changes in the dev branch.

Thus I have built tinygo in the dev branch locally, as outlined in Build from source.
Now, how do I make sure that my new tinygo-dev uses also the standard libraries from dev branch?
I am asking because, according to my research, replace directives in go.mod do not work for standard libraries...

Thank you for any clue sticks!

[eliasnaur]

I use the Nix flake to produce self-contained development builds.

[hb9cwp]

Just came across another clue here where $TINYGOROOT points to my local clone of the TinyGo repo:

rs@x230:~$ tinygo env
GOOS="linux"
GOARCH="amd64"
GOROOT="/usr/local/go"
GOPATH="/home/rs/go"
GOCACHE="/home/rs/.cache/tinygo"
CGO_ENABLED="1"
TINYGOROOT="/home/rs/tinygo/tinygo"
rs@x230:~$ 

"TINYGOROOT is the path to the final location for checking tinygo files. If unset (by a -X ldflag), then sourceDir() will fallback to the original build directory." from Package goenv.

[aykevl]

Now, how do I make sure that my new tinygo-dev uses also the standard libraries from dev branch?

TinyGo built from source should automatically pick up the standard library packages that belong to it, no other configuration is needed. Just make sure you use the right tinygo binary.

[hb9cwp]

Thanks for your hints. Should some, if not all three, occurrances of GOROOT in section "GOROOT packages" of the page "Compiler packages" be replaced by TINYGOROOT for clarity?

As the output of $ tinygo env further up shows, $GOROOT and $TINYGOROOT coexist in TinyGo. They point to different directories: The first to standard libraries etc. of big Go, and the second to standard libraries etc. of TinyGo.

The first two scentences seem to support such a modification of the documentation:
"The GOROOT TINYGOROOT directory in TinyGo is actually a constructed GOROOT, cached and reused over different compiler invocations. It is a combination of Go TinyGo standard library packages and special TinyGo modified packages."

The modified section would be a direct answer to my question how (versions of) TinyGo find their standard library etc.

[aykevl]

This is all a bit confusing, I agree.

There are 3 different package roots:

• The original Go GOROOT. On my system, that is /usr/local/go1.24.0.
• The TINYGOROOT, which is the TinyGo source tree. On my system, that is /home/ayke/src/tinygo/tinygo.
• The cached GOROOT, which is a directory tree that has the two roots above merged. On my system, that is /home/ayke/.cache/tinygo/goroot-c063e1a94e4335527097e6c503b434aa3eb8dd4a486a3280294b3e609612e125.

What you are referring to in the documentation is the cached GOROOT, not TINYGOROOT. The existing phrasing is correct, tough the initial "The GOROOT directory in TinyGo..." refers to the cached GOROOT which might be confusing.

[aykevl]

In any case, this should all work out of the box and no environment variables should be set (in fact, I recommend against setting GOROOT/TINYGOROOT manually unless you really know what you're doing - I basically never set those). If it doesn't work out of the box, I'd like to hear about it.

[hb9cwp]

@aykevl Thank you very much for your detailed replies! You confirm what I have pieced together so far from the documentation and source code. Now, I am looking forward to port my bug fixes for issue #4690 from release v0.37.0 to current v0.38.0-dev and test them again.

The existence of $TINYGOROOT was the missing piece in my puzzle. Until I spotted it in the sample output of tinygo env cited above.

It might be worthwhile to mention it explicitly in the documentation Web site along the lines of your bullet points and recommendation above. Currently, the single mention of $TINYGOROOT is burried in a section about WebAssembly.

[hb9cwp]

Yesterday, I managed to build LLVM and TinyGo from sources. During make llvm-build, C++ had terminated compilation a handful of times on an old Thinkpad x230 with 4 cores and 12 GB RAM that runs Ubuntu 24.04 LTS "Noble":

But it ran to completion after I restarted the make repeatedly. Once LLVM was done, TinyGo built fine in one go.

I repeated the manual build on a Chromebook with 8 cores and 8 GB RAM that runs Crostini (e.b. Debian 12 "Bookworm") inside current ChromeOS. Although Make (actually Ninja) used all 8 cores, I noticed that build slowed down increasingly and terminated more and more frequently as the build progressed over several hours.

Finally, I edited tinygo/GNUmakefile and replaced the two occurrences of ninja by ninja -j 4 to limit scheduling of build tasks to 4 out of 8 cores. This speeds up the build of LLVM by several factors and it runs the 3200+ steps to completion much faster and in one go!

On both platforms, it did not look like a hardware issue like thermal throttling, but rather some I/O bandwidth limitation. I did not observe much swapping. Eventually, some dependencies are not built in the correct order using (many) parallel builds.
Have you experienced something similar while building LLVM from source?

[aykevl]

I have a 10 core / 32GB system which doesn't have this problem. It's mostly a problem on systems where each job has less than 1-2GB of RAM. (I did run into this when I built TinyGo on a Raspberry Pi 3, with 4 cores but just 1GB RAM).

This is a known issue and the Ninja developers refuse to do anything to help (like adding an environment variable to set the number of cores). See: ninja-build/ninja#1482. The best workaround is probably to modify the Makefile to limit the number of cores :/