go/crypto/sha3 : embedding SHA3 and HKDF drawbacks (NIST Standard not fulfilled)

[paocalvi]

Go version

go1.24 all versions

Output of go env in your module/workspace:

C:\Users\paoca>go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=[...]
set GOENV=C:\Users\[...]\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\[...]\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\[...]\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLCHAIN=auto
set GOTOOLDIR=C:\go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.24
set GODEBUG=

What did you do?

I tried to use golang/x/crypto/sha3 (and shake) against the NIST bit-oriented test vectors, i.e.
https://csrc.nist.gov/CSRC/media/Projects/Cryptographic-Algorithm-Validation-Program/documents/sha3/sha3vs.pdf
chapters 6.2.1.1, 6.2.2.1, 6.3.1.1, 6.3.2.1, that are no less important than the byte oriented cases and integrant part of the NIST standard.

What did you see happen?

The embedding of SHA3 and HKDF in golang/crypto is definitely a great thing, but there is a major drawback: the implementation always lacked the "non-byte-complete" feature, implementing partially the NIST specification.
As external x package is was not straight but relatively easy to complete the missing code, implementing correctly the padding function, which is currently compliant only if with the "entire number of bytes" subcase.
Personally I inserted a new primitive "WriteLastByteFractionAndSum" in SHA3, SHAKE, KMAC, HKDF, etc....
Now that the code is "inside", there is a big obstacle in releasing code doing that,

What did you expect to see?

The insertion of primitives to add BITS to the stream to hash or at least to complete the last byte fraction before summing.

[gabyhelp]

Related Issues

• x/crypto/sha3: bytepad function is not safe #69169 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[seankhliao]

Given how as a language Go operates on bytes, it doesn't look like the referenced sections "... Bit-Oriented Input Message Implementations" are really relevant to x/crypto/sha3.

[paocalvi]

Not understood how the way 'go operates on bytes' justifies not being able to interoperate with NIST standard at bit level. Why to embed half of a standard in the core libraries making impossible to use that library to calculate the sha3 of '001', for example? It may be needed to verify the output of non-go software....