-
Notifications
You must be signed in to change notification settings - Fork 18k
runtime: crash on ARMv5TE when running tests #5466
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Owner changed to @minux. |
Comment 3 by [email protected]: Sorry, I already started the compilation froma clean slate. I will let you know when it finishes. |
Comment 4 by [email protected]: Ok, it crashed again, but now in a different point. Also, it did not abort the tests and continued for a while until it seems it crashed again: First crash (formatting was weird): [...] ok regexp 0.845s ok regexp/syntax 4.786s --- FAIL: TestCgoSignalDeadlock (1.12 seconds) crash_cgo_test.go:21: expected "OK\n", but got "SIGILL: illegal instruction\nPC=0x4e5414\n\n\ngoroutine 1 [semacquire]:\nruntime.park(0x62fa8, 0x4ec6b8, 0x4dd018)\n\t/usr/local/go/src/pkg/runtime/proc.c:1175 +0x40\nsemacquireimpl(0x1074f810, 0x1)\n\t/usr/local/go/src/pkg/runtime/zsema_linux_arm.c:113 +0x120\nsync.runtime_Semacquire(0x1074f810)\n\t/usr/local/go/src/pkg/runtime/zsema_linux_arm.c:165 +0x30\nsync.(*WaitGroup).Wait(0x107969a0)\n\t/usr/local/go/src/pkg/sync/waitgroup.go:109 +0x118\nmain.(*builder).do(0x106f4940, 0x1076b310)\n\t/usr/local/go/src/cmd/go/build.go:715 +0x428\nmain.runRun(0x4df3e0, 0x10668010, 0x1, 0x1)\n\t/usr/local/go/src/cmd/go/run.go:86 +0x838\nmain.main()\n\t/usr/local/go/src/cmd/go/main.go:156 +0x4ec\nruntime.main()\n\t/usr/local/go/src/pkg/runtime/proc.c:182 +0x78\nruntime.goexit()\n\t/usr/local/go/src/pkg/runtime/proc.c:1223\n\ngoroutine 2 [syscall]:\nruntime.goexit()\n\t/usr/local/go/src/pkg/runtime/proc.c:1223\n\ngoroutine 3 [syscall]:\nruntime.entersyscallblock()\n\t/usr/local/go/src/pkg/runtime/proc.c:1333 +0xc0\nruntime.MHeap_Scavenger()\n\t/usr/local/go/src/pkg/runtime/mheap.c:453 +0xfc\nruntime.goexit()\n\t/usr/local/go/src/pkg/runtime/proc.c:1223\ncreated by runtime.main\n\t/usr/local/go/src/pkg/runtime/proc.c:165\n\ngoroutine 4 [syscall]:\nruntime.entersyscallblock()\n\t/usr/local/go/src/pkg/runtime/proc.c:1333 +0xc0\ncreated by os/signal.init·1\n\t/usr/local/go/src/pkg/os/signal/signal_unix.go:27 +0x44\n\ngoroutine 6 [running]:\nsyscall.Syscall()\n\t/usr/local/go/src/pkg/syscall/asm_linux_arm.s:15 +0x8\ncreated by main.(*builder).do\n\t/usr/local/go/src/cmd/go/build.go:712 +0x40c\n\ntrap 0x6\nerror 0x0\noldmask 0x0\nr0 0x1\nr1 0xfff8335f\nr2 0x40dfc954\nr3 0x57daa3c4\nr4 0x12fe1d91\nr5 0x4e5154\nr6 0x3e\nr7 0xf0\nr8 0x2\nr9 0x1066f400\nr10 0x1065f360\nfp 0x2b\nip 0x400b59f0\nsp 0x40dfc95c\nlr 0x4e51d6\npc 0x4e5414\ncpsr 0x20000010\nfault 0x0\n" FAIL FAIL runtime 55.997s ? runtime/cgo [no test files] ok runtime/debug 0.061s ok runtime/pprof 1.887s ? runtime/race [no test files] [...] And second crash: [...] ok time 5.640s ok unicode 0.066s ok unicode/utf16 0.057s SIGILL: illegal instruction PC=0x4e5238 goroutine 1 [semacquire]: runtime.park(0x62fa8, 0x4ec478, 0x4dd018) /usr/local/go/src/pkg/runtime/proc.c:1175 +0x40 semacquireimpl(0x10b86798, 0x1) /usr/local/go/src/pkg/runtime/zsema_linux_arm.c:113 +0x120 sync.runtime_Semacquire(0x10b86798) /usr/local/go/src/pkg/runtime/zsema_linux_arm.c:165 +0x30 sync.(*WaitGroup).Wait(0x10b8aec0) /usr/local/go/src/pkg/sync/waitgroup.go:109 +0x118 main.(*builder).do(0x1085a6c0, 0x10b6da80) /usr/local/go/src/cmd/go/build.go:715 +0x428 main.runTest(0x4df438, 0x10668010, 0x3, 0x3) /usr/local/go/src/cmd/go/test.go:330 +0x1ff8 main.main() /usr/local/go/src/cmd/go/main.go:156 +0x4ec runtime.main() /usr/local/go/src/pkg/runtime/proc.c:182 +0x78 runtime.goexit() /usr/local/go/src/pkg/runtime/proc.c:1223 goroutine 2 [syscall]: runtime.goexit() /usr/local/go/src/pkg/runtime/proc.c:1223 goroutine 3 [syscall]: runtime.entersyscallblock() /usr/local/go/src/pkg/runtime/proc.c:1333 +0xc0 runtime.MHeap_Scavenger() /usr/local/go/src/pkg/runtime/mheap.c:453 +0xfc runtime.goexit() /usr/local/go/src/pkg/runtime/proc.c:1223 created by runtime.main /usr/local/go/src/pkg/runtime/proc.c:165 goroutine 4 [syscall]: runtime.entersyscallblock() /usr/local/go/src/pkg/runtime/proc.c:1333 +0xc0 created by os/signal.init·1 /usr/local/go/src/pkg/os/signal/signal_unix.go:27 +0x44 goroutine 6 [running]: syscall.Syscall() /usr/local/go/src/pkg/syscall/asm_linux_arm.s:15 +0x8 created by main.(*builder).do /usr/local/go/src/cmd/go/build.go:712 +0x40c goroutine 13 [syscall]: runtime.entersyscallblock() /usr/local/go/src/pkg/runtime/proc.c:1333 +0xc0 timerproc() /usr/local/go/src/pkg/runtime/ztime_linux_arm.c:194 +0xd4 runtime.goexit() /usr/local/go/src/pkg/runtime/proc.c:1223 created by addtimer /usr/local/go/src/pkg/runtime/ztime_linux_arm.c:82 goroutine 14 [chan receive]: runtime.park(0x62fa8, 0x10be0688, 0x4e1baa) /usr/local/go/src/pkg/runtime/proc.c:1175 +0x40 runtime.chanrecv(0x293488, 0x10be0660, 0x405f7fd0, 0x0, 0x0, ...) /usr/local/go/src/pkg/runtime/chan.c:366 +0x258 runtime.chanrecv1() /usr/local/go/src/pkg/runtime/chan.c:458 +0x30 main.func·026() /usr/local/go/src/cmd/go/signal.go:21 +0x34 runtime.goexit() /usr/local/go/src/pkg/runtime/proc.c:1223 created by main.processSignals /usr/local/go/src/cmd/go/signal.go:23 +0xc0 goroutine 280 [finalizer wait]: runtime.park(0x0, 0x0, 0x4e2d91) /usr/local/go/src/pkg/runtime/proc.c:1175 +0x40 runfinq() /usr/local/go/src/pkg/runtime/mgc0.c:2182 +0x68 runtime.goexit() /usr/local/go/src/pkg/runtime/proc.c:1223 created by runtime.gc /usr/local/go/src/pkg/runtime/mgc0.c:1886 trap 0x6 error 0x0 oldmask 0x0 r0 0x0 r1 0x1 r2 0x4e6fd0 r3 0x0 r4 0x0 r5 0x4e5154 r6 0x1 r7 0xf0 r8 0x0 r9 0x1066f400 r10 0x1065f360 fp 0x4ea0fc ip 0x400b59f0 sp 0x40dfc94c lr 0x7f0e0 pc 0x4e5238 cpsr 0x40000010 fault 0x0 |
Comment 6 by [email protected]: Something looks fishy it seems: bga@blindio:~/go/src$ gdb ../bin/go GNU gdb (GDB) 7.0.1-debian Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>; This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "arm-linux-gnueabi". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>;... Reading symbols from /home/bga/go/bin/go...done. (gdb) x/10i 0x4e5230 0x4e5230 <go/build.Default>: andeq r0, r0, r0 0x4e5234 <go/build.Default+4>: andeq r0, r0, r0 0x4e5238 <go/build.Default+8>: andeq r0, r0, r0 0x4e523c <go/build.Default+12>: andeq r0, r0, r0 0x4e5240 <go/build.Default+16>: andeq r0, r0, r0 0x4e5244 <go/build.Default+20>: andeq r0, r0, r0 0x4e5248 <go/build.Default+24>: andeq r0, r0, r0 0x4e524c <go/build.Default+28>: andeq r0, r0, r0 0x4e5250 <go/build.Default+32>: andeq r0, r0, r0 0x4e5254 <go/build.Default+36>: andeq r0, r0, r0 bga@blindio:~/go/src$ objdump -d --start-address=0x4e5230 --stop-address=0x4e5254 ../bin/go ../bin/go: file format elf32-littlearm |
The crash you are seeing is not the go tool, but the crash of the test binary (the name of the package that failed will follow directly after the crash dump, matching the string ^FAIL). If you cd to that package, run go test -c, then run the resulting $PKG.test binary you should seen an identical crash, as well as be able to use gdb to dissect the instruction. As you go binary appears to be working, could you also include the output of `go env` when replying. |
Comment 11 by [email protected]: Ok. Based on the crash, it seem it is the runtime package that crashed, right? So I did: bga@blindio:~/go/src/pkg/runtime$ go version go version go1.1 linux/arm bga@blindio:~/go/src/pkg/runtime$ go test -c bga@blindio:~/go/src/pkg/runtime$ ./runtime.test PASS No crashes. As requested: bga@blindio:~/go/src/pkg/runtime$ go env GOARCH="arm" GOBIN="" GOCHAR="5" GOEXE="" GOHOSTARCH="arm" GOHOSTOS="linux" GOOS="linux" GOPATH="/home/bga/go-dev" GORACE="" GOROOT="/usr/local/go" GOTOOLDIR="/usr/local/go/pkg/tool/linux_arm" CC="gcc" GOGCCFLAGS="-g -O2 -fPIC -marm -pthread" CGO_ENABLED="1" In case it is relevant, I was compiling go from my home dir and setting GOROOT_FINAL to /usr/local/go so, to be able to compile the test and run it, I created a symlink /usr/local/go pointing to /home/bga/go |
> In case it is relevant, I was compiling go from my home dir and setting GOROOT_FINAL to /usr/local/go so, to be able to compile the test and run it, I created a symlink /usr/local/go pointing to /home/bga/go ^ I wouldn't do _any_ of that. For simplicities sake, remove any reference to any go installation on your system, including unsetting GOROOT and try again. |
Comment 13 by [email protected]: Wait. I only created the symlink to do what you suggested (compiling and running the tests) otherwise it would not find any packages. other than that, i was simply following the instructions on how to compile go from source without doing anything out of the ordinary. But I will do as you suggest anyway. Will report back soon. |
I am sorry that I left you with the suggestion you should symlink things. Reviewing the history of this ticket, I cannot see where I suggested that. I am concerned that you are not following the source installation procedure as outlined on the website, and this may be compounding the problem you are experiencing. Specifically if you have a mixture of go 1.0.x and 1.1 on your system, this might explain why you are finding binaries with armv6 instructions in them |
Comment 15 by [email protected]: You misunderstood me. What you suggested was that I compile the test manually and execute it. I had GOROOT set to /usr/local/go from a previous installation but this directory does not exist anymore as I wiped it clean before trying to compile Go. Just to make it clear, there is no go installation on my system other than the directory where I am compiling it. In any case, I did as you suggested and removed all Go related environment variables: bga@blindio:~/go/src$ set | grep GO bga@blindio:~/go/src$ bga@blindio:~/go/src$ ../bin/go env GOARCH="arm" GOBIN="" GOCHAR="5" GOEXE="" GOHOSTARCH="arm" GOHOSTOS="linux" GOOS="linux" GOPATH="" GORACE="" GOROOT="/home/bga/go" GOTOOLDIR="/home/bga/go/pkg/tool/linux_arm" CC="gcc" GOGCCFLAGS="-g -O2 -fPIC -marm -pthread" CGO_ENABLED="1" bga@blindio:~/go/src$ Then I tried to compile again.: [...] ok html/template 0.513s ok image 0.676s ok image/color 0.104s ok image/draw 0.242s ok image/gif 0.065s ok image/jpeg 19.569s ok image/png 0.309s ok index/suffixarray 0.106s SIGILL: illegal instruction PC=0x4e5414 goroutine 1 [semacquire]: sync.runtime_Semacquire(0x10e71da0) /home/bga/go/src/pkg/runtime/zsema_linux_arm.c:165 +0x30 sync.(*WaitGroup).Wait(0x10a1e360) /home/bga/go/src/pkg/sync/waitgroup.go:109 +0x118 main.(*builder).do(0x10841080, 0x10b59a10) /home/bga/go/src/cmd/go/build.go:715 +0x428 main.runTest(0x4df438, 0x10668010, 0x3, 0x3) /home/bga/go/src/cmd/go/test.go:330 +0x1ff8 main.main() /home/bga/go/src/cmd/go/main.go:156 +0x4ec goroutine 2 [syscall]: goroutine 4 [syscall]: created by os/signal.init·1 /home/bga/go/src/pkg/os/signal/signal_unix.go:27 +0x44 goroutine 6 [running]: syscall.Syscall() /home/bga/go/src/pkg/syscall/asm_linux_arm.s:15 +0x8 created by main.(*builder).do /home/bga/go/src/cmd/go/build.go:712 +0x40c goroutine 74 [finalizer wait]: goroutine 14 [chan receive]: main.func·026() /home/bga/go/src/cmd/go/signal.go:21 +0x34 created by main.processSignals /home/bga/go/src/cmd/go/signal.go:23 +0xc0 trap 0x6 error 0x0 oldmask 0x0 r0 0x1 r1 0x40dfcd80 r2 0xcb0f1f r3 0x53e6049f r4 0x12fe28d3 r5 0x4e5154 r6 0x3e r7 0xf0 r8 0x2 r9 0x1066f400 r10 0x1065f360 fp 0x2b ip 0x400b59f0 sp 0x40dfc95c lr 0x7f0e0 pc 0x4e5414 cpsr 0x20000010 fault 0x0 Anything else I should try? |
Comment 18 by [email protected]: I can provide you with access to my NAS. |
I have been unable to reproduce the problem on my QNAP 119P-II (http://build.golang.org/ the linux-arm-arm5 builder) |
Comment 21 by [email protected]: Hi minux. Did you figure anything out? Do you still need access to the NAS? |
i can't reproduce the SIGILL. but running "go test misc/cgo/test" could trigger this: SIGSEGV: segmentation violation PC=0x3ff971cc goroutine 1 [running]: syscall.Syscall() /home/minux/go/src/pkg/syscall/asm_linux_arm.s:15 +0x8 goroutine 2 [syscall]: runtime.entersyscallblock() /home/minux/go/src/pkg/runtime/proc.c:1333 +0xc0 runtime.MHeap_Scavenger() /home/minux/go/src/pkg/runtime/mheap.c:453 +0xfc runtime.goexit() /home/minux/go/src/pkg/runtime/proc.c:1223 created by runtime.main /home/minux/go/src/pkg/runtime/proc.c:165 goroutine 33 [finalizer wait]: runtime.park(0x0, 0x0, 0x27e059) /home/minux/go/src/pkg/runtime/proc.c:1175 +0x40 runfinq() /home/minux/go/src/pkg/runtime/mgc0.c:2182 +0x68 runtime.goexit() /home/minux/go/src/pkg/runtime/proc.c:1223 created by runtime.gc /home/minux/go/src/pkg/runtime/mgc0.c:1886 trap 0xe error 0x80000005 oldmask 0x0 r0 0x1 r1 0x27f420 r2 0x10352b88 r3 0xaacb5759 r4 0x12fef105 r5 0x27f424 r6 0x1 r7 0xf0 r8 0x2 r9 0x10345000 r10 0x103361b0 fp 0x6 ip 0x1037bffd sp 0x10352b90 lr 0x68e34 pc 0x3ff971cc cpsr 0x80000010 fault 0x3ff971cc note: it's cmd/go that panics, not the test. the pc is incorrect, but i'm still not sure what smashes pc. @dave, do you want to take a look? could you please run "go test ../misc/cgo/test" in a loop and see if you can reproduce the crash? |
@minux will do. As a side note, my arm 5 builder, which is the same hardware, has been running builds successfully (see the dashboard). @OP, as an experiment, grab the binary distribution from here, http://dave.cheney.net/unofficial-arm-tarballs, and see if you can run the tests. go test std should do the job. |
Comment 24 by [email protected]: I downloaded the pre-compiled package and installed it. Ran the tests and it did not crash, but I got 3 timeouts: *** Test killed: ran too long (10m0s). FAIL archive/zip 600.019s *** Test killed: ran too long (10m0s). FAIL crypto/dsa 600.005s *** Test killed: ran too long (10m0s). FAIL math/big 600.006s Considering the crashes I was seeing seemed to be related to semaphores, could these timeouts be a deadlock that may have the same root cause of the crashes? |
Comment 26 by [email protected]: Ran it with -short and it worked except fr this: 2013/05/21 09:14:50 cgi: copy error: write tcp 127.0.0.1:45865: connection reset by peer --- FAIL: TestCopyError (15.12 seconds) host_test.go:351: post-conn.Close, expected child to be gone FAIL FAIL net/http/cgi 16.883s Any idea why compiling Go on my system could result on the crashes I was seeing? |
perhaps a buggy gcc? in fact, on netbsd/arm i do encounter a case where gcc -O2 generates bad code for cmd/gc and in turn causes test failures. i suggest you replace the toolchain in pkg/tools/linux_arm with those from the binary distribution and rerun the test to see if this makes a difference. i originally suspect bad interaction between kernel and sync/atomic, but if "go test -short std" could pass with dave's prebuilt package then i don't think it is the case. |
Comment 28 by [email protected]: I ran the tests again )no changes whatsoever, just executed a second run with exactly the same setup as teh previous one. I got a crash again now: ok net/http/cgi 1.858s ok net/http/cookiejar 0.137s ok net/http/fcgi 0.111s ok net/http/httptest 0.128s SIGSEGV: segmentation violation PC=0x4e5154 goroutine 1 [semacquire]: sync.runtime_Semacquire(0x10d43730) /usr/local/go/src/pkg/runtime/zsema_linux_arm.c:165 +0x30 sync.(*WaitGroup).Wait(0x10ddfa40) /usr/local/go/src/pkg/sync/waitgroup.go:109 +0x118 main.(*builder).do(0x10837300, 0x10b42150) /usr/local/go/src/cmd/go/build.go:715 +0x428 main.runTest(0x4df438, 0x10668010, 0x2, 0x2) /usr/local/go/src/cmd/go/test.go:330 +0x1ff8 main.main() /usr/local/go/src/cmd/go/main.go:156 +0x4ec goroutine 2 [syscall]: goroutine 4 [syscall]: created by os/signal.init·1 /usr/local/go/src/pkg/os/signal/signal_unix.go:27 +0x44 goroutine 6 [running]: syscall.Syscall() /usr/local/go/src/pkg/syscall/asm_linux_arm.s:15 +0x8 created by main.(*builder).do /usr/local/go/src/cmd/go/build.go:712 +0x40c goroutine 14 [chan receive]: main.func·026() /usr/local/go/src/cmd/go/signal.go:21 +0x34 created by main.processSignals /usr/local/go/src/cmd/go/signal.go:23 +0xc0 goroutine 259 [finalizer wait]: trap 0xe error 0x5 oldmask 0x0 r0 0x1 r1 0x40dfcd80 r2 0x40dfc954 r3 0xb7dfac8f r4 0x13002aa2 r5 0x4e5154 r6 0x1 r7 0xf0 r8 0x2 r9 0x1066f400 r10 0x1065f360 fp 0x2b ip 0x400b59f0 sp 0x40dfc95c lr 0x7f0e0 pc 0x4e5154 cpsr 0x80000010 fault 0xb84e515c Looks like exactly the same crash, so it is still there even with the precompiled packages. |
Comment 30 by [email protected]: It is Debian Squeeze running on a QNAP 219P. The kernel is: 2.6.32-5-kirkwood Note the change is a straight Debian Squeeze for the QNAP device in question. There are no customizations (other than adding/removing some packages) including no custom-compiled kernel. |
Comment 33 by [email protected]: I will go ahead and upgrade to Wheezy. Will let you know how it turns out. |
Comment 34 by [email protected]: Ok. After upgrading I ran the tests 3 times. No crashes whatsoever. I guess this was it. |
@bga, glad that you found the solution. i will update the https://code.google.com/p/go-wiki/wiki/GoArm page about the potentially buggy kernel (2.6.32-5-kirkwood) on QNAP 219P. btw, please remove my account on your machine if you haven't done so, and thank you for providing the help. Status changed to Fixed. |
by [email protected]:
The text was updated successfully, but these errors were encountered: