debug: can't hit breakpoints when symlinks are involved

[SidKwok]

What version of Go, VS Code & VS Code Go extension are you using?

• Run go version to get version of Go from the VS Code integrated terminal.
  • go version go1.15.10 linux/amd64
• Run gopls -v version to get version of Gopls from the VS Code integrated terminal.
  • v0.6.10
• Run code -v or code-insiders -v to get version of VS Code or VS Code Insiders.
  • 1.55.2
• Check your installed extensions to get the version of the VS Code Go extension
  • v0.24.2
• Run Ctrl+Shift+P (Cmd+Shift+P on Mac OS) > Go: Locate Configured Go Tools command.
  The machine is own by our company, sorry that I can't provide all details here. But if you guys need anything specific, feel free to ask me to add more.

Share the Go related settings you have added/edited

None. Cuz I download the new VSCode, It has no other plugins except Romote-SSH and Go

Describe the bug

The breakpoints I add doesn't work at Remote-SSH, but work just fine in my local machine.

Steps to reproduce the behavior:

1. Add a break point at line 4:
   

2. Try to run the test file
   

3. The debug console runs just fine

API server listening at: 127.0.0.1:49915
2021-04-28T15:17:13+08:00 info layer=debugger launching process with args: [/absolute/path/debug.test -test.run TestCount]
2021-04-28T15:17:13+08:00 debug layer=debugger continuing
ETCD: use bagent to access ETCD
Info 2021-04-28 15:17:13,473 v1(7) client.go:67 xx.xx.xx.xx - - default - 0 not prod environment, skip collect version info
Info 2021-04-28 15:17:13,473 v1(7) client.go:67  xx.xx.xx.xx - - default - 0 not prod environment, skip collect version info
PASS
2021-04-28T15:17:13+08:00 debug layer=debugger halting
Process exiting with code: 0

4. But it doesn't hit the breakpoint I add at line 4

5. Same code, same enviorment, but when I run the debugger at my local machine(macos), it works just fine.
   

[suzmue]

Could you create a launch.json with the following configuration, and upload the output in the debug console? Thank you!

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
       {
            "name": "Launch test function",
            "type": "go",
            "request": "launch",
            "mode": "test",
            "program": "${workspaceFolder}",
            "args": [
                "-test.run",
                "TestCount"
            ],
            "trace": "verbose"
        },
    ]
}

[SidKwok]

Sorry for the late reply, here is the output:

Verbose logs are written to:
/tmp/vscode-go-debug.txt
2021-5-2, 03:12:24.259 UTC
From client: initialize({"clientID":"vscode","clientName":"Visual Studio Code","adapterID":"go","pathFormat":"path","linesStartAt1":true,"columnsStartAt1":true,"supportsVariableType":true,"supportsVariablePaging":true,"supportsRunInTerminalRequest":true,"locale":"zh-cn","supportsProgressReporting":true,"supportsInvalidatedEvent":true})
InitializeRequest
To client: {"seq":0,"type":"response","request_seq":1,"command":"initialize","success":true,"body":{"supportsConditionalBreakpoints":true,"supportsConfigurationDoneRequest":true,"supportsSetVariable":true}}
InitializeResponse
From client: launch({"name":"Launch Test","type":"go","request":"launch","mode":"test","program":"/home/guoxianglin/vimo_api","showLog":true,"args":["-test.run","TestCount"],"trace":"verbose","__configurationTarget":5,"packagePathToGoModPathMap":{"/home/guoxianglin/vimo_api":"/data00/home/guoxianglin/vimo_api"},"apiVersion":2,"dlvLoadConfig":{"followPointers":true,"maxVariableRecurse":1,"maxStringLen":64,"maxArrayValues":64,"maxStructFields":-1},"showGlobalVariables":false,"debugAdapter":"legacy","dlvToolPath":"/home/guoxianglin/go/bin/dlv","env":{"MAIL":"/var/mail/guoxianglin","USER":"guoxianglin","SSH_CLIENT":"10.255.230.182 58868 22","SHLVL":"3","HOME":"/home/guoxianglin","GOROOT":"/usr/local/go","LOGNAME":"guoxianglin","_":"/data00/home/guoxianglin/.vscode-server/bin/3c4e3df9e89829dce27b7b5c24508306b151f30d/node","XDG_SESSION_ID":"808","PATH":"/usr/local/go/bin:/data00/home/guoxianglin/.vscode-server/bin/3c4e3df9e89829dce27b7b5c24508306b151f30d/bin:/opt/tiger/toutiao/lib:/opt/tiger/jdk/jdk1.8/bin:/opt/tiger/go/go/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/opt/tiger/ss_bin:/usr/local/jdk/bin:/usr/sbin/:/opt/tiger/ss_lib/bin:/opt/tiger/ss_lib/python_package/lib/python2.7/site-packages/django/bin:/opt/tiger/yarn_deploy/hadoop/bin/:/opt/tiger/yarn_deploy/hive/bin/:/opt/tiger/yarn_deploy/jdk/bin/:/opt/tiger/hadoop_deploy/jython-2.5.2/bin:/opt/tiger/dev_toolkit/bin","VSCODE_AGENT_FOLDER":"/home/guoxianglin/.vscode-server","XDG_RUNTIME_DIR":"/run/user/1001","L[...]
LaunchRequest
Using GOPATH: /home/guoxianglin/go
Using GOROOT: /usr/local/go
Using PATH: /usr/local/go/bin:/data00/home/guoxianglin/.vscode-server/bin/3c4e3df9e89829dce27b7b5c24508306b151f30d/bin:/opt/tiger/toutiao/lib:/opt/tiger/jdk/jdk1.8/bin:/opt/tiger/go/go/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/opt/tiger/ss_bin:/usr/local/jdk/bin:/usr/sbin/:/opt/tiger/ss_lib/bin:/opt/tiger/ss_lib/python_package/lib/python2.7/site-packages/django/bin:/opt/tiger/yarn_deploy/hadoop/bin/:/opt/tiger/yarn_deploy/hive/bin/:/opt/tiger/yarn_deploy/jdk/bin/:/opt/tiger/hadoop_deploy/jython-2.5.2/bin:/opt/tiger/dev_toolkit/bin
  export MAIL="/var/mail/guoxianglin"
  export USER="guoxianglin"
  export SSH_CLIENT="10.255.230.182 58868 22"
  export SHLVL="3"
  export HOME="/home/guoxianglin"
  export GOROOT="/usr/local/go"
  export LOGNAME="guoxianglin"
  export _="/data00/home/guoxianglin/.vscode-server/bin/3c4e3df9e89829dce27b7b5c24508306b151f30d/node"
  export XDG_SESSION_ID="808"
  export PATH="/usr/local/go/bin:/data00/home/guoxianglin/.vscode-server/bin/3c4e3df9e89829dce27b7b5c24508306b151f30d/bin:/opt/tiger/toutiao/lib:/opt/tiger/jdk/jdk1.8/bin:/opt/tiger/go/go/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/opt/tiger/ss_bin:/usr/local/jdk/bin:/usr/sbin/:/opt/tiger/ss_lib/bin:/opt/tiger/ss_lib/python_package/lib/python2.7/site-packages/django/bin:/opt/tiger/yarn_deploy/hadoop/bin/:/opt/tiger/yarn_deploy/hive/bin/:/opt/tiger/yarn_deploy/jdk/bin/:/opt/tiger/hadoop_deploy/jython-2.5.2/bin:/opt/tiger/dev_toolkit/bin"
  export VSCODE_AGENT_FOLDER="/home/guoxianglin/.vscode-server"
  export XDG_RUNTIME_DIR="/run/user/1001"
  export LANG="en_US.UTF-8"
  export GOPATH="/home/guoxianglin/go"
  export SHELL="/bin/bash"
  export JAVA_HOME="/opt/tiger/jdk/jdk1.8"
  export PWD="/data00/home/guoxianglin"
  export SSH_CONNECTION="10.255.230.182 58868 10.248.179.94 22"
  export VSCODE_INJECT_NODE_MODULE_LOOKUP_PATH="/data00/home/guoxianglin/.vscode-server/bin/3c4e3df9e89829dce27b7b5c24508306b151f30d/remote/node_modules"
  export VSCODE_CWD="/data00/home/guoxianglin"
  export VSCODE_LOG_NATIVE="false"
  export VSCODE_AMD_ENTRYPOINT="vs/server/remoteExtensionHostProcess"
  export VSCODE_PIPE_LOGGING="true"
  export VSCODE_VERBOSE_LOGGING="true"
  export VSCODE_EXTHOST_WILL_SEND_SOCKET="true"
  export VSCODE_HANDLES_UNCAUGHT_ERRORS="true"
  export VSCODE_LOG_STACK="false"
  export VSCODE_NLS_CONFIG="{"locale":"zh-cn","availableLanguages":{"*":"zh-cn"},"_languagePackId":"b475c8051e44470192b4d0d6d26919c5.zh-cn","_translationsConfigFile":"/home/guoxianglin/.vscode-server/data/clp/b475c8051e44470192b4d0d6d26919c5.zh-cn/tcf.json","_cacheRoot":"/home/guoxianglin/.vscode-server/data/clp/b475c8051e44470192b4d0d6d26919c5.zh-cn","_resolvedLanguagePackCoreLocation":"/home/guoxianglin/.vscode-server/data/clp/b475c8051e44470192b4d0d6d26919c5.zh-cn/3c4e3df9e89829dce27b7b5c24508306b151f30d","_corruptedFile":"/home/guoxianglin/.vscode-server/data/clp/b475c8051e44470192b4d0d6d26919c5.zh-cn/corrupted.info","_languagePackSupport":true}"
  export BROWSER="/data00/home/guoxianglin/.vscode-server/bin/3c4e3df9e89829dce27b7b5c24508306b151f30d/bin/helpers/browser.sh"
  export VSCODE_IPC_HOOK_CLI="/run/user/1001/vscode-ipc-59ea811c-aa65-431c-966d-29312aab645b.sock"
  export APPLICATION_INSIGHTS_NO_DIAGNOSTIC_CHANNEL="true"
  export GOMODCACHE="/home/guoxianglin/go/pkg/mod"
  export GOPROXY="https://goproxy.byted.org|https://goproxy.cn|direct"
Current working directory: /home/guoxianglin/vimo_api
Running: /home/guoxianglin/go/bin/dlv test --headless=true --listen=127.0.0.1:7362 --api-version=2 --log=true -- -test.run TestCount
To client: {"seq":0,"type":"event","event":"output","body":{"category":"stdout","output":"API server listening at: 127.0.0.1:7362\n"}}
API server listening at: 127.0.0.1:7362
To client: {"seq":0,"type":"event","event":"output","body":{"category":"stderr","output":"2021-05-02T11:12:25+08:00 info layer=debugger launching process with args: [/data00/home/guoxianglin/vimo_api/debug.test -test.run TestCount]\n"}}
2021-05-02T11:12:25+08:00 info layer=debugger launching process with args: [/data00/home/guoxianglin/vimo_api/debug.test -test.run TestCount]
To client: {"seq":0,"type":"event","event":"initialized"}
InitializeEvent
To client: {"seq":0,"type":"response","request_seq":2,"command":"launch","success":true}
From client: setBreakpoints({"source":{"name":"counter.go","path":"/home/guoxianglin/vimo_api/counter.go"},"lines":[6],"breakpoints":[{"line":6}],"sourceModified":false})
SetBreakPointsRequest
Debuggee is not running. Setting breakpoints without halting.
All cleared
Creating on: /home/guoxianglin/vimo_api/counter.go:6
Error on CreateBreakpoint: could not find file /home/guoxianglin/vimo_api/counter.go
All set:[null]
To client: {"seq":0,"type":"response","request_seq":3,"command":"setBreakpoints","success":true,"body":{"breakpoints":[{"verified":false,"line":6}]}}
SetBreakPointsResponse
From client: configurationDone(undefined)
ConfigurationDoneRequest
Changing DebugState from Halted to Running
To client: {"seq":0,"type":"response","request_seq":4,"command":"configurationDone","success":true}
ConfigurationDoneResponse {"seq":72,"type":"response","request_seq":4,"command":"configurationDone","success":true}
To client: {"seq":0,"type":"event","event":"output","body":{"category":"stderr","output":"2021-05-02T11:12:26+08:00 debug layer=debugger continuing\n"}}
2021-05-02T11:12:26+08:00 debug layer=debugger continuing
To client: {"seq":0,"type":"event","event":"output","body":{"category":"stdout","output":"ETCD: use bagent to access ETCD\n"}}
ETCD: use bagent to access ETCD
To client: {"seq":0,"type":"event","event":"output","body":{"category":"stdout","output":"Info 2021-05-02 11:12:26,084 v1(7) client.go:67 10.248.179.94 - - default - 0 not prod environment, skip collect version info\nInfo 2021-05-02 11:12:26,084 v1(7) client.go:67 10.248.179.94 - - default - 0 not prod environment, skip collect version info\n"}}
Info 2021-05-02 11:12:26,084 v1(7) client.go:67 10.248.179.94 - - default - 0 not prod environment, skip collect version info
Info 2021-05-02 11:12:26,084 v1(7) client.go:67 10.248.179.94 - - default - 0 not prod environment, skip collect version info
To client: {"seq":0,"type":"event","event":"output","body":{"category":"stdout","output":"PASS\n"}}
PASS
continue state {"Running":false,"Recording":false,"Threads":null,"NextInProgress":false,"exited":true,"exitStatus":0,"When":""}
handleReenterDebug(breakpoint).
To client: {"seq":0,"type":"event","event":"terminated"}
TerminatedEvent
From client: threads(undefined)
To client: {"seq":0,"type":"response","request_seq":5,"command":"threads","success":true,"body":{"threads":[]}}
From client: disconnect({"restart":false})
DisconnectRequest
Closing Delve.
HaltRequest
To client: {"seq":0,"type":"event","event":"output","body":{"category":"stderr","output":"2021-05-02T11:12:26+08:00 debug layer=debugger halting\n"}}
2021-05-02T11:12:26+08:00 debug layer=debugger halting
HaltResponse
Failed to halt - Process 369127 has exited with status 0
DetachRequest
DisconnectRequest to parent to shut down protocol server.
To client: {"seq":0,"type":"response","request_seq":6,"command":"disconnect","success":true}
DisconnectResponse
Process exiting with code: 0
Sending TerminatedEvent as delve is closed
To client: {"seq":0,"type":"event","event":"terminated"}

[hyangah]

Thanks for sharing the trace @SidKwok
I see two different directory paths

• /home/guoxianglin/vimo_api
• /data00/home/guoxianglin/vimo_api

From the environment variables, I think the current directory is the second, but the go command also sees the second directory path. But, the vscode is seeing the first directory path. By any chance, are you using symlinks?

[SidKwok]

I think the answer is no, here is the output:

guoxianglin@n248-179-094:~$ ls -al
total 56
drwxr-xr-x  9 guoxianglin guoxianglin 4096 Apr 28 15:14 .
drwxr-xr-x  3 root        root        4096 Apr 27 18:05 ..
-rw-------  1 guoxianglin guoxianglin 1864 Apr 28 16:13 .bash_history
drwxr-xr-x  4 guoxianglin guoxianglin 4096 Apr 27 21:56 .cache
drwxr-xr-x  4 guoxianglin guoxianglin 4096 Apr 27 22:04 .config
-rw-r--r--  1 guoxianglin guoxianglin  154 Apr 27 21:46 .gitconfig
drwxr-xr-x  5 guoxianglin guoxianglin 4096 Apr 28 15:45 go
-rw-r--r--  1 guoxianglin guoxianglin   55 Apr 27 18:05 .k5login
drwxr-xr-x  2 guoxianglin guoxianglin 4096 Apr 27 21:22 .ssh
-rw-------  1 guoxianglin guoxianglin 2133 Apr 27 21:46 .viminfo
drwxr-xr-x 10 guoxianglin guoxianglin 4096 May  2 11:12 vimo_api
drwxr-xr-x  5 guoxianglin guoxianglin 4096 May  4 01:57 .vscode-server
drwxr-xr-x  5 guoxianglin guoxianglin 4096 Apr 28 15:14 .vscode-server-insiders
-rw-r--r--  1 guoxianglin guoxianglin  183 Apr 27 21:25 .wget-hsts

It doesn't seem to have any symlinks.

But if the path is wrong, can we just simply change the path being used by vscode? Like a map or sth?

[hyangah]

Thanks @SidKwok The path is coming from the vscode.
Delve and go are though seeing /data00/home/quoxianlin/vimo_api, which I don't understand why.
(I wondered if the entire /home or /home/quoxianlin is symlinked)

From the project directory (/home/quoxianlin/vimo_api) can you run go env and share the output?

[SidKwok]

guoxianglin@n248-179-094:~/vimo_api$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/guoxianglin/.cache/go-build"
GOENV="/home/guoxianglin/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/guoxianglin/go/pkg/mod"
GONOPROXY="*.byted.org,*.everphoto.cn,git.smartisan.com"
GONOSUMDB="*.byted.org,*.everphoto.cn,git.smartisan.com"
GOOS="linux"
GOPATH="/home/guoxianglin/go"
GOPRIVATE="*.byted.org,*.everphoto.cn,git.smartisan.com"
GOPROXY="https://goproxy.byted.org|https://goproxy.cn|direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.google.cn"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/guoxianglin/vimo_api/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build025444414=/tmp/go-build -gno-record-gcc-switches"
guoxianglin@n248-179-094:~/vimo_api$ 

Let's say if the entire /home is symlinked, what should be done to due with such situation?

[SidKwok]

@hyangah Your guess is true!

guoxianglin@n248-179-094:/home$ ls -al
total 16
drwxr-xr-x  4 root   root   4096 Apr 27 18:05 .
drwxr-xr-x 22 root   root   4096 Apr 27 21:30 ..
drwxr-xr-x  3 falcon falcon 4096 Oct 22  2019 falcon
lrwxrwxrwx  1 root   root     24 Apr 27 18:05 guoxianglin -> /data00/home/guoxianglin
drwxr-xr-x  3 tiger  tiger  4096 Apr 27 18:12 tiger

[SidKwok]

@hyangah When I try to launch vscode from /data00/home/guoxianglin, everything works just fine! Thank you!

But I would like to figure out that how to make it work when launching vscode from a symlink. Any solution on this?

[hyangah]

@SidKwok Thanks for confirming. Currently the recommendation is to use substitutePath (https://github.com/golang/vscode-go/blob/master/docs/debugging.md#launch-configurations) if you can configure 'launch.json'.

That doesn't currently work with 'debug test' codelenses though. Today we discussed this in our triage meeting and we are currently looking into solutions like:

1. allow substitutePath settings through settings.json, or
2. try to detect this symlink case and populate substitutePath automatically - this would be possible when the main module is symlinked. It will be a bit tricky for dependencies.

cc @suzmue @polinasok

[SidKwok]

Both solutions seem good to me. But personally I prefer the first one, because we can mark the true path using this flag. Let me know if there are any updates.

Thanks everyone.

[hyangah]

Thanks! I filed a separate feature request #1467.