src debug command

[DaedalusG]

Src Debug

src debug is a little pet project I've been working on for a long time now as a way to learn golang. The intention here is to create a cli command that collects the data from common diagnostic commands to speed up debugging and make sharing information about a Sourcegraph instance easier for our support team.

The command is divided into three separate sub commands (server,compose, and kube) each specific to their deployment type and with some unique flags and values.

Here is an example of the return from a src debug server command:

warrengifford@Warrens-MacBook-Pro src-debug-test % src -v debug server -c vibrant_ritchie -o serv3
This command will archive docker-cli data for container: vibrant_ritchie
 SRC_ENDPOINT: https://cse-aws-test.sgdev.org
 Output filename: serv3.zip
Do you want to start writing to an archive? [y/N]: y
archiving file "serv3/inspect-vibrant_ritchie.txt" with 12294 bytes
archiving file "serv3/vibrant_ritchie.log" with 571587 bytes
archiving file "serv3/top-vibrant_ritchie.txt" with 84951 bytes
archiving file "serv3/config/siteConfig.json" with 3772 bytes
archiving file "serv3/config/external_services.txt" with 9480 bytes

This is the simplest version of the command and output

For an abridged example of kubernetes --

warrengifford@Warrens-MacBook-Pro src-debug-test % src debug kube -n ns-sourcegraph -o kube-test
Archiving kubectl data for 33 pods
 SRC_ENDPOINT: https://cse-aws-test.sgdev.org
 Context: gke_beatrix-test-overlay_us-central1-c_beatrix-test
 Namespace: ns-sourcegraph
 Output filename: kube-test.zip
Do you want to start writing to an archive? [y/N]: y

... after unzip

warrengifford@Warrens-MacBook-Pro src-debug-test % cd kube-test
warrengifford@Warrens-MacBook-Pro kube-test % ls
config  kubectl
warrengifford@Warrens-MacBook-Pro kube-test % ls config
external_services.txt   siteConfig.json
warrengifford@Warrens-MacBook-Pro kube-test % ls kubectl
events.txt                      persistent-volume-claims.txt    pods
getPods.txt                     persistent-volumes.txt
warrengifford@Warrens-MacBook-Pro kube-test % ls kubectl/pods
cadvisor-44nvf                                  jaeger-56fbb74f8-jmtjs
cadvisor-55b9d                                  minio-54b558cdd6-hlgbt
cadvisor-79pgg                                  pgsql-8565fd54ff-6n8tk
cadvisor-7s8gw                                  precise-code-intel-worker-67946948c5-f9282
cadvisor-9dxct                                  precise-code-intel-worker-67946948c5-xq8pf
cadvisor-cwmpm                                  prometheus-79f68b8d76-x8skc
cadvisor-gnnbq                                  redis-cache-bb84b65cf-j6gfj
cadvisor-lmd6q                                  redis-store-75ffcfb74b-b8pxf
cadvisor-nq75p                                  repo-updater-978f46dff-27b52
cadvisor-zkspf                                  searcher-67fd7f64cc-l6dqs
codeinsights-db-78b8747cdd-bzj72                searcher-67fd7f64cc-tpjj9
codeintel-db-8fb75ddfd-xhncd                    sourcegraph-frontend-6869dbb7d-tf4v8
github-proxy-7bfd897445-tgtxv                   sourcegraph-frontend-6869dbb7d-tr7rk
gitserver-0                                     symbols-95d9c59d9-v6brz
grafana-0                                       syntect-server-6b4478987-tpvsg
indexed-search-0                                worker-6dc84c8cf5-bkmdh
indexed-search-1
warrengifford@Warrens-MacBook-Pro kube-test % ls kubectl/pods/gitserver-0
describe-gitserver-0.txt        jaeger-agent.log                prev-gitserver.log
gitserver.log                   manifest-gitserver-0.yaml

Here you can see the general directory structure outputted in the zip archive and get an idea about the data captured by the command.

Test plan

I'd love to get some testing from friends here 🙏

I tested each command with every iteration against the two AER test environments cse-aws (the command was tested against a local compose instance for docker cli outputs) and cse-k8s. These instances are fairly small so it might be a good idea to test out the kube command against a much larger instance to test out the semaphore throttling, I don't have kubectl access to .com but that would be a good test ground.

Testing amounted to executing the command and checking to make sure that expected outputs were received and written to a zip file.

To run the command checkout the src-debugger and run the command from there:

wglaptop@Warrens-MacBook-Pro-2 src-cli % pwd
/Users/wglaptop/src-cli
wglaptop@Warrens-MacBook-Pro-2 src-cli % go run ./cmd/src debug
'src debug' gathers and bundles debug data from a Sourcegraph deployment for troubleshooting.

Usage:

	src debug command [command options]

The commands are:

	kube                 dumps context from k8s deployments
	compose              dumps context from docker-compose deployments
	server               dumps context from single-container deployments


Use "src debug [command] -h" for more information about a subcommands.
src debug has access to flags on src -- Ex: src -v kube -o foo.zip

Note that the command makes use of your SRC_ENDPOINT to gather siteconfig.json and external service config json. It also uses your local machine kubectl context to target the Sourcegraph kubernetes cluster, and a host machines docker cli.

Feel free to hit me here or DM me with any questions!

[bobheadxi]

This looks great! I wonder if this might be a good opportunity to remove the report a bug page entirely and ask admins to provide a dump using this new command instead?

[DaedalusG]

@bobheadxi Maybe we can synthesize the page and this command. I'd like to eventually add some more functionality to this command. In particular I'd love for it to include a dump of the prometheus data, with a supporting way to reconstruct our grafana interface from that data. I like the alerts that are listed In the site admin debug page though, and I'll def review those PRS for further Ideas here.

Listing for later:
https://github.com/sourcegraph/sourcegraph/pull/10704
https://github.com/sourcegraph/sourcegraph/issues/7657

[bobheadxi]

Happy to chat about the alerts summary implementation if you need help/more context! It's been a looong time but I also find the "smart summary" concept behind it neat

[mucles]

@DaedalusG Great work 💯 ! this looks great 🔥 I think it would also be helpful to be able to see if there are any alerts being fired in the instance from the Prometheus data to be in the data dump. it would also be great to have a docs page we could share with our site admins.

[DaedalusG]

@mucles Definitly wanting to build out som further features and usability here-- for the docs page, I believe src commands have a doc page automatically generated, but I'll probably flesh things out a bit after merging this.

[LawnGnome]

Much like @ericfritz, I don't think there's anything here that blocks merge, other than maybe one missing error return. I've tried to call out places where I think Go conventions would normally lead to something slightly different, or where there might be some room for future improvements, but none of that requires fixes in this PR.

Great job conceiving and leading this!

[LawnGnome]

It's unusual in Go to bundle an error field into a structure, as Go tends to strongly prefer returning error values as separate return parameters. Normally, the "constructor" would return something like *archiveFile, error, and let callers handle err that way, rather than having to introspect the returned *archiveFile. This tends to look something like this in practice:

type myType struct {
  derivedFoo any
  derivedBar any
}

func NewMyType(foo any, bar any) (*MyType, error) {
  derivedFoo, err := doSomethingWithFoo(foo)
  if err != nil {
    return nil, errors.Wrap(err, "could not derive foo")
  }

  derivedBar, err := doSomethingWithBar(bar)
  if err != nil {
    return nil, errors.Wrap(err, "could not derive bar")
  }

  return &MyType{
    derivedFoo: derivedFoo,
    derivedBar: derivedBar,
  }, nil
}

Where we do tend to break this in Sourcegraph is where the error actually needs to persist across invocations — the most common case is where a GraphQL resolver does some sort of one time internal call to retrieve things from the database, but then might need to return the same error for each field that was requested in the GraphQL query.

You don't need to change this, but it's worth being aware of for future Go projects.

[DaedalusG]

Will investigate after merge, ty ty

[efritz]

@LawnGnome FYI these values go through a channel which is why it's packaged as a struct in the first place. Does that change this advice?

[efritz]

Small follow-up: #743