Skip to content

src debug command #731

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 114 commits into from
May 13, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
114 commits
Select commit Hold shift + click to select a range
dfa7592
setup debug command scaffold
DaedalusG Jul 2, 2021
e729fa0
added test kubectl events logging
DaedalusG Jul 2, 2021
eb9f057
debug: Create ZIP archive
tsenart Jul 9, 2021
b48f332
debug: Fail if zip file already exists
tsenart Jul 9, 2021
dabaacb
debug: introduce directory structure in zip
tsenart Jul 9, 2021
74a28c3
added --all-naamespaces to get events command
DaedalusG Jul 14, 2021
88abc6f
Merge branch 'main' into src-debugger
DaedalusG Aug 9, 2021
77f06d2
populated with TODO tasks, add get pods output
DaedalusG Aug 10, 2021
746fce8
setting up k8s get pods to enable grabbing logs
DaedalusG Aug 10, 2021
0dff03b
got pod names parsed to slice
DaedalusG Aug 11, 2021
b7c4625
pulling logs from all pods, TODO: handle subcontainers
DaedalusG Aug 12, 2021
d4821a0
add missing .txt string to log archive names
DaedalusG Aug 12, 2021
3de8d87
used json get pods out, refactored k8s logs into their own spot
DaedalusG Aug 13, 2021
53932e4
refactor get logs into savek8sLogs
DaedalusG Aug 18, 2021
fd4ad22
working on logs for previous pods, need to only write when command re…
DaedalusG Aug 19, 2021
9415d56
archives prev-logs if exists
DaedalusG Aug 20, 2021
c216923
refactoring getPods to handler scope
DaedalusG Aug 22, 2021
e392fbb
refactored all kubectl call functions to be declared outside of handler
DaedalusG Aug 22, 2021
9ef8036
added pod manifests and describes
DaedalusG Aug 22, 2021
ae4c240
added some more TODOs
DaedalusG Aug 22, 2021
9d09cb2
added a little error handling on archives
DaedalusG Aug 23, 2021
247d0a3
corrected some error handling
DaedalusG Aug 24, 2021
8115b72
added validation on out flag, and setup for deployment flag
DaedalusG Aug 24, 2021
3631b8a
improved logic for deployment flag
DaedalusG Aug 25, 2021
5db3a39
refactored deployment script
DaedalusG Aug 25, 2021
dd52bf3
added get PV and PVC, used out flag as baseDir for archive filestructure
DaedalusG Aug 26, 2021
5c6987c
changed archive filestructure to be oriented around pod directories c…
DaedalusG Aug 26, 2021
dc32e6f
cleaned up some finished TODOs, shortened selector values for -d flag
DaedalusG Aug 26, 2021
26ebd78
refactored kubectl archiving into -d flag switch
DaedalusG Aug 27, 2021
53e8ecc
working get containers
DaedalusG Aug 27, 2021
928d61f
Concurrency!!
tsenart Aug 27, 2021
1a655d9
made deploy level kubectl functions concurrent
DaedalusG Aug 30, 2021
572218e
figured out bug in getLogs caused by immediately invoked function and…
DaedalusG Aug 30, 2021
62d915c
all kubectl functions refactored to run as goroutines, bug present ca…
DaedalusG Aug 31, 2021
6943f52
added some prints for debugging
DaedalusG Aug 31, 2021
0336668
clean up some flag validating prints and comments
DaedalusG Aug 31, 2021
f26f592
A bunch of improvements
Aug 31, 2021
533c545
add todo for logging
DaedalusG Aug 31, 2021
88b2a4e
Merge branch 'src-debugger' of https://github.com/sourcegraph/src-cli…
DaedalusG Aug 31, 2021
2d91a35
fixed hardcoding 999999 in setOpenFileLimits
DaedalusG Aug 31, 2021
a3c9977
removed plural k8s func names, started work on docker commands
DaedalusG Sep 2, 2021
709817c
working log archival for docker containers
DaedalusG Sep 2, 2021
edbe953
added docker inspect
DaedalusG Sep 3, 2021
419d194
added docker container stats
DaedalusG Sep 3, 2021
8a6aa2a
improved logic for get containers
DaedalusG Sep 7, 2021
e2cdff3
cleanup
DaedalusG Sep 30, 2021
877fedc
clarify assumptions
DaedalusG Sep 30, 2021
662c142
introduced debug sub commands kube, comp, and serv -- Ex: src debug k…
DaedalusG Nov 22, 2021
b96532a
for some reason my .gitignore was ignoring these new files
DaedalusG Nov 22, 2021
7981e89
changed flagset to use .StringVar rather than .String in flagSet package
DaedalusG Jan 6, 2022
b54a1af
correct flag typo in serv
DaedalusG Feb 2, 2022
2d46ea8
fixed verbose output in kube command, changed -out flag to -o
DaedalusG Feb 2, 2022
d3d557e
switching to passanger, stating getConfig
DaedalusG Feb 7, 2022
e393535
ignore errors and get extsvc configs
Feb 7, 2022
7dff3b9
add some comments from work with Tomas
DaedalusG Feb 9, 2022
2e3b525
added namespace flag for kube subcommand
DaedalusG Feb 25, 2022
77f62e0
added a little safety check to the kube command
DaedalusG Mar 15, 2022
d1264a9
added a semaphore to kube, added some text safty checks
DaedalusG Apr 15, 2022
47ba3c5
added semphore to all RPC calls in archiveKube
DaedalusG Apr 15, 2022
367de18
added current-context logging before kube command execution
DaedalusG Apr 16, 2022
13a2620
Atempting to debug error inconsistent --previous logs call
DaedalusG Apr 21, 2022
a8505f8
added network validation to comp subcommand
DaedalusG Apr 21, 2022
6a64a3c
added a function to pull site config, cleaned up logging and usage funcs
DaedalusG Apr 22, 2022
487ab74
Merge branch 'main' into src-debugger
DaedalusG Apr 22, 2022
9c81467
moved sub-functions to relevant file, trial utility archiveFileFromCo…
DaedalusG Apr 26, 2022
d080d66
converted all kube commands to , renamed kube getContainers to getPods
DaedalusG Apr 26, 2022
a33444c
refactored compose functions with archiveFileFromCommand
DaedalusG Apr 26, 2022
688ccee
Made archiveFileFromCommand calls aesthetically pleasing, added docke…
DaedalusG Apr 28, 2022
e1fe03d
changes all path package calls to path/filepath calls
DaedalusG Apr 28, 2022
df05363
emptied graveyard
DaedalusG Apr 28, 2022
9c656bd
return early from failed semaphore.Acquire rather than reading error
DaedalusG Apr 28, 2022
38f5ce1
handle errors, remove direct comparison to booleans
DaedalusG Apr 28, 2022
d6d71e5
cleaned up some more linter erros
DaedalusG Apr 28, 2022
023cbe0
refactor adding verify func
DaedalusG Apr 29, 2022
3fafa35
fixed final improperly handled error
DaedalusG Apr 29, 2022
7ecd7de
use semaphore in debug comp
DaedalusG Apr 29, 2022
daa38e8
working errgroups in comp
DaedalusG Apr 29, 2022
e5c3f8c
refactor to errgroup in kube
DaedalusG Apr 29, 2022
4bdba62
refactored writer in archive functions
DaedalusG Apr 30, 2022
6b35078
corrected all complex filepath calls using fmt.Sprintf()
DaedalusG May 2, 2022
94a6538
filter docker ps output to appropriate network
DaedalusG May 2, 2022
f759711
removed setup debug as will as openfile limit adjustment
DaedalusG May 2, 2022
06640e5
normalize site config
May 2, 2022
8ecdff3
correctly process json
DaedalusG May 2, 2022
4396daa
fleshed out serv command
DaedalusG May 2, 2022
c42c855
Update .gitignore
DaedalusG May 5, 2022
9e12b5f
Apply suggestions from code review
DaedalusG May 5, 2022
ed870e5
implement and repair infered refactors in erics suggestions
DaedalusG May 5, 2022
dc7aef6
addressed many suggested code improvements for readability and style;…
DaedalusG May 5, 2022
bd1366c
refactor baseDir processor
DaedalusG May 6, 2022
91c28d6
went over errors, still probably needs another run since I widely use…
DaedalusG May 6, 2022
109f49e
clarify all string serializations
DaedalusG May 6, 2022
877a333
Update cmd/src/debug_comp.go
DaedalusG May 6, 2022
edf6100
Update cmd/src/debug_common.go
DaedalusG May 6, 2022
14f3441
Update cmd/src/debug_common.go
DaedalusG May 6, 2022
1b09789
Update cmd/src/debug_kube.go
DaedalusG May 6, 2022
7eee487
Update cmd/src/debug_kube.go
DaedalusG May 6, 2022
3ab8775
Update cmd/src/debug_kube.go
DaedalusG May 6, 2022
066fd39
finishing touches
DaedalusG May 6, 2022
10fa745
refactored run command, handled some errors
DaedalusG May 6, 2022
0f2f878
bring branch up to main
DaedalusG May 6, 2022
c54fd23
final error handling cleanup
DaedalusG May 6, 2022
feb5cf4
add changelog entry
DaedalusG May 6, 2022
da74d2b
Update cmd/src/debug_serv.go
DaedalusG May 13, 2022
f0c57a4
Update cmd/src/debug_comp.go
DaedalusG May 13, 2022
9a2c7b0
Update cmd/src/debug_kube.go
DaedalusG May 13, 2022
4aa5180
Update cmd/src/debug_kube.go
DaedalusG May 13, 2022
1a6076b
Update cmd/src/debug.go
DaedalusG May 13, 2022
91fef39
address harvey suggestions
DaedalusG May 13, 2022
4805a7c
Merge branch 'src-debugger' of https://github.com/sourcegraph/src-cli…
DaedalusG May 13, 2022
8eb1d8f
finalize changes from harvey's suggestions
DaedalusG May 13, 2022
6b11205
fix go.mod
DaedalusG May 13, 2022
48ffc5f
really fix go.mod
DaedalusG May 13, 2022
cc29b45
Merge branch 'main' into src-debugger
efritz May 13, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
./src
./cmd/src/src
*.zip
release
./vendor
.idea
Expand Down
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ All notable changes to `src-cli` are documented in this file.
## Unreleased

### Added

- New command `src debug`. [#731](https://github.com/sourcegraph/src-cli/pull/731)
- `src lsif upload` now supports the `-gitlab-token` flag. [#721](https://github.com/sourcegraph/src-cli/pull/721)
- Batch Changes can be applied to Bitbucket Cloud when `src` is used with Sourcegraph 3.40 or later. [#725](https://github.com/sourcegraph/src-cli/pull/725)

Expand Down
42 changes: 42 additions & 0 deletions cmd/src/debug.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
package main

import (
"flag"
"fmt"
)

var debugCommands commander

func init() {
usage := `'src debug' gathers and bundles debug data from a Sourcegraph deployment for troubleshooting.

Usage:

src debug command [command options]

The commands are:

kube dumps context from k8s deployments
compose dumps context from docker-compose deployments
server dumps context from single-container deployments


Use "src debug command -h" for more information about a subcommands.
src debug has access to flags on src -- Ex: src -v kube -o foo.zip

`

flagSet := flag.NewFlagSet("debug", flag.ExitOnError)
handler := func(args []string) error {
debugCommands.run(flagSet, "src debug", usage, args)
return nil
}

// Register the command.
commands = append(commands, &command{
flagSet: flagSet,
aliases: []string{},
handler: handler,
usageFunc: func() { fmt.Println(usage) },
})
}
123 changes: 123 additions & 0 deletions cmd/src/debug_common.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
package main

import (
"archive/zip"
"context"
"encoding/json"
"fmt"
"log"
"os"
"path/filepath"
"strings"

"github.com/sourcegraph/src-cli/internal/exec"

"github.com/sourcegraph/sourcegraph/lib/errors"
)

type archiveFile struct {
name string
data []byte
err error
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's unusual in Go to bundle an error field into a structure, as Go tends to strongly prefer returning error values as separate return parameters. Normally, the "constructor" would return something like *archiveFile, error, and let callers handle err that way, rather than having to introspect the returned *archiveFile. This tends to look something like this in practice:

type myType struct {
  derivedFoo any
  derivedBar any
}

func NewMyType(foo any, bar any) (*MyType, error) {
  derivedFoo, err := doSomethingWithFoo(foo)
  if err != nil {
    return nil, errors.Wrap(err, "could not derive foo")
  }

  derivedBar, err := doSomethingWithBar(bar)
  if err != nil {
    return nil, errors.Wrap(err, "could not derive bar")
  }

  return &MyType{
    derivedFoo: derivedFoo,
    derivedBar: derivedBar,
  }, nil
}

Where we do tend to break this in Sourcegraph is where the error actually needs to persist across invocations — the most common case is where a GraphQL resolver does some sort of one time internal call to retrieve things from the database, but then might need to return the same error for each field that was requested in the GraphQL query.

You don't need to change this, but it's worth being aware of for future Go projects.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will investigate after merge, ty ty

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@LawnGnome FYI these values go through a channel which is why it's packaged as a struct in the first place. Does that change this advice?

}

func archiveFileFromCommand(ctx context.Context, path, cmd string, args ...string) *archiveFile {
f := &archiveFile{name: path}
f.data, f.err = exec.CommandContext(ctx, cmd, args...).CombinedOutput()
if f.err != nil {
f.err = errors.Wrapf(f.err, "executing command: %s %s: received error: %s", cmd, strings.Join(args, " "), f.data)
}
return f
}

// verify prompts the user to confirm they want to run the command
func verify(confirmationText string) (bool, error) {
input := ""
for strings.ToLower(input) != "y" && strings.ToLower(input) != "n" {
fmt.Printf("%s [y/N]: ", confirmationText)
if _, err := fmt.Scanln(&input); err != nil {
return false, err
}
}

return strings.ToLower(input) == "y", nil
}

func processBaseDir(base string) (string, string) {
if !strings.HasSuffix(base, ".zip") {
return base + ".zip", base
}

return base, strings.TrimSuffix(base, ".zip")
}

// write all the outputs from an archive command passed on the channel to to the zip writer
func writeChannelContentsToZip(zw *zip.Writer, ch <-chan *archiveFile, verbose bool) error {
for f := range ch {
if verbose {
log.Printf("archiving file %q with %d bytes", f.name, len(f.data))
}

if f.err != nil {
return f.err
}

zf, err := zw.Create(f.name)
if err != nil {
return errors.Wrapf(err, "failed to create %q", f.name)
}

if _, err := zf.Write(f.data); err != nil {
return errors.Wrapf(err, "failed to write to %q", f.name)
}
}
return nil
}

// TODO: Currently external services and site configs are pulled using the SRC_ENDPOINT env var,
// if theres a way to validate that the env var is pointing at the same instance as the docker and kubectl commands,
// it should be implemented.

// TODO: file issue on the existence of OAuth signKey which needs to be redacted

// getExternalServicesConfig calls src extsvc list with the format flag -f,
// and then returns an archiveFile to be consumed
func getExternalServicesConfig(ctx context.Context, baseDir string) *archiveFile {
const fmtStr = `{{range .Nodes}}{{.id}} | {{.kind}} | {{.displayName}}{{"\n"}}{{.config}}{{"\n---\n"}}{{end}}`
return archiveFileFromCommand(
ctx,
filepath.Join(baseDir, "config", "external_services.txt"),
os.Args[0], "extsvc", "list", "-f", fmtStr,
)
}

// getSiteConfig calls src api -query=... to query the api for site config json
func getSiteConfig(ctx context.Context, baseDir string) *archiveFile {
const siteConfigStr = `query { site { configuration { effectiveContents } } }`
f := archiveFileFromCommand(ctx,
filepath.Join(baseDir, "config", "siteConfig.json"),
os.Args[0], "api", "-query", siteConfigStr,
)

if f.err != nil {
return f
}

var siteConfig struct {
Data struct {
Site struct {
Configuration struct {
EffectiveContents string
}
}
}
}

if err := json.Unmarshal(f.data, &siteConfig); err != nil {
f.err = err
return f
}

f.data = []byte(siteConfig.Data.Site.Configuration.EffectiveContents)
return f
}
Loading