Skip to content

Commit ee83879

Browse files
DaedalusGtsenartTomás SenartefritzLawnGnome
authored
src debug command (#731)
* setup debug command scaffold * added test kubectl events logging * debug: Create ZIP archive * debug: Fail if zip file already exists * debug: introduce directory structure in zip * added --all-naamespaces to get events command * populated with TODO tasks, add get pods output * setting up k8s get pods to enable grabbing logs * got pod names parsed to slice * pulling logs from all pods, TODO: handle subcontainers * add missing .txt string to log archive names * used json get pods out, refactored k8s logs into their own spot * refactor get logs into savek8sLogs * working on logs for previous pods, need to only write when command returns without error * archives prev-logs if exists * refactoring getPods to handler scope * refactored all kubectl call functions to be declared outside of handler * added pod manifests and describes * added some more TODOs * added a little error handling on archives * corrected some error handling * added validation on out flag, and setup for deployment flag * improved logic for deployment flag * refactored deployment script * added get PV and PVC, used out flag as baseDir for archive filestructure * changed archive filestructure to be oriented around pod directories containing logs, past logs, and manifests * cleaned up some finished TODOs, shortened selector values for -d flag * refactored kubectl archiving into -d flag switch * working get containers * Concurrency!! * made deploy level kubectl functions concurrent * figured out bug in getLogs caused by immediately invoked function and variable scope in for loop * all kubectl functions refactored to run as goroutines, bug present causing some zip writes to write empty files * added some prints for debugging * clean up some flag validating prints and comments * A bunch of improvements 1. Set open file limits in process to support this much concurrency. This could also be done in the shell with `ulimits -n 99999`. 2. Pass in `context.Context` so we cancel any pending go-routines when returning early. 3. Change getXXX signatures from returning a slice to a single *archiveFile (since we were never actually returning more than one file) 4. Check for error in the file archiving loop and abort if there's any. 5. Verbose logging support. * add todo for logging * fixed hardcoding 999999 in setOpenFileLimits * removed plural k8s func names, started work on docker commands * working log archival for docker containers * added docker inspect * added docker container stats * improved logic for get containers * cleanup * clarify assumptions * introduced debug sub commands kube, comp, and serv -- Ex: src debug kube -out=test * for some reason my .gitignore was ignoring these new files * changed flagset to use .StringVar rather than .String in flagSet package * correct flag typo in serv * fixed verbose output in kube command, changed -out flag to -o * switching to passanger, stating getConfig * ignore errors and get extsvc configs * add some comments from work with Tomas * added namespace flag for kube subcommand * added a little safety check to the kube command * added a semaphore to kube, added some text safty checks * added semphore to all RPC calls in archiveKube * added current-context logging before kube command execution * Atempting to debug error inconsistent --previous logs call * added network validation to comp subcommand * added a function to pull site config, cleaned up logging and usage funcs * moved sub-functions to relevant file, trial utility archiveFileFromCommand in src extsvc function * converted all kube commands to , renamed kube getContainers to getPods * refactored compose functions with archiveFileFromCommand * Made archiveFileFromCommand calls aesthetically pleasing, added docker ps, and get pods commands as a sort of index * changes all path package calls to path/filepath calls * emptied graveyard * return early from failed semaphore.Acquire rather than reading error * handle errors, remove direct comparison to booleans * cleaned up some more linter erros * refactor adding verify func * fixed final improperly handled error * use semaphore in debug comp * working errgroups in comp * refactor to errgroup in kube * refactored writer in archive functions * corrected all complex filepath calls using fmt.Sprintf() * filter docker ps output to appropriate network * removed setup debug as will as openfile limit adjustment * normalize site config * correctly process json * fleshed out serv command * Update .gitignore Co-authored-by: Eric Fritz <[email protected]> * Apply suggestions from code review Implementation of Eric's review suggestions; untested, requiring duplication in other files Co-authored-by: Eric Fritz <[email protected]> * implement and repair infered refactors in erics suggestions * addressed many suggested code improvements for readability and style; still needs run through for error handling, and potentially some more refactors * refactor baseDir processor * went over errors, still probably needs another run since I widely used errors.Wrapf * clarify all string serializations * Update cmd/src/debug_comp.go Co-authored-by: Eric Fritz <[email protected]> * Update cmd/src/debug_common.go Co-authored-by: Eric Fritz <[email protected]> * Update cmd/src/debug_common.go Co-authored-by: Eric Fritz <[email protected]> * Update cmd/src/debug_kube.go Co-authored-by: Eric Fritz <[email protected]> * Update cmd/src/debug_kube.go Co-authored-by: Eric Fritz <[email protected]> * Update cmd/src/debug_kube.go Co-authored-by: Eric Fritz <[email protected]> * finishing touches * final error handling cleanup * add changelog entry * Update cmd/src/debug_serv.go Co-authored-by: Adam Harvey <[email protected]> * Update cmd/src/debug_comp.go Co-authored-by: Adam Harvey <[email protected]> * Update cmd/src/debug_kube.go Co-authored-by: Adam Harvey <[email protected]> * Update cmd/src/debug_kube.go Co-authored-by: Adam Harvey <[email protected]> * Update cmd/src/debug.go Co-authored-by: Adam Harvey <[email protected]> * address harvey suggestions * finalize changes from harvey's suggestions * fix go.mod * really fix go.mod Co-authored-by: Tomás Senart <[email protected]> Co-authored-by: Tomás Senart <[email protected]> Co-authored-by: Eric Fritz <[email protected]> Co-authored-by: Adam Harvey <[email protected]>
1 parent f6ade94 commit ee83879

File tree

8 files changed

+896
-2
lines changed

8 files changed

+896
-2
lines changed

.gitignore

+2
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
11
./src
2+
./cmd/src/src
3+
*.zip
24
release
35
./vendor
46
.idea

CHANGELOG.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ All notable changes to `src-cli` are documented in this file.
1212
## Unreleased
1313

1414
### Added
15-
15+
- New command `src debug`. [#731](https://github.com/sourcegraph/src-cli/pull/731)
1616
- `src lsif upload` now supports the `-gitlab-token` flag. [#721](https://github.com/sourcegraph/src-cli/pull/721)
1717
- Batch Changes can be applied to Bitbucket Cloud when `src` is used with Sourcegraph 3.40 or later. [#725](https://github.com/sourcegraph/src-cli/pull/725)
1818

cmd/src/debug.go

+42
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
package main
2+
3+
import (
4+
"flag"
5+
"fmt"
6+
)
7+
8+
var debugCommands commander
9+
10+
func init() {
11+
usage := `'src debug' gathers and bundles debug data from a Sourcegraph deployment for troubleshooting.
12+
13+
Usage:
14+
15+
src debug command [command options]
16+
17+
The commands are:
18+
19+
kube dumps context from k8s deployments
20+
compose dumps context from docker-compose deployments
21+
server dumps context from single-container deployments
22+
23+
24+
Use "src debug command -h" for more information about a subcommands.
25+
src debug has access to flags on src -- Ex: src -v kube -o foo.zip
26+
27+
`
28+
29+
flagSet := flag.NewFlagSet("debug", flag.ExitOnError)
30+
handler := func(args []string) error {
31+
debugCommands.run(flagSet, "src debug", usage, args)
32+
return nil
33+
}
34+
35+
// Register the command.
36+
commands = append(commands, &command{
37+
flagSet: flagSet,
38+
aliases: []string{},
39+
handler: handler,
40+
usageFunc: func() { fmt.Println(usage) },
41+
})
42+
}

cmd/src/debug_common.go

+123
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
package main
2+
3+
import (
4+
"archive/zip"
5+
"context"
6+
"encoding/json"
7+
"fmt"
8+
"log"
9+
"os"
10+
"path/filepath"
11+
"strings"
12+
13+
"github.com/sourcegraph/src-cli/internal/exec"
14+
15+
"github.com/sourcegraph/sourcegraph/lib/errors"
16+
)
17+
18+
type archiveFile struct {
19+
name string
20+
data []byte
21+
err error
22+
}
23+
24+
func archiveFileFromCommand(ctx context.Context, path, cmd string, args ...string) *archiveFile {
25+
f := &archiveFile{name: path}
26+
f.data, f.err = exec.CommandContext(ctx, cmd, args...).CombinedOutput()
27+
if f.err != nil {
28+
f.err = errors.Wrapf(f.err, "executing command: %s %s: received error: %s", cmd, strings.Join(args, " "), f.data)
29+
}
30+
return f
31+
}
32+
33+
// verify prompts the user to confirm they want to run the command
34+
func verify(confirmationText string) (bool, error) {
35+
input := ""
36+
for strings.ToLower(input) != "y" && strings.ToLower(input) != "n" {
37+
fmt.Printf("%s [y/N]: ", confirmationText)
38+
if _, err := fmt.Scanln(&input); err != nil {
39+
return false, err
40+
}
41+
}
42+
43+
return strings.ToLower(input) == "y", nil
44+
}
45+
46+
func processBaseDir(base string) (string, string) {
47+
if !strings.HasSuffix(base, ".zip") {
48+
return base + ".zip", base
49+
}
50+
51+
return base, strings.TrimSuffix(base, ".zip")
52+
}
53+
54+
// write all the outputs from an archive command passed on the channel to to the zip writer
55+
func writeChannelContentsToZip(zw *zip.Writer, ch <-chan *archiveFile, verbose bool) error {
56+
for f := range ch {
57+
if verbose {
58+
log.Printf("archiving file %q with %d bytes", f.name, len(f.data))
59+
}
60+
61+
if f.err != nil {
62+
return f.err
63+
}
64+
65+
zf, err := zw.Create(f.name)
66+
if err != nil {
67+
return errors.Wrapf(err, "failed to create %q", f.name)
68+
}
69+
70+
if _, err := zf.Write(f.data); err != nil {
71+
return errors.Wrapf(err, "failed to write to %q", f.name)
72+
}
73+
}
74+
return nil
75+
}
76+
77+
// TODO: Currently external services and site configs are pulled using the SRC_ENDPOINT env var,
78+
// if theres a way to validate that the env var is pointing at the same instance as the docker and kubectl commands,
79+
// it should be implemented.
80+
81+
// TODO: file issue on the existence of OAuth signKey which needs to be redacted
82+
83+
// getExternalServicesConfig calls src extsvc list with the format flag -f,
84+
// and then returns an archiveFile to be consumed
85+
func getExternalServicesConfig(ctx context.Context, baseDir string) *archiveFile {
86+
const fmtStr = `{{range .Nodes}}{{.id}} | {{.kind}} | {{.displayName}}{{"\n"}}{{.config}}{{"\n---\n"}}{{end}}`
87+
return archiveFileFromCommand(
88+
ctx,
89+
filepath.Join(baseDir, "config", "external_services.txt"),
90+
os.Args[0], "extsvc", "list", "-f", fmtStr,
91+
)
92+
}
93+
94+
// getSiteConfig calls src api -query=... to query the api for site config json
95+
func getSiteConfig(ctx context.Context, baseDir string) *archiveFile {
96+
const siteConfigStr = `query { site { configuration { effectiveContents } } }`
97+
f := archiveFileFromCommand(ctx,
98+
filepath.Join(baseDir, "config", "siteConfig.json"),
99+
os.Args[0], "api", "-query", siteConfigStr,
100+
)
101+
102+
if f.err != nil {
103+
return f
104+
}
105+
106+
var siteConfig struct {
107+
Data struct {
108+
Site struct {
109+
Configuration struct {
110+
EffectiveContents string
111+
}
112+
}
113+
}
114+
}
115+
116+
if err := json.Unmarshal(f.data, &siteConfig); err != nil {
117+
f.err = err
118+
return f
119+
}
120+
121+
f.data = []byte(siteConfig.Data.Site.Configuration.EffectiveContents)
122+
return f
123+
}

0 commit comments

Comments
 (0)