Skip to content

Optionally capture Lambda request and response payloads #222

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Oct 1, 2021

Conversation

astuyve
Copy link
Contributor

@astuyve astuyve commented Sep 10, 2021

What does this PR do?

Uses span tags to recursively tag attrs of the request and response object.

  • Max depth of 5 (not configurable)
  • Max length of 5,000 chars (backend limit for span tags)
  • Filters out common auth patterns
  • Allows for more specific patterns with datadog.yaml obfuscations

Motivation

Testing Guidelines

Additional Notes

Types of Changes

  • Bug fix
  • New feature
  • Breaking change
  • Misc (docs, refactoring, dependency upgrade, etc.)

Check all that apply

  • This PR's description is comprehensive
  • This PR contains breaking changes that are documented in the description
  • This PR introduces new APIs or parameters that are documented and unlikely to change in the foreseeable future
  • This PR impacts documentation, and it has been updated (or a ticket has been logged)
  • This PR's changes are covered by the automated tests
  • This PR collects user input/sensitive content into Datadog
  • This PR passes the integration tests (ask a Datadog member to run the tests)

TODO

  • Documentation update

@codecov-commenter
Copy link

codecov-commenter commented Sep 10, 2021

Codecov Report

Merging #222 (25b7c65) into main (fefe509) will decrease coverage by 0.41%.
The diff coverage is 72.72%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #222      +/-   ##
==========================================
- Coverage   88.06%   87.65%   -0.42%     
==========================================
  Files          31       32       +1     
  Lines        1198     1231      +33     
  Branches      240      249       +9     
==========================================
+ Hits         1055     1079      +24     
- Misses         86       91       +5     
- Partials       57       61       +4     
Impacted Files Coverage Δ
src/trace/listener.ts 87.32% <ø> (ø)
src/index.ts 77.87% <42.85%> (-2.32%) ⬇️
src/utils/tag-object.ts 80.00% <80.00%> (ø)
src/utils/index.ts 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fefe509...25b7c65. Read the comment docs.

@astuyve astuyve marked this pull request as ready for review September 14, 2021 17:31
@astuyve astuyve requested a review from a team as a code owner September 14, 2021 17:31
@astuyve astuyve changed the title [WIP] Optionally capture Lambda request and response payloads Optionally capture Lambda request and response payloads Sep 14, 2021
@@ -0,0 +1,34 @@
const redactableKeys = ["authorization", "x-authorization", "password", "token"];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this could be customizable by the customer?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh good question - we actually already allow the customer to define obfuscation rules via datadog.yaml https://docs.datadoghq.com/tracing/setup_overview/configure_data_security/?tab=datadogyaml#replace-rules-for-tag-filtering

This is just a hardcoded list to prevent non-configured payload capture from indexing basic/common secrets. It's not something we want to automatically index. If a user wants to capture these, they'll have to manually set a tag.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh I see, so maybe we should make a link/note about this config option when we'll release this new env var?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup! And it'll be in the blog post :)

Copy link
Contributor

@DarcyRaynerDD DarcyRaynerDD left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, left some comments.

@@ -3,3 +3,4 @@ export { wrap, promisifiedHandler } from "./handler";
export { Timer } from "./timer";
export { logError, logDebug, Logger, setLogLevel, setLogger, LogLevel } from "./log";
export { get, post } from "./request";
export { tagObject } from "./tagObject";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we typically name files like this tag-object.ts

if (typeof obj === "string") {
let parsed: string;
try {
parsed = JSON.parse(obj);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's a string, does it need to be parsed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question. This is needed to deeply tag nested JSON into separate items, otherwise it just tags an object (ala: setTag('request.body', { foo: { blah: 'ahhh' }}), which doesn't display nicely in the app.

@astuyve astuyve merged commit 1ef5527 into main Oct 1, 2021
@astuyve astuyve deleted the aj/capture-request-and-response branch October 1, 2021 18:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants