Skip to content

refactor: Add log_std(out|err) bools to repo_utils that execute a subprocess #2817

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 24, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,9 @@ END_UNRELEASED_TEMPLATE

{#v0-0-0-added}
### Added
* Nothing added.
* Repo utilities `execute_unchecked`, `execute_checked`, and `execute_checked_stdout` now
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The repo_utils apis are internal, so no need to document this in the user-facing changelog.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack 👍, I'll keep that in mind for next time.

support `log_stdout` and `log_stderr` keyword arg booleans. When these are `True`
(the default), the subprocess's stdout/stderr will be logged.

{#v0-0-0-removed}
### Removed
Expand Down
31 changes: 24 additions & 7 deletions python/private/repo_utils.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,8 @@ def _execute_internal(
arguments,
environment = {},
logger = None,
log_stdout = True,
log_stderr = True,
Comment on lines +101 to +102
Copy link
Collaborator Author

@dougthor42 dougthor42 Apr 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One alternative API option is to use a single string arg. Eg:

log_std = "both"  # one of ["both", "stdout", "stderr"]

(I'd prefer an enum but this is starlark not python 🫠)

LMK which API people would prefer or if you have another option in mind. And please suggest a better arg name than log_std haha.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the log_xxx API is fine. These are internal helpers, so easy to change if we want.

**kwargs):
"""Execute a subprocess with debugging instrumentation.

Expand All @@ -116,6 +118,10 @@ def _execute_internal(
logger: optional `Logger` to use for logging execution details. Must be
specified when using module_ctx. If not specified, a default will
be created.
log_stdout: If True (the default), write stdout to the logged message. Setting
to False can be useful for large stdout messages or for secrets.
log_stderr: If True (the default), write stderr to the logged message. Setting
to False can be useful for large stderr messages or for secrets.
**kwargs: additional kwargs to pass onto rctx.execute

Returns:
Expand Down Expand Up @@ -160,7 +166,7 @@ def _execute_internal(
cwd = _cwd_to_str(mrctx, kwargs),
timeout = _timeout_to_str(kwargs),
env_str = _env_to_str(environment),
output = _outputs_to_str(result),
output = _outputs_to_str(result, log_stdout = log_stdout, log_stderr = log_stderr),
))
elif _is_repo_debug_enabled(mrctx):
logger.debug((
Expand All @@ -171,7 +177,7 @@ def _execute_internal(
op = op,
status = "success" if result.return_code == 0 else "failure",
return_code = result.return_code,
output = _outputs_to_str(result),
output = _outputs_to_str(result, log_stdout = log_stdout, log_stderr = log_stderr),
))

result_kwargs = {k: getattr(result, k) for k in dir(result)}
Expand All @@ -183,6 +189,8 @@ def _execute_internal(
mrctx = mrctx,
kwargs = kwargs,
environment = environment,
log_stdout = log_stdout,
log_stderr = log_stderr,
),
**result_kwargs
)
Expand Down Expand Up @@ -220,7 +228,16 @@ def _execute_checked_stdout(*args, **kwargs):
"""Calls execute_checked, but only returns the stdout value."""
return _execute_checked(*args, **kwargs).stdout

def _execute_describe_failure(*, op, arguments, result, mrctx, kwargs, environment):
def _execute_describe_failure(
*,
op,
arguments,
result,
mrctx,
kwargs,
environment,
log_stdout = True,
log_stderr = True):
return (
"repo.execute: {op}: failure:\n" +
" command: {cmd}\n" +
Expand All @@ -236,7 +253,7 @@ def _execute_describe_failure(*, op, arguments, result, mrctx, kwargs, environme
cwd = _cwd_to_str(mrctx, kwargs),
timeout = _timeout_to_str(kwargs),
env_str = _env_to_str(environment),
output = _outputs_to_str(result),
output = _outputs_to_str(result, log_stdout = log_stdout, log_stderr = log_stderr),
)

def _which_checked(mrctx, binary_name):
Expand Down Expand Up @@ -331,11 +348,11 @@ def _env_to_str(environment):
def _timeout_to_str(kwargs):
return kwargs.get("timeout", "<default timeout>")

def _outputs_to_str(result):
def _outputs_to_str(result, log_stdout = True, log_stderr = True):
lines = []
items = [
("stdout", result.stdout),
("stderr", result.stderr),
("stdout", result.stdout if log_stdout else "<log_stdout = False; skipping>"),
("stderr", result.stderr if log_stderr else "<log_stderr = False; skipping>"),
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any tips on how this should be tested?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we have any unit tests of these.

]
for name, content in items:
if content:
Expand Down