Skip to content

gh-126835: Rename AST optimization related stuff after moving const folding to the peephole optimizier #131830

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,7 @@ Include/internal/pycore_time.h @pganssle @abalkin

# AST
Python/ast.c @isidentical @JelleZijlstra @eclips4
Python/ast_opt.c @isidentical @eclips4
Python/ast_process.c @isidentical @eclips4
Parser/asdl.py @isidentical @JelleZijlstra @eclips4
Parser/asdl_c.py @isidentical @JelleZijlstra @eclips4
Lib/ast.py @isidentical @JelleZijlstra @eclips4
Expand Down
6 changes: 3 additions & 3 deletions Include/internal/pycore_compile.h
Original file line number Diff line number Diff line change
Expand Up @@ -34,16 +34,16 @@ PyAPI_FUNC(PyCodeObject*) _PyAST_Compile(
int optimize,
struct _arena *arena);

/* AST optimizations */
extern int _PyCompile_AstOptimize(
/* AST processing */
extern int _PyCompile_AstProcess(
struct _mod *mod,
PyObject *filename,
PyCompilerFlags *flags,
int optimize,
struct _arena *arena,
int syntax_check_only);

extern int _PyAST_Optimize(
extern int _PyAST_Process(
struct _mod *,
struct _arena *arena,
PyObject *filename,
Expand Down
4 changes: 2 additions & 2 deletions InternalDocs/compiler.md
Original file line number Diff line number Diff line change
Expand Up @@ -505,8 +505,8 @@ Important files
* [Python/ast.c](../Python/ast.c):
Used for validating the AST.

* [Python/ast_opt.c](../Python/ast_opt.c):
Optimizes the AST.
* [Python/ast_process.c](../Python/ast_process.c):
Processes the AST before compiling.

* [Python/ast_unparse.c](../Python/ast_unparse.c):
Converts the AST expression node back into a string (for string annotations).
Expand Down
2 changes: 1 addition & 1 deletion Makefile.pre.in
Original file line number Diff line number Diff line change
Expand Up @@ -426,7 +426,7 @@ PYTHON_OBJS= \
Python/asdl.o \
Python/assemble.o \
Python/ast.o \
Python/ast_opt.o \
Python/ast_process.o \
Python/ast_unparse.o \
Python/bltinmodule.o \
Python/brc.o \
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Move constant folding to the peephole optimizer. Rename AST optimization
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you say which files were renamed? we have a 3.13 news entry that mentions ast_opt so maybe it's a good idea to say that Python/ast_opt.c is renamed Python/ast_process.c

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I will.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I think it's good to mention the files I don't know if we ever mentioned those structs and functions publicly. If not, no need to mention their renames. Strictly speaking, Python/* files are also all internals so maybe it's not needed to mention them in the end but I found a 3.13 NEWS entry that mentioned it (which is why I suggested this)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Struct maybe yes because it's private, but _PyAST_Optimize and _PyCompile_AstOptimize are in internal API, aren't they?

Copy link
Member

@picnixz picnixz Mar 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We actually made the structs private in 3.13:

Make ``_PyASTOptimizeState`` internal to ast_opt.c. Make ``_PyAST_Optimize``
take two integers instead of a pointer to this struct. This avoids the need
to include pycore_compile.h in ast_opt.c.

But we also mentioned _PyAST_Optimize, so we might want to mention the change even if no one should use them.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can include it in the news item. No need for what's new.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Im a bit confused. This is news entry, not what's new. Am I missing something?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I think Irit just wanted to say "keep what you wrote but no need for a What's New entry" (maybe I wasn't clear but the changelog I mentioned also was a simple blurb, not a full-fledged What's New entry)

related files (Python/ast_opt.c -> Python/ast_process.c), structs (_PyASTOptimizeState -> _PyASTProcessState)
and functions (_PyAST_Optimize -> _PyAST_Process, _PyCompile_AstOptimize -> _PyCompile_AstProcess).
2 changes: 1 addition & 1 deletion PCbuild/_freeze_module.vcxproj
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,7 @@
<ClCompile Include="..\Python\asdl.c" />
<ClCompile Include="..\Python\assemble.c" />
<ClCompile Include="..\Python\ast.c" />
<ClCompile Include="..\Python\ast_opt.c" />
<ClCompile Include="..\Python\ast_process.c" />
<ClCompile Include="..\Python\ast_unparse.c" />
<ClCompile Include="..\Python\bltinmodule.c" />
<ClCompile Include="..\Python\brc.c" />
Expand Down
2 changes: 1 addition & 1 deletion PCbuild/_freeze_module.vcxproj.filters
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@
<ClCompile Include="..\Python\ast.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\Python\ast_opt.c">
<ClCompile Include="..\Python\ast_process.c">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="..\Python\ast_unparse.c">
Expand Down
2 changes: 1 addition & 1 deletion PCbuild/pythoncore.vcxproj
Original file line number Diff line number Diff line change
Expand Up @@ -571,7 +571,7 @@
<ClCompile Include="..\Python\asdl.c" />
<ClCompile Include="..\Python\assemble.c" />
<ClCompile Include="..\Python\ast.c" />
<ClCompile Include="..\Python\ast_opt.c" />
<ClCompile Include="..\Python\ast_process.c" />
<ClCompile Include="..\Python\ast_unparse.c" />
<ClCompile Include="..\Python\bltinmodule.c" />
<ClCompile Include="..\Python\bootstrap_hash.c" />
Expand Down
2 changes: 1 addition & 1 deletion PCbuild/pythoncore.vcxproj.filters
Original file line number Diff line number Diff line change
Expand Up @@ -1301,7 +1301,7 @@
<ClCompile Include="..\Python\ast.c">
<Filter>Python</Filter>
</ClCompile>
<ClCompile Include="..\Python\ast_opt.c">
<ClCompile Include="..\Python\ast_process.c">
<Filter>Python</Filter>
</ClCompile>
<ClCompile Include="..\Python\ast_unparse.c">
Expand Down
77 changes: 38 additions & 39 deletions Python/ast_opt.c → Python/ast_process.c
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
/* AST Optimizer */
#include "Python.h"
#include "pycore_ast.h" // _PyAST_GetDocString()
#include "pycore_c_array.h" // _Py_CArray_EnsureCapacity()
Expand All @@ -23,7 +22,7 @@ typedef struct {

_Py_c_array_t cf_finally; /* context for PEP 678 check */
int cf_finally_used;
} _PyASTOptimizeState;
} _PyASTProcessState;

#define ENTER_RECURSIVE() \
if (Py_EnterRecursiveCall(" during compilation")) { \
Expand All @@ -33,14 +32,14 @@ if (Py_EnterRecursiveCall(" during compilation")) { \
#define LEAVE_RECURSIVE() Py_LeaveRecursiveCall();

static ControlFlowInFinallyContext*
get_cf_finally_top(_PyASTOptimizeState *state)
get_cf_finally_top(_PyASTProcessState *state)
{
int idx = state->cf_finally_used;
return ((ControlFlowInFinallyContext*)state->cf_finally.array) + idx;
}

static int
push_cf_context(_PyASTOptimizeState *state, stmt_ty node, bool finally, bool funcdef, bool loop)
push_cf_context(_PyASTProcessState *state, stmt_ty node, bool finally, bool funcdef, bool loop)
{
if (_Py_CArray_EnsureCapacity(&state->cf_finally, state->cf_finally_used+1) < 0) {
return 0;
Expand All @@ -56,14 +55,14 @@ push_cf_context(_PyASTOptimizeState *state, stmt_ty node, bool finally, bool fun
}

static void
pop_cf_context(_PyASTOptimizeState *state)
pop_cf_context(_PyASTProcessState *state)
{
assert(state->cf_finally_used > 0);
state->cf_finally_used--;
}

static int
control_flow_in_finally_warning(const char *kw, stmt_ty n, _PyASTOptimizeState *state)
control_flow_in_finally_warning(const char *kw, stmt_ty n, _PyASTProcessState *state)
{
PyObject *msg = PyUnicode_FromFormat("'%s' in a 'finally' block", kw);
if (msg == NULL) {
Expand All @@ -77,7 +76,7 @@ control_flow_in_finally_warning(const char *kw, stmt_ty n, _PyASTOptimizeState *
}

static int
before_return(_PyASTOptimizeState *state, stmt_ty node_)
before_return(_PyASTProcessState *state, stmt_ty node_)
{
if (state->cf_finally_used > 0) {
ControlFlowInFinallyContext *ctx = get_cf_finally_top(state);
Expand All @@ -91,7 +90,7 @@ before_return(_PyASTOptimizeState *state, stmt_ty node_)
}

static int
before_loop_exit(_PyASTOptimizeState *state, stmt_ty node_, const char *kw)
before_loop_exit(_PyASTProcessState *state, stmt_ty node_, const char *kw)
{
if (state->cf_finally_used > 0) {
ControlFlowInFinallyContext *ctx = get_cf_finally_top(state);
Expand Down Expand Up @@ -366,7 +365,7 @@ optimize_format(expr_ty node, PyObject *fmt, asdl_expr_seq *elts, PyArena *arena
}

static int
fold_binop(expr_ty node, PyArena *arena, _PyASTOptimizeState *state)
fold_binop(expr_ty node, PyArena *arena, _PyASTProcessState *state)
{
if (state->syntax_check_only) {
return 1;
Expand All @@ -390,18 +389,18 @@ fold_binop(expr_ty node, PyArena *arena, _PyASTOptimizeState *state)
return 1;
}

static int astfold_mod(mod_ty node_, PyArena *ctx_, _PyASTOptimizeState *state);
static int astfold_stmt(stmt_ty node_, PyArena *ctx_, _PyASTOptimizeState *state);
static int astfold_expr(expr_ty node_, PyArena *ctx_, _PyASTOptimizeState *state);
static int astfold_arguments(arguments_ty node_, PyArena *ctx_, _PyASTOptimizeState *state);
static int astfold_comprehension(comprehension_ty node_, PyArena *ctx_, _PyASTOptimizeState *state);
static int astfold_keyword(keyword_ty node_, PyArena *ctx_, _PyASTOptimizeState *state);
static int astfold_arg(arg_ty node_, PyArena *ctx_, _PyASTOptimizeState *state);
static int astfold_withitem(withitem_ty node_, PyArena *ctx_, _PyASTOptimizeState *state);
static int astfold_excepthandler(excepthandler_ty node_, PyArena *ctx_, _PyASTOptimizeState *state);
static int astfold_match_case(match_case_ty node_, PyArena *ctx_, _PyASTOptimizeState *state);
static int astfold_pattern(pattern_ty node_, PyArena *ctx_, _PyASTOptimizeState *state);
static int astfold_type_param(type_param_ty node_, PyArena *ctx_, _PyASTOptimizeState *state);
static int astfold_mod(mod_ty node_, PyArena *ctx_, _PyASTProcessState *state);
static int astfold_stmt(stmt_ty node_, PyArena *ctx_, _PyASTProcessState *state);
static int astfold_expr(expr_ty node_, PyArena *ctx_, _PyASTProcessState *state);
static int astfold_arguments(arguments_ty node_, PyArena *ctx_, _PyASTProcessState *state);
static int astfold_comprehension(comprehension_ty node_, PyArena *ctx_, _PyASTProcessState *state);
static int astfold_keyword(keyword_ty node_, PyArena *ctx_, _PyASTProcessState *state);
static int astfold_arg(arg_ty node_, PyArena *ctx_, _PyASTProcessState *state);
static int astfold_withitem(withitem_ty node_, PyArena *ctx_, _PyASTProcessState *state);
static int astfold_excepthandler(excepthandler_ty node_, PyArena *ctx_, _PyASTProcessState *state);
static int astfold_match_case(match_case_ty node_, PyArena *ctx_, _PyASTProcessState *state);
static int astfold_pattern(pattern_ty node_, PyArena *ctx_, _PyASTProcessState *state);
static int astfold_type_param(type_param_ty node_, PyArena *ctx_, _PyASTProcessState *state);

#define CALL(FUNC, TYPE, ARG) \
if (!FUNC((ARG), ctx_, state)) \
Expand Down Expand Up @@ -437,7 +436,7 @@ stmt_seq_remove_item(asdl_stmt_seq *stmts, Py_ssize_t idx)
}

static int
astfold_body(asdl_stmt_seq *stmts, PyArena *ctx_, _PyASTOptimizeState *state)
astfold_body(asdl_stmt_seq *stmts, PyArena *ctx_, _PyASTProcessState *state)
{
int docstring = _PyAST_GetDocString(stmts) != NULL;
if (docstring && (state->optimize >= 2)) {
Expand Down Expand Up @@ -467,7 +466,7 @@ astfold_body(asdl_stmt_seq *stmts, PyArena *ctx_, _PyASTOptimizeState *state)
}

static int
astfold_mod(mod_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
astfold_mod(mod_ty node_, PyArena *ctx_, _PyASTProcessState *state)
{
switch (node_->kind) {
case Module_kind:
Expand All @@ -489,7 +488,7 @@ astfold_mod(mod_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
}

static int
astfold_expr(expr_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
astfold_expr(expr_ty node_, PyArena *ctx_, _PyASTProcessState *state)
{
ENTER_RECURSIVE();
switch (node_->kind) {
Expand Down Expand Up @@ -607,14 +606,14 @@ astfold_expr(expr_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
}

static int
astfold_keyword(keyword_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
astfold_keyword(keyword_ty node_, PyArena *ctx_, _PyASTProcessState *state)
{
CALL(astfold_expr, expr_ty, node_->value);
return 1;
}

static int
astfold_comprehension(comprehension_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
astfold_comprehension(comprehension_ty node_, PyArena *ctx_, _PyASTProcessState *state)
{
CALL(astfold_expr, expr_ty, node_->target);
CALL(astfold_expr, expr_ty, node_->iter);
Expand All @@ -623,7 +622,7 @@ astfold_comprehension(comprehension_ty node_, PyArena *ctx_, _PyASTOptimizeState
}

static int
astfold_arguments(arguments_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
astfold_arguments(arguments_ty node_, PyArena *ctx_, _PyASTProcessState *state)
{
CALL_SEQ(astfold_arg, arg, node_->posonlyargs);
CALL_SEQ(astfold_arg, arg, node_->args);
Expand All @@ -636,7 +635,7 @@ astfold_arguments(arguments_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
}

static int
astfold_arg(arg_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
astfold_arg(arg_ty node_, PyArena *ctx_, _PyASTProcessState *state)
{
if (!(state->ff_features & CO_FUTURE_ANNOTATIONS)) {
CALL_OPT(astfold_expr, expr_ty, node_->annotation);
Expand All @@ -645,7 +644,7 @@ astfold_arg(arg_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
}

static int
astfold_stmt(stmt_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
astfold_stmt(stmt_ty node_, PyArena *ctx_, _PyASTProcessState *state)
{
ENTER_RECURSIVE();
switch (node_->kind) {
Expand Down Expand Up @@ -800,7 +799,7 @@ astfold_stmt(stmt_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
}

static int
astfold_excepthandler(excepthandler_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
astfold_excepthandler(excepthandler_ty node_, PyArena *ctx_, _PyASTProcessState *state)
{
switch (node_->kind) {
case ExceptHandler_kind:
Expand All @@ -814,15 +813,15 @@ astfold_excepthandler(excepthandler_ty node_, PyArena *ctx_, _PyASTOptimizeState
}

static int
astfold_withitem(withitem_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
astfold_withitem(withitem_ty node_, PyArena *ctx_, _PyASTProcessState *state)
{
CALL(astfold_expr, expr_ty, node_->context_expr);
CALL_OPT(astfold_expr, expr_ty, node_->optional_vars);
return 1;
}

static int
fold_const_match_patterns(expr_ty node, PyArena *ctx_, _PyASTOptimizeState *state)
fold_const_match_patterns(expr_ty node, PyArena *ctx_, _PyASTProcessState *state)
{
if (state->syntax_check_only) {
return 1;
Expand Down Expand Up @@ -863,7 +862,7 @@ fold_const_match_patterns(expr_ty node, PyArena *ctx_, _PyASTOptimizeState *stat
}

static int
astfold_pattern(pattern_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
astfold_pattern(pattern_ty node_, PyArena *ctx_, _PyASTProcessState *state)
{
// Currently, this is really only used to form complex/negative numeric
// constants in MatchValue and MatchMapping nodes
Expand Down Expand Up @@ -905,7 +904,7 @@ astfold_pattern(pattern_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
}

static int
astfold_match_case(match_case_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
astfold_match_case(match_case_ty node_, PyArena *ctx_, _PyASTProcessState *state)
{
CALL(astfold_pattern, expr_ty, node_->pattern);
CALL_OPT(astfold_expr, expr_ty, node_->guard);
Expand All @@ -914,7 +913,7 @@ astfold_match_case(match_case_ty node_, PyArena *ctx_, _PyASTOptimizeState *stat
}

static int
astfold_type_param(type_param_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
astfold_type_param(type_param_ty node_, PyArena *ctx_, _PyASTProcessState *state)
{
switch (node_->kind) {
case TypeVar_kind:
Expand All @@ -936,11 +935,11 @@ astfold_type_param(type_param_ty node_, PyArena *ctx_, _PyASTOptimizeState *stat
#undef CALL_SEQ

int
_PyAST_Optimize(mod_ty mod, PyArena *arena, PyObject *filename, int optimize,
int ff_features, int syntax_check_only)
_PyAST_Process(mod_ty mod, PyArena *arena, PyObject *filename, int optimize,
int ff_features, int syntax_check_only)
{
_PyASTOptimizeState state;
memset(&state, 0, sizeof(_PyASTOptimizeState));
_PyASTProcessState state;
memset(&state, 0, sizeof(_PyASTProcessState));
state.filename = filename;
state.optimize = optimize;
state.ff_features = ff_features;
Expand Down
2 changes: 1 addition & 1 deletion Python/bltinmodule.c
Original file line number Diff line number Diff line change
Expand Up @@ -847,7 +847,7 @@ builtin_compile_impl(PyObject *module, PyObject *source, PyObject *filename,
goto error;
}
int syntax_check_only = ((flags & PyCF_OPTIMIZED_AST) == PyCF_ONLY_AST); /* unoptiomized AST */
if (_PyCompile_AstOptimize(mod, filename, &cf, optimize,
if (_PyCompile_AstProcess(mod, filename, &cf, optimize,
arena, syntax_check_only) < 0) {
_PyArena_Free(arena);
goto error;
Expand Down
8 changes: 4 additions & 4 deletions Python/compile.c
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ compiler_setup(compiler *c, mod_ty mod, PyObject *filename,
c->c_optimize = (optimize == -1) ? _Py_GetConfig()->optimization_level : optimize;
c->c_save_nested_seqs = false;

if (!_PyAST_Optimize(mod, arena, filename, c->c_optimize, merged, 0)) {
if (!_PyAST_Process(mod, arena, filename, c->c_optimize, merged, 0)) {
return ERROR;
}
c->c_st = _PySymtable_Build(mod, filename, &c->c_future);
Expand Down Expand Up @@ -1449,8 +1449,8 @@ _PyAST_Compile(mod_ty mod, PyObject *filename, PyCompilerFlags *pflags,
}

int
_PyCompile_AstOptimize(mod_ty mod, PyObject *filename, PyCompilerFlags *cf,
int optimize, PyArena *arena, int no_const_folding)
_PyCompile_AstProcess(mod_ty mod, PyObject *filename, PyCompilerFlags *cf,
int optimize, PyArena *arena, int no_const_folding)
{
_PyFutureFeatures future;
if (!_PyFuture_FromAST(mod, filename, &future)) {
Expand All @@ -1460,7 +1460,7 @@ _PyCompile_AstOptimize(mod_ty mod, PyObject *filename, PyCompilerFlags *cf,
if (optimize == -1) {
optimize = _Py_GetConfig()->optimization_level;
}
if (!_PyAST_Optimize(mod, arena, filename, optimize, flags, no_const_folding)) {
if (!_PyAST_Process(mod, arena, filename, optimize, flags, no_const_folding)) {
return -1;
}
return 0;
Expand Down
2 changes: 1 addition & 1 deletion Python/pythonrun.c
Original file line number Diff line number Diff line change
Expand Up @@ -1498,7 +1498,7 @@ Py_CompileStringObject(const char *str, PyObject *filename, int start,
}
if (flags && (flags->cf_flags & PyCF_ONLY_AST)) {
int syntax_check_only = ((flags->cf_flags & PyCF_OPTIMIZED_AST) == PyCF_ONLY_AST); /* unoptiomized AST */
if (_PyCompile_AstOptimize(mod, filename, flags, optimize, arena, syntax_check_only) < 0) {
if (_PyCompile_AstProcess(mod, filename, flags, optimize, arena, syntax_check_only) < 0) {
_PyArena_Free(arena);
return NULL;
}
Expand Down
1 change: 0 additions & 1 deletion Tools/c-analyzer/cpython/ignored.tsv
Original file line number Diff line number Diff line change
Expand Up @@ -347,7 +347,6 @@ Objects/unicodeobject.c unicode_translate_call_errorhandler argparse -
Parser/parser.c - reserved_keywords -
Parser/parser.c - soft_keywords -
Parser/lexer/lexer.c - type_comment_prefix -
Python/ast_opt.c fold_unaryop ops -
Python/ceval.c - _PyEval_BinaryOps -
Python/ceval.c - _Py_INTERPRETER_TRAMPOLINE_INSTRUCTIONS -
Python/codecs.c - Py_hexdigits -
Expand Down
Loading