Skip to content

bpo-46541: Replace core use of _Py_IDENTIFIER() with statically initialized global objects. #30928

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 117 commits into from
Feb 8, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
117 commits
Select commit Hold shift + click to select a range
ec0c931
Add _Py_STRING_LENGTH().
ericsnowcurrently Dec 2, 2021
cea1335
Add PyInterpreterState._preallocated.strings (identifiers-only for now).
ericsnowcurrently Dec 2, 2021
a5513fc
Move the strings to _PyRuntimeState.
ericsnowcurrently Jan 19, 2022
dd084ee
Initialize the global singleton strings (identifiers-only for now).
ericsnowcurrently Dec 2, 2021
472b0a6
Add the global (non-identifier) string literals.
ericsnowcurrently Dec 2, 2021
8a3096b
Add _PyObject_CallMethod().
ericsnowcurrently Dec 2, 2021
38300b3
Get rid of _Py_IDENTIFIER() in tokenizer.c.
ericsnowcurrently Dec 2, 2021
fb9fe24
Add _PyDict_GetItemWithError().
ericsnowcurrently Dec 3, 2021
7fa8e83
Set the hash if not set yet.
ericsnowcurrently Dec 4, 2021
9563c27
Add _PySys_GetAttr().
ericsnowcurrently Dec 3, 2021
674e40b
Get rid of _Py_IDENTIFIER() in _warnings.c.
ericsnowcurrently Dec 3, 2021
e5e7483
_PyObject_LookupSpecial() -> _PyObject_LookupSpecialId().
ericsnowcurrently Dec 4, 2021
07ac6ba
Add _PyObject_LookupSpecial().
ericsnowcurrently Dec 4, 2021
249e830
Get rid of _Py_IDENTIFIER() in bltinmodule.c.
ericsnowcurrently Dec 4, 2021
3ea3c09
Get rid of _Py_IDENTIFIER() in ceval.c.
ericsnowcurrently Dec 4, 2021
5713dab
Add a TODO.
ericsnowcurrently Jan 19, 2022
77a21d6
Get rid of _Py_IDENTIFIER() in specialize.c.
ericsnowcurrently Jan 19, 2022
704d440
Get rid of _Py_IDENTIFIER() in traceback.c.
ericsnowcurrently Jan 19, 2022
2800db6
Get rid of _Py_IDENTIFIER() in codecs.c.
ericsnowcurrently Jan 20, 2022
db72429
Get rid of _Py_IDENTIFIER() in compile.c.
ericsnowcurrently Jan 24, 2022
737391f
Get rid of _Py_IDENTIFIER() in importdl.c.
ericsnowcurrently Jan 24, 2022
9e3ff70
Get rid of _Py_IDENTIFIER() in marshal.c.
ericsnowcurrently Jan 24, 2022
71fc362
Get rid of _Py_IDENTIFIER() in errors.c.
ericsnowcurrently Jan 24, 2022
2046b53
Get rid of _Py_IDENTIFIER() in import.c.
ericsnowcurrently Jan 24, 2022
a4c1aa9
Add _PySys_SetAttr().
ericsnowcurrently Jan 24, 2022
48fcb06
Get rid of _Py_IDENTIFIER() in pylifecycle.c.
ericsnowcurrently Jan 24, 2022
c9bc124
Get rid of _Py_IDENTIFIER() in pythonrun.c.
ericsnowcurrently Jan 24, 2022
816c630
Get rid of _Py_IDENTIFIER() in sysmodule.c.
ericsnowcurrently Jan 24, 2022
ef2b061
Get rid of _Py_IDENTIFIER() in typeobject.c.
ericsnowcurrently Jan 25, 2022
4e03f95
Get rid of _Py_IDENTIFIER() in abstract.c.
ericsnowcurrently Jan 25, 2022
814ec4a
Add _PyEval_GetBuiltin().
ericsnowcurrently Jan 25, 2022
a91fe82
Get rid of _Py_IDENTIFIER() in descrobject.c.
ericsnowcurrently Jan 25, 2022
25a2ec4
Get rid of _Py_IDENTIFIER() in moduleobject.c.
ericsnowcurrently Jan 25, 2022
865cc8a
Get rid of _Py_IDENTIFIER() in object.c.
ericsnowcurrently Jan 25, 2022
e4c2f0d
Get rid of _Py_IDENTIFIER() in bytearrayobject.c.
ericsnowcurrently Jan 25, 2022
f82baa8
Get rid of _Py_IDENTIFIER() in bytesobject.c.
ericsnowcurrently Jan 25, 2022
4b10323
Get rid of _Py_IDENTIFIER() in classobject.c.
ericsnowcurrently Jan 25, 2022
ffd9493
Get rid of _Py_IDENTIFIER() in complexobject.c.
ericsnowcurrently Jan 25, 2022
ddd887b
Get rid of _Py_IDENTIFIER() in dictobject.c.
ericsnowcurrently Jan 25, 2022
0a02936
Get rid of _Py_IDENTIFIER() in enumobject.c.
ericsnowcurrently Jan 25, 2022
d157842
Get rid of _Py_IDENTIFIER() in exceptions.c.
ericsnowcurrently Jan 25, 2022
297240c
Get rid of _Py_IDENTIFIER() in fileobject.c.
ericsnowcurrently Jan 25, 2022
875353d
Get rid of _Py_IDENTIFIER() in frameobject.c.
ericsnowcurrently Jan 25, 2022
9138b2e
Get rid of _Py_IDENTIFIER() in funcobject.c.
ericsnowcurrently Jan 25, 2022
5e576f0
Get rid of _Py_IDENTIFIER() in genericaliasobject.c.
ericsnowcurrently Jan 25, 2022
7591bf1
Get rid of _Py_IDENTIFIER() in genobject.c.
ericsnowcurrently Jan 25, 2022
f27547e
Get rid of _Py_IDENTIFIER() in iterobject.c.
ericsnowcurrently Jan 25, 2022
dd3900f
Get rid of _Py_IDENTIFIER() in listobject.c.
ericsnowcurrently Jan 25, 2022
ecd568d
Get rid of _Py_IDENTIFIER() in longobject.c.
ericsnowcurrently Jan 25, 2022
187133d
Get rid of _Py_IDENTIFIER() in methodobject.c.
ericsnowcurrently Jan 25, 2022
e04947e
Get rid of _Py_IDENTIFIER() in odictobject.c.
ericsnowcurrently Jan 25, 2022
91ef303
Get rid of _Py_IDENTIFIER() in rangeobject.c.
ericsnowcurrently Jan 25, 2022
21c564a
Get rid of _Py_IDENTIFIER() in setobject.c.
ericsnowcurrently Jan 25, 2022
d833719
Get rid of _Py_IDENTIFIER() in structseq.c.
ericsnowcurrently Jan 25, 2022
329f8fd
Get rid of _Py_IDENTIFIER() in tupleobject.c.
ericsnowcurrently Jan 25, 2022
0cbeb2d
Get rid of _Py_IDENTIFIER() in unicodeobject.c.
ericsnowcurrently Jan 25, 2022
972044a
Get rid of _Py_IDENTIFIER() in unionobject.c.
ericsnowcurrently Jan 25, 2022
e55556f
Get rid of _Py_IDENTIFIER() in weakrefobject.c.
ericsnowcurrently Jan 25, 2022
5e32d13
Get rid of _Py_IDENTIFIER() in _pickle.c.
ericsnowcurrently Jan 25, 2022
6081cf1
Add _PyObject_CallMethodObj().
ericsnowcurrently Jan 25, 2022
6aa5674
Use _PyObject_CallMethodObj().
ericsnowcurrently Jan 25, 2022
37c0200
_PyObject_CallMethod() -> _PyObject_CallMethodFormat().
ericsnowcurrently Jan 25, 2022
10f82cd
_PyObject_CallMethodObj() -> _PyObject_CallMethod().
ericsnowcurrently Jan 25, 2022
85f16a4
Get rid of _Py_IDENTIFIER() in the _io module.
ericsnowcurrently Jan 25, 2022
aa641a3
Get rid of _Py_IDENTIFIER() in _threadmodule.c.
ericsnowcurrently Jan 26, 2022
dda08ad
Get rid of _Py_IDENTIFIER() in _abc.c.
ericsnowcurrently Jan 25, 2022
4d75c6b
Get rid of _Py_IDENTIFIER() in faulthandler.c.
ericsnowcurrently Jan 26, 2022
9d36c02
Get rid of _Py_IDENTIFIER() in _collectionsmodule.c.
ericsnowcurrently Jan 26, 2022
1f42649
Get rid of _Py_IDENTIFIER() in itertoolsmodule.c.
ericsnowcurrently Jan 26, 2022
c2f1cfb
Get rid of _Py_IDENTIFIER() in posixmodule.c.
ericsnowcurrently Jan 26, 2022
3ed68a7
Get rid of _Py_IDENTIFIER() in _operator.c.
ericsnowcurrently Jan 26, 2022
d86ceb1
Get rid of _Py_IDENTIFIER() in timemodule.c.
ericsnowcurrently Jan 26, 2022
c0c6777
Get rid of _Py_IDENTIFIER() in main.c.
ericsnowcurrently Jan 26, 2022
c9f4563
Get rid of _Py_static_string() in typeobject.c.
ericsnowcurrently Jan 26, 2022
0796475
Get rid of _Py_static_string() in ast_opt.c.
ericsnowcurrently Jan 26, 2022
3e5fac3
Get rid of _Py_static_string() in compile.c.
ericsnowcurrently Jan 26, 2022
77d423e
Disallow _Py_IDENTIFIER in core.
ericsnowcurrently Jan 26, 2022
308fb9e
Leave all the functions.
ericsnowcurrently Jan 26, 2022
2136577
Auto-generate the global strings.
ericsnowcurrently Jan 27, 2022
b14d0a8
Drop _Py_unicode_state.empty_string.
ericsnowcurrently Jan 27, 2022
61abdab
Do not include the latin-1 characters (for now).
ericsnowcurrently Jan 27, 2022
db9de0a
Use a better prefix for the field names.
ericsnowcurrently Jan 27, 2022
1074b77
Clean up the declarations a little.
ericsnowcurrently Jan 27, 2022
d4f58fd
Add _Py_IS_POINTER_FROM_VAR().
ericsnowcurrently Jan 27, 2022
09c2a51
Consider all global strings as singletons.
ericsnowcurrently Jan 27, 2022
983dda0
_Py_IS_POINTER_FROM_VAR() -> _PyObject_IsSingleton().
ericsnowcurrently Jan 27, 2022
3fbc75f
Add _PyUnicode_InternSafe().
ericsnowcurrently Jan 27, 2022
bd8b74c
Intern the global strings when used.
ericsnowcurrently Jan 27, 2022
b305ded
Limit when we check the global strings for consistency.
ericsnowcurrently Jan 27, 2022
3da2d65
Drop _PyUnicode_InternSafe().
ericsnowcurrently Jan 27, 2022
3673bbd
Add a --check option to generate_global_objects.py.
ericsnowcurrently Jan 28, 2022
9a419a7
Drop unused global strings.
ericsnowcurrently Jan 28, 2022
c6a292d
Check the global objects in CI.
ericsnowcurrently Jan 28, 2022
ffbd515
Fix _PyObject_CallMethod() for size_t.
ericsnowcurrently Jan 31, 2022
878059b
Dump the exception if normal printing fails.
ericsnowcurrently Feb 1, 2022
780e61f
Regenerate test_frozenmain.h.
ericsnowcurrently Feb 1, 2022
876aa77
Add some identifiers that are only used in non-debug builds.
ericsnowcurrently Feb 1, 2022
f0bd78d
Drop _PyObject_CallMethod_SizeT().
ericsnowcurrently Feb 1, 2022
d575057
Merge branch 'main' into global-strings
ericsnowcurrently Feb 1, 2022
db92384
Cast to void* before casting to uint64_t.
ericsnowcurrently Feb 1, 2022
955d26a
Explicitly mark each module that still uses _Py_IDENTIFIER().
ericsnowcurrently Feb 1, 2022
3c66484
_testembed still uses _Py_IDENTIFIER().
ericsnowcurrently Feb 1, 2022
05fd034
Fix name collisions from Windows.
ericsnowcurrently Feb 1, 2022
78482cf
_Py_GET_GLOBAL_IDENTIFIER() -> _Py_ID().
ericsnowcurrently Feb 4, 2022
72316f4
_Py_GET_GLOBAL_STRING() -> _Py_STR().
ericsnowcurrently Feb 4, 2022
75b6a83
Use a simpler field prefix than spamspamspam_.
ericsnowcurrently Feb 4, 2022
7b670c1
ASCII_STR() -> STRUCT_FOR_ASCII_STR().
ericsnowcurrently Feb 4, 2022
144ed0f
LITERAL() -> STRUCT_FOR_STR().
ericsnowcurrently Feb 4, 2022
f3aa225
ID() -> STRUCT_FOR_ID().
ericsnowcurrently Feb 4, 2022
050b34b
Use _PyObject_IMMORTAL_INIT().
ericsnowcurrently Feb 4, 2022
dbec341
Drop _Py_STRING_LENGTH().
ericsnowcurrently Feb 4, 2022
7a075ac
Inline uses of _Py_ID().
ericsnowcurrently Feb 4, 2022
6fdba80
Inline uses of _Py_STR().
ericsnowcurrently Feb 4, 2022
4e00c87
Drop _PyObject_IsSingleton(). (C99 undefined makes it too tricky.)
ericsnowcurrently Feb 4, 2022
fbe8f4a
Drop address-taking from _PY_ID() and _Py_STR().
ericsnowcurrently Feb 4, 2022
f2d9b81
Add a compatibility shim for _PyObject_LookupSpecialId().
ericsnowcurrently Feb 4, 2022
7781025
Merge branch 'main' into global-strings
ericsnowcurrently Feb 4, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,8 @@ jobs:
run: make smelly
- name: Check limited ABI symbols
run: make check-limited-abi
- name: Check global objects
run: make check-global-objects

build_win32:
name: 'Windows (x86)'
Expand Down
4 changes: 4 additions & 0 deletions Include/cpython/abstract.h
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,10 @@ PyObject_CallMethodOneArg(PyObject *self, PyObject *name, PyObject *arg)
2 | PY_VECTORCALL_ARGUMENTS_OFFSET, NULL);
}

PyAPI_FUNC(PyObject *) _PyObject_CallMethod(PyObject *obj,
PyObject *name,
const char *format, ...);

/* Like PyObject_CallMethod(), but expect a _Py_Identifier*
as the method name. */
PyAPI_FUNC(PyObject *) _PyObject_CallMethodId(PyObject *obj,
Expand Down
1 change: 1 addition & 0 deletions Include/cpython/ceval.h
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ PyAPI_FUNC(int) _PyEval_SetAsyncGenFinalizer(PyObject *);
PyAPI_FUNC(PyObject *) _PyEval_GetAsyncGenFinalizer(void);

/* Helper to look up a builtin object */
PyAPI_FUNC(PyObject *) _PyEval_GetBuiltin(PyObject *);
PyAPI_FUNC(PyObject *) _PyEval_GetBuiltinId(_Py_Identifier *);
/* Look at the current frame's (if any) code's co_flags, and turn on
the corresponding compiler flags in cf->cf_flags. Return 1 if any
Expand Down
1 change: 1 addition & 0 deletions Include/cpython/dictobject.h
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ typedef struct {

PyAPI_FUNC(PyObject *) _PyDict_GetItem_KnownHash(PyObject *mp, PyObject *key,
Py_hash_t hash);
PyAPI_FUNC(PyObject *) _PyDict_GetItemWithError(PyObject *dp, PyObject *key);
PyAPI_FUNC(PyObject *) _PyDict_GetItemIdWithError(PyObject *dp,
struct _Py_Identifier *key);
PyAPI_FUNC(PyObject *) _PyDict_GetItemStringWithError(PyObject *, const char *);
Expand Down
13 changes: 12 additions & 1 deletion Include/cpython/object.h
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,16 @@ typedef struct _Py_Identifier {
Py_ssize_t index;
} _Py_Identifier;

#if defined(NEEDS_PY_IDENTIFIER) || !defined(Py_BUILD_CORE)
// For now we are keeping _Py_IDENTIFIER for continued use
// in non-builtin extensions (and naughty PyPI modules).

#define _Py_static_string_init(value) { .string = value, .index = -1 }
#define _Py_static_string(varname, value) static _Py_Identifier varname = _Py_static_string_init(value)
#define _Py_IDENTIFIER(varname) _Py_static_string(PyId_##varname, #varname)

#endif /* NEEDS_PY_IDENTIFIER */

typedef int (*getbufferproc)(PyObject *, Py_buffer *, int);
typedef void (*releasebufferproc)(PyObject *, Py_buffer *);

Expand Down Expand Up @@ -249,7 +255,12 @@ typedef struct _heaptypeobject {
PyAPI_FUNC(const char *) _PyType_Name(PyTypeObject *);
PyAPI_FUNC(PyObject *) _PyType_Lookup(PyTypeObject *, PyObject *);
PyAPI_FUNC(PyObject *) _PyType_LookupId(PyTypeObject *, _Py_Identifier *);
PyAPI_FUNC(PyObject *) _PyObject_LookupSpecial(PyObject *, _Py_Identifier *);
PyAPI_FUNC(PyObject *) _PyObject_LookupSpecialId(PyObject *, _Py_Identifier *);
#ifndef Py_BUILD_CORE
// Backward compatibility for 3rd-party extensions
// that may be using the old name.
#define _PyObject_LookupSpecial _PyObject_LookupSpecialId
#endif
PyAPI_FUNC(PyTypeObject *) _PyType_CalculateMetaclass(PyTypeObject *, PyObject *);
PyAPI_FUNC(PyObject *) _PyType_GetDocFromInternalDoc(const char *, const char *);
PyAPI_FUNC(PyObject *) _PyType_GetTextSignatureFromInternalDoc(const char *, const char *);
Expand Down
2 changes: 2 additions & 0 deletions Include/cpython/sysmodule.h
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
# error "this header file must not be included directly"
#endif

PyAPI_FUNC(PyObject *) _PySys_GetAttr(PyThreadState *tstate,
PyObject *name);
PyAPI_FUNC(PyObject *) _PySys_GetObjectId(_Py_Identifier *key);
PyAPI_FUNC(int) _PySys_SetObjectId(_Py_Identifier *key, PyObject *);

Expand Down
3 changes: 3 additions & 0 deletions Include/internal/pycore_call.h
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ PyAPI_FUNC(PyObject *) _PyObject_Call(
PyObject *args,
PyObject *kwargs);

extern PyObject * _PyObject_CallMethodFormat(
PyThreadState *tstate, PyObject *callable, const char *format, ...);


// Static inline variant of public PyVectorcall_Function().
static inline vectorcallfunc
Expand Down
4 changes: 4 additions & 0 deletions Include/internal/pycore_global_objects.h
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ extern "C" {
# error "this header requires Py_BUILD_CORE define"
#endif

#include "pycore_global_strings.h" // struct _Py_global_strings


// These would be in pycore_long.h if it weren't for an include cycle.
#define _PY_NSMALLPOSINTS 257
Expand Down Expand Up @@ -36,6 +38,8 @@ struct _Py_global_objects {
PyBytesObject ob;
char eos;
} bytes_characters[256];

struct _Py_global_strings strings;
} singletons;
};

Expand Down
Loading