Add helpers to populate modules in C

[tiran]

BPO	42376
Nosy	@tiran, @serhiy-storchaka, @shihai1991, @erlend-aasland
PRs	gh-86542: New C-APIs to simplify module attribute declaration #23286

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tiran'
closed_at = None
created_at = <Date 2020-11-16.19:27:21.162>
labels = ['expert-C-API', 'type-feature', '3.10']
title = 'Add helpers to populate modules in C'
updated_at = <Date 2021-11-16.17:23:51.530>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2021-11-16.17:23:51.530>
actor = 'erlendaasland'
assignee = 'christian.heimes'
closed = False
closed_date = None
closer = None
components = ['C API']
creation = <Date 2020-11-16.19:27:21.162>
creator = 'christian.heimes'
dependencies = []
files = []
hgrepos = []
issue_num = 42376
keywords = ['patch']
message_count = 1.0
messages = ['381144']
nosy_count = 4.0
nosy_names = ['christian.heimes', 'serhiy.storchaka', 'shihai1991', 'erlendaasland']
pr_nums = ['23286']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue42376'
versions = ['Python 3.10']

[tiran]

It's currently inconvenient and sometimes error-prone to initialize module members in C. Even stdlib modules are sometimes missing error checks or have reference counting issues in error path. I propose add three helpers to deal with common cases:

1. declaration of simple attributes (int, str, bool, float) in an array of structs
2. creation and addition of a new type from a type spec
3. creation and addition a new exception object

(1) Simple attribute declaration uses a NULL terminated array of PyModuleConst_Def. The internal definition of the type can be hidden by macros. Example:

static PyModuleConst_Def example_constants[] = {
PyModuleConst_None("none_value"),
PyModuleConst_Long("integer", 42),
PyModuleConst_Bool("false_value", 0),
PyModuleConst_Bool("true_value", 1),
PyModuleConst_String("somestring", "Hello"),
PyModuleConst_LongMacro(EXAMPLE_INT),
PyModuleConst_StringMacro(EXAMPLE_STRING),
{NULL},
}

A new function "int PyModule_AddConstants(PyObject *module, PyModuleConst_Def *def)" populates the module object with attributes.

(2) Type initialization

The function "PyTypeObject * PyModule_AddNewTypeFromSpec(PyObject *module, PyType_Spec *spec, PyObject *base)" is PyType_FromModuleAndSpec() + PyModule_AddType(). Additionally to PyType_FromModuleAndSpec() it also handles "base" argument with single type instance. The extra feature avoids "PyTuple_Pack(1, baseclass)" in the caller. The function adds a strong reference to the module object and returns a borrowed reference. The borrowed reference is designed module state assignment.

(3) exception creation

The function "PyObject* PyModule_AddNewException(PyObject *module, const char *name, const char *doc, PyObject *base, PyObject *dict)" creates a new exception, calculates the attribute name for the qualified exception name, and adds the exception to the module. It's PyErr_NewExceptionWithDoc() + PyModule_AddObjectRef().

Note about (1): I first proposed to add PyModuleDef.m_constants, however we cannot extend the struct easily becaues it conflicts with stable ABI. Petr mentioned that he has plans to deal with the issue. Also see discussion https://discuss.python.org/t/define-module-constants-in-pymoduledef/5749

[encukou]

With 2 years of extra experience, I see issues in this proposal -- see capi-workgroup/decisions#6 (comment)

If anyone would like to move any part of this proposal forward, feel free to bounce ideas off me. I recommend tackling adding types separately from the module constants.

[rruuaanng]

With 2 years of extra experience, I see issues in this proposal -- see capi-workgroup/decisions#6 (comment)

If anyone would like to move any part of this proposal forward, feel free to bounce ideas off me. I recommend tackling adding types separately from the module constants.

Can you explain your viewpoint? That is, separate types from module constants. When I was working on #127435, I had to review the content of the proposal. In other words, I want to take over part of the proposal.

[encukou]

I suggest to come up with a different mechanism for constants (where creating the Python object is rather trivial and unchanging) and for types (every few releases we get a new PyType_From* type-creation function). I don't think they should be bundled together in one issue like here.