PEP 670: Convert _PyObject_SIZE() and _PyObject_VAR_SIZE() macros to functions #99845
Description
The _PyObject_SIZE() and _PyObject_VAR_SIZE() macros should be converted to functions: see PEP 670 for the rationale.
My problem is that I don't know if the return type should be signed (Py_ssize_t) or unsigned (size_t).
CPython usage of _PyObject_SIZE():
- Signed: 18. Implementation of
__sizeof__()methods. - Unsigned: 0
- Implicit cast to unsigned: 2. Calls
PyObject_Malloc(_PyObject_SIZE(tp))andgc_alloc(_PyObject_SIZE(tp), presize)where the first argument type issize_t.
CPython usage of _PyObject_VAR_SIZE():
- Signed: 5
- Unsigned: 1
- Implicit cast to unsigned: 1. Call
_PyDebugAllocatorStats(..., _PyObject_VAR_SIZE(&PyTuple_Type, len))where the argument type issize_t.
To get a container length, the C API uses signed type (Py_ssize_t): PyList_Size(), PyDict_Size(), Py_SIZE(), etc.
To allocate memory, the C API prefers unsigned type (size_t): PyMem_Malloc(), PyObject_Realloc(), etc.
Python allocator functions reject size greater than PY_SSIZE_T_MAX:
void *
PyMem_RawMalloc(size_t size)
{
/*
* Limit ourselves to PY_SSIZE_T_MAX bytes to prevent security holes.
* Most python internals blindly use a signed Py_ssize_t to track
* things without checking for overflows or negatives.
* As size_t is unsigned, checking for size < 0 is not required.
*/
if (size > (size_t)PY_SSIZE_T_MAX)
return NULL;
return _PyMem_Raw.malloc(_PyMem_Raw.ctx, size);
}
Some "sizeof" functions freely mix signed and unsigned types. Example:
static PyObject *
deque_sizeof(dequeobject *deque, void *unused)
{
Py_ssize_t res;
Py_ssize_t blocks;
res = _PyObject_SIZE(Py_TYPE(deque));
blocks = (size_t)(deque->leftindex + Py_SIZE(deque) + BLOCKLEN - 1) / BLOCKLEN;
assert(deque->leftindex + Py_SIZE(deque) - 1 ==
(blocks - 1) * BLOCKLEN + deque->rightindex);
res += blocks * sizeof(block);
return PyLong_FromSsize_t(res);
}
blocks and sizeof(block) are unsigned, but res is signed.
Another problem is that _PyObject_VAR_SIZE() has an undefined behavior on integer overflow. Maybe it should return SIZE_MAX on oveflow, to make sure that Python allocator function fail (return NULL)?
Linked PRs
- gh-99845: Use size_t type in __sizeof__() methods #99846
- gh-99845: Clean up _PyObject_VAR_SIZE() usage #99847
- gh-99845: Change _PyDict_KeysSize() return type to size_t #99848
- gh-99845: PEP 670: Convert PyObject macros to functions #99850
- gh-99845: _PySys_GetSizeOf() uses size_t #99903
- [3.11] gh-99845: _PyObject_DictPointer(): fix dictoffset cast #99922
- [3.10] gh-99845: _PyObject_DictPointer(): fix dictoffset cast (GH-99922) #99924
Activity
vstinner commentedon Nov 28, 2022
Aha,
_PySys_GetSizeOf()returnssize_t, butbytesio_sizeof()stores its result in aPy_ssize_tvariable.vstinner commentedon Nov 28, 2022
@corona10 @erlend-aasland @serhiy-storchaka: Are you ok to use an unsigned type for _PyObject_SIZE() and _PyObject_VAR_SIZE() result? See PR #99846 and PR #99847 for concrete changes to prepare converting these macros to functions.
vstinner commentedon Nov 28, 2022
In PyPI top 5000 projects, _PyObject_SIZE() and _PyObject_VAR_SIZE() are used with signed and unsigned types.
numpy uses unsigned:
Cython gdb plugin treats _PyObject_VAR_SIZE() macro result as unsigned:
JPype uses signed and unsigned types:
frozendict uses signed type (copy of CPython Objects/dictobject.c which also uses signed types currently):
pickle uses signed:
recordclass uses signed:
Nuitka uses unsigned:
pyobjc-core casts implicitly
_PyObject_SIZE()result to unsigned (size_t):vstinner commentedon Nov 28, 2022
I created PR #99848 for Objects/dictobject.c.
vstinner commentedon Nov 28, 2022
The final change should be PR #99850 which convert the two macros to functions.
gh-99845: Change _PyDict_KeysSize() return type to size_t (#99848)
gh-99845: Clean up _PyObject_VAR_SIZE() usage (#99847)
vstinner commentedon Nov 30, 2022
I was confused by the "SIZE" name:
Py_SIZE()returns the number of elements in a container like a list or a dict: it has no unit._PyObject_SIZE()is different: it returns an object size in bytes: it has an unit.IMO for bytes, it's better to use unsigned numbers. In the Python API, it's not an issue, it's an object. For example,
_PySys_GetSizeOf()and_PyType_PreHeaderSize()return types are unsigned (size_t).gh-99845: Use size_t type in __sizeof__() methods (#99846)
gh-99845: PEP 670: Convert PyObject macros to functions (#99850)
gh-99845: _PyObject_DictPointer(): fix dictoffset cast (#99922)
pythongh-99845: _PyObject_DictPointer(): fix dictoffset cast (pythonG…
gh-99845: _PyObject_DictPointer(): fix dictoffset cast (GH-99922)
Merge branch 'main' into fixinspect
vstinner commentedon Dec 5, 2022
Fixed by 131801d