tracemalloc raw-domain allocator hook violates PYMEM_DOMAIN_RAW contract

Bug description:

tracemalloc's raw-domain allocator hook requires an attached thread
state (PyGILState_Ensure(): tracemalloc_alloc / tracemalloc_realloc) , which violates the documented PYMEM_DOMAIN_RAW contract:

MRE (Minimal Reproducible Example):

Pure Python MRE

import ctypes
import sys
import threading
import tracemalloc

if not sys.platform.startswith("linux"):
    msg = "This MRE requires Linux"
    raise RuntimeError(msg)

pythonapi = ctypes.pythonapi
pythonapi.PyMem_RawMalloc.restype = ctypes.c_void_p
pythonapi.PyMem_RawMalloc.argtypes = [ctypes.c_size_t]
pythonapi.PyMem_RawFree.argtypes = [ctypes.c_void_p]


# this is only needed to creat a native thread
# without a `PyThreadState` attached to it
libc_cdll = ctypes.CDLL(None)
libc_cdll.pthread_create.restype = ctypes.c_int
libc_cdll.usleep.argtypes = [ctypes.c_uint]

# unlike CDLL, PyDLL does NOT release the GIL around the call
# this is intentional, so that the main thread holds the GIL for
# the full 5 seconds creating conditions under which the raw-allocator
# hook cannot acquire the GIL, even if it were to request it
blocking_libc = ctypes.PyDLL(None)
blocking_libc.sleep.argtypes = [ctypes.c_uint]

THREAD_FN = ctypes.CFUNCTYPE(ctypes.c_void_p, ctypes.c_void_p)
done = threading.Event()


@THREAD_FN
def native_thread_entry(_arg: ctypes.c_void_p) -> ctypes.c_void_p:
    # this thread was created using `pthread_create()`, not
    # `threading.Thread` it does NOT have an associated `PyThreadState`
    libc_cdll.usleep(500_000)

    # according to the `PyMem_SetAllocator` contract for `PYMEM_DOMAIN_RAW`
    # ("a thread state is not attached when the allocator is called"),
    # the call below MUST execute without blocking on the GIL
    ptr = pythonapi.PyMem_RawMalloc(64)
    pythonapi.PyMem_RawFree(ptr)

    # if the allocator internally calls `PyGILState_Ensure()`,
    # then this code will never reach this point while the GIL is busy
    done.set()


def main() -> None:
    # tracemalloc.start() sets a hook on `PYMEM_DOMAIN_RAW`
    tracemalloc.start()
    thread_id = ctypes.c_ulong()
    libc_cdll.pthread_create(ctypes.byref(thread_id), None, native_thread_entry, None)
    # The main thread holds the GIL continuously for 5 seconds (`PyDLL.sleep` does not
    # release the GIL). If the raw-allocator hook requires the GIL, a deadlock occurs
    blocking_libc.sleep(5)

    # deadlock
    print("DEADLOCK" if not done.is_set() else "OK")


main()

mre.c:

#define PY_SSIZE_T_CLEAN
#include <Python.h>

#include <pthread.h>
#include <stdatomic.h>
#include <stdbool.h>
#include <unistd.h>

static atomic_bool done = false;

static void *native_thread_entry(void *arg)
{
    (void)arg;

    usleep(500000);

    void *ptr = PyMem_RawMalloc(64);
    PyMem_RawFree(ptr);

    atomic_store(&done, true);
    return NULL;
}

static PyObject *mre_run(PyObject *self, PyObject *args)
{
    (void)self;
    (void)args;

    atomic_store(&done, false);

    pthread_t thread;
    int err = pthread_create(&thread, NULL, native_thread_entry, NULL);
    if (err != 0) {
        return PyErr_SetFromErrno(PyExc_OSError);
    }

    sleep(5);

    printf("%s\n", atomic_load(&done) ? "OK" : "DEADLOCK");

    pthread_detach(thread);

    Py_RETURN_NONE;
}

static PyMethodDef methods[] = {
    {
        "run",
        mre_run,
        METH_NOARGS,
        "Run the raw allocator MRE.",
    },
    {NULL, NULL, 0, NULL},
};

static struct PyModuleDef module = {
    PyModuleDef_HEAD_INIT,
    "mre",
    NULL,
    -1,
    methods,
};

PyMODINIT_FUNC PyInit_mre(void)
{
    return PyModule_Create(&module);
}

setup.py:

from setuptools import Extension, setup

setup(
    name="mre",
    ext_modules=[
        Extension(
            "mre",
            sources=["mre.c"],
            extra_compile_args=["-std=c11"],
        ),
    ],
)

test.py:

import tracemalloc

import mre

tracemalloc.start()
mre.run()

CPython versions tested on:

3.10, 3.11, 3.12, 3.13, 3.14, 3.15

Operating systems tested on:

Linux

Linked PRs

The definition of tracemalloc's raw allocator is here:

cpython/Python/tracemalloc.c

Lines 688 to 692 in 37e98da

    
           static void* 
        
           tracemalloc_raw_malloc(void *ctx, size_t size) 
        
           { 
        
               return tracemalloc_alloc(1, 0, ctx, 1, size); 
        
           }

In tracemalloc_alloc, this sets need_gil to 1, which will in turn call PyGILState_Ensure to get an attached thread state:

cpython/Python/tracemalloc.c

Lines 499 to 501 in 37e98da

    
           static void* 
        
           tracemalloc_alloc(int need_gil, int use_calloc, 
        
                             void *ctx, size_t nelem, size_t elsize)

So, the problem isn't that tracemalloc's raw allocator needs the caller to hold the GIL (if it did, I suspect you'd see a crash at runtime when calling PyMem_RawMalloc). What's the expected behavior of the MRE? I see it prints "DEADLOCK", but the program finishes.

What's the expected behavior of the MRE?

DEADLOCK

The PYMEM_DOMAIN_RAW contract is still violated: the documentation explicitly requires that "a thread state is not attached when the allocator is called", and calling PyGILState_Ensure() to obtain that state is a hidden, undocumented block on the GIL inside a call that is supposed to be safe without it. In the MRE, this manifests not as a "permanent deadlock" but as:

An unpredictable delay in PyMem_RawMalloc() / PyMem_RawFree(), lasting up to however long another thread holds the GIL (up to 5 seconds in the example), which by itself violates the contract that the allocator "can operate without an attached thread state," i.e., without blocking on the GIL at all.
A potential genuine deadlock in more complex scenarios, for example, if the thread holding the GIL is itself waiting on the result of an operation that depends on PyMem_RawMalloc() completing in the native thread (a classic lock-ordering inversion deadlock), or if the GIL is held indefinitely (not just 5 seconds, but say, until a long blocking operation in a C extension finishes).

The contract doesn't specify that a thread state can't be attached inside the callback. It only specifies that the callback isn't called with one.

A potential genuine deadlock in more complex scenarios, for example, if the thread holding the GIL is itself waiting on the result of an operation that depends on PyMem_RawMalloc() completing in the native thread (a classic lock-ordering inversion deadlock), or if the GIL is held indefinitely (not just 5 seconds, but say, until a long blocking operation in a C extension finishes).

A thread shouldn't be holding the GIL at all if it's waiting on some operation (except in some very special circumstances).

It seems more accurate to say not that the contract is "violated," but that the implementation makes optional a blocking call for something the documentation explicitly names as the reason the raw domain exists in the first place - an allocator that "can operate without an attached thread state". In other words, the hook does technically work (it doesn't crash, and there's no UB), but it stops being non-blocking / GIL-independent in cases where the GIL is busy - this is a regression in expected behavior/performance, not undefined behavior or a literal violation of the letter of the spec.

Yes, but looking at the code, changing tracemalloc to not need the GIL would likely be a significant change. I guess we could try swapping out PyGILState_Ensure calls for PyMutex calls to a static lock, but I don't know what implications that would have, or if that's even possible (if tracemalloc is calling into something that needs the GIL, such as incrementing an object's reference count, then there's definitely no easy way to fix this).

changing tracemalloc to not need the GIL would likely be a significant change

FWIW I expect tracemalloc will need to be re-architected for free-threaded and subinterpreters. There are thread-safety issues (e.g. #126315) and issues with sub-interpreters (e.g. #134604). I expect to fix both tracemalloc will need a fair bit of work to be functional in those scenarios.

Honestly, I think it would be better to deprecate and remove tracemalloc. It's slow, buggy, and not very useful. Memray is better in basically every way.

Yeah. Even with a re-architecture, unless we are okay with a lot more <unknown> frames, an attached thread state is needed to get the current thread's Python stack frames. I suppose we could queue acquiring the stack trace until the next time the interpreter re-attaches? But I'd need to think more if that can be done safely.

I do think an allocation in another thread causing a deadlock with tracemalloc is pretty bad. We should at very minimum document that this may happen and ideally conservatively collect Python frame information for PYMEM_DOMAIN_RAW (either not require a threadstate or defer collecting the trace).

I'm still not sure I understand the scenario where this will deadlock. Why is a thread waiting on an allocation without releasing the GIL?

PyMem_RawMalloc isn't documented as a possibly-blocking API. It's also not blocking in-practice except when tracemalloc is enabled. I would only think about discipline around releasing the GIL if I'm calling a possibly-blocking API.

Usually when an API is documented as being usable without an attached thread state you can also use it with a thread state. In this case that's not true if tracemalloc is enabled.

Usually when an API is documented as being usable without an attached thread state you can also use it with a thread state. In this case that's not true if tracemalloc is enabled.

I think I'm missing some context. Even if tracemalloc is enabled, PyMem_RawMalloc doesn't deadlock when called with an attached thread state.

Would that mean that PyMem_Raw* allocations aren't traced by tracemalloc?

No, they would be traced and be reflected in total traced memory, you just won't get traceback for them if raw allocator is called without an active thread state.

So we're just making tracemalloc even less useful than it already is? I guess it's not a huge deal, but I think this is a bad long-term solution. We shouldn't strip away features from tracemalloc when they become inconvenient for someone; let's follow process and deprecate and remove tracemalloc entirely.

But anyway, it would be ideal if PyO3 didn't rely on the allocator being non-blocking. Tools like Memray are also completely capable of acquiring the GIL in their raw allocator hook. We can't just pretend that this has always been a part of the contract for PYMEM_DOMAIN_RAW.

In case it isn't clear, this latest round of comments comes from gh-155725, where Kumar tried to use RawMalloc in the ssl module.

pythongh-155725: Capture tracemalloc tracebacks without a thread stat…

pythongh-155725: Revert Doc/c-api/memory.rst changes

So we're just making tracemalloc even less useful than it already is? I guess it's not a huge deal, but I think this is a bad long-term solution. We shouldn't strip away features from tracemalloc when they become inconvenient for someone; let's follow process and deprecate and remove tracemalloc entirely.

I've added tracebacks for raw allocations back, an active thread state was required earlier because tracemalloc stores file names as python objects, now it is stored as a raw string so thread state is not needed. With that, there is no behavior change on my latest version of PR.

FWIW I don't care too much about memory tracking using tracemalloc, but it is blocker for using raw allocator in ssl module so I want to fix it rather than going through the route of deprecation and wait 2 releases.

pythongh-155725: Hold a finalization guard while capturing a detached…

Note that the "pure Python minimal reproducer" is invalid. ctypes CFUNCTYPE callbacks call PyGILState_Ensure() themselves.

gh-155725: Remove `PyGILState_Ensure` usage from `tracemalloc` (#156409)

[3.15] gh-155725: Remove `PyGILState_Ensure` usage from `tracemalloc` (…

I've fixed this in main and it is backported to 3.15.

pythongh-155725: Remove `PyGILState_Ensure` usage from `tracemalloc` (p…

Navigation Menu

Repository navigation

`tracemalloc` raw-domain allocator hook violates `PYMEM_DOMAIN_RAW` contract #155725

Related:

Bug description:

MRE (Minimal Reproducible Example):

CPython versions tested on:

Operating systems tested on:

Linked PRs

7 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	static void*
	tracemalloc_raw_malloc(void *ctx, size_t size)
	{
	return tracemalloc_alloc(1, 0, ctx, 1, size);
	}

	static void*
	tracemalloc_alloc(int need_gil, int use_calloc,
	void *ctx, size_t nelem, size_t elsize)

tracemalloc raw-domain allocator hook violates PYMEM_DOMAIN_RAW contract #155725

Description

Related:

Bug description:

MRE (Minimal Reproducible Example):

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

ZeroIntensity commented on Aug 13, 2026

chirizxc commented on Aug 13, 2026

chirizxc commented on Aug 13, 2026

ZeroIntensity commented on Aug 13, 2026

chirizxc commented on Aug 13, 2026

ZeroIntensity commented on Aug 13, 2026

emmatyping commented on Aug 13, 2026

ZeroIntensity commented on Aug 14, 2026

emmatyping commented on Aug 14, 2026

emmatyping commented on Aug 14, 2026

ZeroIntensity commented on Aug 14, 2026

ngoldbaum commented on Aug 14, 2026

ZeroIntensity commented on Aug 15, 2026

7 remaining items

kumaraditya303 commented on Aug 26, 2026

ZeroIntensity commented on Aug 26, 2026

ngoldbaum commented on Aug 26, 2026

kumaraditya303 commented on Aug 26, 2026

gpshead commented on Aug 30, 2026

kumaraditya303 commented on Aug 31, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

`tracemalloc` raw-domain allocator hook violates `PYMEM_DOMAIN_RAW` contract #155725