Skip to content

Segfault in _Unpickler_MemoPut (_pickle.c) from Unpickler.memo assignment with free-threading #150505

New issue
New issue
Open
#150550
Open
Segfault in _Unpickler_MemoPut (_pickle.c) from Unpickler.memo assignment with free-threading#150505
#150550
Labels
interpreter-core(Objects, Python, Grammar, and Parser dirs)topic-free-threadingtype-bugAn unexpected behavior, bug, or error

Description

@Naserume
Naserume
opened on May 27, 2026

Bug report

Bug description:

Unpickler_set_memo modifies self->memo and self->memo_size without any critical section.

When two threads set .memo on the same Unpickler instance concurrently, one can set self->memo = NULL while another is still indexing into it via _Unpickler_MemoPut, leading to a NULL-pointer dereference.

Reproducer:

import io
import pickle
from threading import Thread

shared_unpickler = pickle.Unpickler(io.BytesIO(b''))

def assign():
    for _ in range(10000):
        shared_unpickler.memo = {j: j for j in range(100)}

if __name__ == "__main__":
    threads = [Thread(target=assign) for _ in range(8)]
    for t in threads: t.start()
    for t in threads: t.join()

TSAN report:

WARNING: ThreadSanitizer: data race (pid=214122)
  Read of size 8 at 0x7dca529b9130 by thread T2:
    #0 _Unpickler_MemoPut /home/cpython/./Modules/_pickle.c:1631:22 (_pickle.cpython-316t-x86_64-linux-gnu.so+0x2122f) 
    #1 Unpickler_set_memo /home/cpython/./Modules/_pickle.c:7804:17 (_pickle.cpython-316t-x86_64-linux-gnu.so+0x26b77) 
    #2 getset_set /home/cpython/Objects/descrobject.c:250:16 (python3.16t+0x22d8fb) 
    #3 _PyObject_GenericSetAttrWithDict /home/cpython/Objects/object.c:2049:19 (python3.16t+0x2d844b) 
    #4 PyObject_GenericSetAttr /home/cpython/Objects/object.c:2120:12 (python3.16t+0x2d8d4b) 
    #5 Unpickler_setattr /home/cpython/./Modules/_pickle.c:7857:12 (_pickle.cpython-316t-x86_64-linux-gnu.so+0x24eca) 
    #6 PyObject_SetAttr /home/cpython/Objects/object.c:1533:15 (python3.16t+0x2d480f) 
    #7 _PyEval_EvalFrameDefault /home/cpython/Python/generated_cases.c.h:11536:27 (python3.16t+0x45231a) 
    #8 _PyEval_EvalFrame /home/cpython/./Include/internal/pycore_ceval.h:122:16 (python3.16t+0x436463) 
    #9 _PyEval_Vector /home/cpython/Python/ceval.c:2134:12 (python3.16t+0x436463)
    #10 _PyFunction_Vectorcall /home/cpython/Objects/call.c (python3.16t+0x215013) 
    #11 _PyObject_VectorcallTstate /home/cpython/./Include/internal/pycore_call.h:144:11 (python3.16t+0x216b11) 
    #12 _PyObject_VectorcallPrepend /home/cpython/Objects/call.c:855:20 (python3.16t+0x216b11)
    #13 method_vectorcall /home/cpython/Objects/classobject.c:55:12 (python3.16t+0x21a132) 
    #14 _PyObject_VectorcallTstate /home/cpython/./Include/internal/pycore_call.h:144:11 (python3.16t+0x48ec15) 
    #15 context_run /home/cpython/Python/context.c:728:29 (python3.16t+0x48ec15)
    #16 method_vectorcall_FASTCALL_KEYWORDS /home/cpython/Objects/descrobject.c:421:24 (python3.16t+0x22e84c) 
    #17 _PyObject_VectorcallTstate /home/cpython/./Include/internal/pycore_call.h:144:11 (python3.16t+0x2149a8) 
    #18 PyObject_Vectorcall /home/cpython/Objects/call.c:327:12 (python3.16t+0x2149a8)
    #19 _Py_VectorCallInstrumentation_StackRefSteal /home/cpython/Python/ceval.c:766:11 (python3.16t+0x4370c8) 
    #20 _PyEval_EvalFrameDefault /home/cpython/Python/generated_cases.c.h:1846:35 (python3.16t+0x43d022) 
    #21 _PyEval_EvalFrame /home/cpython/./Include/internal/pycore_ceval.h:122:16 (python3.16t+0x436463) 
    #22 _PyEval_Vector /home/cpython/Python/ceval.c:2134:12 (python3.16t+0x436463)
    #23 _PyFunction_Vectorcall /home/cpython/Objects/call.c (python3.16t+0x215013) 
    #24 _PyObject_VectorcallTstate /home/cpython/./Include/internal/pycore_call.h:144:11 (python3.16t+0x216b11) 
    #25 _PyObject_VectorcallPrepend /home/cpython/Objects/call.c:855:20 (python3.16t+0x216b11)
    #26 method_vectorcall /home/cpython/Objects/classobject.c:55:12 (python3.16t+0x21a132) 
    #27 _PyVectorcall_Call /home/cpython/Objects/call.c:273:16 (python3.16t+0x214ca2) 
    #28 _PyObject_Call /home/cpython/Objects/call.c:348:16 (python3.16t+0x214ca2)
    #29 PyObject_Call /home/cpython/Objects/call.c:373:12 (python3.16t+0x214d07) 
    #30 thread_run /home/cpython/./Modules/_threadmodule.c:388:21 (python3.16t+0x61a708) 
    #31 pythread_wrapper /home/cpython/Python/thread_pthread.h:234:5 (python3.16t+0x5473bb) 

  Previous write of size 8 at 0x7dca529b9130 by thread T1:
    #0 Unpickler_set_memo /home/cpython/./Modules/_pickle.c:7816:21 (_pickle.cpython-316t-x86_64-linux-gnu.so+0x26cc4) 
    #1 getset_set /home/cpython/Objects/descrobject.c:250:16 (python3.16t+0x22d8fb) 
    #2 _PyObject_GenericSetAttrWithDict /home/cpython/Objects/object.c:2049:19 (python3.16t+0x2d844b) 
    #3 PyObject_GenericSetAttr /home/cpython/Objects/object.c:2120:12 (python3.16t+0x2d8d4b) 
    #4 Unpickler_setattr /home/cpython/./Modules/_pickle.c:7857:12 (_pickle.cpython-316t-x86_64-linux-gnu.so+0x24eca) 
    #5 PyObject_SetAttr /home/cpython/Objects/object.c:1533:15 (python3.16t+0x2d480f) 
    #6 _PyEval_EvalFrameDefault /home/cpython/Python/generated_cases.c.h:11536:27 (python3.16t+0x45231a) 
    #7 _PyEval_EvalFrame /home/cpython/./Include/internal/pycore_ceval.h:122:16 (python3.16t+0x436463) 
    #8 _PyEval_Vector /home/cpython/Python/ceval.c:2134:12 (python3.16t+0x436463)
    #9 _PyFunction_Vectorcall /home/cpython/Objects/call.c (python3.16t+0x215013) 
    #10 _PyObject_VectorcallTstate /home/cpython/./Include/internal/pycore_call.h:144:11 (python3.16t+0x216b11) 
    #11 _PyObject_VectorcallPrepend /home/cpython/Objects/call.c:855:20 (python3.16t+0x216b11)
    #12 method_vectorcall /home/cpython/Objects/classobject.c:55:12 (python3.16t+0x21a132) 
    #13 _PyObject_VectorcallTstate /home/cpython/./Include/internal/pycore_call.h:144:11 (python3.16t+0x48ec15) 
    #14 context_run /home/cpython/Python/context.c:728:29 (python3.16t+0x48ec15)
    #15 method_vectorcall_FASTCALL_KEYWORDS /home/cpython/Objects/descrobject.c:421:24 (python3.16t+0x22e84c) 
    #16 _PyObject_VectorcallTstate /home/cpython/./Include/internal/pycore_call.h:144:11 (python3.16t+0x2149a8) 
    #17 PyObject_Vectorcall /home/cpython/Objects/call.c:327:12 (python3.16t+0x2149a8)
    #18 _Py_VectorCallInstrumentation_StackRefSteal /home/cpython/Python/ceval.c:766:11 (python3.16t+0x4370c8) 
    #19 _PyEval_EvalFrameDefault /home/cpython/Python/generated_cases.c.h:1846:35 (python3.16t+0x43d022) 
    #20 _PyEval_EvalFrame /home/cpython/./Include/internal/pycore_ceval.h:122:16 (python3.16t+0x436463) 
    #21 _PyEval_Vector /home/cpython/Python/ceval.c:2134:12 (python3.16t+0x436463)
    #22 _PyFunction_Vectorcall /home/cpython/Objects/call.c (python3.16t+0x215013) 
    #23 _PyObject_VectorcallTstate /home/cpython/./Include/internal/pycore_call.h:144:11 (python3.16t+0x216b11) 
    #24 _PyObject_VectorcallPrepend /home/cpython/Objects/call.c:855:20 (python3.16t+0x216b11)
    #25 method_vectorcall /home/cpython/Objects/classobject.c:55:12 (python3.16t+0x21a132) 
    #26 _PyVectorcall_Call /home/cpython/Objects/call.c:273:16 (python3.16t+0x214ca2) 
    #27 _PyObject_Call /home/cpython/Objects/call.c:348:16 (python3.16t+0x214ca2)
    #28 PyObject_Call /home/cpython/Objects/call.c:373:12 (python3.16t+0x214d07) 
    #29 thread_run /home/cpython/./Modules/_threadmodule.c:388:21 (python3.16t+0x61a708) 
    #30 pythread_wrapper /home/cpython/Python/thread_pthread.h:234:5 (python3.16t+0x5473bb) 

SUMMARY: ThreadSanitizer: data race /home/cpython/./Modules/_pickle.c:1631:22 in _Unpickler_MemoPut
==================

==214455==ERROR: ThreadSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7afb5f96f25a bp 0x7afb187fd6f0 sp 0x7afb187fd690 T214460)
==214455==The signal is caused by a READ memory access.
==214455==Hint: address points to the zero page.
    #0 _Unpickler_MemoPut /home/cpython/./Modules/_pickle.c:1657:16 (_pickle.cpython-316t-x86_64-linux-gnu.so+0x2125a) 
    #1 Unpickler_set_memo /home/cpython/./Modules/_pickle.c:7804:17 (_pickle.cpython-316t-x86_64-linux-gnu.so+0x26b77) 
    #2 getset_set /home/cpython/Objects/descrobject.c:250:16 (python3.16t+0x22d8fb) 
    #3 _PyObject_GenericSetAttrWithDict /home/cpython/Objects/object.c:2049:19 (python3.16t+0x2d844b) 
    #4 PyObject_GenericSetAttr /home/cpython/Objects/object.c:2120:12 (python3.16t+0x2d8d4b) 
    #5 Unpickler_setattr /home/cpython/./Modules/_pickle.c:7857:12 (_pickle.cpython-316t-x86_64-linux-gnu.so+0x24eca) 
    #6 PyObject_SetAttr /home/cpython/Objects/object.c:1533:15 (python3.16t+0x2d480f) 
    #7 _PyEval_EvalFrameDefault /home/cpython/Python/generated_cases.c.h:11536:27 (python3.16t+0x45231a) 
    #8 _PyEval_EvalFrame /home/cpython/./Include/internal/pycore_ceval.h:122:16 (python3.16t+0x436463) 
    #9 _PyEval_Vector /home/cpython/Python/ceval.c:2134:12 (python3.16t+0x436463)
    #10 _PyFunction_Vectorcall /home/cpython/Objects/call.c (python3.16t+0x215013) 
    #11 _PyObject_VectorcallTstate /home/cpython/./Include/internal/pycore_call.h:144:11 (python3.16t+0x216b11) 
    #12 _PyObject_VectorcallPrepend /home/cpython/Objects/call.c:855:20 (python3.16t+0x216b11)
    #13 method_vectorcall /home/cpython/Objects/classobject.c:55:12 (python3.16t+0x21a132) 
    #14 _PyObject_VectorcallTstate /home/cpython/./Include/internal/pycore_call.h:144:11 (python3.16t+0x48ec15) 
    #15 context_run /home/cpython/Python/context.c:728:29 (python3.16t+0x48ec15)
    #16 method_vectorcall_FASTCALL_KEYWORDS /home/cpython/Objects/descrobject.c:421:24 (python3.16t+0x22e84c) 
    #17 _PyObject_VectorcallTstate /home/cpython/./Include/internal/pycore_call.h:144:11 (python3.16t+0x2149a8) 
    #18 PyObject_Vectorcall /home/cpython/Objects/call.c:327:12 (python3.16t+0x2149a8)
    #19 _Py_VectorCallInstrumentation_StackRefSteal /home/cpython/Python/ceval.c:766:11 (python3.16t+0x4370c8) 
    #20 _PyEval_EvalFrameDefault /home/cpython/Python/generated_cases.c.h:1846:35 (python3.16t+0x43d022) 
    #21 _PyEval_EvalFrame /home/cpython/./Include/internal/pycore_ceval.h:122:16 (python3.16t+0x436463) 
    #22 _PyEval_Vector /home/cpython/Python/ceval.c:2134:12 (python3.16t+0x436463)
    #23 _PyFunction_Vectorcall /home/cpython/Objects/call.c (python3.16t+0x215013) 
    #24 _PyObject_VectorcallTstate /home/cpython/./Include/internal/pycore_call.h:144:11 (python3.16t+0x216b11) 
    #25 _PyObject_VectorcallPrepend /home/cpython/Objects/call.c:855:20 (python3.16t+0x216b11)
    #26 method_vectorcall /home/cpython/Objects/classobject.c:55:12 (python3.16t+0x21a132) 
    #27 _PyVectorcall_Call /home/cpython/Objects/call.c:273:16 (python3.16t+0x214ca2) 
    #28 _PyObject_Call /home/cpython/Objects/call.c:348:16 (python3.16t+0x214ca2)
    #29 PyObject_Call /home/cpython/Objects/call.c:373:12 (python3.16t+0x214d07) 
    #30 thread_run /home/cpython/./Modules/_threadmodule.c:388:21 (python3.16t+0x61a708) 
    #31 pythread_wrapper /home/cpython/Python/thread_pthread.h:234:5 (python3.16t+0x5473bb) 
    #32 __tsan_thread_start_func <null> (python3.16t+0xf9182) 
    #33 start_thread nptl/pthread_create.c:447:8 (libc.so.6+0x9caa3) (BuildId: 8e9fd827446c24067541ac5390e6f527fb5947bb)
    #34 clone3 misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 (libc.so.6+0x129c6b) (BuildId: 8e9fd827446c24067541ac5390e6f527fb5947bb)

ThreadSanitizer can not provide additional info.
SUMMARY: ThreadSanitizer: SEGV /home/cpython/./Modules/_pickle.c:1657:16 in _Unpickler_MemoPut
==214455==ABORTING

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

Activity

  1. Naserume
    added
    type-bugAn unexpected behavior, bug, or error
    on May 27, 2026
  2. StanFromIreland
    added
    interpreter-core(Objects, Python, Grammar, and Parser dirs)
    topic-free-threading
    on May 27, 2026
  3. StanFromIreland
    added this to Pickle and copy issues 🥒on May 27, 2026
  4. AhmedMagdy9876
    added 4 commits that reference this issue on May 28, 2026

    pythongh-150505: fix data race in Unpickler_set_memo under free-threa…

    be6a4a2

    pythongh-150505: fix data race in Unpickler_set_memo under free-threa…

    194ca8e

    pythongh-150505: fix data race in Unpickler_set_memo under free-threa…

    486a6e4

    pythongh-150505: fix data race in Unpickler_set_memo under free-threa…

    61839ca

  5. kumaraditya303 commented on May 28, 2026

    @kumaraditya303
    kumaraditya303
    on May 28, 2026
    Contributor

    I think there are other issues as well if pickler objects are shared across threads, is this actually supposed to work?

  6. AhmedMagdy9876 commented on May 28, 2026

    @AhmedMagdy9876
    AhmedMagdy9876
    on May 28, 2026

    @kumaraditya303 the fix addresses the specific race in memo assignment, not Unpickler's other mutable state. thay would also race if the object were shared across threads during .load().
    The question of whether sharing an Unpickler across threads shall be a supported use case seems worth settling first. If the answer is no, a documentation note and/or annotation would be more appropriate than scattered locking. If the answer is yes, then a more comprehensive audit of all mutable fields would be needed beyond just memo.
    What do you think on the intended thread-safety contract for Unpickler? Happy to adjust the approach based on that.

  7. kumaraditya303 commented on May 28, 2026

    @kumaraditya303
    kumaraditya303
    on May 28, 2026
    Contributor

    I don't think pickler objects are expected to work across threads, @serhiy-storchaka what do you think?

  8. serhiy-storchaka commented on May 28, 2026

    @serhiy-storchaka
    serhiy-storchaka
    on May 28, 2026
    Member

    I do not see what can be a reason of sharing pickler or unpickler objects between threads. Neither dump() nor load() can work concurrently, because they change the internal state (memo) depending on input data and timing, and the order of changes matters, even if we make them atomic.

  9. AhmedMagdy9876
    added 3 commits that reference this issue on May 28, 2026

    pythongh-150505: Revert "fix data race in Unpickler_set_memo under fr…

    6f465f9

    pythongh-150505: document that Pickler and Unpickler are not thread-safe

    aef265f

    pythongh-150505: remove unnecessary NEWS entry for docs-only change

    a540d41

  10. devdanzin commented on Jul 23, 2026

    @devdanzin
    devdanzin
    on Jul 23, 2026
    Member

    Corroborating @kumaraditya303's "there are other issues as well if pickler objects are shared across threads" — the Pickler side has the same class of problem (this issue is the Unpickler _Unpickler_MemoPut). A shared Pickler takes no per-object critical section, so concurrent dump() / clear_memo() crash it: PyMemoTable_Clear does Py_XDECREF(me_key) + memset over mt_table (Modules/_pickle.c:816-819) with no lock, racing _PyMemoTable_Lookup / PyMemoTable_Size reads (:848 / :807, via Pickler.dump()), plus the shared output buffer (_Pickler_Write :1106). The result is a double-DECREF of a memo'd key (_Py_NegativeRefcount) and a _Pickler_Write SIGSEGV.

    Reproducer (free-threaded build, PYTHON_GIL=0) — crashes 6/6 on a plain free-threaded debug build:

    import _pickle, io, threading
p = _pickle.Pickler(io.BytesIO())
N = 30000
def dumper():
    for _ in range(N):
        try: p.dump([1, 2, 3, "x", {"k": 1}])
        except Exception: pass
def clearer():
    for _ in range(N):
        try: p.clear_memo()
        except Exception: pass
ts = [threading.Thread(target=dumper) for _ in range(4)] + \
     [threading.Thread(target=clearer) for _ in range(4)]
for t in ts: t.start()
for t in ts: t.join()

    Given @serhiy-storchaka's and @kumaraditya303's point that sharing pickler/unpickler objects across threads isn't a scenario that can work (both dump() and load() mutate the memo), we're not filing this separately — just recording it here so the Pickler-side "other issues" are documented next to the Unpickler.memo fix, in case it's useful when deciding how far to take the hardening.

    Found with fusil's --tsan mode; reproducer reduced with AI assistance (Claude Code), verified by hand on free-threaded debug/TSan builds of main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    interpreter-core(Objects, Python, Grammar, and Parser dirs)topic-free-threadingtype-bugAn unexpected behavior, bug, or error

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions