list iterator __setstate__() differs for exhausted iterator GIL vs. non-GIL #129139

Activity

1. picnixz

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   interpreter-core(Objects, Python, Grammar, and Parser dirs)(Objects, Python, Grammar, and Parser dirs)

   topic-free-threading

   on Jan 21, 2025
2. eendebakpt commented on Jan 21, 2025

   

   eendebakpt

   on Jan 21, 2025

   Contributor

   More actions

   @Yhg1s Your changes in #128637 might result in a similar behavior change for tuple iterators as is described here for list iterators.
3. Yhg1s commented on Jan 22, 2025

   

   Yhg1s

   on Jan 22, 2025

   Member

   More actions

   Yes, indeed. It's not difficult to fix, by the way, it just requires an extra check in the relevant __setstate__ methods. I'm not sure it's worth it.
4. thalesmourabh commented on Nov 6, 2026

   

   thalesmourabh

   on Nov 6, 2026

   More actions

   Triage: this affects three iterator types, not one, and one of the symptoms is present in the default (GIL-enabled) build.

   Both builds below are the same commit, 534605904dfa7db25bf19a1b959411b8562c2b80, differing only in --disable-gil. Released versions are the python-build-standalone builds.

   Version	iter(list)	reversed(list)	iter(tuple)
   3.12.16	ok	ok	ok
   3.13.18 (GIL)	ok	resurrects	ok
   3.13.18t	resurrects	resurrects	ok
   3.14.10 (GIL)	ok	resurrects	ok
   3.14.10t	resurrects	resurrects	resurrects
   3.16.0a1+dev (GIL)	ok	resurrects	ok
   3.16.0a1+dev free-threaded	resurrects	resurrects	resurrects

   "resurrects" means list(it) returns an element again after it.__setstate__(0) on an already exhausted iterator. Not affected in any build: str, bytes, bytearray, range, and the generic sequence iterator.

   reversed() is affected in the GIL-enabled build, since 3.13. That part is not free-threading specific:

   >>> it = reversed([1]); list(it)
   [1]
   >>> it.__setstate__(0); list(it)
   [1]          # 3.13+, GIL-enabled; 3.12 returns []

   Root cause. __setstate__ uses it_seq == NULL as the "exhausted" sentinel, while the thread-safety work changed _next to use it_index < 0. The __setstate__ methods were not updated.

   In 3.12, listreviter_next always cleared it_seq when exhausted. After a2d4281 it returns early instead:

   Py_ssize_t index = FT_ATOMIC_LOAD_SSIZE_RELAXED(it->it_index);
   if (index < 0) {
       return NULL;          /* returns before it_seq = NULL below */
   }
   ...
   FT_ATOMIC_STORE_SSIZE_RELAXED(it->it_index, -1);
   #ifndef Py_GIL_DISABLED
       it->it_seq = NULL;    /* unreachable on the normal exhaustion path */
   #endif

   For reversed([1]): it_index starts at 0, the first next() yields the item and stores -1, the second next() takes the early return, and it_seq is never cleared. __setstate__(0) then sees it_seq != NULL, runs its body and rewinds the iterator.

   The forward list iterator escapes this in the GIL build because its exhaustion is detected by list_get_item_ref returning NULL, a path that still reaches it->it_seq = NULL. Under --disable-gil that assignment is compiled out, which is why the forward iterator is affected there.

   Culprits:

   • reversed(list), GIL build: a2d4281 ("Make list objects thread-safe in --disable-gil builds #112087: Make __sizeof__ and listiter_{len, next} to be threadsafe", first in v3.13.0a4)
   • iter(tuple), free-threaded: 388e1ca ("Make the specializing interpreter thread-safe in --disable-gil builds #115999: Make list and tuple iteration more thread-safe", first in v3.14.0a6)

   The second one confirms what @eendebakpt predicted in the first comment on this issue: gh-128637 did produce the equivalent behaviour for tuple iterators.

   The correct shape already exists in the tree. Objects/iterobject.c:112 checks both sentinels, which is why the generic sequence iterator does not appear in the table:

   if (index >= 0 && it->it_seq != NULL) {

   The three affected sites check only one: Objects/listobject.c:4198, Objects/listobject.c:4350, Objects/tupleobject.c:1206. This matches @Yhg1s's comment that it "just requires an extra check in the relevant __setstate__ methods".

   For completeness: bytesobject.c:3625/3641 and unicodeobject.c:15100/15121 have the same it_seq != NULL guard but do not reproduce, so I have left them out.

   I have no opinion on whether this is worth fixing — that is the open question in this thread. I'm only putting the scope on record, since it is larger than the issue title suggests.

   Reproduction, version matrix and root-cause analysis done with assistance from Claude Code, reviewed by a human.