[mame]

Under the M:N thread scheduler, a Ruby thread in a non-main Ractor may be resumed on a different native thread after rb_thread_call_without_gvl() returns. The OpenSSL error queue is thread-local, so errors pushed by the function without the GVL are left on the previous native thread: the exception loses its reason, and the stray errors appear in an unrelated exception raised later on that native thread.

8.times.map {
  Ractor.new {
    4.times.map {
      Thread.new {
        2000.times.map do
          OpenSSL::KDF.pbkdf2_hmac("", salt: "", iterations: 0, length: 1, hash: "sha256")
        rescue OpenSSL::KDF::KDFError => e
          e.message
        end.uniq
      }
    }.flat_map(&:value).uniq
  }
}.flat_map(&:value).uniq
#=> ["PKCS5_PBKDF2_HMAC: invalid iteration count", "PKCS5_PBKDF2_HMAC"]

Save the error queue at the end of the function and restore it after the GVL is reacquired, using OSSL_ERR_STATE (OpenSSL 3.2+).

What do you think? @ko1 @rhenium

[rhenium]

I hadn't thought about this. Thanks for catching it!

ossl_pkey.c uses rb_thread_call_with_gvl() to reacquire GVL in the middle to check whether the thread has been interrupted. Is it possible to resume on a different native thread at that point as well?

I think it would also be acceptable for these methods to clear the queue, since I expect they fail only because of invalid argument values.

I suspect this might not be the only place ruby/openssl breaks if the native thread changes. In particular, some SSLContext callbacks may invoke Ruby code while the OpenSSL error queue is non-empty. I'll look into similar cases later.