Activity

1. aisk

   added

   type-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error

   topic-SSL

   on Sep 28, 2026
2. kevinprimeraquecreo-stack

   added this to Project_VKon Oct 18, 2026
3. github-project-automation

   moved this to In progress in Project_VKon Oct 18, 2026
4. kevinprimeraquecreo-stack

   added this to Project_VKon Oct 18, 2026
5. jonathan-simms commented on Oct 23, 2026

   

   jonathan-simms

   on Oct 23, 2026

   More actions

   Confirming this from production, with a different library and a clean repro of the mechanism.

   Environment: python:3.12-slim → Python 3.12.15, OpenSSL 3.5.7, slack_sdk 3.33.4 using its dependency-free builtin websocket client (plain ssl.SSLSocket, one long-lived receive thread reused across reconnects).

   Symptom: after one TLS write failure on 2026-10-18 the receive thread never recovered: every reconnect completed the handshake, read the server hello, and then the first non-blocking recv that should have been EAGAIN raised BrokenPipeError(32). The library's auto-reconnect looped like that for five days.

   Evidence that it's the stale error queue, not the socket:

   • strace -f on the thread for one cycle: read returns -1 EAGAIN, no EPIPE anywhere in the trace, no write failing, and Python raises BrokenPipeError immediately after that EAGAIN. The errno in the exception comes from ERR_GET_REASON of the queued ERR_LIB_SYS entry (tls_retry_write_records failure, errno 32 from the original failed write), not from the syscall.
   • _ssl.c PySSL_SetError returns via PyErr_SetFromErrno on the ERR_LIB_SYS branch before reaching ERR_clear_error(), and _ssl__SSLSocket_read_impl does not clear the queue before SSL_read_ex, so SSL_get_error keeps reporting SSL_ERROR_SYSCALL from ERR_peek_error() on every later read from that thread.
   • A fresh SocketModeClient (new threads) started inside the same process/container connected and stayed up while the original thread was still looping. Same token, same host, same SSL context. Only the thread differed.

   Workaround we shipped: a watchdog that discards the client and its threads when the connection stays down, since there is no way to clear the per-thread queue from Python. Happy to provide the strace excerpt if useful.
6. zoc

   added a commit that references this issue on Oct 24, 2026

   fix: clear the thread's OpenSSL error queue before router API use

   ...

   6dbd15a
7. ifor commented on Nov 3, 2026

   

   ifor

   on Nov 3, 2026

   More actions

   Another production confirmation, and a trigger that needs no failed write by the application.

   Environment: python:3.12-slim images: 3.12.13 (OpenSSL 3.5.6) and 3.12.14, 3.12.15, 3.12.16 (OpenSSL 3.5.7), Linux x86_64. All are affected. Ubuntu 24.04's 3.12.3 with OpenSSL 3.0.13 is not.

   1. A server that cuts a response off is enough. Suppose the peer sends FIN and then RST in the middle of a response, with no close_notify. (We see this from a council portal that ends long downloads after about 25 s.) OpenSSL records SSL_R_UNEXPECTED_EOF_WHILE_READING and tries to send a fatal alert. That write fails with EPIPE and adds an ERR_LIB_SYS entry. PySSL_SetError reads ERR_peek_last_error(), which is the ERR_LIB_SYS entry, and takes the early return in the SSL_ERROR_SSL case. So recv() raises BrokenPipeError instead of SSLEOFError, and both entries stay queued. From then on, every recv() in that thread that has to wait, on any connection, raises BrokenPipeError at once. Loopback only, anonymous ECDH, so no certificate is needed:

   """No failed write by the application: a server that cuts a response off (FIN, then RST,
   no close_notify) makes recv() raise BrokenPipeError and poisons the thread."""
   import socket, ssl, struct, sys, threading, time
   
   def ctx(server):
       c = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER if server else ssl.PROTOCOL_TLS_CLIENT)
       c.maximum_version = ssl.TLSVersion.TLSv1_2
       c.set_ciphers("aNULL:@SECLEVEL=0")
       if not server:
           c.check_hostname, c.verify_mode = False, ssl.CERT_NONE
       return c
   
   listener = socket.socket()
   listener.bind(("127.0.0.1", 0))
   listener.listen(5)
   held = []
   
   def serve():
       sctx = ctx(True)
       while True:
           raw, _ = listener.accept()
           tls = sctx.wrap_socket(raw, server_side=True)
           if tls.recv(16) == b"cut":
               tls.sendall(b"part of a body")
               s = socket.socket(fileno=tls.detach())   # no close_notify
               s.shutdown(socket.SHUT_WR)               # FIN ...
               s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack("ii", 1, 0))
               s.close()                                # ... then RST
           else:
               held.append(tls)                         # stays open and silent
   
   threading.Thread(target=serve, daemon=True).start()
   cctx = ctx(False)
   
   def connect(first):
       c = cctx.wrap_socket(socket.create_connection(listener.getsockname()))
       c.settimeout(1)
       c.send(first)
       return c
   
   def show(label, op):
       t = time.time()
       try:
           r = op()
           print(f"{label}: returned {r!r} after {time.time() - t:.2f}s")
       except Exception as e:
           print(f"{label}: {type(e).__name__}({getattr(e, 'errno', None)}) after {time.time() - t:.2f}s")
   
   print(sys.version.split()[0], ssl.OPENSSL_VERSION, sys.platform)
   healthy = connect(b"idle")
   show("healthy.recv() before         ", lambda: healthy.recv(100))     # TimeoutError: correct
   cut = connect(b"cut")
   time.sleep(0.3)                                                      # FIN and RST are in
   show("cut.recv() (the part)         ", lambda: cut.recv(100))
   show("cut.recv() (FIN+RST, no alert)", lambda: cut.recv(100))         # expected: an EOF
   for i in range(3):
       show(f"healthy.recv() after, try {i + 1} ", lambda: healthy.recv(100))  # expected: TimeoutError

   Python 3.12.16, OpenSSL 3.5.7:

   healthy.recv() before         : TimeoutError(None) after 1.00s
   cut.recv() (the part)         : returned b'part of a body' after 0.00s
   cut.recv() (FIN+RST, no alert): BrokenPipeError(32) after 0.00s
   healthy.recv() after, try 1 : BrokenPipeError(32) after 0.00s
   healthy.recv() after, try 2 : BrokenPipeError(32) after 0.00s
   healthy.recv() after, try 3 : BrokenPipeError(32) after 0.00s
   

   Python 3.12.3, OpenSSL 3.0.13:

   cut.recv() (FIN+RST, no alert): returned b'' after 0.00s
   healthy.recv() after, try 1 : TimeoutError(None) after 1.00s
   

   2. Connection pools make it last. We used one boto3 client shared by a thread pool. A single reset during an S3 upload left two threads failing every request about 0.25 ms after sending it, for eight hours, with ConnectionClosedError: Connection was closed before we received a valid response. Other threads were using the same pooled connections without trouble. Each failure throws away a healthy pooled connection, but the other threads keep the pool full. So the poisoned thread rarely opens a connection of its own, and that is the only thing that clears it (newPySSLSocket's ERR_clear_error(), or the handshake state machine).

   A workaround that holds: call ERR_clear_error() before each request, through ctypes on _ssl's own libcrypto handle (a botocore before-send handler, or requests.Session.send). The fix suggested in the report would cover both triggers: ERR_clear_error() before the two ERR_LIB_SYS returns, at Modules/_ssl.c lines 655–659 and 685–689 on v3.12.16 (755–759 and 791–795 on main). For trigger 1 there is a further question. When the queue holds SSL_R_UNEXPECTED_EOF_WHILE_READING followed by the ERR_LIB_SYS from the failed alert, the EOF is arguably the error to report.