Skip to content

SocketModeClient (aiohttp backend): connect() retries forever against a closed ClientSession — orphaned retry loop survives close() #1913

Description

@eBeat-developers

Environment

  • slack_sdk 3.40.1, slack_bolt 1.27.0 (AsyncSocketModeHandler), aiohttp 3.14.1
  • Python 3.11, macOS, long-running gateway process under launchd

Summary

slack_sdk.socket_mode.aiohttp.SocketModeClient creates its aiohttp.ClientSession once in __init__ (line 130) and never recreates it. The connect() retry loop (lines 347–409) catches every exception, sleeps ping_interval (default 10s), and retries — without ever checking self.closed or whether the session object is still usable. Once that loop runs against a closed session, ws_connect() raises RuntimeError: Session is closed immediately on every attempt, so the loop can never succeed and never exits: a permanent retry loop for the lifetime of the process.

ERROR slack_bolt.AsyncApp: Failed to connect (error: Session is closed); Retrying...
Traceback (most recent call last):
  File ".../slack_sdk/socket_mode/aiohttp/__init__.py", line 377, in connect
    self.current_session = await self.aiohttp_client_session.ws_connect(
  File ".../aiohttp/client.py", line 1244, in _ws_connect
    resp = await self.request(
  File ".../aiohttp/client.py", line 581, in _request
    raise RuntimeError("Session is closed")
RuntimeError: Session is closed

How the loop becomes orphaned

close() (lines 446–457) cancels current_session_monitor and message_receiver fire-and-forget — it calls .cancel() but does not await the tasks — and then immediately closes the ClientSession. Both tasks are created via asyncio.ensure_future as independent top-level tasks, and both can call connect_to_new_endpoint() (the monitor on staleness, the receiver on a CLOSE frame). If one of them is inside a reconnect attempt when the cancel is requested, its in-flight connect() call keeps running against the now-closed session. At that point nothing references the task anymore: the owning application has replaced its handler/client, but the retry loop lives on.

Observed in production (three long-running gateways):

  • exact 10.000s cadence (= default ping_interval), never accelerating or backing off;
  • one host accumulated 270,136 RuntimeError: Session is closed events over 46 days;
  • another host showed two parallel orphaned loops (event pairs 10–13ms apart, both at 10s cadence, stable for 1.5h+);
  • the count of parallel loops never changes once established — consistent with rare races each leaking one loop;
  • only a full process restart clears it. An application-level watchdog (which rebuilds the handler) cannot, because the orphaned task is unreachable.

Suggested fixes (any one of these would break the failure mode)

  1. In connect()'s exception handler: if self.closed (or self.aiohttp_client_session.closed), log once and return instead of retrying.
  2. At the top of connect(): recreate self.aiohttp_client_session when it is closed (mirrors what the Node.js @slack/socket-mode SDK effectively does by building a fresh connection per attempt).
  3. In close(): after cancelling current_session_monitor / message_processor / message_receiver, await asyncio.wait([...], timeout=...) so no cancelled task is still mid-connect when the session closes.

Workaround we deployed

Application-side: cancel and await client.current_session_monitor / client.message_processor / client.message_receiver before calling close_async(), plus a logging-filter-based storm detector that restarts the process (supervisor restarts it) if the "Failed to connect (error: Session is closed)" cadence is detected anyway. Happy to turn any of the suggested fixes into a PR if maintainers indicate a preferred direction.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:asyncbugM-T: A confirmed bug report. Issues are confirmed when the reproduction steps are documented

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions