Skip to content

Free threading memory leak #145615

@colesbury

Description

@colesbury

This bug was originally found by a colleague. Memory usage (RSS) keeps increasing even though allocated blocks stays essentially constant. Self contained reproducer below:

import gc
import sys
import os
import concurrent.futures

def get_rss_mb():
    with open("/proc/self/statm") as f:
        return int(f.read().split()[1]) * os.sysconf("SC_PAGE_SIZE") / (1024 * 1024)

def work(_):
    return [bytearray(64) for _ in range(4000)]


NUM_ITERATIONS = int(sys.argv[1]) if len(sys.argv) > 1 else 20

pool = concurrent.futures.ThreadPoolExecutor(max_workers=4)
for i in range(NUM_ITERATIONS):
    futures = [pool.submit(work, j) for j in range(100)]
    results = [f.result() for f in futures]
    del futures, results  # free worker-allocated objects on main thread
    gc.collect()
    gc.collect()
    blocks = sys.getallocatedblocks()
    print(f"iter {i:3d}: RSS={get_rss_mb():7.1f} MB {blocks=}")

pool.shutdown()
print(f"after shutdown: RSS={get_rss_mb():7.1f} MB")

The problem is that we end up having orphaned mimalloc pages in the full queue, even though those pages have available blocks or are completely empty. They're "leaked" until the thread exits.

I think #145614 is also related.

I think there are at least two underlying bugs here:

  1. _PyMem_mi_page_maybe_free leaves empty pages in the full page queue. This bug can happen when a page goes from "full" to "all free".
  2. _mi_page_free_collect calls _PyMem_mi_page_clear_qsbr if page->local_free != NULL. This can get erroneously triggered even the by traversing heaps. We only want to clear the page QSBR counter when it becomes empty again.

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.14bugs and security fixes3.15new features, bugs and security fixestopic-free-threadingtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions