Consume the boundary after a multipart _charset_ part - #13640

What do these changes do?

MultipartReader.next() has a special case for the RFC 7578 §4.6 _charset_
field: when the first part of a multipart/form-data body is named _charset_,
its value is read as the default charset and the next part is returned instead.
That branch read the _charset_ part's body and then called fetch_next_part()
directly, so the delimiter line that terminates the _charset_ part was never
consumed. fetch_next_part() therefore fed the boundary line into
HeadersParser, which raised InvalidHeader. These changes release the
_charset_ part and read its delimiter with _read_boundary() before fetching
the next part, and return None when that delimiter turns out to be the closing
one.

Are there changes in behavior for the user?

Yes — a multipart/form-data body containing a _charset_ field is now parsed
instead of raising. Previously the boundary line was consumed as a header, so
await request.post() raised InvalidHeader and the server answered
500 Internal Server Error for every boundary that does not itself parse as a
name: value pair — i.e. for every realistic boundary. The existing coverage
only passed because it uses the boundary :, which makes the delimiter line
--: parse as a (bogus) header named --, silently attached to the following
part's headers. That stray header is gone now too. No API changes.

Is it a substantial burden for the maintainers to support this?

No. It is five lines inside the existing _charset_ branch, reusing
BodyPartReader.release() and MultipartReader._read_boundary() — the same two
steps the ordinary next() path already performs between parts. No new helpers,
no new public surface.

Related issue number

None — found while probing the multipart header/boundary seams. Happy to file one
if you prefer an issue on record.

Checklist

• I think the code is well written
• Unit tests for the changes exist
• Documentation reflects the changes — N/A (no API or documented-behaviour change)
• If you provide code modification, please add yourself to CONTRIBUTORS.txt — already listed
• Add a new news fragment into the CHANGES/ folder

Reproducer

import asyncio, aiohttp
from aiohttp import web

B = "----WebKitFormBoundaryABC"
BODY = (
    f"--{B}\r\n"
    'Content-Disposition: form-data; name="_charset_"\r\n\r\n'
    "utf-8\r\n"
    f"--{B}\r\n"
    'Content-Disposition: form-data; name="field1"\r\n\r\n'
    "foo\r\n"
    f"--{B}--\r\n"
).encode()

async def handler(request):
    return web.json_response(dict(await request.post()))

async def main():
    app = web.Application()
    app.router.add_post("/", handler)
    runner = web.AppRunner(app); await runner.setup()
    site = web.TCPSite(runner, "127.0.0.1", 0); await site.start()
    port = runner.addresses[0][1]
    async with aiohttp.ClientSession() as s:
        async with s.post(f"http://127.0.0.1:{port}/", data=BODY,
                          headers={"Content-Type": f"multipart/form-data; boundary={B}"}) as r:
            print(r.status, await r.text())
    await runner.cleanup()

asyncio.run(main())

Before: 500 500 Internal Server Error with
aiohttp.http_exceptions.InvalidHeader: Invalid HTTP header: b'------WebKitFormBoundaryABC'
raised from multipart.py:832 fetch_next_part → multipart.py:935 _read_headers.
After: 200 {"field1": "foo"}.

$ AIOHTTP_NO_EXTENSIONS=1 PYTHONPATH=. python -m pytest tests/test_multipart.py -k default_encoding -q
tests/test_multipart.py .FFF.                                            [100%]
=================================== FAILURES ===================================
E           AssertionError: assert ['--', 'Content-Disposition'] == ['Content-Disposition']
E             At index 0 diff: '--' != 'Content-Disposition'
E             Left contains one more item: 'Content-Disposition'
tests/test_multipart.py:1188: AssertionError
tests/test_multipart.py:1215:
E               aiohttp.http_exceptions.InvalidHeader: 400, message:
E                 Invalid HTTP header: b'--WebKitFormBoundary'
tests/test_multipart.py:1235:
E               aiohttp.http_exceptions.InvalidHeader: 400, message:
E                 Invalid HTTP header: b'--WebKitFormBoundary--'
=========================== short test summary info ============================
FAILED tests/test_multipart.py::TestMultipartReader::test_read_form_default_encoding
FAILED tests/test_multipart.py::TestMultipartReader::test_read_form_default_encoding_boundary_without_colon
FAILED tests/test_multipart.py::TestMultipartReader::test_read_form_default_encoding_as_last_part

$ AIOHTTP_NO_EXTENSIONS=1 PYTHONPATH=. python -m pytest tests/test_multipart_helpers.py tests/test_multipart.py -q
tests/test_multipart_helpers.py ................s....................... [ 14%]
tests/test_multipart.py ................................................ [ 64%]
======================== 268 passed, 7 skipped in 0.37s ========================

$ AIOHTTP_NO_EXTENSIONS=1 PYTHONPATH=. python -m pytest tests/test_web_request.py tests/test_web_functional.py -q
======================= 292 passed, 14 skipped in 1.78s ========================

$ python -m black --check aiohttp/multipart.py tests/test_multipart.py
All done! 2 files would be left unchanged.
$ python -m isort --check-only aiohttp/multipart.py tests/test_multipart.py   # clean
$ python -m flake8 aiohttp/multipart.py tests/test_multipart.py               # clean
$ python -m mypy aiohttp/multipart.py
Success: no issues found in 1 source file

Drafted with Claude Code (Claude Opus 5 and Fable 5.1); human review by @2sumtech pending before this leaves draft.