moonhttp

The HTTP family for MoonBit: the formats a request and a response are written in, with nothing about sockets in them.

Site · Guide · API · Figures · mooncakes

// An event stream, one frame at a time.
@sse.Event::new(data="{\"seq\":1}", kind="tick", id="1").encode()
@sse.decode(received[:])  // the frames as written
@sse.events(received[:])  // what an EventSource dispatches from them

// A cookie set on the way out and read on the way back.
@cookie.Cookie::new("sid", token, http_only=true, secure=true).encode()
@cookie.get(request_header[:], "sid"[:])

// A posted form, whichever way the browser encoded it.
let form = @form.parse(body[:], content_type[:])
form.field("name")
form.file("avatar")

// A second request for something that has not changed.
let etag = @conditional.hash(body[:], digest=@sha2.Hasher::new())
@conditional.Ask::read(headers[:]).evaluate(verb="GET", etag~)  // Go | Fresh | Failed

// A WebSocket message compressed the way RFC 7692 asks, and the bit that says so.
let terms = @pmd.Extension::agree(@pmd.offers(offer[:])[:]).unwrap()
let (frame, compressed) = @pmd.Writer::new(extension=terms).frame(@ws.Opcode::Text, body[:])
frame.encode_extension(rsv1=compressed)  // one deflate segment, tail stripped

// The same WebSocket over HTTP/3: the extended CONNECT of RFC 9220, allowed by one
// setting the server sends, and answered with 200 or 501.
let wire = @http3.websocket_request("https", "example.com", "/chat", headers)
@http3.Upgrade::of(request, settings=@http3.Settings::new(4096, 16384, 100, enable_connect_protocol=true))
// Accept | Unsupported("...") | Refused("...")

// A stored response, and what may be done with it.
let stored = @cache.Entry::read(status=200, headers, request, sent~, received~)
stored.verdict(request=@cache.control, presented=headers, now~)  // Serve | Validate | Miss
stored.conditional()  // the If-None-Match a cache sends instead

// And where a 302 sends the request next.
@client.follow(status=302, location="/next", from~, verb=@method.Method::Post, visited=[from][:])

Run moon run src/examples/tour for the whole surface in one go. The worked examples run a package each:

moon run src/examples/04-ws-handshake       moon run src/examples/09-http3-frames
moon run src/examples/05-ws-frame           moon run src/examples/10-http3-message
moon run src/examples/06-qpack-primitives   moon run src/examples/11-http3-conn
moon run src/examples/07-qpack-field        moon run src/examples/12-range
moon run src/examples/08-qpack-dynamic      moon run src/examples/13-conditional
moon run src/examples/14-ws-compressed      moon run src/examples/15-cache

src/examples/16-ws-echo is the other kind: a server, not a walkthrough. It serves ws://127.0.0.1:9001 with this library's handshake, frames and compression over a socket from moonbitlang/async, which is what scripts/autobahn/run.sh runs Autobahn against.

The documentation is a site of four pages: a guide that takes the library one message at a time, the API reference generated from the sources' doc comments, and ten figures that draw the byte layouts — a frame header, a chunked body, the DEFLATE stream a WebSocket keeps open.

Packages

Package	What	Specification
sse	Server-Sent Events, both directions	WHATWG HTML, the event-stream format
form	A request body read as the form its Content-Type says it is, multipart/form-data or application/x-www-form-urlencoded	WHATWG HTML form submission
range	Asking for part of a representation: the fields, the arithmetic that resolves a range against a length, and the multi-range body	RFC 9110 §14
conditional	Entity-tags, the five precondition fields, and the order they are weighed in	RFC 9110 §8.8, §13
cache	What may be stored, how long it stays fresh, how old it is, and whether it may answer: the decisions, with no store	RFC 9111
client	Where a redirect sends a request, with which method and which credentials; and Alt-Svc read into the alternatives it names	RFC 9110 §15.4, RFC 7838
cookie	Cookie and Set-Cookie, each read and written	RFC 6265, and its revision for SameSite
ws	WebSocket framing: opcodes, masking, fragment reassembly, close statuses	RFC 6455 §5, §7.4.1
pmd	The permessage-deflate extension: its four parameters, the offer and the answer, and a reader and writer over ws frames	RFC 7692
deflate	Raw DEFLATE shaped for a stream that outlives a message: a window carried across messages, and a ceiling on what one may expand to	RFC 1951
upgrade	The WebSocket opening handshake, both sides	RFC 6455 §4
huffman	The Huffman code both header compressions use	RFC 7541 Appendix B
method	The request method: the nine standard ones and any other token, with whether each is safe and idempotent	RFC 9110 §9, RFC 5789
header	A header field: a name and a value, as octets	RFC 9110 §5
hpack	HTTP/2 header compression: both tables, both ends, and a bound on the list a block decodes to	RFC 7541
qpack	HTTP/3 header compression: both tables, the field lines, both instruction streams	RFC 9204
http1	HTTP/1.1 messages: the request and status lines, field lines, body framing and the chunked coding; Conn, a whole connection for either end on h11's state machine, octets in and events out	RFC 9112
http2	HTTP/2 framing: the nine-octet header, all ten frame types, the connection preface; Conn, a whole connection for either end after python-hyper h2, with stream states, flow control, settings, push and field validation	RFC 9113, RFC 8441
http3	HTTP/3 frames, their placement rules, settings, stream types and connection; the connection error each refusal closes with; a reader for a request stream and one for each unidirectional stream; and WebSocket over HTTP/3 as the extended CONNECT RFC 9220 defines	RFC 9114, RFC 9220

One vocabulary

The same word means the same thing in every package here, and in every other package the organisation publishes. encode and decode are a pair, and parse is for a format with no symmetric writer. code is an enum's value on the wire, so a WebSocket close status is status and never code. A limit passed is Exceeded(limit~, got~), a predicate reads as an adjective or is named is_, and a refusal is Refused.

Configuration

Every bound and every leniency is an argument, and every default is the one the mainstream uses.

// A bound belongs to an endpoint. Either form names one.
let small = @moonmime.Limits::new(parts=8, part_size=64 * 1024)
let same = { ..@moonmime.limits, parts: 8, part_size: 64 * 1024 }

@form.parse(body[:], content_type[:], limits=small)

// By default a body that is not a form is an empty form, and a multipart body
// that stops making sense gives up the parts it read. `strict` says so instead.
@form.parse(body[:], content_type[:], strict=true)   // raises NotAForm, or Malformed(at~)

@sse.Event::of("hello").encode(space=false)          // drop the optional space

// A compressed message is the one place a peer can ask for memory it did not
// pay for, so the reader bounds what one expands to, in octets.
@pmd.Reader::new(limit=1 * 1024 * 1024)

// And a writer can leave a message under this many octets alone, which is worth
// doing: a compressed "OK" is longer than "OK".
@pmd.Writer::new(threshold=64)

Setting	Default	Why that one
limits.parts	1000	Starlette's max_files and max_fields
limits.part_size	1 MiB	python-multipart's in-memory part size
strict	off	A truncated upload is ordinary on a dropped connection, and "this request carried no form" is an answer rather than a failure. On is for the caller who would rather be told
space	on	The space after a colon is optional in the format and universal on the wire; a reader strips it either way
limit (pmd)	16 MiB	The ceiling ws puts on a message anyway. RFC 7692 sets none, and a compressed message is where that absence would cost the most
threshold (pmd)	0	Every message compressed, which is what agreeing to the extension means. A caller compressing a bulletin board and not a chat box raises it

Limits has no per-call mirror, and therefore no precedence rule. A bound belongs to an endpoint rather than to a request — an avatar upload and a spreadsheet import are two endpoints, each with its own — so the record is the only place it comes from and there is nothing to arbitrate. Where a setting can arrive from two places, as in mooncred, the precedence is published with it.

Downloads

A download needs three answers, and each comes from its own place: what the body is and under what name to save it is moonmime's guess_type and content_disposition; resuming it and seeking in it is range; not sending it again is conditional. Reading the file and sending it in pieces is the server's, not this library's — there are no sockets here.

Framing, not transport

Nothing here opens a connection or writes to one. sse turns an event into the bytes of a frame and back; sending each frame as its own chunk is the server's job, and it matters — a stream delivered as one body is not a stream, because a client dispatches an event only when it reads that event's blank line.

A client wants the second of sse's two readings. decode answers each frame as written; events answers what a browser's EventSource hands its listeners — no event for a frame without data, the type defaulting to message, and the id and reconnection delay carried from frame to frame as the stream's own state.

form reads a body that has already arrived. Both of its encodings are bounded by a Limits the caller sets, because the body came from whoever sent it: a million empty parts and one enormous part are two ways of asking a server to allocate more than it has. A form over its bounds is refused whole, not truncated — a handler given the first thousand parts of a larger form would be answering a request nobody sent.

Refusals that close a connection

HTTP/1.1 and HTTP/2 have faults an endpoint answers by dropping one stream and carrying on. HTTP/3 has almost none: a peer that puts DATA before HEADERS, repeats a pseudo-header, or opens a second control stream has a different idea of what the streams mean than this endpoint does, and resetting one of them leaves the two ends still disagreeing. So every refusal in http3 carries the connection error the peer is to be told, and Refused::code is that error.

let reader = @http3.Incoming::new(limit=16384)   // SETTINGS_MAX_FIELD_SECTION_SIZE
let parts = reader.feed(bytes) catch {
  e => {
    // e.code() is what the connection closes with; e.to_string() names the rule.
    transport.close(e.code().code())
    return
  }
}

Code	Number	Raised when
NoError	0x100	the connection ends and nothing went wrong
GeneralProtocolError	0x101	a fault no other code names
InternalError	0x102	the endpoint hit a fault of its own
StreamCreationError	0x103	a second control, encoder or decoder stream; a push stream at a server
ClosedCriticalStream	0x104	the control, encoder or decoder stream ended
FrameUnexpected	0x105	a frame on a stream that does not carry it, or in an order it may not take
FrameError	0x106	a frame that could not be read: a length that does not add up
ExcessiveLoad	0x107	a frame over the size this endpoint advertised
IdError	0x108	a push id never promised, or used twice
SettingsError	0x109	a setting HTTP/2 defined, or the same identifier twice
MissingSettings	0x10a	a control stream that opened with something other than SETTINGS
RequestRejected	0x10b	a request this endpoint will not serve
RequestCancelled	0x10c	a request the client abandoned
RequestIncomplete	0x10d	a request stream that ended before the request did
MessageError	0x10e	a malformed request: a repeated, absent or misplaced pseudo-header, a connection-specific field
ConnectError	0x10f	the CONNECT method or its extended form misused
VersionFallback	0x110	a version this endpoint does not serve
QpackDecompressionFailed	0x200	a field section the table cannot resolve: a static index past the table, a dynamic entry that is not there
QpackEncoderStreamError	0x201	a table capacity over the one advertised, or an instruction that names no entry
QpackDecoderStreamError	0x202	an Insert Count Increment of 0

Three readers enforce the rules the codes name, one per stream that carries state: Incoming for a request stream, Control for the peer's control stream, and Encoder and Decoder for QPACK's two. Each keeps the tail of an instruction whose end has not arrived, so a frame or an instruction split across packets is read once it is whole and refused at the octet that made it wrong.

What is checked

sse is measured against the two example streams the WHATWG specification prints, which are the cases an implementation that trims too much or too little gets wrong; then every frame it writes is read back as the event that wrote it, the exact bytes of each frame are pinned, all three line endings are accepted, and the things a reader must ignore rather than refuse are ignored. events is read against the same streams, and each rule by which it differs from decode is checked by a test that turns red when the rule is removed.

form is measured on both encodings by content type, upper case and a charset parameter included, on what strict turns into an error, and on one bound holding for both. The decoders behind it are measured where they live: moonmime against a browser's multipart body, moonurl against the parser and serialiser cases of web-platform-tests.

conditional follows §13, which states rules rather than printing vectors: the comparison table of §8.8.3.2 entry for entry, then §13.2.2's steps one at a time, then against each other — the order is the part that gets got wrong. The content tag is pinned to SHA-256's vector for hello.

hpack is measured against all twelve worked examples of RFC 7541 Appendix C, in both directions: the encoder reproduces the published blocks octet for octet and the decoder reproduces the published header lists, with the three requests of a series sharing one dynamic table and the three responses evicting under a 256-octet ceiling. huffman is measured against the coded strings those same blocks contain.

http3 is measured section by section: every frame type round-trips, every placement rule of §7.2 is checked on each of the three stream kinds, the settings and the unidirectional stream types round-trip, and a request stream decodes into its pseudo-headers, its fields and its body.

The rules a server keeps are measured one code at a time. On a request stream: DATA before HEADERS and after the trailers, a third HEADERS, and a frame a control stream owns; the four ways a field section is malformed — a repeated pseudo-header, one this protocol does not define, one after the regular fields, and the missing mandatory set — with a http scheme still owing an authority to :authority or to Host; the six connection-specific fields, with te: trailers still accepted; a field over the advertised section size; and the two QPACK faults a field section can carry. On the control stream: SETTINGS first and once, DATA and HEADERS refused there, the HTTP/2 identifiers and a repeated one, a payload ending between an identifier and its value, a frame over the size limit refused before its payload is held, and the stream not being allowed to end. On the unidirectional streams: each of the three critical ones refused a second time, a push stream refused at a server, all four refused closure, and the two QPACK faults — a capacity over the advertised one, an Insert Count Increment of zero.

Against h3spec 0.1.14's "HTTP/3 servers" section, a server built on this package passes all 34 cases.

The extended CONNECT of RFC 9220 is measured beside it. The connect switch is written only when it is on and survives the control stream; the client's request carries the five pseudo-headers and parses back into them; a classic CONNECT names only an authority, and one with :scheme and :path but no :protocol is malformed; and a server answers a WebSocket with 200, an unknown :protocol with 501, and a request that used :protocol before the setting with a refusal.

qpack is measured against the instruction vectors of RFC 9204 Appendix B and, section by section, against every representation §4.5 defines — indexed and post-base, relative and absolute, with a name reference and with a literal name.

ws is measured against all five frames RFC 6455 §5.7 prints: the unmasked and masked text messages, the fragmented one, the ping with its pong, and the two long length forms. upgrade is measured against the key and proof §1.3 prints, which is the one computation in it.

deflate is read against streams zlib wrote, because those are the ones a WebSocket endpoint actually meets: messages at every length where DEFLATE or the frame layer changes shape, a stored block, a dynamic block, a window negotiated down to nine bits, and four messages that only decode if the window of the first survives into the last. The other direction is checked by giving what it writes to another DEFLATE implementation to read. Every split of a compressed message is read as two pieces, which is what a message arriving in fragments is.

Whether to code a message at all is settled before the search. The encoder prices a code for the message's own alphabet — an octet histogram, costed as the dynamic header and symbol widths would cost it — and copies the message into stored blocks only when that price, and a sample of three-octet repeats, both read it as noise. So a stream of random 128 KiB frames stays linear instead of coded octet by coded octet, while a payload of tokens or base64, drawn from ninety-five symbols none of which ever repeats, still comes out at 83% of its size rather than the 100% copying it would have cost. A message that is coded gets the widths its own symbols deserve: the encoder counts what the parse produces, then writes a dynamic block or the fixed one, whichever costs fewer bits. On a 128 KiB message drawn from twenty-six letters — the shape a text frame has — that is 66% of the input where the fixed code alone spends 95%, for the same time. What either writes is decoded through a small table rather than a bit at a time, and the fixed codes are reversed once rather than per symbol, so the two directions stay close in cost at the lengths a WebSocket carries.

pmd is measured on §7.1's four parameters — read, written and negotiated, each with the values the grammar forbids — and then against Autobahn, whose 12.x and 13.x groups are the conformance suite for this extension: scripts/autobahn/run.sh starts the endpoint and points the fuzzingclient at it, and CI runs it. The endpoint answers server_no_context_takeover, so a message that reads as noise is carried stored rather than searched against the one before it. The last run passes 250 cases with three informational and none failed; the thousand-message 128 KiB case 12.2.10 takes about 9 to 13 seconds on the shared runners, and the binary 12.3.15 about 11 to 16, where they were 78 and 199 seconds before. The same behaviours are replayed frame by frame in the unit tests, and scripts/autobahn/interop.py is the check that needs no Docker: a permessage-deflate client written against zlib, which is what a browser is.

cache follows §4.2.1's order for a freshness lifetime, §4.2.2's tenth, §4.2.3's arithmetic on four clock readings, §3's list of reasons not to store — each as a test that would pass if the rule were dropped — and §4's list of reasons not to reuse, including the ones a Vary mismatch and a disconnect turn on. The cache-tests suite's cases that a decision is the whole answer to are replayed as well (scripts/cache-tests.json); the dozen it reads differently are named with the reading this package follows in scripts/gen_cache_vectors.mjs.

client is measured on §15.4's method rewrites (a 301, a 302 and a 303; a 307 and a 308 that keep both method and body), on the credentials that stop at the origin they were meant for, on relative Location resolution, on loops written two ways, and on the Alt-Svc examples §3 of RFC 7838 prints.

What lives elsewhere

URI references and data URLs are moonurl; media types, multipart and Content-Disposition are moonmime.

The variable-length integers are not here either. HTTP/3's frames are counted in QUIC's varint and both header compressions count in RFC 7541's prefixed integer; neither belongs to HTTP, and both are written in more than one protocol. They are moonvar, which this depends on and which depends on nothing.

TLS is moontls and QUIC is moonquic, so that parsing an HTTP message does not mean carrying a handshake. Content codings are moonzip; deflate here is the other shape — a stream that is stopped and resumed inside, which is what RFC 7692 §7.2 needs and what a file compressor has no reason to offer.

Install

moon add moonbitstack/moonhttp

Licence

Apache-2.0. See LICENSE.