Add callable support for `decimal_format` option by NyanFisher · Pull Request #978 · msgspec/msgspec

NyanFisher · 2026-02-11T11:34:59Z

Hello!

Description of the problem solved by this PR

The msgspec library has many useful features, but the current version lacks the ability to correctly quantize Decimal values during encoding. In applications related to finance and precise calculations, it is critical to take into account maximum accuracy and return values rounded to a specified precision. Without implementing this functionality, a complete transition from pydantic to msgspec is not possible.

Changes implemented in this PR

Core Functionality

Added DECIMAL_FORMAT_CALLABLE enum value to support callable decimal_format
Added decimal_callable field to EncoderState and Encoder structs
Added in_decimal_callable field to EncoderState struct
Added recursion protection in mpack_encode_decimal() and json_encode_decimal()
Implemented callable invocation for both JSON and MessagePack encoders

Validation & Safety

Added runtime check preventing callable from returning Decimal (avoids infinite recursion)
- Direct Decimal returns: lambda d: d
- Nested structures: lambda d: [d], lambda d: {"key": d}

Type Hints

Updated json.pyi and msgpack.pyi stubs to include callable type hints:

decimal_format: Union[
    Literal["string", "number"],
    Callable[[decimal.Decimal], Any],
]

Examples

Rounding to 2 Decimal Places

import msgspec
import decimal

enc = msgspec.json.Encoder(
    decimal_format=lambda d: str(d.quantize(decimal.Decimal("0.01")))
)

value = decimal.Decimal("123.456789")
print(enc.encode(value))  # b'"123.46"'

MessagePack with Rounding

import msgspec
import decimal

# MessagePack with custom rounding
enc = msgspec.msgpack.Encoder(
    decimal_format=lambda d: float(d.quantize(decimal.Decimal("0.001")))
)

value = decimal.Decimal("3.14159265")
msg = enc.encode(value)
print(msgspec.msgpack.decode(msg))  # 3.142

Error: Returning Decimal from Callable

import msgspec
import decimal

# INVALID: callable must not return Decimal
enc = msgspec.json.Encoder(
    decimal_format=lambda d: d.quantize(decimal.Decimal("0.01"))  # Error!
)

try:
    enc.encode(decimal.Decimal("1.234"))
except TypeError as e:
    print(e)  # callable returned a value containing a Decimal

I would appreciate any comments on improving or restructuring the code, as I don't often write in C.

Fix my issue - Closes #848

NyanFisher · 2026-04-08T04:22:34Z

CI failures are unrelated to this change:

"Profile Windows (ARM64)" - runner infra issue ("Install command runner" step)
"build" (documentation) - link checker fails on a pre-existing broken link (Broken link to ravyn.dev encoders example in docs #976)

All build, test, and wheel jobs pass across all platforms.

Siyet · 2026-04-10T12:28:57Z

Code looks solid and CI is green across the matrix - nice work, especially the test coverage in test_common.py.

One API design question I'd like to raise before this moves forward: the current shape places decimal_quantize / decimal_rounding on the encoder itself, which means every Decimal field in every struct passing through that encoder gets the same scale and rounding mode. In financial code it's common to have heterogeneous Decimal fields in the same payload (e.g. price at scale 4, quantity at scale 0, tax_rate at scale 6) - with an encoder-level setting you'd need separate encoders per shape, which defeats most of the ergonomic win.

An alternative would be to attach quantization to the type via Annotated[Decimal, Meta(...)], e.g.

Price = Annotated[Decimal, Meta(decimal_quantize="0.0001", decimal_rounding="ROUND_HALF_EVEN")]

That composes naturally with per-field configuration, lives next to the type where the constraint is logically defined, and matches how gt/ge/pattern etc. already work today. The downside is more plumbing through TypeNode instead of one encoder kwarg.

Did you consider the Meta-based approach? If so, what made you land on encoder-level? Both have trade-offs and I'd rather get the API right before merge.

cc @jcrist @ofek — this expands the encoder API surface, so I'd like your read on whether the encoder-kwarg shape is the one we want, or whether Meta-based quantization is preferable.

jcrist · 2026-04-10T17:13:30Z

Instead of two new options for quantization, how about adding a single decimal_format option to Encoder? This would take either a string to pass to quantize (something like decimal.quantize(Decimal(decimal_format))), or a callable that takes in the decimal and returns a new value to encode. A few examples:

# Uses default rounding
enc = Encoder(decimal_format="0.0001")

# Custom rounding
enc = Encoder(decimal_format=lambda d: d.quanitize(decimal.Decimal("0.001"), "ROUND_DOWN"))

I like this since it's more flexible, and also only adds a single new option. Otherwise I'd worry about other users needing further customization, resulting in a number of decimal_* kwargs.

I wouldn't expect a callable here to have a perf cost - calling into python here is negligible, most of the time will be in the quantize call itself.

Did you consider the Meta-based approach? If so, what made you land on encoder-level? Both have trade-offs and I'd rather get the API right before merge.

In msgspec, (currently) encoding doesn't have any type-level information, it only has the values. This means customization for encoding cannot rely on information in annotations, it has to rely on the actual object instances themselves. This is admittedly less flexible in cases where you might want to encode different values differently, but keeps the encoder simple and supports values that exist outside of containers with attached annotations (e.g. encode(decimal_object) wouldn't have annotations, but encode(struct_with_a_decimal_field) would).

For now a single setting on an Encoder is both straightforward to implement, and matches the current conventions.

NyanFisher · 2026-04-13T09:17:16Z

@Siyet @jcrist Hello! Thanks for the review!

@Siyet

Did you consider the Meta-based approach? If so, what made you land on encoder-level? Both have trade-offs and I'd rather get the API right before merge.

I hadn't considered using Meta, but I think that approach would result in a large number of TypeNode. I work at a bank and know that a single Price isn't enough, since it's too general a concept. But it's a good idea for future 😃

@jcrist

Instead of two new options for quantization, how about adding a single decimal_format option to Encoder?

I like this idea, but the decimal_format parameter already exists. If you plan to extend the interface with additional types, I don’t think this is the best solution, as it will confuse users. I suggest using a separate additional parameter called decimal_quantize with the types Decimal | Callable[[Decimal], Decimal], which would be responsible exclusively for quantization.
This way, we’ll retain the ability to convert Decimal to “string”/“number”, add quantization, and maintain backward compatibility.

Siyet · 2026-04-15T10:04:03Z

After thinking it through I'm coming around to @jcrist's single-kwarg shape. One slot for everything is, in my view, the right call here.

Encoder(decimal_format=lambda d: d.quantize(Decimal("0.001"), ROUND_DOWN))  # custom
Encoder(decimal_format="string")                                            # existing
Encoder(decimal_format="number")                                            # existing

We could split it along dataclasses.field(default=..., default_factory=...) lines (value in one kwarg, callable in another), but that split exists specifically to disambiguate "the value is a callable" from "call this to produce the value", and neither "string" nor "number" is callable. Introducing a separate decimal_hook just to satisfy a pattern we do not actually need feels like overcomplicating the interface.

There is also the naming angle: decimal_format reads as a verb just as naturally as it reads as a noun ("how to format the decimal"), which makes "pass a callable that does the formatting" fit the name rather than fight it.

@NyanFisher regarding your concern about overloading an existing kwarg: the three shapes ("string" / "number" / callable) dispatch unambiguously on type (string vs. callable), so the dispatch logic in C stays simple and the user-facing docs just enumerate the three accepted shapes in one place.

NyanFisher · 2026-04-28T05:38:12Z

@Siyet

Please review this PR when you have a moment 🙂 I changed the implementation.

Siyet

Re-checked after the rework. CI is green across the matrix, design matches what we landed on. Ran some scenarios beyond the existing tests locally (WSL Ubuntu 22.04, Python 3.10, build at 4810273), three blockers inline.

Plus docs: docs/supported-types.rst:595-606 only mentions 'string'/'number', would be good to add a callable example covering the use case from #848.

Nit: test_encoder_decimal_callable_raise_error_if_fn_return_decimal should use match="must not return a Decimal" to pin the message rather than any TypeError.

NyanFisher · 2026-04-29T07:07:15Z

@Siyet Hello!

I have corrected the comments. Please re-check this PR when you have a moment.

NyanFisher · 2026-05-20T09:45:05Z

Hi @Siyet ! Just gently bumping this pull request. Let me know if there’s anything I can fix or rebase to help get it merged! 🙂

Siyet · 2026-06-15T19:52:55Z

Looks good. A few non-blocking things before merge:

Needs a rebase (currently conflicting with main).
Test gap: test_encoder_decimal_callable_convert_value_to_string and ..._into_different_types take the proto fixture but hardcode msgspec.json.Encoder, so the msgpack callable encode path is not actually exercised. Switching them to proto.Encoder would cover it.
Minor: the Encoder/JSONEncoder docstrings only say "should return a supported object"; worth a line that the result must not contain a Decimal (it is in supported-types.rst but not the docstrings). The decimal_format=1 ValueError -> TypeError change is also worth a CHANGELOG note.

NyanFisher · 2026-06-16T19:22:49Z

@Siyet, I have corrected the comments.

The decimal_format=1 ValueError -> TypeError

I was thinking, maybe we should keep ValueError for backward compatibility? I can swap the conditions.

if (decimal_format == NULL) {
    ...
}
else if (PyCallable_Check(decimal_format)) {
    ...
}
else {
    bool ok = false;
    if (PyUnicode_CheckExact(decimal_format)) {
        if (PyUnicode_CompareWithASCIIString(decimal_format, "string") == 0) {
            self->decimal_format = DECIMAL_FORMAT_STRING;
            ok = true;
        }
        else if (PyUnicode_CompareWithASCIIString(decimal_format, "number") == 0) {
            self->decimal_format = DECIMAL_FORMAT_NUMBER;
            ok = true;
        }
        if (!ok) {
            PyErr_Format(
                PyExc_ValueError,
                "`decimal_format` must be 'string', 'number', or a callable, got %R",
                decimal_format
            );
            return -1;
        }
    }
}

What do you think?

NyanFisher · 2026-06-29T15:28:33Z

@sobolevn, hello! Could you please join the review of this PR? I want to finish it. 🙂

codspeed-hq · 2026-06-29T16:00:27Z

Merging this PR will improve performance by 11.74%

⚡ 5 improved benchmarks
✅ 134 untouched benchmarks
⏩ 135 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
⚡	Simulation	`test_decode_type[arm-msgpack-False]`	849.6 µs	741.6 µs	+14.56%
⚡	Simulation	`test_decode_type[arm-msgpack-True]`	841 µs	754.9 µs	+11.4%
⚡	Simulation	`test_decode_type[arm-msgpack-enum]`	857.4 µs	770.1 µs	+11.34%
⚡	Simulation	`test_decode_type[arm-msgpack-None]`	754.5 µs	679.1 µs	+11.1%
⚡	Simulation	`test_decode_type[arm-json-None]`	925.7 µs	839 µs	+10.33%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing NyanFisher:decimal-quantize (8c52acf) with main (4440c96)}

135 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

sobolevn

Thanks! Looks really promising!

sobolevn · 2026-06-29T16:02:15Z

            PyErr_Format(
                PyExc_ValueError,
-                "`decimal_format` must be 'string' or 'number', got %R",
+                "`decimal_format` must be 'string', 'number', or a callable, got %R",


Let's not repeat this error twice :)

Removed the duplication.

I have changed the terms to maintain backward compatibility. I wrote about it in a comment.

Yes, we should not change the error type here. Otherwise, we can break something for users.

sobolevn · 2026-06-29T16:15:10Z

+        ):
+            enc.encode(decimal.Decimal("1"))
+
+    def test_encoder_decimal_callable_raise_error(self, proto):


Question (maybe for another PR): do we explicitly test _pydecimal types? where decimal.c was not available / compiled?

Do you mean such tests?

@pytest.fixture def force_pydecimal(monkeypatch): monkeypatch.delitem(sys.modules, "_decimal", raising=False) monkeypatch.delitem(sys.modules, "decimal", raising=False) monkeypatch.setitem(sys.modules, "_decimal", None) if "decimal" in sys.modules: importlib.reload(sys.modules["decimal"]) yield monkeypatch.delitem(sys.modules, "_decimal", raising=False) if "decimal" in sys.modules: importlib.reload(sys.modules["decimal"]) def test_pydecimal(force_pydecimal): import decimal

I have never written such tests. 😃

We can do that in a separate PR.

- Add in_decimal_callable flag to EncoderState to track when we're inside a decimal_format callable - Raise TypeError if decimal_format callable returns a value containing a Decimal (would cause infinite recursion) - Add documentation with examples for decimal_format callable usage - Update stubs to allow Any return type from decimal_format callable

sobolevn

Awesome! Just one comment from me :)

sobolevn · 2026-06-30T17:40:51Z

+            PyErr_SetString(
+                PyExc_TypeError,
+                "callable returned a value containing a Decimal"
+            );
+            return -1;


Suggested change

PyErr_SetString(

PyExc_TypeError,

"callable returned a value containing a Decimal"

);

return -1;

return ms_decimal_format_error();

this can also be an inline function.

Thx! Fixed it.

sobolevn

Great feature! Thank you! Let's keep it open for a while to potentially get more feedback from others.

Siyet

Second review after @sobolevn's approval, and it's a solid one - LGTM.

What I checked on 8c52acf (deep pass, including an empirical run):

The recursion guard survived the post-review refactors: a callable returning a Decimal (direct or nested arbitrarily deep in dict/list/tuple, json and msgpack alike) raises the clean TypeError, never a RecursionError or crash. Exceptions from the callable propagate as-is, KeyboardInterrupt included.
Refcounting and GC are clean: Py_VISIT/Py_CLEAR on the callable are wired into both encoder types, encoder-in-a-cycle-with-its-callable collects, and 600k encodes / 300k error-path iterations / 100k encoder create-destroy cycles all show zero RSS growth.
All four EncoderState creation sites (encode, encode_into, encode_lines, module-level) initialize the new fields; string/number output is byte-identical to 0.21.1 for both protocols.
Stubs (_DecimalFormatSig) match runtime validation, getter returns the callable by identity, enc_hook is not consulted for Decimal in any mode.
All review threads are addressed in code; the one deferred item for the record is _pydecimal test coverage, agreed to land as a separate PR.

Two tiny wording nits, fine to fix in a follow-up push or leave as is:

The msgpack Encoder docstring and the supported-types.rst note both say the callable "must return a JSON-serializable value" - for msgpack that reads oddly; "a value encodable by the encoder" would cover both.
The docs warning says returning a Decimal "would cause infinite recursion" and then that a TypeError is raised - the guard means there's no recursion at all, the second half is what actually happens.

Since this is a feature, the final call is @ofek's per our merge policy - from my side it's ready.

Siyet · 2026-07-02T16:42:00Z

Correcting myself from the review above: I keep forgetting our maintainer pool has grown. With @sobolevn's approval and mine this PR has two maintainer reviews, and the API shape itself was originally suggested by @jcrist - that's plenty for a feature. No need to gate on one specific person, so merging with a clear conscience.

Thanks @NyanFisher for seeing this through!

Batches the changelog entries accumulated since the 0.21.1 release, following the usual pre-release batching practice (like msgspec#1020). Covers merged user-facing changes through msgspec#1109/msgspec#978/msgspec#1105; internal CI/benchmarking/test-infra changes are intentionally omitted. Also records the move to the msgspec GitHub organization. --------- Co-authored-by: Siyet <Siyet@users.noreply.github.com>

NyanFisher force-pushed the decimal-quantize branch from 2357ed5 to 076b58c Compare February 11, 2026 12:06

NyanFisher changed the title ~~Implement quantization for Decimal type when encode~~ Draft: Implement quantization for Decimal type when encode Feb 11, 2026

NyanFisher force-pushed the decimal-quantize branch 2 times, most recently from 2ee9741 to 31effee Compare February 11, 2026 13:51

NyanFisher changed the title ~~Draft: Implement quantization for Decimal type when encode~~ Implement quantization for Decimal type when encode Feb 11, 2026

NyanFisher force-pushed the decimal-quantize branch 2 times, most recently from 117001b to b5ff6b7 Compare February 11, 2026 15:32

NyanFisher force-pushed the decimal-quantize branch from b5ff6b7 to 09b8d01 Compare April 9, 2026 10:07

NyanFisher force-pushed the decimal-quantize branch from 09b8d01 to f1d9799 Compare April 27, 2026 17:36

NyanFisher closed this Apr 27, 2026

NyanFisher reopened this Apr 27, 2026

NyanFisher changed the title ~~Implement quantization for Decimal type when encode~~ Draft: Implement quantization for Decimal type when encode Apr 27, 2026

NyanFisher changed the title ~~Draft: Implement quantization for Decimal type when encode~~ Add callable support for decimal_format option Apr 28, 2026

NyanFisher force-pushed the decimal-quantize branch from 9485ce8 to 4810273 Compare April 28, 2026 05:26

Siyet reviewed Apr 28, 2026

View reviewed changes

Comment thread src/msgspec/_core.c

Comment thread src/msgspec/json.pyi Outdated

Comment thread src/msgspec/_core.c Outdated

NyanFisher force-pushed the decimal-quantize branch from e0fbd2a to b11ce2b Compare April 29, 2026 06:52

NyanFisher requested a review from Siyet April 29, 2026 07:16

NyanFisher force-pushed the decimal-quantize branch from c70348b to 06a3a3f Compare June 16, 2026 19:10

NyanFisher temporarily deployed to docs-preview June 16, 2026 19:11 — with GitHub Actions Inactive

NyanFisher force-pushed the decimal-quantize branch from 06a3a3f to 468dea0 Compare June 16, 2026 19:12

NyanFisher temporarily deployed to docs-preview June 16, 2026 19:14 — with GitHub Actions Inactive

sobolevn reviewed Jun 29, 2026

View reviewed changes

NyanFisher had a problem deploying to docs-preview June 30, 2026 17:04 — with GitHub Actions Failure

NyanFisher and others added 7 commits June 30, 2026 22:06

Add callable support for decimal_format option

3136dc4

Reduce the number of examples in the documentation

fd12197

Correct comments after the review

bad19c7

Add new _DecimalFormatSig type alias

fee188f

Flip the compare sign

1503574

Change the conditions for backward compatibility

a3f03ce

NyanFisher force-pushed the decimal-quantize branch from a2d7766 to a3f03ce Compare June 30, 2026 17:06

NyanFisher had a problem deploying to docs-preview June 30, 2026 17:06 — with GitHub Actions Failure

sobolevn reviewed Jun 30, 2026

View reviewed changes

Extract ms_decimal_format_error helper

045a25c

NyanFisher had a problem deploying to docs-preview June 30, 2026 19:02 — with GitHub Actions Failure

sobolevn approved these changes Jun 30, 2026

View reviewed changes

NyanFisher had a problem deploying to docs-preview July 2, 2026 13:24 — with GitHub Actions Failure

NyanFisher temporarily deployed to docs-preview July 2, 2026 14:48 — with GitHub Actions Inactive

Merge main

8c52acf

Siyet deployed to docs-preview July 2, 2026 15:16 — with GitHub Actions View deployment

Siyet mentioned this pull request Jul 2, 2026

CodSpeed: test_pickle_load[arm] is bimodal and randomly flags unrelated PRs #1110

Open

Siyet approved these changes Jul 2, 2026

View reviewed changes

Siyet added this pull request to the merge queue Jul 2, 2026

Merged via the queue into msgspec:main with commit 68f53a7 Jul 2, 2026
27 checks passed

Siyet mentioned this pull request Jul 3, 2026

Add changelog entries for changes since 0.21.1 #1112

Merged

Uh oh!

Conversation

NyanFisher commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of the problem solved by this PR

Changes implemented in this PR

Core Functionality

Validation & Safety

Type Hints

Examples

Rounding to 2 Decimal Places

MessagePack with Rounding

Error: Returning Decimal from Callable

Uh oh!

NyanFisher commented Apr 8, 2026

Uh oh!

Siyet commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jcrist commented Apr 10, 2026

Uh oh!

NyanFisher commented Apr 13, 2026

Uh oh!

Siyet commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NyanFisher commented Apr 28, 2026

Uh oh!

Siyet left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

NyanFisher commented Apr 29, 2026

Uh oh!

NyanFisher commented May 20, 2026

Uh oh!

Siyet commented Jun 15, 2026

Uh oh!

NyanFisher commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NyanFisher commented Jun 29, 2026

Uh oh!

codspeed-hq Bot commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will improve performance by 11.74%

Performance Changes

Footnotes

Uh oh!

sobolevn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sobolevn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sobolevn left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

NyanFisher commented Feb 11, 2026 •

edited

Loading

Siyet commented Apr 10, 2026 •

edited

Loading

Siyet commented Apr 15, 2026 •

edited

Loading

NyanFisher commented Jun 16, 2026 •

edited

Loading

codspeed-hq Bot commented Jun 29, 2026 •

edited

Loading

sobolevn left a comment •

edited

Loading