Conversation
Also adds tests
This takes an optional callable of the form:
```python
def ext_hook(code: int, data: memoryview) -> Any:
pass
```
If provided, this will be used to decode any extension types found in a
message (except those that are marked via type-annotation to remain as
`Ext` objects). Note that `data` is a memoryview into the larger message
buffer, minimizing data copies. To prevent the whole message buffer from
persisting in memory for too long, the user should ensure they don't
keep a reference to the `data` buffer in the decoded object.
Also renames `ExtType` to `Ext`.
Member
Author
|
Edit: this was due to python 3.8 seemingly not fully supporting vectorcall constructors for all builtin types. This worked for struct types since they used a custom metaclass (so the call was defined on the class's type), but doesn't work for vectorcall defined on the class itself until python 3.9 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds support for both encoding and decoding msgpack extension types. This comes with 2 additional apis:
msgspec.Exttype, representing the tuple of(code: int, data: bytes_like)that describes a msgpack extension.ext_hookparameter tomsgspec.Decoder/msgspec.decode.By default msgspec will deserialize all extension objects into
msgspec.Extobjects. However, if a user specifies aext_hooktomsgspec.Decoder/msgspec.decode, this callback will be used instead to deserialize the extension type. Together with the existingdefaulthook inmsgspec.Encoder/msgspec.encode, this lets users fully add extra types to be serialized/deserialized.For example, say we wanted to define a new extension type with code
42that contains a buffer of data serialized viapickle. If we wanted to apply this to all non-builtin types serialized by msgspec, we could do this as follows:Note that
ext_hookhas the signatureext_hook(code: int, data: memoryview) -> Any. Heredatais a memoryview into larger message buffer. This lets us avoid an extra copy when forwarding the data buffer toext_hook, but it means that if the returned object holds on to a reference to that buffer (instead of doing a copy itself) then the larger message will persist in memory until that object is freed. For most use cases ofext_hookI wouldn't expect this issue to come up, but it's worth noting. The performance improvement from avoiding the copy is worth it to add a tiny rare footgun IMO.Fixes #26.
Still a few todos:
Extaccept amemoryviewfordata, rather than justbytesorbytearray