Skip to content

Add Extension Type support - #31

Merged
jcrist merged 12 commits into
masterfrom
exttype
Jun 12, 2021
Merged

Add Extension Type support#31
jcrist merged 12 commits into
masterfrom
exttype

Conversation

@jcrist

@jcrist jcrist commented Jun 11, 2021

Copy link
Copy Markdown
Member

This adds support for both encoding and decoding msgpack extension types. This comes with 2 additional apis:

  • A new msgspec.Ext type, representing the tuple of (code: int, data: bytes_like) that describes a msgpack extension.
  • A new ext_hook parameter to msgspec.Decoder/msgspec.decode.

By default msgspec will deserialize all extension objects into msgspec.Ext objects. However, if a user specifies a ext_hook to msgspec.Decoder/msgspec.decode, this callback will be used instead to deserialize the extension type. Together with the existing default hook in msgspec.Encoder/msgspec.encode, this lets users fully add extra types to be serialized/deserialized.

For example, say we wanted to define a new extension type with code 42 that contains a buffer of data serialized via pickle. If we wanted to apply this to all non-builtin types serialized by msgspec, we could do this as follows:

import msgspec
import pickle
from typing import Any

def default(obj: Any) -> Any:
    """Serialize the object using pickle, and mark it as an extension type with code 42"""
    return msgspec.Ext(42, pickle.dumps(obj))

def ext_hook(code: int, data: memoryview) -> Any:
    if code == 42:
        return pickle.loads(data)
    raise TypeError(f"Unknown extension type with code {code}")

enc = msgspec.Encoder(default=default)
dec = msgspec.Decoder(ext_hook=ext_hook)

buf = enc.encode(range)  # functions normally aren't serializable with msgspec
out = dec.decode(buf)
assert out == range

Note that ext_hook has the signature ext_hook(code: int, data: memoryview) -> Any. Here data is a memoryview into larger message buffer. This lets us avoid an extra copy when forwarding the data buffer to ext_hook, but it means that if the returned object holds on to a reference to that buffer (instead of doing a copy itself) then the larger message will persist in memory until that object is freed. For most use cases of ext_hook I wouldn't expect this issue to come up, but it's worth noting. The performance improvement from avoiding the copy is worth it to add a tiny rare footgun IMO.

Fixes #26.

Still a few todos:

  • Make Ext accept a memoryview for data, rather than just bytes or bytearray
  • Add docs

jcrist added 8 commits June 9, 2021 08:52
This takes an optional callable of the form:

```python
def ext_hook(code: int, data: memoryview) -> Any:
    pass
```

If provided, this will be used to decode any extension types found in a
message (except those that are marked via type-annotation to remain as
`Ext` objects). Note that `data` is a memoryview into the larger message
buffer, minimizing data copies. To prevent the whole message buffer from
persisting in memory for too long, the user should ensure they don't
keep a reference to the `data` buffer in the decoded object.

Also renames `ExtType` to `Ext`.
@jcrist jcrist mentioned this pull request Jun 11, 2021
@jcrist

jcrist commented Jun 12, 2021

Copy link
Copy Markdown
Member Author

Hmmm, looks like there's a segfault somewhere, haven't hit that locally. Will debug further later.

Edit: this was due to python 3.8 seemingly not fully supporting vectorcall constructors for all builtin types. This worked for struct types since they used a custom metaclass (so the call was defined on the class's type), but doesn't work for vectorcall defined on the class itself until python 3.9

jcrist added 4 commits June 11, 2021 21:54
Vectorcall for constructors doesn't work universally in python 3.8
(afaict). Dropping for now.
@jcrist jcrist changed the title [WIP] Add Extension Type support Add Extension Type support Jun 12, 2021
@jcrist
jcrist merged commit 7107fef into master Jun 12, 2021
@jcrist
jcrist deleted the exttype branch June 12, 2021 23:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEA] A fallback mechanism

1 participant