Description
Is there an equivalent to from_builtins, but for objects? Or maybe msgspec.obj.encode / msgspec.obj.decode ?
I'd like to be able to encode/decode into regular Python objects, specifically SQLAlchemy.
Given:
tables.py
from sqlalchemy import ForeignKey
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship
from typing import List
# SQLAlchemy 2.0 Models
class Base(DeclarativeBase):
pass
class GrommetTable(Base):
__tablename__ = 'grommet'
id: Mapped[int] = mapped_column(primary_key=True)
name: Mapped[str]
widget_id: Mapped[int] = mapped_column(ForeignKey("widget.id"))
widget: Mapped["WidgetTable"] = relationship(back_populates="grommets")
class WidgetTable(Base):
__tablename__ = 'widget'
id: Mapped[int] = mapped_column(primary_key=True)
name: Mapped[str]
grommets: Mapped[List["GrommetTable"]] = relationship(back_populates="widget")
and:
models.py
from msgspec import Struct
class Grommet(Struct):
id: int
name: str
widget_id: int
class Widget(Struct):
id: int
name: str
grommets: List[Grommet]
and (create test data):
from sqlalchemy import create_engine
from sqlalchemy.orm import Session
from models import Grommet, Widget
from tables import GrommetTable, WidgetTable
engine = create_engine("sqlite+pysqlite:///:memory:", echo=True)
# Create table structure
Base.metadata.create_all(engine)
# Create demo data
g1 = GrommetTable(id=1, name="Grommet 1")
g2 = GrommetTable(id=2, name="Grommet 2")
g3 = GrommetTable(id=3, name="Grommet 3")
g4 = GrommetTable(id=4, name="Grommet 4")
g5 = GrommetTable(id=5, name="Grommet 5")
g6 = GrommetTable(id=6, name="Grommet 6")
w1 = WidgetTable(id=1, name="Widget 1", grommets=[g1, g2, g3])
w2 = WidgetTable(id=2, name="Widget 2", grommets=[g4, g5, g6])
with Session(engine) as session:
session.add_all([w1, w2])
session.commit()
It would be great to be able to do something like the below:
from sqlalchemy import select
from sqlalchemy.orm import selectinload
from models import Widget
from tables import WidgetTable
with Session(engine) as session:
query = select(WidgetTable).options(selectinload(WidgetTable.grommets))
result = session.execute(query)
for widget in result.scalars():
widget = from_obj(widget, Widget) # NEW FEATURE
print(widget)
Output:
Widget(id=1, name='Widget 1', grommets=[Grommet(id=1, name='Grommet 1', widget_id=1), Grommet(id=2, name='Grommet 2', widget_id=1), Grommet(id=3, name='Grommet 3', widget_id=1)])
Widget(id=2, name='Widget 2', grommets=[Grommet(id=4, name='Grommet 4', widget_id=2), Grommet(id=5, name='Grommet 5', widget_id=2), Grommet(id=6, name='Grommet 6', widget_id=2)])
The current way to do this would be something like the below implementation, which does work, but feels like there should be something more elegant.
from msgspec import from_builtins
from msgspec.inspect import ListType, type_info
def from_obj(obj, type, _data_only=False):
data = {}
for field in type_info(type).fields:
if isinstance(field.type, ListType):
value = [from_obj(o, field.type.item_type.cls, _data_only=True) for o in getattr(obj, field.name)]
else:
value = getattr(obj, field.name)
data[field.name] = value
if _data_only:
return data
else:
return from_builtins(data, type)
I think it's most closely related to the Encode/Decode because the input could have more attributes than the Struct, just as JSON input could have more keys than the Struct when utilizing Schema Evolution
This feature would be useful because it allows a common, very fast internal data structure to be used when reading/writing from common data serialization formats (json, toml, yaml msgpack), but also from a large number of common data storage engines (SQLite, MySQL / MariaDB, PostgreSQL, etc..)
Pydantic v1 has a similar system with from_orm(), which I believe has now been replaced with model_validate() in v2.
Description
Is there an equivalent to
from_builtins, but for objects? Or maybemsgspec.obj.encode/msgspec.obj.decode?I'd like to be able to encode/decode into regular Python objects, specifically SQLAlchemy.
Given:
tables.pyand:
models.pyand (create test data):
It would be great to be able to do something like the below:
Output:
The current way to do this would be something like the below implementation, which does work, but feels like there should be something more elegant.
I think it's most closely related to the Encode/Decode because the input could have more attributes than the
Struct, just as JSON input could have more keys than theStructwhen utilizing Schema EvolutionThis feature would be useful because it allows a common, very fast internal data structure to be used when reading/writing from common data serialization formats (
json,toml,yamlmsgpack), but also from a large number of common data storage engines (SQLite,MySQL/MariaDB,PostgreSQL, etc..)Pydantic v1 has a similar system with from_orm(), which I believe has now been replaced with model_validate() in v2.