Skip to content

from_builtins, but for objects? (e.g instantiate from SQLAlchemy models) #416

Description

@dmckeone

Description

Is there an equivalent to from_builtins, but for objects? Or maybe msgspec.obj.encode / msgspec.obj.decode ?

I'd like to be able to encode/decode into regular Python objects, specifically SQLAlchemy.

Given:

tables.py

from sqlalchemy import ForeignKey
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship
from typing import List


# SQLAlchemy 2.0 Models
class Base(DeclarativeBase):
    pass


class GrommetTable(Base):
    __tablename__ = 'grommet'

    id: Mapped[int] = mapped_column(primary_key=True)
    name: Mapped[str]
    widget_id: Mapped[int] = mapped_column(ForeignKey("widget.id"))

    widget: Mapped["WidgetTable"] = relationship(back_populates="grommets")


class WidgetTable(Base):
    __tablename__ = 'widget'

    id: Mapped[int] = mapped_column(primary_key=True)
    name: Mapped[str]

    grommets: Mapped[List["GrommetTable"]] = relationship(back_populates="widget")

and:

models.py

from msgspec import Struct


class Grommet(Struct):
    id: int
    name: str
    widget_id: int


class Widget(Struct):
    id: int
    name: str
    grommets: List[Grommet]

and (create test data):

from sqlalchemy import create_engine
from sqlalchemy.orm import Session

from models import Grommet, Widget
from tables import GrommetTable, WidgetTable

engine = create_engine("sqlite+pysqlite:///:memory:", echo=True)

# Create table structure
Base.metadata.create_all(engine)

# Create demo data
g1 = GrommetTable(id=1, name="Grommet 1")
g2 = GrommetTable(id=2, name="Grommet 2")
g3 = GrommetTable(id=3, name="Grommet 3")
g4 = GrommetTable(id=4, name="Grommet 4")
g5 = GrommetTable(id=5, name="Grommet 5")
g6 = GrommetTable(id=6, name="Grommet 6")

w1 = WidgetTable(id=1, name="Widget 1", grommets=[g1, g2, g3])
w2 = WidgetTable(id=2, name="Widget 2", grommets=[g4, g5, g6])

with Session(engine) as session:
    session.add_all([w1, w2])
    session.commit()

It would be great to be able to do something like the below:

from sqlalchemy import select
from sqlalchemy.orm import selectinload

from models import Widget
from tables import WidgetTable

with Session(engine) as session:
    query = select(WidgetTable).options(selectinload(WidgetTable.grommets))
    result = session.execute(query)

    for widget in result.scalars():
        widget = from_obj(widget, Widget)   # NEW FEATURE
        print(widget)

Output:

Widget(id=1, name='Widget 1', grommets=[Grommet(id=1, name='Grommet 1', widget_id=1), Grommet(id=2, name='Grommet 2', widget_id=1), Grommet(id=3, name='Grommet 3', widget_id=1)])
Widget(id=2, name='Widget 2', grommets=[Grommet(id=4, name='Grommet 4', widget_id=2), Grommet(id=5, name='Grommet 5', widget_id=2), Grommet(id=6, name='Grommet 6', widget_id=2)])

The current way to do this would be something like the below implementation, which does work, but feels like there should be something more elegant.

from msgspec import from_builtins
from msgspec.inspect import ListType, type_info

def from_obj(obj, type, _data_only=False):
    data = {}
    for field in type_info(type).fields:
        if isinstance(field.type, ListType):
            value = [from_obj(o, field.type.item_type.cls, _data_only=True) for o in getattr(obj, field.name)]
        else:
            value = getattr(obj, field.name)

        data[field.name] = value

    if _data_only:
        return data
    else:
        return from_builtins(data, type)

I think it's most closely related to the Encode/Decode because the input could have more attributes than the Struct, just as JSON input could have more keys than the Struct when utilizing Schema Evolution

This feature would be useful because it allows a common, very fast internal data structure to be used when reading/writing from common data serialization formats (json, toml, yaml msgpack), but also from a large number of common data storage engines (SQLite, MySQL / MariaDB, PostgreSQL, etc..)

Pydantic v1 has a similar system with from_orm(), which I believe has now been replaced with model_validate() in v2.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions