Compatibility with SQLA2.1 - #61229
Conversation
|
According to an old discussion in the SQLA repo,
In SQLA2.1 this api was finally marked as private and
Thoughts? |
|
Thanks for runnig it @Dev-iL -> really cool you are doing it !
We only seem to use from sqlalchemy_utils import UUIDTypeI guess in sqlalchemy 2.+ there should be already an alternative type we can use ? https://docs.sqlalchemy.org/en/21/core/custom_types.html#backend-agnostic-guid-type
Not sure about compatibility, but I would imagine it should be quite easy to replace it |
|
@potiuk Thanks for the tip! I think we might be able to remove both of these dependencies: |
02201e3 to
1fcfcb9
Compare
|
@jscheffl Do you see any issue with modifying the old migrations so we can move to new ORM type hints? The DB should stay the same (if I did everything right). The SQLA docs mention that:
So here's what I think:
|
1fcfcb9 to
01552b8
Compare
I do not see an issue in this except (1) testing and (2) review. But all looks like simple subsitutes. So OK for me. |
How to test this besides adding the |
Mhm, yeah taking a Airflow 2.7.0 setup and run the migration :-) Such integrative tests are not in CI in my view. |
Isn't that what the migration tests (e.g. this) do? |
Ah... yeah ... 🤦 |
26174d5 to
a88ad9e
Compare
SQLAlchemy 2.1 deprecated the `noload` lazy loading strategy (sqlalchemy/sqlalchemy#11045). `noload` silently returns `None`/empty collections — essentially incorrect results — and will be removed in a future release. This PR replaces all 5 occurrences of `lazy="noload"` with `lazy="raise"`, which raises `InvalidRequestError` if the relationship is accessed without an explicit eager load (e.g. `joinedload`). All affected relationships are already properly loaded via `joinedload()` wherever they're accessed, so this is a safe drop-in that also catches missing eager loads at development time instead of silently returning `None`. Two callers needed fixes to work correctly with `lazy="raise"`: - `TaskInstance.rendered_task_instance_fields` and `TaskInstance.hitl_detail` needed `passive_deletes=True` to tell SQLAlchemy to rely on the DB-level `ON DELETE CASCADE` rather than attempting ORM-level cascade processing (which would fail since FK columns are also PK columns on RTIF, and `lazy="raise"` prevents the ORM from loading the collection to clear them). - The HITL API endpoints needed `joinedload(TI.rendered_task_instance_fields)` added to their queries, since `TaskInstanceResponse` accesses `rendered_task_instance_fields` during Pydantic serialization. The `get_hitl_detail` endpoint also needed an explicit `model_validate()` call so serialization happens while the session is still active. **Changed models:** - `Log.task_instance` - `TaskInstance.rendered_task_instance_fields` - `TaskInstance.hitl_detail` - `TaskInstanceHistory.hitl_detail` - `XComModel.task` related: apache#61229
SQLAlchemy 2.1 deprecated the `noload` lazy loading strategy (sqlalchemy/sqlalchemy#11045). `noload` silently returns `None`/empty collections — essentially incorrect results — and will be removed in a future release. This PR replaces all 5 occurrences of `lazy="noload"` with `lazy="raise"`, which raises `InvalidRequestError` if the relationship is accessed without an explicit eager load (e.g. `joinedload`). All affected relationships are already properly loaded via `joinedload()` wherever they're accessed, so this is a safe drop-in that also catches missing eager loads at development time instead of silently returning `None`. Two callers needed fixes to work correctly with `lazy="raise"`: - `TaskInstance.rendered_task_instance_fields` and `TaskInstance.hitl_detail` needed `passive_deletes=True` to tell SQLAlchemy to rely on the DB-level `ON DELETE CASCADE` rather than attempting ORM-level cascade processing (which would fail since FK columns are also PK columns on RTIF, and `lazy="raise"` prevents the ORM from loading the collection to clear them). - The HITL API endpoints needed `joinedload(TI.rendered_task_instance_fields)` added to their queries, since `TaskInstanceResponse` accesses `rendered_task_instance_fields` during Pydantic serialization. The `get_hitl_detail` endpoint also needed an explicit `model_validate()` call so serialization happens while the session is still active. **Changed models:** - `Log.task_instance` - `TaskInstance.rendered_task_instance_fields` - `TaskInstance.hitl_detail` - `TaskInstanceHistory.hitl_detail` - `XComModel.task` related: apache#61229
SQLAlchemy 2.1 deprecated the `noload` lazy loading strategy (sqlalchemy/sqlalchemy#11045). `noload` silently returns `None`/empty collections — essentially incorrect results — and will be removed in a future release. This PR replaces all 5 occurrences of `lazy="noload"` with `lazy="raise"`, which raises `InvalidRequestError` if the relationship is accessed without an explicit eager load (e.g. `joinedload`). All affected relationships are already properly loaded via `joinedload()` wherever they're accessed, so this is a safe drop-in that also catches missing eager loads at development time instead of silently returning `None`. Two callers needed fixes to work correctly with `lazy="raise"`: - `TaskInstance.rendered_task_instance_fields` and `TaskInstance.hitl_detail` needed `passive_deletes=True` to tell SQLAlchemy to rely on the DB-level `ON DELETE CASCADE` rather than attempting ORM-level cascade processing (which would fail since FK columns are also PK columns on RTIF, and `lazy="raise"` prevents the ORM from loading the collection to clear them). - The HITL API endpoints needed `joinedload(TI.rendered_task_instance_fields)` added to their queries, since `TaskInstanceResponse` accesses `rendered_task_instance_fields` during Pydantic serialization. The `get_hitl_detail` endpoint also needed an explicit `model_validate()` call so serialization happens while the session is still active. **Changed models:** - `Log.task_instance` - `TaskInstance.rendered_task_instance_fields` - `TaskInstance.hitl_detail` - `TaskInstanceHistory.hitl_detail` - `XComModel.task` related: apache#61229
SQLAlchemy 2.1 deprecated the `noload` lazy loading strategy (sqlalchemy/sqlalchemy#11045). `noload` silently returns `None`/empty collections — essentially incorrect results — and will be removed in a future release. This PR replaces all 5 occurrences of `lazy="noload"` with `lazy="raise"`, which raises `InvalidRequestError` if the relationship is accessed without an explicit eager load (e.g. `joinedload`). All affected relationships are already properly loaded via `joinedload()` wherever they're accessed, so this is a safe drop-in that also catches missing eager loads at development time instead of silently returning `None`. Two callers needed fixes to work correctly with `lazy="raise"`: - `TaskInstance.rendered_task_instance_fields` and `TaskInstance.hitl_detail` needed `passive_deletes=True` to tell SQLAlchemy to rely on the DB-level `ON DELETE CASCADE` rather than attempting ORM-level cascade processing (which would fail since FK columns are also PK columns on RTIF, and `lazy="raise"` prevents the ORM from loading the collection to clear them). - The HITL API endpoints needed `joinedload(TI.rendered_task_instance_fields)` added to their queries, since `TaskInstanceResponse` accesses `rendered_task_instance_fields` during Pydantic serialization. The `get_hitl_detail` endpoint also needed an explicit `model_validate()` call so serialization happens while the session is still active. **Changed models:** - `Log.task_instance` - `TaskInstance.rendered_task_instance_fields` - `TaskInstance.hitl_detail` - `TaskInstanceHistory.hitl_detail` - `XComModel.task` related: apache#61229
SQLAlchemy 2.1 deprecated the `noload` lazy loading strategy (sqlalchemy/sqlalchemy#11045). `noload` silently returns `None`/empty collections — essentially incorrect results — and will be removed in a future release. This PR replaces all 5 occurrences of `lazy="noload"` with `lazy="raise"`, which raises `InvalidRequestError` if the relationship is accessed without an explicit eager load (e.g. `joinedload`). All affected relationships are already properly loaded via `joinedload()` wherever they're accessed, so this is a safe drop-in that also catches missing eager loads at development time instead of silently returning `None`. Two callers needed fixes to work correctly with `lazy="raise"`: - `TaskInstance.rendered_task_instance_fields` and `TaskInstance.hitl_detail` needed `passive_deletes=True` to tell SQLAlchemy to rely on the DB-level `ON DELETE CASCADE` rather than attempting ORM-level cascade processing (which would fail since FK columns are also PK columns on RTIF, and `lazy="raise"` prevents the ORM from loading the collection to clear them). - The HITL API endpoints needed `joinedload(TI.rendered_task_instance_fields)` added to their queries, since `TaskInstanceResponse` accesses `rendered_task_instance_fields` during Pydantic serialization. The `get_hitl_detail` endpoint also needed an explicit `model_validate()` call so serialization happens while the session is still active. **Changed models:** - `Log.task_instance` - `TaskInstance.rendered_task_instance_fields` - `TaskInstance.hitl_detail` - `TaskInstanceHistory.hitl_detail` - `XComModel.task` related: apache#61229
) SQLAlchemy 2.1 deprecated the `noload` lazy loading strategy (sqlalchemy/sqlalchemy#11045). `noload` silently returns `None`/empty collections — essentially incorrect results — and will be removed in a future release. This PR replaces all 5 occurrences of `lazy="noload"` with `lazy="raise"`, which raises `InvalidRequestError` if the relationship is accessed without an explicit eager load (e.g. `joinedload`). All affected relationships are already properly loaded via `joinedload()` wherever they're accessed, so this is a safe drop-in that also catches missing eager loads at development time instead of silently returning `None`. Two callers needed fixes to work correctly with `lazy="raise"`: - `TaskInstance.rendered_task_instance_fields` and `TaskInstance.hitl_detail` needed `passive_deletes=True` to tell SQLAlchemy to rely on the DB-level `ON DELETE CASCADE` rather than attempting ORM-level cascade processing (which would fail since FK columns are also PK columns on RTIF, and `lazy="raise"` prevents the ORM from loading the collection to clear them). - The HITL API endpoints needed `joinedload(TI.rendered_task_instance_fields)` added to their queries, since `TaskInstanceResponse` accesses `rendered_task_instance_fields` during Pydantic serialization. The `get_hitl_detail` endpoint also needed an explicit `model_validate()` call so serialization happens while the session is still active. **Changed models:** - `Log.task_instance` - `TaskInstance.rendered_task_instance_fields` - `TaskInstance.hitl_detail` - `TaskInstanceHistory.hitl_detail` - `XComModel.task` related: #61229
…che#64262) SQLAlchemy 2.1 deprecated the `noload` lazy loading strategy (sqlalchemy/sqlalchemy#11045). `noload` silently returns `None`/empty collections — essentially incorrect results — and will be removed in a future release. This PR replaces all 5 occurrences of `lazy="noload"` with `lazy="raise"`, which raises `InvalidRequestError` if the relationship is accessed without an explicit eager load (e.g. `joinedload`). All affected relationships are already properly loaded via `joinedload()` wherever they're accessed, so this is a safe drop-in that also catches missing eager loads at development time instead of silently returning `None`. Two callers needed fixes to work correctly with `lazy="raise"`: - `TaskInstance.rendered_task_instance_fields` and `TaskInstance.hitl_detail` needed `passive_deletes=True` to tell SQLAlchemy to rely on the DB-level `ON DELETE CASCADE` rather than attempting ORM-level cascade processing (which would fail since FK columns are also PK columns on RTIF, and `lazy="raise"` prevents the ORM from loading the collection to clear them). - The HITL API endpoints needed `joinedload(TI.rendered_task_instance_fields)` added to their queries, since `TaskInstanceResponse` accesses `rendered_task_instance_fields` during Pydantic serialization. The `get_hitl_detail` endpoint also needed an explicit `model_validate()` call so serialization happens while the session is still active. **Changed models:** - `Log.task_instance` - `TaskInstance.rendered_task_instance_fields` - `TaskInstance.hitl_detail` - `TaskInstanceHistory.hitl_detail` - `XComModel.task` related: apache#61229
…che#64262) SQLAlchemy 2.1 deprecated the `noload` lazy loading strategy (sqlalchemy/sqlalchemy#11045). `noload` silently returns `None`/empty collections — essentially incorrect results — and will be removed in a future release. This PR replaces all 5 occurrences of `lazy="noload"` with `lazy="raise"`, which raises `InvalidRequestError` if the relationship is accessed without an explicit eager load (e.g. `joinedload`). All affected relationships are already properly loaded via `joinedload()` wherever they're accessed, so this is a safe drop-in that also catches missing eager loads at development time instead of silently returning `None`. Two callers needed fixes to work correctly with `lazy="raise"`: - `TaskInstance.rendered_task_instance_fields` and `TaskInstance.hitl_detail` needed `passive_deletes=True` to tell SQLAlchemy to rely on the DB-level `ON DELETE CASCADE` rather than attempting ORM-level cascade processing (which would fail since FK columns are also PK columns on RTIF, and `lazy="raise"` prevents the ORM from loading the collection to clear them). - The HITL API endpoints needed `joinedload(TI.rendered_task_instance_fields)` added to their queries, since `TaskInstanceResponse` accesses `rendered_task_instance_fields` during Pydantic serialization. The `get_hitl_detail` endpoint also needed an explicit `model_validate()` call so serialization happens while the session is still active. **Changed models:** - `Log.task_instance` - `TaskInstance.rendered_task_instance_fields` - `TaskInstance.hitl_detail` - `TaskInstanceHistory.hitl_detail` - `XComModel.task` related: apache#61229
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Updates CI and tests to accommodate SQLAlchemy 2.1 pre-release installs, while working around known incompatibilities in snowflake-sqlalchemy.
Changes:
- Force-install SQLAlchemy 2.1 pre-release and
snowflake-sqlalchemyfrom Gitmainin “upgrade SQLAlchemy” CI paths. - Skip/ignore Snowflake-related tests when
snowflake-sqlalchemybreaks under SQLAlchemy 2.1. - Filter/skip known failing import errors in DAG serialization and example DAG import tests.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/docker/entrypoint_ci.sh | Adds prerelease SQLAlchemy + git snowflake-sqlalchemy install step in upgrade check. |
| Dockerfile.ci | Mirrors the CI install step for SQLAlchemy 2.1 prerelease + git snowflake-sqlalchemy. |
| providers/snowflake/tests/conftest.py | Ignores Snowflake provider tests when snowflake-sqlalchemy import fails under SQLA 2.1. |
| providers/common/sql/tests/unit/common/sql/operators/test_sql.py | Skips Snowflake-specific tests when snowflake-sqlalchemy is incompatible/unimportable. |
| airflow-core/tests/unit/serialization/test_dag_serialization.py | Filters a known Snowflake/SQLA 2.1 import error from assertions. |
| airflow-core/tests/unit/always/test_example_dags.py | Skips example DAG import checks when Snowflake/SQLA mismatch causes a known error. |
a88ad9e to
5d42c11
Compare
|
2.1.0b2 was just released. |
5d42c11 to
f4084e4
Compare
|
@Dev-iL This draft PR has had no activity for 7 weeks. Closing to keep the queue clean. You are welcome to reopen and continue when you're ready. If you'd like to pick it back up, please rebase onto the current Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you resume work, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you. |
f4084e4 to
4d86847
Compare
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. |
4d86847 to
75221dc
Compare
- Ignore import error in `TestStringifiedDAGs`, `TestSQLCheckOperatorDbHook`, and `test_should_be_importable`. - Skip all snowflake provider tests (since the foundational `SnowflakeHook` is not importable).
75221dc to
fe9cd17
Compare
This PR is intended to ensure compatibility with SQLA2.1.
The plan is as follows:
Latest SQLAlchemy test: coreandLatest SQLAlchemy test: providerstasks.After the 2.1 release: monitor for dependencies holding us back.
ScalarAttributeImplon SQLA2.1 kvesteri/sqlalchemy-utils#804psycopg2in driverless postgres URLs #62159lazy="noload"withlazy="raise"#64262Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.