Fix DatabricksSqlHook sqlalchemy_url missing http_path from connection extra by Vamsi-klu · Pull Request #69747 · apache/airflow

Vamsi-klu · 2026-07-11T06:53:29Z

Problem

DatabricksSqlHook.sqlalchemy_url and get_uri() did not include http_path when that path was configured through the Databricks connection extra JSON. That is a common UI configuration path, and it made SQLAlchemy usage fail even though get_conn() could connect.

What Changed

Added shared http_path resolution for SQLAlchemy URL generation.
Kept sqlalchemy_url offline-safe by allowing explicit http_path and connection-extra http_path, but not making a REST API call for sql_endpoint_name.
Added a follow-up commit that preserves the existing get_conn() precedence: explicit http_path, then sql_endpoint_name, then connection extra.
Added tests for endpoint precedence and offline SQLAlchemy URL behavior.

Why The Follow-Up Commit Exists

The feedback was correct: the first helper implementation resolved connection extras before named SQL endpoints, which would have changed get_conn() behavior. The follow-up keeps the SQLAlchemy fix while preserving the established connection precedence.

Impact

SQLAlchemy URL and URI generation now work when http_path lives in the Databricks connection extra.
get_conn() continues to prefer a named SQL endpoint over connection-extra http_path when both are provided.
sqlalchemy_url does not perform a network/API lookup.

Fixes: #69031

Testing

UV_CACHE_DIR=/tmp/uv-cache-69747-databricks uv run --python 3.11 --project providers/databricks ruff format providers/databricks/src/airflow/providers/databricks/hooks/databricks_sql.py providers/databricks/tests/unit/databricks/hooks/test_databricks_sql.py
UV_CACHE_DIR=/tmp/uv-cache-69747-databricks uv run --python 3.11 --project providers/databricks ruff check --fix providers/databricks/src/airflow/providers/databricks/hooks/databricks_sql.py providers/databricks/tests/unit/databricks/hooks/test_databricks_sql.py
AIRFLOW_HOME=/tmp/airflow-home-69747 UV_CACHE_DIR=/tmp/uv-cache-69747-databricks uv run --python 3.11 --project providers/databricks pytest providers/databricks/tests/unit/databricks/hooks/test_databricks_sql.py::test_sqlachemy_url_property providers/databricks/tests/unit/databricks/hooks/test_databricks_sql.py::test_get_conn_prefers_sql_endpoint_name_over_connection_extra_http_path providers/databricks/tests/unit/databricks/hooks/test_databricks_sql.py::test_sqlalchemy_url_uses_connection_extra_http_path_without_endpoint_lookup -xvs
Result: 3 passed, 1 warning
Commit hooks passed for the follow-up commit.

Was generative AI tooling used to co-author this PR?

Yes — Codex (GPT-5)

Generated-by: Codex (GPT-5) following the guidelines

Drafted-by: Codex (GPT-5); reviewed by @Vamsi-klu before posting

Vamsi-klu · 2026-07-11T06:55:03Z

Reviewers: @potiuk @kaxil (providers) + databricks maintainers. Fixes http_path not propagated to sqlalchemy_url.

Fixes #69031

Drafted-by: Muse Spark 1.1; reviewed by @Vamsi-klu before posting

Vamsi-klu · 2026-07-11T07:46:50Z

Tested this PR branch locally in the Airflow UI with Breeze.

Scenario exercised:

Created a databricks connection named pr_69747_databricks_extra_http_path from the UI.
Used host dbc-pr69747.cloud.databricks.com and a dummy token only.
Put http_path only in Extra JSON: sql/protocolv1/o/1234567890123456/0123-456789-pr69747.
Triggered a scratch Dag that calls DatabricksSqlHook(databricks_conn_id="pr_69747_databricks_extra_http_path").get_uri().
No real Databricks credentials or network call were required for this check.

Result:

Dag run succeeded.
Task log shows the SQLAlchemy URI includes the encoded http_path from connection extra:

databricks://token:***@dbc-pr69747.cloud.databricks.com?http_path=sql%2Fprotocolv1%2Fo%2F1234567890123456%2F0123-456789-pr69747

Screenshots are hosted on a dedicated assets-only branch in the fork.

Connection Extra JSON configured

Connection saved and filtered in the UI

Scratch Dag run succeeded

Task log showing http_path in the URI

Drafted-by: Codex (GPT-5.5)

eladkal · 2026-07-12T08:02:46Z

cc @moomindani

Vamsi-klu · 2026-07-12T09:10:55Z

@moomindani following up on the earlier UI-only/dummy-token validation, I also tested this PR branch against a real Databricks workspace connection through Breeze and the Airflow UI.

This should give reviewers a real Databricks round-trip check for the http_path propagation into DatabricksSqlHook.sqlalchemy_url.

Setup:

Airflow connection: pr_69747_databricks_extra_http_path
Connection type: databricks
Workspace host: dbc-c1a3fd5e-8c66.cloud.databricks.com
SQL warehouse http_path: /sql/1.0/warehouses/8b008239c0249091
http_path was supplied through connection Extra JSON.
A temporary Databricks PAT was used only for this validation and revoked afterward.

Result:

Scratch Dag: pr_69747_databricks_sqlalchemy_url_check
Run ID: manual__2026-07-12T08:57:44.649879+00:00
Dag run state: success
Query executed: select 1 as pr_69747_result

Task log evidence:

PR69747_DATABRICKS_REAL_QUERY_VALIDATED
rendered_uri=databricks://token:***@dbc-c1a3fd5e-8c66.cloud.databricks.com?http_path=%2Fsql%2F1.0%2Fwarehouses%2F8b008239c0249091
http_path=/sql/1.0/warehouses/8b008239c0249091
records=[[1]]

Screenshots:

Airflow connection filtered to the real Databricks workspace

Successful Dag run

Task log showing masked URI, http_path, and records=[[1]]

No credentials are included in the screenshots or log excerpt.

Drafted-by: Codex (GPT-5)

…n extra get_conn correctly resolved http_path from explicit arg, sql_endpoint_name API, and extra_dejson, but sqlalchemy_url only used explicit arg. This caused get_uri() to return URL without http_path when connection defined via UI extra, breaking SQLAlchemy usage. Centralize resolution in _resolve_http_path helper, allow extra lookup in property without triggering API call. Add tests for extra path. Fixes: apache#69031

The SQLAlchemy URL path should be available from connection extras without changing the existing get_conn precedence for named SQL endpoints, so the resolution path needs explicit coverage for both behaviors.

moomindani

Reviewed the updated revision (fde316d) and validated it against a real Databricks workspace with live SQL warehouse round-trips:

sqlalchemy_url / get_uri() resolve http_path from the connection extra with zero API calls, and a live query over the resolved path succeeds (the #69031 fix).
An explicit http_path argument still wins over the extra.
Endpoint-name-only configuration: the property stays fully offline (no REST call — the regression flagged on #69037 is avoided), and get_conn() then resolves via the warehouses API and connects.
With both sql_endpoint_name and an extra http_path set, get_conn() still prefers the endpoint lookup (one API call) — precedence preserved vs main, confirmed live. Thanks for adding the second commit and the tests covering exactly this.
Existing and new unit tests pass locally (60 passed).

One tiny behavior diff I noticed, fine by me: an empty-string http_path in the connection extra used to be passed through to the connector; it now fails fast with the "http_path should be provided" error.

Two non-blocking inline notes below. LGTM.

Drafted-by: Claude Code (Fable 5); reviewed by @moomindani before posting

Vamsi-klu · 2026-07-13T02:24:23Z

addressed both review notes from @moomindani, simplified the guard and added the docstring note. tests still pass.

…fy guard Addressed review feedback on apache#69747. When both sql_endpoint_name and an http_path in the connection extra are set, sqlalchemy_url (which must stay offline) uses the extra value, while get_conn() prefers the named endpoint. This is intentional. - Added clarifying note to the _resolve_http_path docstring. - Simplified the now-redundant guard in get_conn() (the helper already handles the early return). The two tests that cover this exact scenario continue to pass. Refs: apache#69747

moomindani · 2026-07-13T04:27:02Z

Verified the follow-up commit (c56cfcf) — both notes are addressed: the docstring now documents the sqlalchemy_url/get_conn() warehouse-selection asymmetry, and the guard is deduplicated. Re-ran the real-workspace validation suite (19/19) and the unit tests (60 passed) against this head — all green. Thanks!

Drafted-by: Claude Code (Fable 5); reviewed by @moomindani before posting

…n extra (apache#69747) * Fix DatabricksSqlHook sqlalchemy_url missing http_path from connection extra get_conn correctly resolved http_path from explicit arg, sql_endpoint_name API, and extra_dejson, but sqlalchemy_url only used explicit arg. This caused get_uri() to return URL without http_path when connection defined via UI extra, breaking SQLAlchemy usage. Centralize resolution in _resolve_http_path helper, allow extra lookup in property without triggering API call. Add tests for extra path. Fixes: apache#69031 * Preserve Databricks SQL endpoint precedence The SQLAlchemy URL path should be available from connection extras without changing the existing get_conn precedence for named SQL endpoints, so the resolution path needs explicit coverage for both behaviors. * DatabricksSqlHook: document http_path precedence asymmetry and simplify guard Addressed review feedback on apache#69747. When both sql_endpoint_name and an http_path in the connection extra are set, sqlalchemy_url (which must stay offline) uses the extra value, while get_conn() prefers the named endpoint. This is intentional. - Added clarifying note to the _resolve_http_path docstring. - Simplified the now-redundant guard in get_conn() (the helper already handles the early return). The two tests that cover this exact scenario continue to pass. Refs: apache#69747 --------- Co-authored-by: probe <probe@example.com>

boring-cyborg Bot added area:providers provider:databricks labels Jul 11, 2026

Vamsi-klu marked this pull request as ready for review July 11, 2026 07:50

Vamsi-klu and others added 2 commits July 12, 2026 19:42

Preserve Databricks SQL endpoint precedence

fde316d

The SQLAlchemy URL path should be available from connection extras without changing the existing get_conn precedence for named SQL endpoints, so the resolution path needs explicit coverage for both behaviors.

Vamsi-klu force-pushed the fix/databricks-sqlalchemy-url-69031 branch from 4d452c8 to fde316d Compare July 12, 2026 19:56

moomindani approved these changes Jul 12, 2026

View reviewed changes

Comment thread providers/databricks/src/airflow/providers/databricks/hooks/databricks_sql.py

Comment thread providers/databricks/src/airflow/providers/databricks/hooks/databricks_sql.py

eladkal approved these changes Jul 13, 2026

View reviewed changes

eladkal merged commit 06f8c47 into apache:main Jul 13, 2026
83 checks passed

eladkal mentioned this pull request Jul 13, 2026

Fix DatabricksSqlHook sqlalchemy_url to include http_path from connection extra #69037

Closed

1 task

shahar1 mentioned this pull request Jul 24, 2026

Status of testing Providers that were prepared on July 22, 2026 #70355

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix DatabricksSqlHook sqlalchemy_url missing http_path from connection extra - #69747

Fix DatabricksSqlHook sqlalchemy_url missing http_path from connection extra#69747
eladkal merged 3 commits into
apache:mainfrom
Vamsi-klu:fix/databricks-sqlalchemy-url-69031

Vamsi-klu commented Jul 11, 2026 •

edited

Loading

Uh oh!

Vamsi-klu commented Jul 11, 2026

Uh oh!

Vamsi-klu commented Jul 11, 2026 •

edited

Loading

Uh oh!

eladkal commented Jul 12, 2026

Uh oh!

Vamsi-klu commented Jul 12, 2026 •

edited

Loading

Uh oh!

moomindani left a comment

Uh oh!

Uh oh!

Uh oh!

Vamsi-klu commented Jul 13, 2026

Uh oh!

moomindani commented Jul 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Vamsi-klu commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

What Changed

Why The Follow-Up Commit Exists

Impact

Testing

Was generative AI tooling used to co-author this PR?

Uh oh!

Vamsi-klu commented Jul 11, 2026

Uh oh!

Vamsi-klu commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eladkal commented Jul 12, 2026

Uh oh!

Vamsi-klu commented Jul 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

moomindani left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Vamsi-klu commented Jul 13, 2026

Uh oh!

moomindani commented Jul 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Vamsi-klu commented Jul 11, 2026 •

edited

Loading

Vamsi-klu commented Jul 11, 2026 •

edited

Loading

Vamsi-klu commented Jul 12, 2026 •

edited

Loading