Skip to content

Fix SnowflakeSqlApiOperator polling to avoid needless sleeps - #69450

Merged
amoghrajesh merged 2 commits into
apache:mainfrom
astronomer:small-precursor-snowflake-api-pr
Jul 6, 2026
Merged

Fix SnowflakeSqlApiOperator polling to avoid needless sleeps#69450
amoghrajesh merged 2 commits into
apache:mainfrom
astronomer:small-precursor-snowflake-api-pr

Conversation

@amoghrajesh

Copy link
Copy Markdown
Contributor

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

What

SnowflakeSqlApiOperator.poll_on_queries() polls the status of every submitted SQL API statement handle in a loop and waits poll_interval seconds between checks. It exists to avoid hammering the Snowflake SQL API while a query is still running.

Current behaviour

The time.sleep(self.poll_interval) call sits inside the per-handle for loop, so it fires after every handle is checked, regardless of whether that handle (or any other) is still running. For a batch of N statement handles, a single call to poll_on_queries() can sleep for up to N * poll_interval seconds even when every handle already resolved to success or error on this pass. This adds pointless latency: a batch that just finished still waits before the operator can move on and report the result (or raise on failure).

Proposed change

Move the sleep out of the per-handle loop to run at most once per call to poll_on_queries(), and only when at least one handle is still running after checking all of them. A cycle where everything already resolved (all succeeded, or one failed) now returns immediately instead of sleeping first.


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Comment thread providers/snowflake/tests/unit/snowflake/operators/test_snowflake.py Outdated
Comment thread providers/snowflake/tests/unit/snowflake/operators/test_snowflake.py Outdated
@amoghrajesh
amoghrajesh merged commit 669579e into apache:main Jul 6, 2026
82 checks passed
@amoghrajesh
amoghrajesh deleted the small-precursor-snowflake-api-pr branch July 6, 2026 13:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:snowflake Issues related to Snowflake provider

Projects

Development

Successfully merging this pull request may close these issues.

4 participants