The inner loop: Python, PySpark, dbt, Airflow

The inner loop is the cycle an engineer runs dozens of times per hour: edit, run, inspect, fix, repeat. Nothing else affects productivity more. A 5-second loop produces shipped features. A 5-minute loop produces context-switching and Slack scrolling.

This guide walks through the inner loop for each stack a data creator works in, with concrete keystrokes and command lines. The target is seconds per loop, never minutes.

The principle: never leave the editor

The moment you tab out of VS Code, the loop breaks. Attention fragments, typos multiply, and the cost of each iteration doubles. Every stack in this guide has been configured so the entire loop — compile, run, inspect, test — happens inside VS Code.

Python (local, pure)

The baseline. A unit-testable function, no cloud dependencies.

1. Edit the .py file. Pylance autocompletes as you type, Ruff formats on save.
2. Press Cmd+; to run all tests, or Cmd+; Cmd+F to run the current file's tests.
3. Read the result in the Test Explorer panel or the integrated terminal. Failures hyperlink back to the failing line.
4. If a test fails, click the ▷ next to the test in the gutter, pick Debug Test. Breakpoint hits; you inspect.
5. Fix, save, repeat.

A typical loop: 2–5 seconds per iteration.

PySpark via Databricks Connect

The pattern for any work against real Spark: local driver, remote cluster.

Setup once

• Install databricks-connect matching your cluster's DBR version: uv add databricks-connect==15.4.*.
• Authenticate the Databricks extension via OAuth.
• Attach a cluster (or serverless).

The loop

1. Edit the .py file. Write a PySpark expression: df.groupBy("region").count().show().
2. Click Run on Databricks → Debug current file with Databricks Connect, or press the run shortcut bound in keybindings.json.
3. Code runs locally until it hits a DataFrame operation; the operation executes on the remote cluster; results stream back to the integrated terminal.
4. Inspect df in the Variables pane if a breakpoint is set. The inspector shows schema, row count, and a sampled preview.
5. Fix, save, repeat.

A typical loop: 5–10 seconds for a small transform. Longer if the cluster is auto-starting — keep it warm during active work.

When the loop is slow

• Cold cluster start. 3–5 minutes. Wake the cluster before your first edit of the day.
• Network latency. Each DataFrame operation round-trips. Chain operations before calling .show() or .collect() — a single df.filter(...).groupBy(...).count() is one round trip, not three.
• collect() on large results. The full result DataFrame streams to the laptop. Use show() or limit().collect() for inspection, not collect().

dbt

The dbt inner loop has two phases: edit-and-inspect (local) and build-and-test (remote-compute-on-dev-schema).

Setup once

• Install dbt Fusion LSP (dbtLabsInc.dbt) or dbt Power User (innoverio.vscode-dbt-power-user).
• Configure profiles.yml with a dev target that writes to your personal dev schema.
• dbt deps from the integrated terminal.

The edit-and-inspect loop

1. Open a model file.
2. As you edit, the LSP (or Power User) shows compiled SQL in a side panel live. CTE previews render inline.
3. Hover any {{ ref('...') }} to see the resolved table name.
4. Save. dbt parse runs in the background; syntax and DAG errors surface in the Problems pane.

This loop is sub-second — you catch half of dbt bugs before you ever run dbt build.

The build-and-test loop

1. In the integrated terminal: dbt build --select model_name.
2. Build and tests run on the warehouse against your dev schema. Typical completion: 10–30 seconds for a small model.
3. Failures report as "FAIL 1 unique_fct_orders_order_id" with a click-through URL to the query.
4. For a faster iteration, dbt build --select state:modified+ --defer --state target/. This runs only what changed and everything downstream that depends on it, unioned with prod state for upstreams.

When the loop is slow

• Warehouse cold-start (Databricks SQL, Snowflake auto-suspend). Keep the warehouse warm during active work.
• Unnecessary upstream recompiles. Use --defer + --state so only the modified model and its true descendants recompute.
• Slow tests. Profile dbt_expectations tests; some (like expect_row_values_to_be_in_set against a large reference table) are surprisingly expensive.

Airflow

Local Airflow iterates fast. Production Airflow is for production.

Setup once

• Install Astro CLI on PATH.
• In the project directory: astro dev init (or clone an existing Astro project).
• astro dev start — this spins up a local Airflow stack (scheduler, triggerer, webserver, Postgres) in Docker.

The DAG-author loop

1. Edit a DAG .py file. Hot reload picks up the change within 30 seconds.
2. Open the Airflow UI (localhost:8080) in a browser tab only when you need to trigger a run manually. For pure DAG-render checks, the file tree in VS Code and astro dev run dags list are enough.
3. Run a test execution headlessly from the terminal: astro dev run dags test my_dag 2026-04-20. This runs the entire DAG synchronously with full stack traces.
4. Read the logs in the terminal, not the UI.

A typical loop: 10–20 seconds for a small DAG; longer for DAGs with real external work.

The task-callable loop

Task callables are pure Python. Test them the same way as any Python function:

1. Extract the callable from the DAG module.
2. Write a pytest unit test.
3. Run via Test Explorer. Seconds per iteration.

Notebooks (locally-authored, remotely-executed)

Some data exploration still warrants notebooks. VS Code's notebook experience is competitive:

1. Open a .ipynb in VS Code.
2. Select the remote Databricks kernel (the Databricks extension exposes a kernel per configured cluster).
3. Run cells with Shift+Enter. Each cell executes on the remote cluster.
4. Inspect outputs inline, including DataFrames with pagination.

Or use # %% cell markers in a plain .py file for "notebook mode without .ipynb":

# %%
df = spark.table("main.silver.events")
df.count()

# %%
df.groupBy("region").count().show()

# %% cells run independently; state persists across cells in the same kernel session. The files live in Git cleanly (unlike .ipynb).

Multi-stack projects

A real data product mixes stacks: dbt models, Airflow DAGs that orchestrate them, Python fixtures that feed both, PySpark jobs for the heavy bronze→silver lift. The inner loop across stacks needs discipline:

• Split panes in the integrated terminal. One pane per tool: dbt, astro, python REPL. Name them (right-click → Rename) so you do not lose track.
• Tasks in .vscode/tasks.json for the commands you run dozens of times a day. Bind them to keyboard shortcuts. See the commands reference.
• One workspace per project. Multi-root workspaces help when two repos need coordinated edits; do not use them to paper over a monorepo issue that wants a monorepo solution.

The no-browser rule

Every browser tab you open during the inner loop costs a 30-second attention tax. Enforce a rule: no browser tab until the loop succeeds.

• Compile: inline in the dbt LSP.
• Logs: integrated terminal.
• Query results: debug console or df.show().
• Diffs: Source Control panel.
• PR review: GitHub Pull Requests extension.

The only browser tab that earns its keep is the one you open at the end to read a dashboard or confirm a DAG ran in production. Every other one is a leak.

Typical loop times: the target

A well-configured setup hits these:

If your real loop times are 3× these, the setup is wrong, not the work.

Anti-patterns

• Running dbt build with no --select. Every iteration compiles every model. Loop time explodes.
• Running pytest with no filter. Same problem, in Python. Use pytest -k <expr> or the Test Explorer.
• Using the Databricks workspace UI to iterate. The UI's save-and-run cycle is slower than the Connect + debugger path, and diffs do not land in Git.
• Hot-reloading Airflow DAGs by restarting the whole stack. astro dev start hot-reloads automatically. No restart needed.
• Inspecting DataFrames by running collect(). Streams the whole result. show() or limit().collect() almost always works.

See also

• Debugging — the launch configs that make these loops fast.
• Workspace standards — how the paved path configures this.
• The extension pack reference — the extensions that power each loop.
• Commands reference — the keystrokes and Tasks that shave seconds off each iteration.