How Render Workflows powers our internal data pipeline

August 25, 2026

· Mac McCarthy

Background

I watch our data pipeline notifications channel like a hawk, and it’s not really meant for conversation. So it definitely caught my attention when Anurag, our CEO, dropped in asking “Do we need to run dbt again?” It’s a fair question that comes from a legitimate eagerness to see fresh numbers. But it's specific enough to raise flags for the data team: why does the CEO feel the need to nudge us toward manually rerunning the pipeline? What does that say about gaps in our processes or messaging?

When I joined Render, the data team doubled in size, growing from one member to two. We quickly upgraded dbt-core from v1.4 to v1.11, introduced incremental tables and a new layered architecture, cut costs, and brought our analytics engineering capabilities up to modern standards. But one vestige of the olden times stayed in our architecture and caused near-constant frustration: a late-arriving but incredibly important dataset. This particular dataset contained invoice projections throughout the month, and was built daily using a Golang job that Anurag wrote eight years ago. Because it was tightly intertwined with our billing process, it couldn’t simply be plucked out and integrated into the rest of the pipeline.

To cover for this late-arriving data, our dbt project ran twice each day on a cron job hosted on Render: once early in the morning, and once again mid-morning. If the Golang job or our data extraction and loading tool failed or ran slowly, dbt might need to run a third time. This would raise BigQuery costs and require significant manual checking and rerunning by the data team. To boot, there was little observability across the whole system aside from per-system or per-job logs in different places — on Render, locally, inside software UIs. When someone asked whether the data was ready, estimating the timeline meant checking several systems, from BigQuery to Stitch to dbt.

All in all, the 7 or 8 processes in our data pipeline were spaced just far enough apart that, on a normal day, everything would finish, albeit redundantly and in a way that was frustrating to debug. In Metabase, our primary analytics platform, a small progress bar compared the sizes of two datasets, one dependent on the Go projection job and one not, and served as a rough proxy for data pipeline completion. But after enough time stuck at 44%, our CEO, who was watching the dashboard more closely than we were, came straight to the channel and asked, quite resolutely, “dbt again?”

Approaching a solution

The easiest dbt-specific solution was to split our models into two tagged sets: one that depended on the projection data, and one that did not. If the projection had not finished, been extracted, or loaded yet, we could run the projection-independent models first. Once the projection was ready, we could run the dependent models. On days when the projection was ready before dbt started, we could run the full project as usual.

This alone would cut our dbt costs in half and remove the guesswork around whether the projection job had finished. All we needed was a sentinel from the projection job, which did not exist at the time.

But even this simple fix required branching logic, and dbt was only one part of the problem. We still had other tasks to coordinate like waiting for extraction and loading, pushing documentation into Metabase, running cost monitoring packages, running Hightouch, and sending the daily report email. These steps were separated mostly by time and loosely monitored.

We needed a system that could:

• wait for an external task to complete,
• run with branching logic,
• orchestrate individual tasks that could be investigated and rerun, and
• pass packets of information between tasks that would inform their configuration, or specify the branch to take.

I investigated solutions like Prefect and Dagster. They offer per-model task-like blocks, but we would have gotten the most value from them with a paid contract. I thought of Airflow, but with a two-person team and a single dbt project, it seemed like too much overhead.

Our new Workflows product, then in beta, was the perfect solution. I could chain individual tasks together, configure them with decorators, and pass small packets of information between them. Tasks could also retry natively for long stretches while waiting for another process to finish. Plus, I’d be dogfooding our own product.

Engineering walkthrough

The nightly DAG and task ecosystem

Not Ready, Retries

Ready

Projection Ready

Projection Not Ready

Transient

Non-Retryable or Still Failing

Recovered

Still Failing

Clean

Succeeded

Clean

Drift

Cron Trigger (09:00 UTC Daily)

Run Pipeline

Check Projection Ready

Alert Data Science (Unconditional Heartbeat)

Check Source Freshness

Wait for AWS Billing Transfer Ready (Hard Gate)

AWS Ready?

Deduplicate AWS Cost Data (Non-Blocking)

Build dbt: Full

Build dbt: Partial (No Projection)

Wait for Projection Ready (Durable Poll)

Build dbt: Projection Only

Merge Partial and Projection Results

Analyze dbt Failures

Retry Failed Models (Native dbt Retry)

Alert Data Science and Halt

Run BigQuery Monitoring and Cost Checks (Parallel)

Generate dbt Docs

Downstream Fan-Out: Cleanup, Metabase Sync, Cache Warming, Schema Cleanup (Parallel)

Send Metabase Dashboard

Publish Pipeline Summary (Slack)

Deploy Hightouch

Compare Compiled SQL to Live Hightouch SQL

Done, Nothing Alerted

Alert Data Science, One per Drifted Model

Detect Hightouch Drift (Independent Schedule)

A cron job starts the workflow every morning. It also polls for completion, which prevents the workflow from restarting mid-run. From there, a few gates determine which dbt build command should run: checking whether the data transfer is complete, confirming that the projection job is done, and waiting for the projection data to be extracted and loaded:

if projection_result["projection_ready"]:
    summary["path"] = "full_build"
    build_result = await dbt_build_full(run_config=run_config)
else:
    summary["path"] = "not_ready"
    partial_result = await dbt_build_partial_no_projection(run_config=run_config)

if partial_result["status"] != "success":
        summary["path"] = "partial_only"
        build_result = partial_result
    else:
        await wait_for_projection_ready(run_date=projection_result["run_date"], run_config=run_config)
        projection_build_result = await dbt_build_projection_only(run_config=run_config)
        summary["path"] = "partial_plus_projection"
        build_result = merge_dbt_build_results(partial_result, projection_build_result)

The workflow checks early whether the full dbt build can run right away. If not, it still runs the projection-independent work first and waits only for the projection-dependent step. Then dbt runs via subprocess in the task, which allows for complete observability:

def run_command(command, *, cwd, timeout_seconds):
    process = subprocess.Popen(
        command, cwd=str(cwd), text=True,
        stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1,
    )

def _stream(proc):
        for line in proc.stdout:
            logger.info(line.rstrip())

reader = threading.Thread(target=_stream, args=(process,), daemon=True)
    reader.start()
    process.wait(timeout=timeout_seconds)
    reader.join()
    return {"command": command, "returncode": process.returncode}

A background thread drains the subprocess’s stdout line by line into the task’s own logger while the process runs, so a 20-minute dbt build shows up in the dashboard log stream in real time instead of as one wall of text at the end. The task writes dbt’s files like manifest.json, catalog.json, and run_results.json to a GCS bucket for later retrieval by downstream events. Task completion events, with some metadata, write to a small BigQuery table called workflow_run_events, enabling both historical observability and a remarkably accurate estimate of when the data will be available. dbt’s native retry function resolves transient errors in the pipeline automatically. There’s an opportunity to use an agent to resolve other errors and either retry, or open a PR with a diagnosis and solution. That’s v2.

After dbt is done, the workflow runs several downstream tasks in parallel: syncing documentation, running cost monitoring pipelines, and invalidating and re-warming the Metabase cache for faster, more efficient dashboard loads. Each task reports its own status, so failures don’t disappear silently or block the others:

cleanup_result, sync_result, warm_result, schema_result = await asyncio.gather(
    cleanup_orphaned_models(manifest_uri=manifest_uri, run_config=run_config),
    sync_dbt_metabase_docs(manifest_uri=manifest_uri, run_config=run_config),
    warm_metabase_dashboard_caches(run_config=run_config),
    cleanup_stale_bq_schemas(run_config=run_config),
    return_exceptions=True,
)
for name, result in [
    ("cleanup_orphaned_models", cleanup_result),
    ("dbt_metabase_sync", sync_result),
    ("metabase_cache_warming", warm_result),
    ("stale_bq_schema_cleanup", schema_result),
]:
    mark_downstream(summary, name, result)

return_exceptions=True makes this non-blocking, but it comes with a real trap: without an explicit, logged home for each result, a failed task just sits quietly inside the returned list as an unclassified Exception object, invisible to any dashboard or alert. mark_downstream provides that home, so real failures won’t silently go missing from the pipeline’s completion alert.

Downstream, another daily task sends a Metabase dashboard to everyone at the company with updated data. It also posts a short pipeline summary to the data team’s Slack channel with Anurag: data processed with 100% of projections.

I’ve built other tasks alongside the main chain. One detects drift in Hightouch, our reverse ETL tool, by comparing live queries against the SQL we version control in dbt, then deploys approved changes. Another runs dbt and its downstream tasks manually, but only after a six-point safety check confirms there are no concurrent jobs and that the workflow is not currently building or being deployed. A third independent task gathers warning and error logs from each subprocess of the main run_pipeline orchestration task and displays them for quick error resolution across the entire pipeline. Each of these is diagrammed in the Appendix below, and anyone on the data team can administer all three of these independent tasks via a small dbt governance web service, also hosted on Render.

Retry-as-poll: waiting without waiting

The “wait for projection ready” branching logic not only constitutes the linchpin of this workflow; it is, in fact, the raison d’être. At the time, the Go projection job did not emit any sort of sentinel signal that it was completed, and the Stitch API (our extract and load tool) does not provide any endpoint for per-table load completion. So we could not start the workflow based on a webhook or external API call. We needed some way to wait and poll until an external signal shows up.

Of course there is a Python-native way of doing this, namely a while not ready: sleep(60) inside a loop. But that approach burns compute the entire time it waits. Fortunately, Render’s retry engine works as a free “wait until ready” primitive.

class ProjectionNotReady(RuntimeError):
    """Raised so Render Workflows retries the sentinel check."""

def wait_for_projection_ready(run_date: str | None = None) -> dict:
    result = check_readiness(run_date)
    if not result["ready"]:
        raise ProjectionNotReady(f"not ready for {run_date}: status={result['status']}")
    return result

@app.task(
    retry=Retry(
        max_retries=PROJECTION_WAIT_MAX_RETRIES,                   # e.g. 120
        wait_duration_ms=PROJECTION_POLL_INTERVAL_SECONDS * 1000,  # e.g. 600_000 (10 min)
        backoff_scaling=1,   # flat interval -- a poll cadence, not error-backoff
    ),
    timeout_seconds=1800,    # per-attempt ceiling, not the total poll budget
)
def wait_for_projection_ready(run_date=None, run_config=None) -> dict:
    with workflow_run_env(run_config):
        return _wait_for_projection_ready(run_date)

I configure the task with a number of retries ( max_retries), a duration between each poll ( wait_duration_ms), and a flat interval for a poll cadence ( backoff_scaling=1). Nowadays, the Go projection job writes a sentinel row to a small BigQuery reports table with the start and finish timestamp, and the number of successful profiles written. The workflow polls for that sentinel row, then checks that the successful profiles have also landed in BigQuery. It burns no compute between polls, and the workflow continues once the data is ready.

Three execution modes for optimized testing and a thin Render surface

One particular pattern worth highlighting is the three workflow execution modes: prod, dry run, and simulate. Together, they support CI/CD, full-suite testing, and the option to move the workflow into another system later. I thought a lot about the future of Render and its data team in this process. Someday, at 1000 employees, this workflow might need to look fairly different and involve other systems. So, how do I thin out the actual Render SDK mechanics while keeping vendor-agnostic functions and testing?

To keep the Render SDK out of most of the codebase, I register the app and each task in main.py instead of in domain-specific files. This makes the task logic easier to test without installing the SDK or running a dev server. The import and task registration in main.py look like this:

from tasks.projection import check_projection_ready as _check_projection_ready

app = Workflows(
    default_retry=Retry(max_retries=2, wait_duration_ms=2000, backoff_scaling=2),
    default_timeout=3600,
    default_plan="standard",
)

@app.task
def check_projection_ready(run_config: dict | None = None) -> dict:
    with workflow_run_env(run_config):
        return _check_projection_ready() 

Every implementation module can be imported and unit-tested in a bare virtualenv with no render_sdk, no cloud SDKs, and no dev server. The SDK-touching surface is reduced to this one file doing registration and thin delegation.

I run this config through environment variables and set up a context manager, which allows me to set the execution path in the correct format and through child subtasks:

def env_bool(name: str, default: bool = False) -> bool:
    value = os.getenv(name)
    return default if value is None else value.strip().lower() in {"1", "true", "yes", "on"}

def dry_run_enabled() -> bool:
    return env_bool("WORKFLOW_DRY_RUN", False)

def simulate_enabled() -> bool:
    return env_bool("WORKFLOW_SIMULATE", False)

@contextmanager
def workflow_run_env(run_config: dict | None):
    overrides = run_config_to_env(run_config)
    previous = {key: os.environ.get(key) for key in overrides}
    os.environ.update(overrides)
    try:
        yield
    finally:
        for key, value in previous.items():
            if value is None:
                os.environ.pop(key, None)
            else:
                os.environ[key] = value

simulate_enabled() is just a boolean that each task checks before doing any real work. If simulation mode is on, the task returns a stubbed result before making network calls; if not, it runs normally. This simple guard clause powers the whole simulate-mode test matrix without requiring a mocking library.

Production runs every night. After each deployment, dry-run mode tests environment variable configuration and new features; it briefly touches the network for real reads, but skips or minimizes writes. Then, simulate runs in CI. It invokes the Render CLI, starts a dev server, exercises the task branches, and verifies Render mechanics like retry and backoff. This way, Render is simply a thin layer on top of an otherwise independent project. I follow a similar pattern for BigQuery calls, so if we ever move off BigQuery, this workflow will not be a difficult part of the migration.

Outcomes

The data pipeline workflow has been running successfully for about 30 days. On most days, it follows the expected path: run the dbt projection-independent partial build → wait for the projection data to be ready → then run the dbt projection-dependent partial build.

Dashboards are loading faster thanks to the warm Metabase cache. And while costs are difficult to predict with a growing data corpus, as it stands now the reduction in unnecessary dbt runs will save us $30K this year, and the efficient Metabase caching will save us $70K.

One of the most exciting realizations from this process was how extensible the framework is.

The Metabase caching task started as a quick brainstorm with my colleague, then slotted easily into an already well-structured workflow. The dbt governance app and its accompanying workflows were even more incidental, not part of the original plan in any capacity. It now gives the team customizable control over the entire system and a faster path to resolving errors. It’s also a testament to the symbiosis of Render products: a workflow behind a web service is becoming an increasingly common pattern across our internal tools.

I took this screenshot around 5:00AM. The run actually finished at 6:23 AM.

On the alerting and messaging side, the workflow_run_events table records historical task timings and wait-check statuses, giving Metabase a remarkably accurate estimate of when the full data pipeline will complete.

And good news for everyone involved: Anurag offered his congratulations to the team on shipping the workflow. Since then, we haven’t heard a peep from him in the channel.

Appendix

Each of the following independent tasks operates outside the main nightly chain and is administered through the dbt-governance web service.

Hightouch drift review

Identical

Different

Repository Is Right

Hightouch Is Right

Unclear

Deploy Hightouch or Detect Hightouch Drift

Compile Repository SQL and Compare to Live Hightouch SQL

Done, Nothing Alerted

Alert Data Science, One Alert per Drifted Model

Human Reviews the Diff

Accept dbt Model

Accept Hightouch, Open a dbt Pull Request

Investigate; Leave Untouched

Force-Push Compiled SQL to That Model

A drift between repository SQL and live Hightouch SQL is genuinely ambiguous, so every drifted model is alerted and left untouched until a human makes an explicit decision.

Manual dbt run safety checks

No

Yes, No Override

Yes, Override Set

Manual dbt Run Request

Evaluate Safety, Six Checks

Any Blocking Reason?

Execute Manual dbt Request

Blocked, Safety Snapshot Returned

Refresh Row Counts

Result Returned With Safety Snapshot

Six checks guard against concurrent dbt jobs, an in-progress nightly run, or a stale deploy, each captured in the returned safety snapshot.

Diagnostic run reports

Found

None or Unknown

First Visit

Pipeline Completion

Check for Warnings or Errors

Attach Run Report Link to Summary

No Link Added

Completion Alert Renders the Link

Human Clicks Through

Open Run Report

Start Report Generation

Poll Until Complete

Render Report With Status and Events

The full report aggregates every warning and error across the run's task tree, generated on demand only when a human opens it.