Pipeline orchestration for Apache Airflow®

Each step when ready,
each rerun the same

We plan, build and upgrade Apache Airflow® pipelines, with each task waiting for what it needs, tasks written so a retry changes nothing it shouldn’t, and every upgrade rehearsed on a copy first, so a rerun gives the same answer as the first run.

Thanks, we’ll be in touch soon.

{{ heroStatus }}

You’re in good company

A few of the companies we’ve worked with.

When the pipelines grew one script at a time

Apache Airflow often arrives with one team’s jobs. It’s open source, and in the project’s own words “a platform created by the community to programmatically author, schedule and monitor workflows”: each pipeline is written in Python as a Dag, the word Airflow uses for a pipeline’s tasks and the order they run in.

The scheduler starts each run on its schedule, or when the data it waits for is updated, and workers run its tasks.

Years on, Dags have been added one project at a time. Some read “the latest” data, so a rerun gives a different answer. Some add rows each time they run, so a retry counts a day twice. Heavy work at the top of Dag files slows the scheduler, failures reach the people who read the report first, and the setup is still on Airflow 2, with the move to Airflow 3 waiting.

This isn’t a sign the setup was wrong. It grew without a plan for reruns, releases and alerts.

So the work begins with each pipeline, what it reads and what it writes, and one change at a time after that.

What we build

Pipelines planned so each task waits for what it needs, tasks that run again cleanly, logins kept out of the code, and the move to Airflow 3 rehearsed on a copy.

  • Pipelines planned task by task

    Each pipeline’s tasks and what each one waits for, written so any task can run again on its own.

  • Schedules and data triggers

    Runs on a timetable, or started when the data they wait for is updated.

  • Tasks that run again cleanly

    Each task reads its run’s own data and replaces what it writes, so a retry or a rerun gives the same answer.

  • Connections and secrets

    Logins kept in the connections Airflow stores, or in a secrets store, never in the code, each with only the rights it needs.

  • Pipeline files that load quickly

    Heavy work moved inside tasks, so the scheduler reads every Dag file fast.

  • Tests before release

    Each Dag loaded and tested, then run on a copy of the setup before it ships.

  • Jobs brought in from timers

    Scripts on timers and jobs from an older scheduler moved into Dags, with their retries and alerts.

  • The major upgrade, rehearsed

    Dags checked for what Airflow 3 changed or removed, the metadata database backed up and the move rehearsed on a copy, then each release after it.

  • Monitoring and alerts

    Failed and late runs noticed by your team first, with alerts that say what to check.

Run it again, get the same answer

Apache Airflow can retry a task when it fails, and once a problem is fixed, a team can run a past day’s tasks again. So each task is written to give the same result however many times it runs for the same day.

A load replaces that day’s rows rather than adding to them. It reads that day’s own data, never “the latest”, since the data can change between runs, and it takes its date from the run it belongs to, never from the clock.

Run Tuesday’s load again, and see what each way of writing it does.

Daily sales report

An example

Written to add

The load adds the day’s rows each time it runs.

  • MondayMonday’s rows
  • TuesdayTuesday’s rowsAgainAgainand once more with every run
  • WednesdayWednesday’s rows

Written to replace

The load replaces the day’s rows each time it runs.

  • MondayMonday’s rows
  • TuesdayTuesday’s rowsSame as before
  • WednesdayWednesday’s rows

Each day has run once.

Run twice. Written to add, Tuesday counts twice. Written to replace, Tuesday is as it was.

Run again. Written to add, Tuesday counts once more. Written to replace, Tuesday is as it was.

Run again. Written to add, Tuesday counts once more with every run. Written to replace, Tuesday is as it was.

{{ rrSaid }}

Every move rehearsed before it’s made

Apache Airflow plans a feature release every few months, with patch releases as fixes are needed. Airflow 2 no longer gets any fixes, and Airflow 3 is the current major version.

The move to Airflow 3 starts on the Airflow 2 release the upgrade guide asks for, or a later one. Each Dag is checked with the project’s own checks for what Airflow 3 changed or removed, the metadata database, where Airflow keeps the state of each run, is backed up, and the move is tried on a copy of the setup first.

After the move, each feature release is taken the same way, tried on a copy before the live setup.

A setup still on Airflow 2 gets a next step: the path to Airflow 3, written down and tried on a copy first.

Start with the pipeline people wait on

The review covers the setup’s release, each pipeline’s schedule and what its tasks wait for, which tasks would count a day twice or change on a rerun, how the Dag files load, where the logins live, what’s tested, and what’s watched today.

The first changes come in the order we’d make them, with the path to Apache Airflow 3 where the setup is still on Airflow 2.

What you receive is a written review and a proposal for that first step.

Questions worth asking

  1. How do we move to the current major version?

    First onto the Apache Airflow 2 release the upgrade guide asks for, or a later one. Then each Dag is checked with the project’s own checks for what Airflow 3 changed or removed, the metadata database is backed up, and the move is rehearsed on a copy, with the Dags paused during it and a few test Dags turned on first after it.

  2. Can a run show success when a task failed?

    Yes, when its last task is set to run whatever happens before it, since the run takes that task’s state. We add a check that fails the run when any task fails, so a run marked a success means every task did its work.

  3. Should a pipeline run on a schedule or when its data arrives?

    When one pipeline’s output is another’s input, the second can start once the task that updates that data has succeeded, rather than at an hour chosen to be late enough. A pipeline that answers to the clock, such as a report due each morning, keeps its schedule.

  4. Can tasks pass data to each other?

    Small messages, yes. Larger data goes to storage, and the next task reads it from there, with only its path passed along.

  5. What does it cost to work with you?

    It depends on the pipelines you run and the first step you choose. Before any work starts, you get a written proposal with the scope, the price or rate, and how you pay.

    Pricing

  6. Can we hire developers for our own team?

    Yes, through a dedicated team: Webair developers who stay with your product over time and work beside your own people.

    Dedicated team

Apache Airflow, Airflow, Apache and the Airflow logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. Webair is independent of The Apache Software Foundation, and isn’t sponsored by, endorsed by or affiliated with it.

Let’s build it better

Tell us which release your setup runs and which pipeline people wait on each morning, and we’ll suggest where to begin, in a short call.

Thanks, we’ll be in touch soon.

{{ ctaStatus }}