At some point in my journey when learning any programming language, I hit a point where I decide to make a directed acyclic graph (DAG) based task runner.

Whether this is a good idea or not, whether I should just use one of:

• OpenWhisk
• Celery
• Gitea Actions
• Jenkins
• CircleCI
• RunDeck
• Airflow

Or one of countless other job runners is kind of irrelevant. I have Opinions^TM

Lets sketch out the basic architecture and we can cover those opinions as they come up.

Architecture

Task and workflow configuration

Celery does this magic where you feed it python tasks to run and it does some pickling/serialising of datastructures and makes the function run on another runner which also has that task defined.

This annoys me for a few reasons.

1. You have to have the functions defined in code
2. In a language Celery supports (Python)
3. Handling the contracts between the tasks can be problematic
4. Tasks are loaded at startup and hotfixing a single task can mean redeploying entire runners.
5. Sometimes you really do just want to cat a file without all that overhead.
6. Dynamically loading tasks is….. lets say interesting at best. Especially with python packaging formats in flux.

In this world, versioning is really important as a task on node 1 with older code for a function might not act in the way you expect.

I much prefer the world where the runner calls scripts in a given environment. This comes with a caveat. You absolutely should run those scripts in a sandbox, even if it’s a lowly docker container. If you do run them outside of a sandbox, expect all kinds of fun as your users discover they can poke and prod random files on the OS filesystem rather than call the expected API endpoint.

So here we will assume:

• Tasks are run in a safe environment.
  • We’ll assume Docker compatible here (I use Podman on most things, which is Docker compatible)
  • We’ll let the runner mount points have some preset and user configurable exclusions (i.e. /etc, /var, /usr, etc)
• Tasks are either executable scripts, or raw stdin writes to the containes open console.
• Docker images are either:
  • Built from a Dockerfile
  • Pulled from a non-custom registry
• The above will want to be revisited and relaxed at a later date.
• The state from each task for each workflow will be passed by reference (Stored in another service and requested by id).
  • This is something that will save us time later.
• There will only be 1 management service for processing workflows and putting tasks onto queues.
  • This will want to be revisited and relaxed at a later date, but ides a lot of cocmplexity we don’t want at the start.
• We will use MQTT as the message broker.
  • Personal preference, I know it well and it has many useful features, such as will messages.

So a quick block diagram of the architecture:

Simplified — the broker is the only thing workers and the Director talk to, and Workflow Configuration talks to the Director over REST. Storage and the rest of the pieces get their own detail as this series goes on.

Lets hit some code next and we can update the architecture diagram as we go. Next stop, the worker.