Dagwood - DAG-Building Shell Orchestration Tool
--------------------------------------------------

Dagwood is a shell script orchestrator that models task dependencies
using Make-like rules (mtime comparisons, etc.), however it also
includes nicer and more explicit semantics for inspecting the directed
acyclic graph (DAG), importing other projects, and so on.

We enjoy Make, and consider ourselves to be adherents of the Unix
Philosophy; however, Dagwood came about with our experience of Make
becoming unmaintainable in large, polyglot, multi-project builds,
especially in the context of simultaneously optimizing concurrent
builds and recursive and dynamic Makefiles. Dagwood was also
influenced by our experience with Bazel, however without the steep
complexity and more aligned to Unix Philosophy standards.

Dagwood is written in pure C with an embedded DSL powered by LCL
(Lexical Command Language), a Tcl-like language with lexical scoping.
LCL itself is pure C89 and depends only on POSIX headers. The DSL
provides a pleasant interface for describing projects and tasks.

Dagwood projects define a number of tasks, and Dagwood builds a DAG
from these tasks by inspecting their inputs, outputs, and explicit
"depends-on" relationships defined in the task definitions. It then
executes the DAG with a ready-queue scheduler: a task launches the
moment its dependencies complete, with the number of concurrently
running tasks capped by `-j` (default: the CPU count; `-j 0` removes
the bound). The graph describes what may run in parallel; `-j`
describes how much of it may run at once.

Projects can be imported and composed with your project's task graph.
This echoes the Bazel influence mentioned earlier: Dagwood can "import"
other projects into your task-graph, with full namespace isolation.
This is nice for polyglot, multi-project builds, or for depending on
third-party projects.

Imported projects can also be mutated from the importer: tasks can be
overridden, extended, or disabled, and project-level config can be
overridden, all without forking the imported project. The relevant DSL
keywords are task-override, task-extend, task-disable, project-override,
and task-get (the read-only counterpart).

Tasks can also be generated programmatically: `task` is an ordinary
Lcl command, so a proc (in the Dag file, or in an imported helper
module under e.g. .dag/) can loop over a glob and define one task per
input/output pair, with ${self::inputs} / ${self::outputs} standing
for the task's own attributes in its run body. This keeps mtime
staleness granular when there are hundreds of small tasks.

Tasks in Dagwood are simply shell commands. Dagwood only cares about
the return code, but the command could be anything. It could be a
Python script, an executable -- whatever. Dagwood doesn't care as long
as the task returns a sensible exit code.

Some features of Dagwood:

- Tasks declare explicit inputs, outputs, and dependencies within
  nested project blocks:

    project myproject {
        let BUILD_DIR build

        task compile {
            inputs src/main.c
            outputs ${myproject::BUILD_DIR}/main.o
            run {
                gcc -c src/main.c -o ${myproject::BUILD_DIR}/main.o
            }
        }
    }

- Dagwood materializes an entire DAG based off information in the
  tasks (inputs, outputs, dependencies) and validates it before
  running, then executes it with bounded parallelism (`-j N`,
  defaulting to the CPU count).

- Tasks can set their own shell, even to something like `perl -e` if
  you're feeling spicy.

- Dagwood projects can import other projects isolated with namespaces.

- Task attribute introspection: tasks and commands expose their
  attributes as namespace variables. ${project::task::run} gives you the
  run script, ${project::task::outputs} gives you the outputs, etc.
  This is particularly useful for wrapper tools:

    project myproject {
        task compile {
            inputs (src/main.c src/util.c)
            outputs build/main
            run {
                gcc src/main.c src/util.c -o build/main
            }
        }

        ;; Generate compile_commands.json using bear
        task compile_db {
            outputs compile_commands.json
            run {
                bear -- ${myproject::compile::run}
            }
        }

        ;; Or wrap with ccache, distcc, timeout, etc.
        task compile_cached {
            outputs build/main
            run {
                ccache ${myproject::compile::run}
            }
        }
    }

- CLI arguments can override project variables:

    arg CC gcc
    arg CFLAGS "-O2"

  Then run: dag CC=clang CFLAGS="-O3" build

- Dagwood ships with a REPL (`dag -r`) that loads your project and
  drops you into an interactive prompt with the loaded namespaces,
  task data, and CLI overrides all in scope. Useful for poking at
  what Dagwood saw before you commit to running it.

- Dagwood can emit Graphviz and you can inspect the DAG without
  running the project.

- Dagwood's own log lines go to stderr; child task output goes to
  stdout, prefixed with the task id (`[project::task] line`). This
  means `dag | sort` works, `dag <task> 2>/dev/null` gives you only
  the child's own output, and structured-output tools (jq, dot, etc.)
  see clean data. The -q flag suppresses child output entirely if you
  only care about exit status.

LCL Scripting Features:

The embedded LCL interpreter provides filesystem operations via the
Io:: and Posix:: namespaces. This is useful for dynamic task
generation:

    Io::read_file path      ;; Read entire file contents
    Io::write_file path str ;; Write string to file
    Io::getenv VAR          ;; Get environment variable
    Posix::getcwd           ;; Get current working directory
    Posix::exists? path     ;; Check if file/directory exists
    Posix::file? path       ;; Check if path is a regular file
    Posix::dir? path        ;; Check if path is a directory
    Posix::glob pattern     ;; POSIX glob pattern matching
    Posix::readdir path     ;; List directory contents
    Posix::file_mtime path  ;; Get modification time (Unix timestamp)

Standard LCL features like lists, dicts, string manipulation, and
control flow are also available. More documentation forthcoming.

*Planned feature*: Expect integration -- LCL includes a reimplementation
of Expect for interactive process control. Future versions will allow
inline expect scripts within task definitions for interactive builds,
license acceptance prompts, and similar use cases.

Dagwood also deliberately lacks some features:

- Dagwood doesn't have a timeout feature.

But it's designed to work with the coreutils "timeout" command.

- Dagwood will not watch files and continuously rerun itself.

But Dagwood has been designed to work with entr or inotifywait.

- Dagwood doesn't cache builds (thank God).

Use Dache (one of our other projects) -- we strongly believe building
and caching are orthogonal concerns. Dache works very well with
Dagwood.

- Dagwood isn't even hermetic!

You're free to use Dagwood to spawn tasks that run Guix, Nix, or Bazel
builds.

- Dagwood isn't really even a build tool!

Correct, it just builds a DAG of shell commands and runs them
concurrently. You will still need a compiler or even a
language-specific build-tool to build whatever tasks Dagwood is
expected to run.

However, this also makes it useful for many things beyond compiling
software. We even use Dagwood internally for running tests, CICD,
etc. There's a lot of power in how stupidly simple this is.

Some FAQs:

Q: How is this different from `make -j`?

A: At execution time they now schedule the same way: jobs launch as
soon as their prerequisites are satisfied, bounded by `-j`. The
difference is everything around that. Make offers no DAG
introspection, silently fails when targets are missing or misdeclared,
and (most importantly for our use-cases) is extremely brittle when
composing across multiple projects.

Dagwood builds the entire graph upfront, validates inputs and outputs,
detects cycles before running anything, and can show you the
dependency layers (`--dry-run`, `--dot`) that the scheduler is
honoring.

Q: I already use recursive make. What's the advantage here?

A: Recursive make often breaks parallelism and requires a ton of
boilerplate to coordinate dependencies. It's also error-prone and can
silently skip sub-builds.

Dagwood supports composing many projects into a unified graph. Tasks
from different projects can depend on each other explicitly, and the
workspace graph can be inspected and validated before execution.

Q: How is this different from Bazel?

A: Although we took some inspiration from Bazel, Dagwood is quite
different and probably closer to Make.

Dagwood is substantially more simple than Bazel, has no built-in
caching, and does not even emit any files by itself. It uses LCL
(a Tcl-like language) instead of Starlark, does not simultaneously
run as a build-server, does not listen to syscalls to figure out
what to cache, etc.

The inspiration Dagwood took from Bazel is importing and namespacing
other projects (even third-party ones), and being able to override
or extend imported tasks without forking them.

This is useful when, say, you want to compile a patch or plugin into a
submoduled project -- but you don't necessarily want to fork the
project or maintain a parallel repository.

We also use this internally because we believe in separating code from
deployment configuration and release management.

Quick Start:

    # Build (requires C compiler, cmake, and xxd)
    cmake -B build && cmake --build build

    # Create a Dag file
    cat > Dag << 'EOF'
    project hello {
        task greet {
            run { echo "Hello, World!" }
        }
    }
    EOF

    # Run it
    ./build/dag

    # List tasks
    ./build/dag --list

    # See the DAG
    ./build/dag --dot | dot -Tpng -o dag.png

    # Run the full test suite (regular + AddressSanitizer)
    ./test/check_all.sh