Ebro

Ebro is a task runner. Tasks are defined inside YAML files, scripted with Bash, and configured with a name, requirements, when to skip execution, and other details. Tasks can also be imported from other files or extend other tasks.

Ebro is distributed as a single binary, including the script interpreter (mvdan/sh).

It's heavily inspired in go-task/task, but originally built around a personal need for configuring servers, although it's not tied to this use case and remains agnostic.

Getting started

The format of Ebro.yaml files is defined here. Here is an example:

tasks:
  default:
    requires: [echoer, producer]

  echoer:
    script: |
      cat cache/A.txt
      cat cache/B.txt
    when:
      output_changes: |
        cat cache/A.txt
        cat cache/B.txt

  producer:
    requires: [produce_a, produce_b]

  produce_a:
    requires: [cache_dir]
    required_by: [echoer]
    script: echo 'this is A'>cache/A.txt
    when:
      check_fails: test -f cache/A.txt

  produce_b:
    requires: [cache_dir]
    required_by: [echoer]
    script: echo 'this is B'>cache/B.txt
    when:
      check_fails: test -f cache/B.txt

  cache_dir:
    script: mkdir -p cache
    when:
      check_fails: test -d cache

To give it a try, create a folder in your system, copy the content above in a file inside it called Ebro.yaml, and also download the ebrow script alongside it.

curl --location --output ebrow 'https://github.com/sirikon/ebro/releases/latest/download/ebrow'

Before running it, read the script and understand what it does (because you shouldn't blindly execute scripts from the internet).

ebrow is Ebro's "workspace script". It is a Bash script that contains a reference to an exact Ebro version and is able to download it, verify its integrity, and place it inside the .ebro directory created next to itself. The next time you execute it, it will use the already-downloaded Ebro binary. It ensures that the correct binary is present in the workspace.

Now, let's give it execution permissions and execute it:

chmod +x ebrow
./ebrow

Ebro on start will check for a file called Ebro.yaml in the working directory and parse it if present, constructing what is called the inventory, a collection of every task available with their definitive configuration for running.

You can check the inventory yourself by calling ./ebrow -inventory, you'll notice extra details like the definitive list of extra environment variables that will be included in each task execution, or the working directory.

Next, it will construct a plan, which is an ordered list of all the tasks that will be executed sequentially in order to reach our target task, which by default is default, but can be any other by passing it as an argument (./ebrow echoer).

Again, check it yourself by running ./ebrow -plan. This plan is deterministic, which means that given the same configuration, it will always be the same.

Finally, it will execute the plan, running tasks sequentially until the end.

During the first execution it will execute everything, with no skips, which should output something like this:

$ ./ebrow
███ [:cache_dir] running
███ [:produce_a] running
███ [:produce_b] running
███ [:echoer] running
this is A
this is B
███ [:producer] satisfied
███ [:default] satisfied

But if we execute ./ebrow again, we'll see this output:

$ ./ebrow
███ [:cache_dir] skipping
███ [:produce_a] skipping
███ [:produce_b] skipping
███ [:echoer] skipping
███ [:producer] satisfied
███ [:default] satisfied

Ebro skips tasks whenever possible, and the task definition is what mandates when a task should be skipped. In our example, the task echoer is skipped whenever the output of running cat cache/A.txt and cat cache/B.txt doesn't change. In the case of the task produce_a, it skips whenever the command test -f cache/A.txt succeeds, because the file cache/A.txt already exists.

Now we'll manually edit the file cache/A.txt, run ./ebrow again, and see the result.

$ echo 'hello world!' > cache/A.txt
$ ./ebrow
███ [:cache_dir] skipping
███ [:produce_a] skipping
███ [:produce_b] skipping
███ [:echoer] running
hello world!
this is B
███ [:producer] satisfied
███ [:default] satisfied

The when.output_changes checker of the echoer task detected that running cat cache/A.txt and cat cache/B.txt produced a different output when compared with the previous execution, hence, the task is executed again.

Importing tasks

The Ebro.yaml file supports importing tasks from other Ebro.yaml files by defining the imports.*.from parameter. It works like this:

# Ebro.yaml
imports:
  something:
    from: ./somewhere # a directory containing an `Ebro.yaml` file.

# somewhere/Ebro.yaml
tasks:
  default:
    script: echo 'something'
  else:
    script: echo 'something else'

$ ebro something
███ [:something:default] running
something

$ ebro something:else
███ [:something:else] running
something else

It's important to note that the contents of an Ebro.yaml file are considered a module. When we import another Ebro.yaml file, we're creating a new module that hangs from the root module and has an explicitly-given name. In this case, something.

Targeting a module by its name is equivalent to targeting the module's default task. As something is a module, it translates to something:default.

Conditional existence

Tasks can be configured to only exist when another task already exists using the if_tasks_exist parameter. Additionally, we can require tasks only if the referenced task exists by using the ? suffix, and ignore the requirement otherwise.

With this configuration, as the task restic doesn't exist, configure-backups will not exist either, but that's okay, because server's reference to it was optional.

tasks:
  server:
    requires: [configure-backups?]
    script: |
      echo 'Configuring server'

  configure-backups:
    if_tasks_exist: [restic]
    requires: [restic]
    script: |
      echo 'Configuring backups'

$ ebro server
███ [:server] running
Configuring server

But as soon as the restic task exists, this happens:

tasks:
  server:
    requires: [configure-backups?]
    script: |
      echo 'Configuring server'

  configure-backups:
    if_tasks_exist: [restic]
    requires: [restic]
    script: |
      echo 'Configuring backups'

  restic:
    script: |
      echo 'Installing restic'

$ ebro server
███ [:restic] running
Installing restic
███ [:configure-backups] running
Configuring backups
███ [:server] running
Configuring server

Task inheritance

Ebro has a system of task inheritance. Tasks can extend other tasks, merging the parent properties with their own. Check the merging strategy in the schema documentation.

Here's an example Ebro.yaml file and what happens when running ebro on it:

tasks:
  default:
    script: echo 'Hello World'

  parent:
    abstract: true
    environment:
      FOO: "foo"
    required_by: [default]

  child:
    extends: [parent]
    script: echo $FOO

Now, running ebro default child (targeting both default and child tasks) has this output:

$ ebro default child
███ [:child] running
foo
███ [:default] running
Hello World

We can see how the child task looks after the merging strategy is applied with ebro -inventory. It inherited parent's required_by and FOO environment variable, but kept its own script.

:child:
  working_directory: /workdir
  environment:
    EBRO_ROOT: /workdir
    FOO: foo
  required_by:
  - :default
  script: echo $FOO
:default:
  working_directory: /workdir
  environment:
    EBRO_ROOT: /workdir
  script: echo 'Hello World'

CLI

Ebro's command line interface is very straightforward, but has a couple of general rules:

• Commands define an specific action. At most, there is one command in a call. Absence of a command means the default command of running. Commands are prefixed with a single hyphen (-command).
• Flags depend on the command being executed. Their mere presence can mean a boolean value (true or false) or be accompanyed with a value. Flags are prefixed with two hyphens (--flag).
• Targets are the names of the tasks that we want to run. When no target is specified, the task default is assumed.

To know Ebro's available commands with their flags and explanations, run ebro -help (or ./ebrow -help if using the workspace script).

ebro [--flags...] [targets...]
  # Run everything
  flags:
    --file value  Specify the file that should be loaded as root module. default: Ebro.yaml
    --force       Ignore when.* conditionals and dont skip any task. default: false
  targets:
    defaults to [default]


ebro -inventory [--flags...]
  or -i
  # Display complete inventory of tasks with their definitive configuration in YAML format
  flags:
    --file value   Specify the file that should be loaded as root module. default: Ebro.yaml
    --query value  Query the inventory using an `expr` expression


ebro -list [--flags...]
  or -l
  # Display only the names of all the tasks in the inventory
  flags:
    --file value  Specify the file that should be loaded as root module. default: Ebro.yaml


ebro -plan [--flags...] [targets...]
  or -p
  # Display the execution plan
  flags:
    --file value  Specify the file that should be loaded as root module. default: Ebro.yaml
  targets:
    defaults to [default]


ebro -version
  or -v
  # Display ebro's version information in YAML format


ebro -help
  or -h
  # Display this help message

Queries

In -inventory

The -inventory (or -i) command supports a flag called --query for using an Expr expression that transforms the output at your will:

$ ebro -i --query 'tasks | filter(.name == "default") | map(.id)'
- :apt:default
- :caddy:default
- :default
- :docker:default
- :docker:plugins:default

The output is serialized as YAML unless the data resulting from the expression is a string, in which case the result is printed to stdout verbatim.

$ ebro -i --query 'tasks | filter(.name == "default") | map(.id) | join(", ")'
:apt:default, :caddy:default, :default, :docker:default, :docker:plugins:default

Here is the environment available during the expression execution apart from Expr's built-in functions:

• tasks: Array of objects with the following properties:
  • id: (string) Task's ID (ex: :module:task)
  • module: (string) Task's module (ex: :module)
  • name: (string) Task's name (ex: name)
  • working_directory: (string)
  • environment: (string -> string dictionary)
  • labels: (string -> string dictionary)
  • requires: (string array)
  • required_by: (string array)
  • script: (string array)
  • quiet: (bool)
  • interactive: (bool)
  • when: Object with the following properties:
    • check_fails: (string array)
    • output_changes: (string array)
• modules: Array of objects with the following properties:
  • id: (string) Modules's ID (ex: :module:submodule)
  • working_directory: (string)
  • environment: (string -> string dictionary)
  • labels: (string -> string dictionary)

In requires and required_by

The task properties requires and required_by also support querying by expression, but with a difference: The Task properties requires and required_by are not available in the expression environment.

It is not possible to define requires or required_by based on the value of other tasks' requires or required_by.

The Ebro.yaml format

This is also available as a JSON Schema.

The contents of the Ebro.yaml represent a module, which has the following properties:

• working_directory (string)

  The default working directory for all the tasks inside the module. Defaults to the parent module working directory or the current working directory if it is the root module. Relative paths are valid and will be added to the default value to compute the final value.

• environment (object)

  Additional environment variables available for all the tasks inside the module. A limited subset of Bash's capabilities for interpolation (like using ${VAR}) are available to interpolate variables that come from the parent environment (previous variables on the map, parent module or Ebro's execution environment).

• labels (object)

  Dictionary of key/value labels (string -> string). Intended for module authors to put labels that might be useful during -inventory --query or by any third-party tool reading Ebro.yaml files.

  Ebro ignores the contents of labels in regular operations and only uses it for querying when asked to.

  A limited subset of Bash's capabilities for interpolation (like using ${VAR}) are available to interpolate variables that come from the environment.

• imports (object)

  Other modules can be imported.

  • * (string)

    Each key in the object will be the module's name when imported.

    • from (string)

      The path to a directory containing an Ebro.yaml file. Relative paths will be added to the module's working directory to compute the final path.

    • environment (object)

      Additional environment variables available for all the tasks inside the imported module. A limited subset of Bash's capabilities for interpolation (like using ${VAR}) are available to interpolate variables that come from the parent environment (previous variables on the map, current module or Ebro's execution environment).

• tasks (object)

  Collection of tasks defined in the module.

  • * (string)

    Each key in the object will be the task's name.

    • labels (object)

      Dictionary of key/value labels (string -> string). Intended for task authors to put labels that might be useful during -inventory --query or by any third-party tool reading Ebro.yaml files.

      Ebro ignores the contents of labels in regular operations and only uses it for querying when asked to.

      A limited subset of Bash's capabilities for interpolation (like using ${VAR}) are available to interpolate variables that come from the environment.

    • working_directory (string)

      The working directory for the task. Defaults to the module working directory. Relative paths are valid and will be added to the default value to compute the final value.

    • if_tasks_exist (array)

      References other tasks. If any of the referenced tasks don't exist, this task will not exist either.

      The task is purged before the inventory phase.

    • abstract (boolean)

      When true, flags the task as abstract. An abstract task is a task that cannot be executed directly and its only purpose is to be extended by other tasks by using the extends property.

    • extends (array)

      References other tasks and extends them, which effectively means merging their properties with the properties of the referenced task. Here's the merging strategy for each property:

      • labels: Merged. Child task values take precedence over parent task values.
      • working_directory: Untouched.
      • abstract: Untouched.
      • environment: Merged. Child task values take precedence over parent task values.
      • requires: Merged and deduped.
      • required_by: Merged and deduped.
      • script: Concatenated. Child's scripts go after parent's scripts.
      • quiet: If present, child task value takes precedence over parent task value.
      • interactive: If present, child task value takes precedence over parent task value.
      • when.check_fails: Concatenated. Child's scripts go after parent's scripts.
      • when.output_changes: Concatenated. Child's scripts go after parent's scripts.
    • environment (object)

      Additional environment variables available for the task. A limited subset of Bash's capabilities for interpolation (like using ${VAR}) are available to interpolate variables that come from the parent environment (previous variables on the map, tasks referenced in extends, module environment or Ebro's execution environment).

    • requires (array)

      List of task names (or expressions resolving to task names) that need to execute before this one does.

      Tasks are added to the execution plan when referenced by requires.

      To reference tasks from submodules, write the full path of the task (module and task name) with each name separated with a colon (:). Example: submodule:task.

      To reference tasks by their absolute path, prepend the whole path with another colon (:). Example: :module:submodule:task.

      To optionally refer to a task in case it exists, but ignore the reference if it doesn't, add a question mark (?) at the end. Example: :module:submodule:task?.

      To reference tasks based on an expression, define an object with the key query and put the expression as its value. Example: query: 'tasks | filter("something" in .labels) | map(.id)'

    • required_by (array)

      List of task names (or expressions resolving to task names) that require this task to be executed before those. The opposite of requires.

      Tasks are not planned to run due to being referenced by required_by. This just serves to indicate that the referenced tasks, if planned to run, need to run after this one.

      To reference tasks from submodules, write the full path of the task (module and task name) with each name separated with a colon (:). Example: submodule:task.

      To reference tasks by their absolute path, prepend the whole path with another colon (:). Example: :module:submodule:task.

      To optionally refer to a task in case it exists, but ignore the reference if it doesn't, add a question mark (?) at the end. Example: :module:submodule:task?.

      To reference tasks based on an expression, define an object with the key query and put the expression as its value. Example: query: 'tasks | filter("something" in .labels) | map(.id)'

    • script ( string or array )

      Bash script of the task. set -euo pipefail is always prepended to the script. Interpreted by mvdan/sh.

      It can be defined as a sequence of strings instead of a single string, in which case each item of the sequence will be executed in order.

    • quiet (boolean)

      When true, flags the task as quiet. Quiet tasks are task that, visually and logging-wise, look almost the same as skipped tasks. This effectively means that their output will be hidden by default and only shown if the task fails. Also, their log line for "running" will be tinted green instead of yellow.

      Tasks cannot be quiet and interactive at the same time.

    • interactive (boolean)

      When true, flags the task as interactive. Interactive tasks inherit the environment's stdin.

      Tasks cannot be quiet and interactive at the same time.

    • when (object)

      Configure ways in which the task could be skipped. These are computed as an OR, meaning: One of them triggering is enough to trigger the task execution.

      • check_fails ( string or array )

        Bash script. set -euo pipefail is always prepended to the script. Interpreted by mvdan/sh.

        After execution, the exit code will be checked. If it succeeded (exit code 0), the task is skipped. If it fails (exit code different than 0), the task is executed.

        It can be defined as a sequence of strings instead of a single string, in which case each item of the sequence will be executed in order.

      • output_changes ( string or array )

        Bash script. set -euo pipefail is always prepended to the script. Interpreted by mvdan/sh.

        After execution, the output (stdout and stderr, combined) will be compared with the output of the last time it executed and the task succeeded. If the output is the same, the task is skipped. If the output is different, the task is executed. If there is no previous output stored, the task is executed.

        It can be defined as a sequence of strings instead of a single string, in which case each item of the sequence will be executed in order.

• modules (object)

  Modules can have other modules inside

  • * (string)

    Each key in the object will be the module's name

Versioning

Ebro follows SemVer, this means that it's important to clarify what the API is in Ebro, or, "the exposed parts that will only break compatibility on major version releases".

Ebro's API in SemVer terms is composed of, but not limited to:

• The Ebro.yaml specification. The main idea is that an Ebro.yaml file that worked on Ebro version 1.x.x MUST work exactly the same on any future version under the 1.x.x major release. This includes, but is not limited to:
  • How Ebro.yaml is interpreted.
  • Environment variables added by Ebro during the execution.
    • New environment variables prefixed with EBRO_ COULD be added, and this would not be considered a breaking change.
  • How scripts are interpreted and executed.
  • How task names are processed and referred to.
  • In which order are tasks executed.
• The CLI actions, meaning how the machine state changes given an Ebro.yaml file and a set of commands and flags.
  • How the internals of the .ebro directory operate IS NOT part of the API, and this could change at any time without being considered a breaking change.
• The CLI stdout output (NOT stderr) of the -version, -plan, -inventory and -list commands. These commands output a structured representation of data. New data could be added in the future, but existing data will remain the same. This doesn't apply to formatting, as it could change as long as it satisfies the output format specification.
  • The CLI stdout and stderr output of any other command apart from the ones explicitly mentioned above are NOT considered part of the API and could change at any time without it being considered a breaking change.

Unintended functionality that gets included in Ebro but is considered a bug afterwards will NOT be considered part of the API. Bugs won't be kept for the sake of API stability and will be removed. If this was the case, a release including a bugfix that is also a breaking change will have proper instructions for updating any affected Ebro.yaml file.