Beyond started with the Guix workflow language

Recap

In the previous section we defined a simple process. Now it's time to look more closely at workflows.

Defining workflows

A workflow describes how processes relate to each other. So before we can write the workflow, we must define some processes. In this example we will create a file with a process named create-file, and we will compress that file using a process named compress-file.

process: create-file
  outputs "/tmp/file.txt"
  run-time
    complexity
      space 20 MiB
      time  10 seconds
  # { echo hello > {{outputs}} }

process: compress-file
  packages "gzip"
  inputs "/tmp/file.txt"
  outputs "/tmp/file.txt.gz"
  run-time
    complexity
      space 20 mebibytes
      time   2 minutes
  # { gzip {{inputs}} -c > {{outputs}} }

With these definitions in place, we can run both in one go by defining a workflow.

workflow: file-workflow
  processes
    auto-connect create-file compress-file

The workflow specifies all processes that should run. The auto-connect procedure links up all inputs and outputs of all specified processes and ensures that the processes are run in the correct order. Later we will see other ways to specify process dependencies.

Process templates

We can parameterize the inputs and outputs for a process, so that the same process template can serve for different inputs and outputs. Here is a process template that is parameterized on input:

process: (compress-file input)
  name
    string-append "compress-file-"
                  basename input
  packages "gzip"
  inputs input
  outputs
    string-append input ".gz"
  run-time
    complexity
      space 20 mebibytes
      time  10 seconds
  # {
    gzip {{input}} -c > {{outputs}}
  }

Dynamic workflows

We can now dynamically create compression processes by instantiating the compress-file template with specific input file names. We use Scheme's let, and map to simplify the work for us:

process: (create-file filename)
  name
    string-append "create-file-"
                  basename filename
  outputs filename
  run-time
    complexity
      space 20 mebibytes
      time  10 seconds
  # { echo "Hello, world!" > {{filename}} }

process: (compress-file input)
  name
    string-append "compress-file-"
                  basename input
  packages "gzip"
  inputs input
  outputs
    string-append input ".gz"
  run-time
    complexity
      space 20 mebibytes
      time  10 seconds
  # { gzip {{inputs}} -c > {{outputs}} }


;; All inputs files.  The leading dot continues the previous line.
define files
  list "/tmp/one.txt"
     . "/tmp/two.txt"
     . "/tmp/three.txt"

;; Map process templates to files to generate a list of processes.
define create-file-processes
  map create-file files

define compress-file-processes
  map compress-file files

workflow: dynamic-workflow
  processes
    auto-connect compress-file-processes create-file-processes

In the GWL, we can define process dependencies explicitly. This is useful when processes don't have explicit outputs or inputs. Processes can do something other than producing output files, such as inserting data in a database, so process dependencies can be specified manually.

Restrictions can be specified as an association list mapping processes to their dependencies, or via the convenient graph syntax.

workflow: graph-example
  processes
    graph
      A -> B C
      B -> D
      C -> B

Reusing workflows in new workflows

On the next page, we will reuse parts of dynamic-workflow above in a new workflow.

Go beyond beyond started