Support for per task images about toast HOT 5 CLOSED

Just FYI

Drone CI has this feature where you only have the option of specifying an image per task, but you cannot specify a global image.

name: default

kind: pipeline
type: docker

steps:
- name: backend
  image: golang
  commands:
    - go get
    - go build
    - go test

- name: frontend
  image: node:6
  commands:
    - npm install
    - npm test

I tried to replicate this in toast and I seem to have been succesful. For this dummy toast file

tasks:
  foo:
    image: ubuntu:20.04
    description: foo
    command: |
      echo "foo"
      cat /etc/*-release
  bar:
    image: debian:buster
    dependencies: [foo]
    description: bar
    command: |
      echo "bar"
      cat /etc/*-release

the output looks like this:

~/toast> ./target/debug/toast --read-local-cache false --write-local-cache false
[INFO] Ready to run 2 tasks: foo and bar.
[INFO] Running task foo… with image ubuntu:20.04…
foo
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.2 LTS"
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
[INFO] Result of run = Ok("toast-6a78367af5c06e9ad37c9953ac861f1d16cf30909339a04d95caf3c89128fdb6")
[INFO] Running task bar… with image debian:buster…
bar
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
[INFO] Result of run = Ok("toast-256d10a1e470de627c3d4b185b515648f55fc0ba3509dcd8f883045fea4cb55b")

You can check this commit - rohan-sircar@e3b1572 for the corresponding changes. Not sure if you'd be interested in merging this into the main repo though.

I had a few questions about toast such as it's design and workings and stuff, where would it be best to ask?

from toast.

Hi @zimme, thank you for opening this issue, and I apologize for the slow response. I sympathize with the annoyance of needing multiple toastfiles to use multiple base images.

I think this would be a tricky feature for Toast, since Docker images don't support multiple inheritance (for good reasons). One idea is that we allow tasks to either specify a base image or specify dependencies (but not both), and require that all the (transitive) dependencies of a task ultimately come from the same base image. This would complicate the codebase (and the documentation), and I'm not sure if that complexity is worth it.

I can't make any promises that anything will happen on this front, but I will leave this issue open for now since I recognize it as a valid feature request.

If you or anyone reading this is interested in contributing this feature in a pull request, I welcome it, but I recommend discussing the design with me first to de-risk it.

from toast.

Thanks @rohan-sircar for that context. Having per-task images does seem useful.

Feel free to ask general questions about Toast and its design by creating issues in this repository with the question label.

Regarding your change, I cannot merge it as is, because it seems to remove one of the main features of Toast: being able to have tasks depend on other tasks and use files generated from those dependencies. This is not a change which I would support, but I am open to exploring a more hybrid approach in which tasks can have either dependencies on other tasks or their own base image. But note that the design would have to be a bit more nuanced, because a task's transitive dependencies must all be based on the same base image.

from toast.

Feel free to ask general questions about Toast and its design by creating issues in this repository with the question label.

Will do!

because it seems to remove one of the main features of Toast: being able to have tasks depend on other tasks and use files generated from those dependencies.

Ah, right, hadn't considered that. The way it works in Drone is that a common directory is shared by all tasks, probably by using a volume mount, and arbitrary files and directories could be shared by additional volume mounts. But that's probably not appropriate for Toast.

My use case is slightly different though, I wanted to use Toast as the base in a project, so we may not want the same things. But I still wanted to know your thoughts.

from toast.

I think this feature would be too complicated / impractical for Toast, so I'm going to close this issue now. I think the best workaround to have independent DAGs of tasks with different base images is simply to use multiple Toastfiles, or to use something like Docker Compose if dependencies between tasks are not needed.

I am open to revisiting this if there is something I haven't considered.

from toast.