tensorchord / envd Goto Github PK

Description

For example, users need to make sure that the code change is committed and pushed before the environment is destroyed. Thus we need to support rules to define lifecycle hooks.

Description

Support jupyter to develop.

Feature lists

• Support jupyter rules
• Support custom config like port, pwd and so on.

Ref #1

Description

Users want to know all the dependencies in the environment. We can provide a command like MIDI dependency list to list them.

• Mirrors
• IDE configs

Description

	bkClient, err := client.New(clicontext.Context, "unix:///run/buildkit/buildkitd.sock")

We should avoid it.

We can create the buildkitd container to boostrap.

Description

Code may be one of the first-class objects in the build language. The code will exist in the resulting container and users develop models and update them. After the code is changed, you can commit and push to remote.

CODE:
    GIT SSH /root/.ssh/github-key
    GIT LOCAL $HOME/private-repo
    GIT CLONE [email protected]/tensorchord/private-repo.git
    # TODO: Support branch and commit

Feature lists

• #23
• Support public repos
• SSH keys to support private repos
• Cache if the repo exists locally

Description

Just like docker build ./subdir, we need to support users to use other dirs instead of the current working dir to build MIDI.

		&cli.PathFlag{
			Name:  "path",
			Usage: "Path to the directory containing the build.MIDI (Default is current directory)",
			Value: ".",
		},

Ref #43 (comment)

Description

Users need to specify os and lang

Acceptance Criteria

• #20
• Support more OS and language

Description

[+] Building 103.2s (11/11) FINISHED                                                                                                                                                                       
 => docker-image://docker.io/nvidia/cuda:11.2.0-cudnn8-devel-ubuntu20.04                                                                                                                              2.8s
 => => resolve docker.io/nvidia/cuda:11.2.0-cudnn8-devel-ubuntu20.04                                                                                                                                  2.6s
 => local://context                                                                                                                                                                                   3.1s
 => => transferring context: 50.52kB                                                                                                                                                                  0.1s
 => CACHED sh -c apt-get update && apt-get install -y --no-install-recommends python3 python3-pip                                                                                                     0.0s
 => CACHED apt install gcc                                                                                                                                                                            0.0s
 => CACHED pip install -i https://mirror.sjtu.edu.cn/pypi/web/simple jupyter ormb                                                                                                                     0.0s
 => CACHED mkdir /var/midi/remote                                                                                                                                                                     0.0s
 => CACHED mkdir /var/midi/bin                                                                                                                                                                        0.0s
 => CACHED copy /examples/ssh_keypairs/public.pub /var/midi/remote/authorized_keys                                                                                                                    0.0s
 => CACHED copy /bin/midi-ssh /var/midi/bin/midi-ssh                                                                                                                                                  0.0s
 => CACHED merge (apt install gcc, pip install -i https://mirror.sjtu.edu.cn/pypi/web/simple jupyter ormb, copy /bin/midi-ssh /var/midi/bin/midi-ssh)                                                 0.0s
 => exporting to oci image format                                                                                                                                                                    96.6s
 => => exporting layers                                                                                                                                                                               0.0s
 => => exporting manifest sha256:32bbf82b70e17ca70b11b88310ef02450db2bed3b94765415650a2227baa63cf                                                                                                     3.1s
 => => exporting config sha256:9bcaf4d291970033f2a6316dbf11912e77de402c81f0b11896f16c8bab19360b                                                                                                       1.2s
 => => sending tarball                                                                                                                                                                               91.6s

The image is built in buildkit, and it does not exist in the docker host. Thus we need to pipe the buildkit build image into the docker host. It takes about 100s for a 20G base image docker load. It is too slow. We need to optimize it.

Description

Support bash/zsh auto complete

Description

Idle environments should be culled.

Description

Users want to define the entrypoints e.g. MIDI serve to run some user-defined logics.

Description

Support ssh with rules to attach into the environment

Ref #1

Feature lists

• Support setting up the sshd server in the container
• Port forward from the container to localhost.

Description

base(os="ubuntu20.04", language="python3")
print(midi.os)

We need to provide some built-in variables to support conditional build like this

if midi.os == "ubuntu":
    ubuntu_apt_source("xxx")

priority/low

up command runs the container and ssh into it.

Feature lists

• docker container creation
• ssh

Description

We should add a real example, maybe a MIDI file for ResNet model training, to illustrate what MIDI does.

priority/low

Description

Support vscode remote-ssh to develop

Feature list

• Support vscode rule
• Support plugin config

Ref #1

Description

scp is not supported for the container. We should support it.

We need to brainstorm the name of the project. MIDI is not friendly for SEO.

Description

Ref #48

We hard-coded the ssh key in MIDI and sshd package. We should remove them.

Description

Investigate if we should or can support distributed training. It is in low priority since most users do not need distributed DL training.

• Design command
• Mount into the container in runtime

Description

We need to init a docker client to load the image into the docker host. But we may got the error here

ERRO[2022-04-21T22:08:35+08:00] failed to load docker image: Error response from daemon: client version 1.42 is too new. Maximum supported API version is 1.41 

We should suggest setting the envvar DOCKER_API_VERSION to avoid the issue when encountering this.

export DOCKER_API_VERSION=1.41

Description

Support MIDIfile as the input, instead of the hard coding file name in the code.

We can prototype the desired README although there is no runnable code, to help us understand the benefit of the project and the features that we need.

Description

We need set up the test process and add more unit/integration test cases. There are many interfaces in the code, thus it should be easy to mock them and test the main logic.

Acceptance Criteria

• test for IR and frontend
• test for cache
• test for docker client

Discuss whether we can expose the lower level primitive of buildkit into starlark

For example, the torch installation logic is complex. https://pytorch.org/get-started/locally/

We can write the logic as below, which is quite straightforward

# expose p as the builtin command
def pip_package(name):
    p.cache("/root/.pip/cache")
    p.exec("pip install {}".format(name))

def install_torch():
    if global.cuda_version == '10.2': 
       pip_package("torch torchvision torchaudio")
    elif global.cuda_version == '11.3':
       pip_package("torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113")
    elif global.cuda_verion is None:
       pip_package("torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cpu")

@gaocegege

Description

Users may install some new dependencies in the environment and they are not tacked by the declarative manifest file. We need to support a command like MIDI sync to sync the change to the manifest file.

Description

I think the sshd server may be not running when the client is trying to connect. We can add some retry to avoid it.

midi up
error: dial to [172.17.0.2:2222](http://172.17.0.2:2222/) failed dial tcp [172.17.0.2:2222](http://172.17.0.2:2222/): connect: connection refused

Additional Info

Description

Description

Now we have a simple way to extract the publisher/version/extension from the frontend func calls. But it does not always work

For example dbaeumer.vscode-eslint-1.1.1. Current implementation gets the version eslint-1.1.1

We need to find the right first -, instead of the left first - for indexExtension

func ParsePlugin(p string) (Plugin, error) {
	indexPublisher := strings.Index(p, ".")
	publisher := p[:indexPublisher]
	indexExtension := strings.Index(p[indexPublisher:], "-") + indexPublisher
	extension := p[indexPublisher+1 : indexExtension]
	version := p[indexExtension+1:]
	logrus.WithFields(logrus.Fields{
		"publisher": publisher,
		"extension": extension,
		"version":   version,
	}).Debug("vscode plugin is parsed")
	return Plugin{
		Publisher: publisher,
		Extension: extension,
		Version:   version,
	}, nil
}

Description

Not sure whether we should choose /root/ as the default directory. What's the common practise for the user when using docker? Does people use root or create other users?

#54 (comment)

Now we use a simple struct to keep the packages and dependencies. It works in most cases. But if users need to run some custom commands, the expression can be hard to support in the current design.

Thus we may need to re-design this in the future.

Currently the SSH is hard-coded at examples/ssh_keypairs. We may need to make it flexible to configure.

Description

Now the resulting image is not kept in the local docker daemon. We should keep it there.

Description

We need to provide built-in support for dependencies.

Feature lists

• System dependencies
• PyPI depencies (and conda)
• Cache

Description

vscode remote, jupyter and terminal are the most used tools for data scientists to develop models. Thus we need to support them as first-class object.

Feature lists

• Support vscode remote, jupyter, ssh, and other possible editors
• Support plugins for editors (e.g. python plugin for vscode remote)
• Support other configurations for the editors (e.g. TCP port used or jupyter password)
• Support per-user config. There is no need for users to config in every project
• Support runtime override. It is to support remote config. Imagine that, you want to create an environment from a manifest file in the remote git repository. But the manifest defines jupyter as the default IDE, but you want to use vscode remote. Runtime override can help you with this.

Acceptance Criteria

• #16
• #17
• #18
• #22

Description

run(command="ls -l")

Users may need to run some specific commands. We should support that.

Description

Users need to get the data to run the training jobs. Thus the build language needs to support it

Feature lists

• Runtime fetch. We should not place data in the output artifact. Instead, we fetch the data when the container is initialized.
• Credentials (per-user maybe) to get the data from S3 or other OSS.

Description

Now the progress is ugly, we should show them in a fancy way.

Description

• CUDA command
• Decide if we install cuda in base image or install it as a dependency

We do not check if the client can connect to the buildkitd in bootstrap command.

priority/medium .

Description

#14 (comment)

IMO, the crucial use of pkg/error is it implements the method Unwrap, and thus we can use errors.Is to see if two errors are the same.

But pkg/error is in the maintain mode, should we use this one instead?

• No command

Description

CUDA and other runtime dependencies are optional. Runtime override also helps us in this scenario like #1

To mount the folder into the container, several options available:

• docker run -v ... We map the host dir to the container at runtime
• sshfs Use volume plugin to mount the fs through ssh(sftp). Worse performance comparing to -v solution, but works when the container runs on the different machines

Now we enable CGO in target build-local, but it is not friendly to develier. We can write a new target build-and-deliver to disable CGO

/chore

Description

We need a new plugin in vscode to lint/auto-complete the MIDI lang.

• Linting
• Auto-complete
• highlight

Description

We need to support mirrors in the frontend lang.

apt_source("""
deb https://mirror.sjtu.edu.cn/ubuntu focal main restricted
deb https://mirror.sjtu.edu.cn/ubuntu focal-updates main restricted
deb https://mirror.sjtu.edu.cn/ubuntu focal universe
deb https://mirror.sjtu.edu.cn/ubuntu focal-updates universe
deb https://mirror.sjtu.edu.cn/ubuntu focal multiverse
deb https://mirror.sjtu.edu.cn/ubuntu focal-updates multiverse
deb https://mirror.sjtu.edu.cn/ubuntu focal-backports main restricted universe multiverse
deb http://archive.canonical.com/ubuntu focal partner
deb https://mirror.sjtu.edu.cn/ubuntu focal-security main restricted universe multiverse
""")