skroutz / mistry Goto Github PK

This might be beneficial when documentation gets bigger and we need page hierarchies and stuff.

Right now, there are no limits enforced on how much time a job could take. This might lead to cases where clients wait on a never-ending job, that also consumes resources (ie concurrency/backlog configuration options).

For example, this happens on some of our projects when a network issue occurs between our internal network and rubygems (ie. fastly CDNs), resulting in CI builds getting stalled indefinitely.

There should be a hard limit as to how much time a job may take so that we safeguard against such situations. Ideally, it should be both a global configuration that applies across all projects AND a per-project configuration (not sure how this should be specified though).

In some scenarios, you want the artifacts fetched from mistry to completely override the previous contents of the --target-path. To support such use cases, we could add a --replace (or replace-target) boolea option that, if enabled will completely replace the given path.

In the case of rsync, it would just pass the --delete option to rsync.

We could report if the build was cloned from a previous build (ie. build cache was used) or not. This will be useful for debugging purposes.

This could be a --force option. This needs to implemented in JobRequest{} too.

https://blog.travis-ci.com/2017-03-22-introducing-auto-cancellation

This could look like this:

$ mistry build --project foo --stream-output
Building zxc...
Building abc...

Sometimes, it's necessary to just trigger a job and not wait for it to finish.

A common scenario is when someone wants to eagerly prepare a build that knows is going to be needed in the future (eg. triggering a build from a github webhook to precompile the assets of a Rails application).

The server should support this option first, and then it has to be implemented in the client.

These would demonstrate the complete flow and would contain real flags. For example:

$ mistry-cli --project rails --group staging --target /tmp

# ... server output

# ... synced artifacts

Currently, if data_path is a symlink, docker on Mac complains with:

$ ./mistry-cli build --host localhost --project simple
2018/04/12 13:21:47 Error creating job: Error building {project=simple params=map[] group= id=3dc923b}: work: could not start docker container; Error response from daemon: invalid mount config for type "bind": bind source path does not exist
, http code: 500

This is due to docker/for-mac#1298.

We should resolve symlinks after reading the configuration, and update it accordingly before booting the server.

Document how to use incremental building (ie. groups). This should be mentioned briefly in the README but explained thoroughly in a new wiki page.

The following naming scheme seems more natural:

• server: mistryd
• client: mistry

This needs not affect the repository/project name, only the generated binaries.

There are already some test helpers like assertEq, assert, failIfError, assertNotEq.

This issue is about standardizing their usage such as removing duplicate behavior or extracting them to a proper package.

Right now, we get a bunch of json lines. We could perhaps https://github.com/moby/moby/blob/master/pkg/streamformatter/streamformatter.go to parse them.

This could be a configuration option in the json.

Error message:

$ ./mistry build --host localhost --port 8462 --project simple_with_directory \
 --group foo --target /tmp/foo --transport scp -- --x=$RANDOM

/tmp/mistry: Is a directory

The project used should produce artifacts with a directory structure depth equal or bigger than 3.

e.g.

files
└──more_files
    └── foo.js

BuildInfo.Err is supposed to contain build error messages, if any. At the moment, we do not populate it. We should, when possible (eg. when a docker build or docker run fails).

If jobs for a new project (that hasn't been bootstrapped yet) happen to arrive at the same time, one of the (2) calls to BootstrapProject() might fall due to an "directory exists" error.

We could introduce some locking when bootstrapping a project.

We already added the job params inside BuildResult{} so it might make sense to add the whole job.

Some times builds are timing out: https://travis-ci.org/skroutz/mistry/jobs/362986195

When using the btrfs filesystem, the mod time of the subvolumes that have been snapshotted is the mod time of the first subvolume. This creates an ambiguity when running stat or ls -al.

We should update the mod time to reflect the actual mod time of the directory.

https://golang.org/pkg/os/#Chtimes may help mitigate this.

https://github.com/skroutz/mistry/wiki/File-system-layout

We should mention latest symlinks (with or without groups) in the aforementioned wiki page.

As it stands the rsync module is hardcoded to be "mistry":

module := "mistry"

The client should be able to set this option.

Right now, on $ mistryd -h we only show the default filesystem adapter ("plain"). We should somehow display all the available ones.

The user should be able to specify for example flags passed to rsync and also the source directories.

Handle non-existing ready and pending directories of jobs so that the index page doesn't break.

This can happen in cases where the BootstrapProject hasn't finished the directories creation when the index handler polls for jobs.

It should document how one can install the package and how to use it.

Right now, we remove containers in StartContainer() after they run successfully. However, it might happen, for example in case of a mistry failure while a container was running (eg. SIGKILL), that containers are left around with nobody to remove them.

We should decide on how to treat such cases.

Backend

• investigate if we can determine when job has started without looking at the path's mod time. Maybe through build_result.json.
• properly display docker build logs (#37)
• properly format "started at" in job/show
• view: dynamically iterate on build result and print it
• make broker package SSE-specific (maybe this is just a rename), also try to make it simpler
• introduce package webtailer (provides the HTTP handlers and uses package tailer & broker)
• extract presentation logic to a separate struct, if necessary
• make job show handler return a consistent JSON (ie. JobInfo type)
• show all job fields when marshalling to JSON
• check for current jobs (job index) in a goroutine, once for all clients

This should make it feasible to prune mistry-specific containers on boot etc.

This layer (let's suppose it is a Storage package) will expose functions like:

func (s *Storage) GetJobs(project string) ([]Job, error)

and

func (s *Storage) GetJob(project, id string) (Job, error)

This will abstract the storage solution away and will enable more intuitive interactions in regard with fetching saved jobs.

Right now, the client will block indefinitely if mistry is down (connect timeout) or a job takes forever to complete (read timeout).

However, there are cases when this is not acceptable. We should have the option to set a read timeout and a connect timeout on the client. Perhaps we only should expose this as a single timeout (how much the whole command should run for).

This information exists in the BuildInfo struct as of #47.

The job marshalling should also be updated accordingly.

It is hard to setup, not going to be used in production environments and overall rsync is a better solution. The only think we decided to add scp was for use in tests and as a more easy-to-setup default than rsync. However, it seems that it's not so easy since it still requires setting up SSH keys and also required more configuration to work as we want it (as opposed to rsync).

If scp doesn't provide the benefits we chose it for, we should drop it completely. For this, #52 should be done first so that we have an alternative (ie. running tests with rsync without having to setup an rsync daemon).

We should add 3-5 use cases for mistry. Some ideas: rake assets:precompile, bundle install, yarn install . We could be creative here 😄

There's an existing work in branch use-cases.

We should handle errors that the user cannot do something about and somehow segregate them from user-land errors.

For example, an container failed to build because another container with the same ID already exists, is not something the user fix.

agis:~/dev/go/src/github.com/skroutz/mistry [master] $ ./mistry-cli build --project simple --target nonexistingpath
agis:~/dev/go/src/github.com/skroutz/mistry [master] $ ls nonexistingpath
nonexistingpath
agis:~/dev/go/src/github.com/skroutz/mistry [master] $ file nonexistingpath
nonexistingpath: empty

Expected: nonexistingpath should be created as a directory, and the build artifacts should be inside it.

I think that it makes more sense not to cache build failures, most of the time those would be transient errors that should be retried next time someone asks for that build.

In that case we might want to keep the build directory (in a failed subdirectory?) for debugging purposes.

There should be a strategy for somehow pruning old builds. This could be in the form of a "retention period" configuration.

This is necessary, otherwise disk usage will grow indefinitely.

This requires putting both of these main packages under a cmd directory (which is a Go convention).