Background information

When changing unit tests/cli/lib, exaslct uses the latest docker image, because check is based on git sha.
Thus, changes are actually not incorporated when executing tests/commands locally.

Acceptance Criteria

In this case, force rebuild of the docker image.

• For installer
• Usage from https://github.com/exasol/script-languages-release
• Add Readme
• Changelog
• Internal working

Background

• Currently, the exaslct starter script only mounts the current working directory, however the user can specify paths to other locations in certain command line parameters of exaslct, such as --flavor-path, --save-path, --export-path, --cache-directory, --temporary-base-directory, --output-directory, ...
• At the moment, the starter script would use paths inside the runner container for them, if they are not relative the current working directory, as such they are only available in the container and don't get synced to the host

Acceptance Criteria

• path used in the command line parameters are get mounted into the runner container
• the permissions of these mounted directories get fixed after or during the run, because the container and the host might use different uids and gids

Background

• Currently, we produce a docker image for latest, but latest won't be always the same as main

Background

• Currently SLCT provides reuse, which means either reuse an existing or start a new one and keep it
• sometimes, you want to get new environment, but you want to inspect or reuse it afterwards

Dependencies:

• exasol/integration-test-docker-environment#24

Acceptance criteria

• Add a command line option to keep the test environment after the test, but not to reuse an existing.

Background

• In the docker starter exaslct_within_docker_container.sh, we start exaslct in a docker container
• If we use exaslct upload customers probably assume most of the time the forwarded ports of the docker-db to be the ports to connect, too
• However, if you run in docker container and have it not on the host network the port forwards are inaccessible from inside the container
• To provide a more natural behavior we should start the docker starter container with --network host

Background

It turned out that #10 is causing issues with bash version < 4.4, for example on CentOS 7.
The check if an element key is within an associative container is not working the same way on bash 4.2 as on bash 4.4 (using -v).

Acceptance

The bash script needs to be compatible with bash 4.2.

Background

• Currently, the integration test take 30+ minutes. That is too long for rapid development.

Ideas for parallelization

• Run a Github matrix job with one test case per matrix config, see for example https://hanxiao.io/2021/01/24/Speedup-CI-Workflow-in-Github-Actions-via-Strategy-Matrix/

Background

• In exasol/script-languages-release#245 was asked if we keep the docker build cache and if we can use it
• When docker builds images each step in a Dockerfile produces an intermediate image which is kept in the build cache
• Because, exaslct copies the build context to a temporary directory, we probably can't reuse these intermediate images, because Docker doesn't associate the new directory with a previous build
• As such, keeping the build cache is likely not useful

Tasks

• Investigate if the docker build cache can be used?
• If it can't be used, check if we keep it.
• If we keep it, change that.

Acceptance Criteria

• Either, the build cache is used and improve the performance of rebuilds of a single images, or we remove the build cache after the build

Background

• exasol/script-languages-release#243

Tasks:

• extract the exaslct_src from https://github.com/exasol/script-languages
• create a proper poetry package out of exaslct_src
• package exaslct as docker image and provide a starter script which can run it in other repositories solely from docker image
• Setup CI

Background

When creating a release we should verify the correctness of the package version (in pyproject.toml).
It happened in the past that a release was created with the wrong version of the python package.

Acceptance Criteria

Create a GH actions which is trigger during release-droid execution and verifies correctness of the version code.

Background

If upload of container to bucket fs fails, exaslct does not show an error message to the user.

Acceptance Criteria

Show an error message if upload fails (for example is bucketfs password is incorrect)

Background

• we currently update the docker image with the label latest for each merge/push to main, but it would be better to have it point to the last release.

Acceptance Criteria

We need a CI task

• publish-main, which builds the image whenever main get updated (merge, push), it then builds the docker image for the main branch and pushes it with the label main.
• publish-release, which builds the image whenever we create a release (release tag gets created), it then builds the docker image for the commit of this release and pushes it with the release name as label to docker hub. Furthermore, it advances the latest tag to release tag and builds the image also with the label latest and pushes this as well to docker hub.

bash starter_scripts/build_docker_runner_image.sh <label>
bash starter_scripts/push_docker_runner_image.sh <label>

Background:

• the latest tag indicates the latest version the installer should use

Background

In #10 we implemented "mount binding" to the docker container for directories which were given in the parameters.
This mechanism is working if the parameter is given in the form of:

exaslct cmd key1 value1 key2 value2

but not when given in the form:

exaslct cmd key1=value1 key2=value2

The second form is documented in the officiel README.md.

Acceptance Criteria

Change implementation for parameter parsing in exaslct_within_docker_container.sh so, that it collects directories given as parameter list in the form:

exaslct cmd key1=value1 key2=value2

Background:

• CI processes, might use the following environment variables to supply the passwords for the docker registry

  TARGET_DOCKER_PASSWORD
  SOURCE_DOCKER_PASSWORD
  

• The docker runner currently, doesn't forward these

Rationals

• We could use docker run --env or docker run --env-file to provide the environment variables
• Because these environment variables are password the file is the better option, because it is less likely we leak them from there to the stdout

Background

• integration-test-environment version 0.3 contains new docker-db versions and new commandline parameters to set db-mem size and db-disk size

Acceptance Criteria

• updated version project.toml and poetry.lock
• add commandline arguments for db-mem size and db-disk size to run-db-test

Background:

• Caused by exasol/script-languages-release#66
• Some customers want to build language container under Windows, to support this, we need a starter script for Windows which hides the docker voodoo

Background

Parameter path_in_bucket is being instantiated with type luigi.Parameter, but is optional (compare test test_docker_upload).
This causes following warning during test execution:

......./usr/local/lib/python3.6/dist-packages/luigi/parameter.py:279: UserWarning: Parameter "path_in_bucket" with value "None" is not of type string.
  warnings.warn('Parameter "{}" with value "{}" is not of type string.'.format(param_name, param_value))

Acceptance Criteria

Handle absence of parameter path_in_bucket gracefully.

Background:

• Currently, user need to edit files to add packages, this is often inconvenient and it also allows no sanity checks.
• CLI commands could manage the package list files and check for conflicts, incompatibilities checks and select the correct file to edit

Example how this could look like

exaslct add-packages --flavor-path flavors/standard-EXASOL-7.0.0 --python3-package "joblib=1.0.1" 
exaslct add-packages --flavor-path flavors/standard-EXASOL-7.0.0 --python2-package "joblib=1.0.1" 
exaslct add-packages --flavor-path flavors/standard-EXASOL-7.0.0 --cran-package "opitools=1.0.3"
man kann sie auch kombinieren
exaslct add-packages --flavor-path flavors/standard-EXASOL-7.0.0 --python3-package "joblib=1.0.1" --python2-package "joblib=1.0.1" --cran-package "opitools=1.0.3" --apt-package "parallel"
man kann mehrere gleichzeitig hinzufügen
exaslct add-packages --flavor-path flavors/standard-EXASOL-7.0.0 --python3-package "joblib=1.0.1" --python3-package "joblib=1.0.1" --python3-package "many_requests"
oder man kann ganze paket listen hinzufügen
exaslct add-packages --flavor-path flavors/standard-EXASOL-7.0.0 --python3-package-list "path/to/package/list" 
wäre für die anderen package arten equivalent
für python könnte man auch noch überlegen requirement files zu erlauben, ein hauptproblem, von denen, ist ihre laxe specification (d.h. wir würden vermutlich nur ein definiertes subset unterstützen )
exaslct add-packages --flavor-path flavors/standard-EXASOL-7.0.0 --python3-requirements-file "path/to/requirements.txt" 

Background

We want to create a new release with integration-test-docker-environment 0.4.0 integrated.

Acceptance Criteria

• Changelog updated
• Ready to create release with release-droid

Background

We used this parameter to disable several tests in the past, however with exasol/script-languages#224 this environment variable is not being used anymore.

Acceptance Criteria

Remove the variable.

Background

The test environment currently has some assumptions about the folder structure of the respective python test files.
At some places the folders are manipulated manually (for example here and here.
This does not work with the mount solution of #10, and hence the mounting for test folders was removed in #26.

Acceptance Criteria

Re-activate the mounting here and adjust the test environment to work correctly.

Background

In #10 we add volume mount points for each directory given by the arguments.
However, if any of the directories contain symbolic links to paths which are not mounted, the container won't have access to those directories which might cause the respective docker run to fail.

For example assume the following scenario
$HOME
├── flavors
│   └── my_flavor
│   └── flavor_base
│   └── build_deps -> ../../../my_deps/
└── my_deps

running

./exaslct export --flavor-path=flavors/my_flavor --export-path ./out

will not work, as $HOME/my_deps will not be visible from within the container.

Acceptance Criteria

In starter_scripts/exaslct_within_docker_container.sh find all symbolic links of all mount point directories (given by command line arguments) and add to this list. As this might be very expansive, declare a new command line parameter --mount_symbolic_link_directories which enable/disables this functionality.

Alternative:

Add a command line parameter --mount-extra-directory which allows the user to specify additional directories which get mounted to the container. This would also solve problems with get_additional_build_directories_mapping configration in the flavors which in theory could point anywhere in the file system.

Background

In #69 we created a slim version of exaslct with some limitations. These limitations were caused due to the old bash version which is pre-installed on MacOsX (license problem).
It turned out that a new version of bash can easily be installed via homebrew, just with brew install bash.

Acceptance Criteria

1. Adapt exaslct scripts to find and use homebrew version of exaslct (and exit with error if not found). Best approach is to define a env variable in ./exaslct which will be used in all subsequent scripts.
2. Adjust documentation

Background:

• Our starter scripts get more complex bash scripts
• Bash is highly error-prone
• shellcheck statically checks the scripts for common pitfalls

Acceptance Criteria

• Add a script to run it locally
• Setup shellcheck https://www.shellcheck.net/ for the starter scripts in CI
• Fix all issues shellcheck recognizes

Background:

• Currently, each build_steps.py has it own import statement for DockerFlavorAnalyzeImageTask
• These import statement are fixed to one package path and one name
• This complicates changes to the structure in script-languages-container-tool, because if we change the module path or name of the class the flavors will break
• If we inject the import statement during the dynamic load of the build_steps.py file, we can implement compatibility mechanisms, for example we could change the name of the class with import, as follows

from path.to.package import NewName as OldName

• With changes to the package path also only require a change in the injected import statement

Background

Today we have one class TestContainer which takes all arguments from client and then spawns test environment.
Independent of the kind of test database it passes all parameters to SpawnTestEnvironment.
SpawnTestEnvironment then distinguishes the kind of database and uses the internal docker db or an external provided db, see here

This causes some problems, as the interface is not well defined, see here.

Acceptance Criteria

The published interface by SpawnTestEnvironment should be clear. One idea is that the actual database is attached via DependencyInjection

Background:

• construct_docker_runner_image_name.sh generate the tag for docker files
• currently it generates the following exasol/script-languages:container-tool-runner-$VERSION
• however, with this the script push_docker_runner_image.sh fails, because this docker repository doesn't exist
• the correct repository is exasol/script-language-container

Avoid running build of script languages container as root.
The container needs to run temporarily as root in order to create user and groups.
After that, root access is not needed and may cause problems.
A potential way could, be to first create group with the same gid as is set for the docker socket and then create a user with the same uid as the caller and add it to the created group. After that, we drop the root user with su to the created user and call exaslct. This way the user can access the docker socket, but isn't root and writes files with the same uid as the caller. We might need to add the user to an additional group basically all active groups of the caller, such that he can access files or directories of the respective groups. Note: We can't change the owner or group of the docker socket, because we would change it on the host as well.

Background:

• The exalsct installer creates a directory exalsct_scripts and a symlink to entry script.
• At the moment, this symlink seems to use an absolute path which works if you install exaslct on your machine, but not, if you want to commit it to a repository and run it a CI environment

Background

We discovered that different versions of bash can cause different behavior of exaslct command line tools, see #48 .
Because of that we need at least a CI check for CentOs 7, which comes with bash 4.2 (the minimum compatible version).

Acceptance Criteria

A new CI check which validates export functionality on push, in a CentOS 7 enviornment.

Background

• we often need to debug setups on user systems
• we typical need information about the system, e.g
  • OS
  • OS Version
  • Docker Daemon Version
  • Docker Client Version
  • Host can access Docker Hub
  • Docker container can access internet (Ubuntu)
  • bash version
  • gnu utils Version
  • installed exaslct Version
  • Project name
  • Repository Status + Sub module status
    • modified?
    • commit id history until first commit in main branch history
    • paths to exaslct.log files for the last X builds
    • Docker images with exasol script languages prefix
  • available disk space on /
  • are checksums ok

Acceptance Criteria

• would be a new command of exaslct (something like "diagnosis")
• add message like "Please send to Exasol"
• develop a bash script which collect the listed information and prints it to terminal and file
• the script needs to work on minimal dependencies
• the script needs to be as error resilient as possible and collect as much information as possible

Add new integration tests in installer scripts, for testing version number.
For that, extract most part of https://github.com/exasol/script-languages-container-tool/blob/main/installer/tests/test_exaslct_install_template_with_current_ref.sh into separate script and include via source (to keep trap/env variable/etc.)

Background:

• Caused by exasol/script-languages-release#66
• MacOS X doesn't support readlink -f, we need to find a replacement for it

Background:

• When executing 'exaslct spawn-test-environment ...' I don't mount any additional path
• In that case, the variable is empty $chown_directories and the chown in following line fails
  

  script-languages-container-tool/starter_scripts/exaslct_within_docker_container.sh

  Line 144 in 6c312bb

  RUN_COMMAND="/script-languages-container-tool/starter_scripts/exaslct_without_poetry.sh $quoted_arguments; RETURN_CODE=\$?; chown -R $(id -u):$(id -g) $chown_directories; chown -R $(id -u):$(id -g) .build_output &> /dev/null; exit \$RETURN_CODE"

Update changelog.md

Background

• We write the standard output and error for the language container tests only after they have finished
• However, for long-running or if the system fails, it would be good to write them as soon the output is available
• Basically, return a stream from the exec_run at the following position and write it directly to a file

• See ContainerLogger https://github.com/exasol/integration-test-docker-environment/blob/master/exasol_integration_test_docker_environment/lib/logging/container_log_handler.py

Background

At generate activation cmd / upload, the value of alter_session contained None (it should have path-in-bucket).

Situation:

• Currently, the short help messages for the commands of exaslct are not helpful

Usage: exaslct.py [OPTIONS] COMMAND [ARGS]...

Options:
--help Show this message and exit.

Commands:
build This command builds all stages of the
script...
clean-all-images This command uploads the whole script...
clean-flavor-images This command uploads the whole script...
export This command exports the whole script...
generate-language-activation This command generates a alter session...
push This command pushes all stages of the
script...
run-db-test This command runs the integration tests in...
save This command pushes all stages of the
script...
spawn-test-environment This command spawn a test environment with
a...
upload This command uploads the whole script...

Task:

• Add short help messages to command, refer to https://click.palletsprojects.com/en/7.x/documentation/#command-short-help

Backgorund

• exaslct will be used in several repositories, such as the script-languages, the legacy-udfclient-* and later also in the repositories for each flavor
• at the moment exaslct only works on linux and needs at least a linux docker container to run on MacOS and likely also Windows
• as such we are going to distribute exaslct as a docker container which gets build for each commit to the main branch and will be published to docker hub
• we need a minimum set of scripts to run the docker container and mount all necessary directories into it
• these scripts need to be installed and updated in each repository that uses exaslct

Acceptance Criteria

• we need a single script which bootstraps the whole installation and which can install past, current and future versions of exaslct
  • the bestway to do this, is probably, by having a super simple bootstrap script which loads the actual install script for the requested version

Background

• before we moved exaslct to this repo it wrote the stdout/err to exaslct.log
• we need at least this functionality, better would be to write it to the job directory, because exaslct.log gets overwritten

Background

• currently, the default for installer script is main, but we don't upload a docker image for main in the CI.
• the default for construct_docker_runner_image_name.sh is probably also incorrect.

Background

We are migrating all script language container tests to Python3.

Acceptance Criteria

exaslct run_db_test must be executed in Python3.

Background

exaslct is not compatible with MacOsX due to two reasons:

1. Command readlink contains only a subset of the features of the linux version. We need to use greadlink (which comes with gtools, can be installed with "brew install coreutils")
2. Bash version is very old (3.6.2), we need at least 4.2 (due to usage of associative arrays)

Solution

We have two options here:

1. Write a slim version of exaslct which is specifically for MacOsX, which does not support the mount point handling. And which uses greadlink instead of readlink
2. Write exaslct in Go