As a user I would like to be able to use the spark cluster for python-based jobs. This requires that pyspark (and python) are made available in the docker image.

Summary

The current spark docker-image contains the entire spark code and can be used with scala (spark-shell) or sql (spark-sql): pyspark is also installed but to use it requires the following:

• python3 (>=3.6)
• pip (during image creation)
• pyspark dependencies, as listed in this list
• (plus other libs TBD)

Notes

Some things which I've quickly tested:

• to install pip and python

RUN microdnf update && \
    ...
    microdnf install python3 python3-pip && \
    ...

• setting paths etc.

RUN ln -s /usr/bin/python3 /usr/bin/python
RUN ln -s /usr/bin/pip3 /usr/bin/pip

ENV SPARK_HOME=/stackable/spark
ENV PATH=$SPARK_HOME:$PATH:/bin:$JAVA_HOME/bin:$JAVA_HOME/jre/bin
ENV PYSPARK_PYTHON=/usr/bin/python
ENV PYTHONPATH=$SPARK_HOME/python:$PYTHONPATH

• installing dependencies with pip: doing this as root triggers a warning, but results in permissions erros otherwise and we are not using any virtualenvs anyway. Maybe there is a better way of doing this:

# pyspark dependencies
USER root
RUN pip install --upgrade pip
RUN pip install --no-cache-dir numpy pandas pyarrow

USER stackable
...

Rationale

In the context MLOps it is planned to provide one or more environments for the following tasks:

• feature engineering
• model training
  • distributed computations (in line or partitioned feature creation)
  • hyperparameter tuning (a grid-search type arrangement where the model + one combination of parameters is sent to a single worker)
• model management (experimentation, versioning etc.)
• model pipelines

Many of these use python libraries.

The fuse binary is not built and shipped by default by the Hadoop project. That means we'll have to build it ourselves and include it in the image.

cd hadoop-hdfs-project/hadoop-hdfs-native-client
mvn clean package -P native

Results in:

[WARNING] make[2]: *** [main/native/libhdfspp/lib/fs/CMakeFiles/fs_obj.dir/build.make:104: main/native/libhdfspp/lib/fs/CMakeFiles/fs_obj.dir/filehandle.cc.o] Error 1
[WARNING] make[2]: *** Waiting for unfinished jobs....
[WARNING] make[1]: *** [CMakeFiles/Makefile2:2629: main/native/libhdfspp/lib/fs/CMakeFiles/fs_obj.dir/all] Error 2
[WARNING] make[1]: *** Waiting for unfinished jobs....
[WARNING] make: *** [Makefile:146: all] Error 2

plus more stuff before that.

Context: For our 22.11 release we wanted to upgrade base images for our older product images. This raised the question of how far back we should do this and whether we should do it for all images etc.

An outcome of this could also be to keep everything updated for now until we find a reason not to do this anymore.

Images should be tagged using the following scheme:

tag: version - dependencies - stackable_version
version: \d+(\.\d+)?
dependencies: dep (-dep)?
dep: name version
name: [a-z]+
stackable_version: "stackable" version

For example, the full name of the Kafka image should be: docker.stackable.tech/stackable/kafka:2.6.1-scala2.13-stackable0.0.1

The current purely tag based system for tracking product and image versions does not allow us to update older product images with newer base images, without also upgrading the product version.

Goal

We implement a system that supports upgrading base images for older product images. This system should be easy to use without much repetitive manual work.

New Branch System

In the new branching system, we will maintain a release branch per platform release, i.e. a branch release-22.09, release-22.11.
On these branches we will tag platform versions, i.e. 22.11.0-rc1, 22.11.0, 22.11.1.

As we now tag a whole platform release instead of individual products, the build system/GitHub actions should build all images from these tags.

Next Steps

• #236
• #235

Roadmap

As this issue turned out to be more complex than anticipated, we decided to divide it into multiple stages.

Keep our next Release current under the new Scheme

• #236
• #235

LTS Strategy and Upgrading older Images

Do we want to upgrade base images for all previous release, all the time, indefinitely? This is to be decided. Upgrading older product images is not strictly necessary for the next release, so we decoupled this.

• #237
• #238
• #239

Lastly, manual steps should be automated as best we can.

➜  docker-images git:(main) ✗ git tag -l | grep -E ".*-stackable[0-9]+\.[0-9]+\.[0-9]+"
airflow-stackable0.2.0
antora-stackable0.1.0
hadoop3.3.3-stackable0.1.0
hbase-stackable0.4.0
hbase2.4.11-stackable0.7.0
hbase2.4.12-stackable0.1.0
hbase2.4.12-stackable0.2.0
hbase2.4.6-stackable0.7.0
hbase2.4.8-stackable0.7.0
hbase2.4.9-stackable0.7.0
hive2.3.9-stackable0.5.0
hive3.1.3-stackable0.1.0
java-base-stackable0.2.1
kafka-stackable0.3.0
kafka3.2.0-stackable0.1.0
nifi1.15.1-stackable0.1.0
nifi1.15.2-stackable0.1.0
nifi1.15.3-stackable0.1.0
nifi1.16.0-stackable0.1.0
nifi1.16.1-stackable0.1.0
nifi1.16.2-stackable0.1.0
nifi1.16.3-stackable0.1.0
opa0.41.0-stackable0.1.0
pyspark-k8s3.3.0-stackable0.1.0
pyspark-k8s3.3.0-stackable0.2.0
spark-k8s-stackable0.2.0
spark-k8s-stackable0.3.0
spark-k8s3.3.0-stackable0.1.0
spark-k8s3.3.0-stackable0.2.0
superset-stackable0.1.0
superset-stackable0.2.0
superset-stackable0.3.0
superset-stackable1.0.0
superset-stackable2.0.0
superset-stackable2.0.1
superset-stackable2.1.0
superset1.5.1-stackable0.1.0
superset1.5.1-stackable0.2.0
testing-tools0.1.0-stackable0.1.0
tools0.2.0-stackable0.3.0
trino387-stackable0.1.0
trino395-stackable0.1.0
trino396-stackable0.1.0
zookeeper-stackable0.4.0
zookeeper-stackable0.7.1
➜  docker-images git:(main) ✗

superset-stackable0.1.0
superset-stackable0.2.0
superset-stackable0.3.0
superset-stackable1.0.0
superset-stackable2.0.0
superset-stackable2.0.1
superset-stackable2.1.0

superset-stackable0.1.0
superset-stackable0.2.0
superset-stackable0.3.0
superset-stackable1.0.0
superset-stackable2.0.1
superset-stackable2.1.0

superset-stackable0.1.1
superset-stackable0.2.1
superset-stackable0.3.1
superset-stackable1.0.1
superset-stackable2.0.2
superset-stackable2.1.1

The PR #226 changed all the java base image versions. This should not be happening. E.g. Druid only supports Java 11.

Description

String manipulation and especially regex matching is a pain in GH actions (bash). Push tag validation down into the Python script.

@soenkeliebau :

(kafka|zookeeper|nifi|druid|opa|hbase|hadoop|trino|airflow|superset|spark-k8s|pyspark-k8s)([0-9.]+)-stackable([0-9]+.[0-9]+.[0-9]+)$

All images must fulfill the requirements for OpenShift (see stackabletech/issues#207). One requirement is that all images must be based on a Red Hat Enterprise Linux or Red Hat Universal Base Image (see https://redhat-connect.gitbook.io/partner-guide-for-red-hat-openshift-and-container/program-on-boarding/technical-prerequisites#dockerfile-requirements). The Stackable Airflow image is based on the upstream Airflow image which in the end is based on Debian. Therefore the base image must be replaced with a Red Hat UBI.

The upstream Airflow image contains an argument PYTHON_BASE_IMAGE (see https://github.com/apache/airflow/blob/2.2.5/Dockerfile#L47). It should be evaluated if this can be set to an image which is based on a Red Hat UBI.

Goal

We want to implement a system that supports upgrading base images for older product images. This system should be easy to use without much repetitive manual work.

LTS Strategy and Upgrading older Images

Do we want to upgrade base images for all previous release, all the time, indefinitely? This is to be decided. Upgrading older product images is not strictly necessary for the next release, so we decoupled this.

• #237
• #238
• #239

Lastly, manual steps should be automated as best we can.

https://github.com/apache/hbase-operator-tools
https://hbase.apache.org/downloads.html

Unfortunately, we need to make sure that the version the operator tools are compiled for matches the HBase version itself.
I think it makes sense to include this directly in the HBase image but a separate client image would also work

To use HBase/Phoenix tables with compression (e.g. snappy) there have to be particular hadoop native libraries present in the product image. These have been provisionally added for all current supported HBase versions (2.4.6, 2.4.8, 2.4.9, 2.4.11, 2.4.12) in this branch: https://github.com/stackabletech/docker-images/tree/compression-native-libs

For HBase 2.5(.3) it is possible to use an HBase image compiled against Hadoop 3 (3.2.4). There is also a corresponding Phoenix library.

Using these libraries adds about 0.2 GB to the image size (from 1.2 GB to 1.4 GB), though this can be reduced if only the native libraries are retained. This ticket covers implementing a strategy for supplying compression capabilities for supported HBase versions.

Open issues

• using the HBase archive that is bundled with Hadoop 3 results in compression libs (as .jars) being present in the /libs folder e.g. snappy, lz4. These .jars however are not detected by HBase when creating a table with compression (classpath issue?)
• most of the HBase documentation regarding this issue assumes that HBase and Hadoop are co-located, which is obviously not the case when running these products in Kubernetes. We have tried compiling Hadoop with the snappy flag and importing the libraries to HBase pods but this did not get around versioning issues.

Links

https://issues.apache.org/jira/browse/HADOOP-17125

Acceptance criteria

• decide how or whether older (i.e. pre-2.5.3) versions of HBase should have these supporting libraries (if so, then that will probably be in the form of copying native libs into the image, as done in the branch mentioned above. This will probably necessitate some conditional logic when using different approaches for different HBase versions).
• verify that HBase 2.5.3 (compiled with Hadoop 3) supports compression
  • Just using HBase-compiled-with-Hadoop may not be enough as the compression codecs are supplied as .jar files, but the hadoop native libraries are still assumed to be present (this must be done manually as described here)
• decide if the Hadoop inclusions should be "trimmed" (i.e. just compression) or not

As discussed in architecture meeting 04.05.2022.
Sometimes an argument is passed in to a Dockerfile which needs to be used in different forms e.g. for --build-arg PYTHON=3.8 we may need 3.8 in places (for e.g. product downloads), but 38 in others (microdnf commands). There are different ways of deriving and persisting one variable from another, and this issue covers documenting them with examples in the README e.g.

• using the moby approach of choosing a specific base image (see here)
• use an inline script wrapped by a RUN so that the derived variable is available for the scope of that command
• source the environment variables and echo the environment to a file
• etc.

With Issue stackabletech/issues#301 we identified the operators using the tools image.

From this List we are going to distribute those tools into their corresponding product images.

Operator to modify

• zookeeper - no change needed
• hdfs - no change needed
• kafka
• nifi
• trino
• druid

Currently we do not properly set all necessary tags on all docker images when they are pushed into the repository by the github actions.

We should extend the github actions to set all needed tags in all stages (pr, nightly, release)

and remove dependabot

All images must fulfill the requirements for OpenShift (see stackabletech/issues#207). One requirement is that all images must be based on a Red Hat Enterprise Linux or Red Hat Universal Base Image (see https://redhat-connect.gitbook.io/partner-guide-for-red-hat-openshift-and-container/program-on-boarding/technical-prerequisites#dockerfile-requirements). The Stackable Superset image is based on the upstream Superset image which in the end is based on Debian. Therefore the base image must be replaced with a Red Hat UBI.

Description

I was exploring an use-case where we want to use Trino in our DAGs. Looking into how Airflow runs and how the dependencies are handled, I have reach the possible conclusion it might quite easy to add with the way you have created the docker images.

I have build and run it locally with the added dependency and was able to configure the Trino connection and run simple queries against a test Trino cluster, which by the looks of it, works.

HBASE 2.4.9 Release Notes

These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements.

• HBASE-26542 | Minor | Apply a `package` to test protobuf files

The protobuf structures used in test are all now scoped by the package name `hbase.test.pb`.

• HBASE-26512 | Major | Make timestamp format configurable in HBase shell scan output

HBASE-23930 changed the formatting of the timestamp attribute on each Cell as displayed by the HBase shell to be formatted as an ISO-8601 string rather that milliseconds since the epoch. Some users may have logic which expects the timestamp to be displayed as milliseconds since the epoch. This change introduces the configuration property hbase.shell.timestamp.format.epoch which controls whether the shell will print an ISO-8601 formatted timestamp (the default "false") or milliseconds since the epoch ("true").

HBASE Changelog

Release 2.4.9 - 2021-12-23

IMPROVEMENTS:

BUG FIXES:

TESTS:

SUB-TASKS:

OTHER:

Pulling the airflow image i noticed that the last layer (200MB) is duplicated. This results in a unnecessary big image and longer download times.

The command that duplicated the whole layer seems to be chown -R stackable:stackable /stackable here

DoD:

• Remove 200MB from the airflow image

Our docker images currently have a hard coded label RELEASE=1 which was required for Openshift certification. Another label is VERSION which currently is set from the product version (e.g. 3.8.0 for a ZooKeeper 3.8.0).

For RELEASE we want to update to our actual naming of our releases. The next one would be 22.09. This should probably be parameterized in the build script instead of each product image.

The version label does not make sense currently. It will always be 3.8.0 (for a ZooKeeper 3.8.0) even if we adapt, add to or change that image.

The VERSION label should become the full tag we create from that image like zookeeper3.8.0-stackable0.1.0 to clearly identify the image.

This is done when:

• The RELEASE label is set "globally" via a CLI parameter in the build script
• The VERSION label is set to the actually name of the created docker images like zookeeper3.8.0-stackable0.1.0

The current situation for e.g. ubi8-rust-builder:
we use FROM registry.access.redhat.com/ubi8/ubi-minimal AS builder as base image, to install stuff we use && microdnf install --disablerepo=* --enablerepo=ubi-8-appstream-rpms --enablerepo=ubi-8-baseos curl findutils gcc gcc-c++ make cmake openssl-devel pkg-config systemd-devel unzip pearl -y \ && rm -rf /var/cache/yum

This leads to the error that ubi-8-appstream is not found

To update ubi8-rust-builder we need to have the following changes to the dockerfile:

• rustup for target architecture:
  ONBUILD RUN \
if [ $(arch) = "aarch64" ]; then \
. $HOME/.cargo/env && rustup target add aarch64-unknown-linux-gnu; \
else \
. $HOME/.cargo/env && rustup target add x86_64-unknown-linux-gnu; \
fi

• set environment to target architecture:
  ONBUILD ENV \
. $HOME/.cargo/env && \
if [ $(arch) = "aarch64" ]; then \
CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc \
CC_aarch64_unknown_linux_gnu=aarch64-linux-gnu-gcc \
CXX_aarch64_unknown_linux_gnu=aarch64-linux-gnu-g++; \
else \
CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=x86_64-linux-gnu-gcc \
CC_X86_64_unknown_linux_gnu=x86_64-linux-gnu-gcc \
CXX_X86_64_unknown_linux_gnu=x86_64-linux-gnu-g++; \
fi

• set --target for cargo
  if [ $(arch) = "aarch64" ]; then \
. $HOME/.cargo/env && cargo build --release --target=aarch64-unknown-linux-gnu; \
else \
. $HOME/.cargo/env && cargo build --release --target=x86_64-unknown-linux-gnu; \
fi

• change search and copy of compiled binary:
  ONBUILD RUN find /src/target \
-regextype egrep \
# The interesting binaries are all directly in ${BUILD_DIR}.
-maxdepth 3 \
# Well, binaries are executable.
-executable \
# Well, binaries are files.
-type f \
# Filter out tests.
! -regex ".*\-[a-fA-F0-9]{16,16}$" \
# Ignore some .so files
! -regex ".*\.so$" \
# Copy the matching files into /app.
-exec cp {} /app \;

This is the preparation for multi-arch operator images to work.

Sidenote: Working branch https://github.com/stackabletech/docker-images/tree/ubi8-rust-builder-multi-arch

To enable cross-platform building for Docker Images of products and operators, we agreed on using docker buildx build.

To make this happen we need some checks and steps:

• Enable choice of platform in build_product_images.py
• If no particular system is chosen, default to the current system (platform.machine() )
• Check availability of dependencies (Java-Base, Tools etc. ), if not build them first

As a user of druid images I want to be able to use images that apply image versioning consistently. Currently we have:

0.22.1-authorizer0.1.0-stackable0
0.22.1-authorizer0.1.0-stackable0.2.0
0.22.1-stackable0
0.22.1-stackable0.1.0

where 0.22.1-stackable0 does not to resolve to 0.22.1-authorizer0.1.0-stackable0.2.0

For the sake of maintainability we should document where additional scripts and metrics exporters used in our docker images are coming from.

Release Notes for Superset 1.4

Superset 1.4 focuses heavily on continuing to polish the core Superset experience. This release has a very very long list of fixes from across the community.

• User Experience
• Database Experience
• Breaking Changes and Full Changelog

User Facing Features

• Charts and dashboards in Superset can now be certified! In addition, the Edit Dataset modal more accurately reflects state of Certification (especially for Calculated Columns). (#17335, #16454)

• Parquet files can now be uploaded into an existing connected database that has Data Upload enabled. Eventually, the contributor hopes that this foundation can be used to accommodate feather and orc files. (#14449)

• Tabs can now be added to Column elements in dashboards. (#16593)

• The experience of using alerts and reports have improved in a few minor ways. (#16335,#16281)

• Drag and drop now has a clickable ghost button for an improved user experience. (#16119)

Database Experience

• Apache Drill: Superset can now connect to Apache Drill (thru ODBC / JDBC) and impersonate the currently logged in user. (#17353).

• Firebolt: Superset now supports the cloud data warehouse Firebolt! (#16903).

• Databricks: Superset now supports the new SQL Endpoints in Databricks. (#16862)

• Apache Druid: Superset Explore now can take advantage of support for JOIN's in Druid (note: the DRUID_JOINS feature flag needs to be enabled). (#16770)

• AWS Aurora: Superset now has a separate db_engine_spec for Amazon Aurora. (#16535)

• Clickhouse: Superset now includes function names in the auto-complete for SQL Lab. (#16234)

• Google Sheets: Better support for private Google Sheets was added. (#16228)

Developer Experience

• The Makefile for Superset has gone through a number of improvements. (#16327, #16533)

• Add Python instrumentation to pages, showing method calls used to build the page & how long each one took. This requires a configuration flag (see PR for more info). (#16136)

Breaking Changes and Full Changelog

Breaking Changes

• 16660: The columns Jinja parameter has been renamed table_columns to make the columns query object parameter available in the Jinja context.
• 16711: The url_param Jinja function will now by default escape the result. For instance, the value O'Brien will now be changed to O''Brien. To disable this behavior, call url_param with escape_result set to False: url_param("my_key", "my default", escape_result=False).

Changelog

To see the complete changelog in this release, head to CHANGELOG.MD. As mentioned earlier, this release has a MASSIVE amount of bug fixes. The full changelog lists all of them!

In #208 we decided to use a new branching system, to be able to update old docker images to use new base images.

In this ticket we want to implement it, and create a branch for the upcoming release. This ticket does not deal with new base images for older releases. Any sort of automation is also not part of this ticket.

New Branching System

In the new branching system, we will maintain a release branch per platform release, i.e. a branch release-22.09, release-22.11.
On these branches we will tag platform versions, i.e. 22.11.0-rc1, 22.11.0, 22.11.1.

As we now tag a whole platform release instead of individual products, the build system/GitHub actions should build all images from these tags.

Acceptance Criteria

• A new branch has been made for the release 22.11
• a new tag for 22.11 release candidate 1 exists
• The GitHub action has been changed, to build images based on the new tag
• We are ready to release the 22.11 images

Requirements

#236 Should be done first!

Description

We need to Dockerize all our products.
These Docker images will probably evolve over time but initially we need them for all our supported versions, they need to include JMX where relevant and any authorizers needed.

We decided to keep the Dockerfiles in one central location (i.e. this repository) and not colocate them with the operators.

We would like to use the RedHat UBI images as a starting point (they do exist with JDKs as well).

One thing I've always been unsure about is a tagging strategy (e.g. when to use "latest") so that should be part of this.
Our "old" tar.gz packages could serve as a starting point but they were created manually and this might be a good time to make this workflow repeatable and automated.

Please also investigate "buildah" as a replacement for Docker.

Questions

Q: Should we add the packages to our Nexus before building the images ?
A: ??? (It would simplify the image pipeline a lot. For example different Nifi versions have different download urls)

Q: Kafka - which Scala version 2.12, 1.13 or both ?
A: Images are now build with both and also tagged with the scala version.

Q: Trino - should the images also contain the trino cli and trino jdbc ?
A: ???

Q: Tagging with latest is not possible with the current tagging scheme. See meeting notes on versions and tags below. Is this a problem ?
A: ??? (operators need to know exact versions when they create pods anyway)

Acceptance criteria

Docker images

• OPA
• Apache ZooKeeper
• Apache NiFi
• Apache Kafka
• Trino
• Apache Hadoop
• Apache Spark
• Apache HBase
• Apache Hive
• Apache Superset
• (Maybe) Apache Druid - Coordinate with Felix

Operators adapted

moved to -> stackabletech/issues#136

Docs updated (thx Felix)

• OPA
• Apache ZooKeeper
• Apache NiFi
• Apache Kafka
• Trino
• Apache Hadoop
• Apache Spark
• Apache HBase
• Apache Hive
• Apache Superset
• Apache Druid

Annotation prometheus.io/scrape: "true" should be added to role services

As a user of our operators I'd like to have low Java DNS cache timeouts so the products don't crash on startup.

See these links for reference:

• https://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html
• https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-jvm-ttl.html

This is done when all Java based products have a java.security file (for me it's in /etc/java/java-11-openjdk/java-11-openjdk-11.0.14.1.1-2.el8_5.x86_64/conf/security but make sure it's stable) that includes the setting networkaddress.cache.ttl and a value of 5.

For this it might be worth creating one or more (for different Java versions) Stackable Java base images that do this once and we can reuse it for all our images.
See also this for reference: stackabletech/hdfs-operator#147

Official Release Notes

This is a patch upgrade, it fixes the log4j issue among others things.

The Superset version is hard coded in the Dockerfile but the version from the configuration file conf.py should be taken.

Superset is similar to AirFlow because both are based on the same Python framework and their images can be built in the same way. The Dockerfile for AirFlow already uses a parameterized approach which should also be used in the Dockerfile for Superset.

We received feedback that our product images are fairly large and have been asked to investigate if this can be reduced with reasonable effort.

The operators are fine, this is just in reference to the product images.

Specifically mentioned were:

• NiFi
• Druid
• Trino

For this specific use case Trino would be the most important one to have a look at.

This requires #237.

Once we know how, when and until when we want to keep older product versions maintained, we should implement this strategy for our previous platform releases.

The Docker images are much larger than the official ones:

$ docker images | grep trino
docker.stackable.tech/stackable/trino        362-stackable0                 4a6cc6f9f9dd   2 weeks ago     4GB
trinodb/trino                                362                            8c4c3cc8b6b5   2 months ago    1.33GB

The size of the images should be decreased to speed up the download. This would also make the integration tests faster and decrease the chance of hitting timeouts.

One possible optimization is to not rename the installation directory in a separate layer:

RUN mkdir /stackable/trino-server && \
    mv /stackable/trino-server-${PRODUCT}/* /stackable/trino-server && \
    rm -rf /stackable/trino-server-${PRODUCT}

I am attempting to install and get an hbase cluster running using the helm charts. The operators install fine but when actually attempting to apply manifests to create the cluster objects (zookeepercluster, zookeepernode etc...) I am met with Warning Failed 4s (x2 over 18s) kubelet Failed to pull image "docker.stackable.tech/stackable/zookeeper:3.8.0-stackable23.1.0": rpc error: code = Unknown desc = no matching manifest for linux/arm64/v8 in the manifest list entries. Is this a known issue?

S3 libs are needed for basically everything (e.g. reading/writing data and history-server) therefore we want to pre-ship them with our images.
Another reason is to put the correct versions in the image, as many users will be confused on which versions to put there

Implementation can be copied from hive image

• Must: Libs added to new Image version. Only adding it to Spark 3.3.0 only is ok, as older Spark versions have older Hadoop versions and need other aws libs
• Must: kuttl test added
• Could: kuttl tests using s3a libs via pvcs or similar are removed
• Documentation & changelog updated to mention the inclusion of these libs

This is about updating everything on main. This is not about updating base images for older product images.

Some products might need to be upgraded directly, others depend on the java-base image. This needs to be upgraded.

The java-base image will need to be split, and versioned differently. Some products require Java 17, while i.e. Druid only supports java 11. The Java base image needs the latest ubi8 base image though, so we will have a java-11 base image, and a java-17 base image (at least).

Acceptance criteria

• Java base image has been split, and all new java base images use the latest ubi8 base
• java based products are linked to their new respective java base image
• all non-java products have been upgraded as well (airflow, opa, superset, others?)
• Use newest image tags in:
  • Operator repository examples
  • Operator documentation
  • Getting started guides
  • Integration tests
  • Optional: tools image in init containers

Progress Tags

• #248
• #250
• #251

Products

• Airflow
  • 2.2.3-stackable0.4.0
  • 2.2.4-stackable0.4.0
  • 2.2.5-stackable0.4.0
  • 2.4.1-stackable0.5.0
• Druid
  • 0.23.0-stackable0.2.0
  • 24.0.0-stackable0.2.0
• Hadoop HDFS
  • 3.2.2-stackable0.6.0
  • 3.3.1-stackable0.6.0
  • 3.3.3-stackable0.2.0
  • 3.3.4-stackable0.2.0
• HBase
  • 2.4.6-stackable0.8.0
  • 2.4.8-stackable0.8.0
  • 2.4.9-stackable0.8.0
  • 2.4.11-stackable0.8.0
  • 2.4.12-stackable0.3.0
• Hive
  • 2.3.9-stackable0.6.0
  • 3.1.3-stackable0.2.0
• Kafka
  • 2.7.1-stackable0.6.0
  • 2.8.1-stackable0.6.0
  • 3.1.0-stackable0.6.0
  • 3.2.0-stackable0.2.0
  • 3.3.1-stackable0.2.0
• NiFi
  • 1.13.2-stackable0.5.0
  • 1.15.0-stackable0.5.0
  • 1.15.1-stackable0.2.0
  • 1.15.2-stackable0.2.0
  • 1.15.3-stackable0.2.0
  • 1.16.0-stackable0.2.0
  • 1.16.1-stackable0.2.0
  • 1.16.2-stackable0.2.0
  • 1.16.3-stackable0.2.0
  • 1.18.0-stackable0.2.0
• pyspark-k8s
  • 3.2.1-stackable0.7.0
  • 3.3.0-stackable0.3.0
• spark-k8s
  • 3.2.1-stackable0.6.0
  • 3.3.0-stackable0.3.0
• Superset
  • 1.3.2-stackable2.2.0
  • 1.4.1-stackable2.2.0
  • 1.5.1-stackable0.3.0
• Trino
  • 377-stackable0.2.0
  • 387-stackable0.2.0
  • 395-stackable0.2.0
  • 396-stackable0.2.0
• ZooKeeper
  • 3.5.8-stackable0.8.0
  • 3.6.3-stackable0.8.0
  • 3.7.0-stackable0.8.0
  • 3.8.0-stackable0.8.0

Internal

• OPA
  • 0.27.1-stackable0.3.0
  • 0.28.0-stackable0.3.0
  • 0.37.2-stackable0.3.0
  • 0.41.0-stackable0.2.0
  • 0.45.0-stackable0.2.0
• Tools
  • 0.2.0-stackable0.4.0

Progress Operator repository image adaptation

Products

• stackabletech/airflow-operator#193
• stackabletech/druid-operator#339
• stackabletech/hdfs-operator#271
• stackabletech/hbase-operator#275
• stackabletech/hive-operator#271
• stackabletech/kafka-operator#513
• stackabletech/nifi-operator#376
• stackabletech/spark-k8s-operator#176
• stackabletech/superset-operator#295
• stackabletech/trino-operator#340
• stackabletech/zookeeper-operator#586

Internal

• stackabletech/opa-operator#374

It is not yet clear if automation will be needed.

From: https://kubernetes.io/blog/2023/03/17/upcoming-changes-in-kubernetes-v1-27/ - any k8s.gcr.io images will be moved to registry.k8s.io. The git-sync implementation for Airflow uses k8s.gcr.io/git-sync/git-sync:v3.6.4 and this should be mirrored in our Nexus.