lightbend / cloudflow Goto Github PK

Describe the bug
Quoted from a Gitter conversation:

V.Zaytsev @unit7
Hi everyone! Can anyone suggest me why for Flink-streamlets kafka consumer-group-id is specified as the name of the input topic? In fact, this means that only one Flink-streamlet can read from a particular topic, that is all Flink-streamlets will be combined in one consumer group. So what was the design idea? For akka-streamlets, consumer-group-id is formed taking into account the streamlet and inlet from which the reading occurs.
Thanks in advance.
The related sources:

• https://github.com/lightbend/cloudflow/blob/master/core/cloudflow-flink/src/main/scala/cloudflow/flink/FlinkStreamletContextImpl.scala#L49
• https://github.com/lightbend/cloudflow/blob/master/core/cloudflow-akka/src/main/scala/cloudflow/akkastream/AkkaStreamletContextImpl.scala#L77

Debasish Ghosh @debasishg 11
@unit7-0 We can surely align the consumer group id to make it similar to akka. But Flink Kafka connectors don’t use the consumer group management functionality because they use lower-level APIs on each parallel instance for more control on individual partition consumption. So, essentially, the “group.id” setting in the Flink Kafka connector is only used for committing offsets back to ZK / Kafka brokers.

V.Zaytsev @unit7
@debasishg, Did I understand correctly that since each flink-streaming-job tracks the offsets individually and saves it by the internal checkpoint mechanism, is there no need to specify a different groupd.id for each streamlet? Accordingly in the case when there are two streamlets reading the same topic but has different business logic will work correctly although the consumers will be in the same consumer-group?

I think they are right and we should adjust our consumer group id to be the same as the one we use for the Akka streamlet runner.

During the installation process, we should have an auto-selection of default as the cluster version.
Users should also be able to pick another version.

Describe the bug
After building the operator docker image, pushing it to a repo, and adjusting the installation script to use it, Gitter user @mrooding ran into a runtime error: java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/Module

To Reproduce

• build the operator docker image using sbt docker in the operator sub-project
• tag and push the resulting image to a registry
• adjust the installation script to use the above image
• install

Check the logs

Expected behavior
No errors

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

To Reproduce
When building several modules I get different type of warnings.
Eg.

BlueprintBuilder.scala:81:3: Could not find any member to link for "StreamletRef".
[warn]   /**
[warn]   ^

or

EndpointActionsSpec.scala:23:19: Unused import
[warn] import skuber.ext.Ingress
[warn]                   ^

Expected behavior
We should remove the warnings.

Describe the bug

• In an application with multiple Flink streamlets and Akka streamlets, only one Flink streamlet gets deployed correctly. All Akka streamlets deploy fine. On doing an undeploy followed by another deploy does the fix and deploys all streamlets correctly.

To Reproduce

Here's the situation that replicates the bug (as reported):

• A multi streamlet application - Flink + Akka
• 2 Flink streamlets
• Akka streamlets deploy fine with kubectl cloudflow deploy ..
• During deploy using the cli, only 1 Flink streamlet deploys successfully.
• In the operator log, the message is: 2020-03-02 15:28:06,817 INFO [ActorSystemImpl] - Executing update action for resource stock-market-app in namespace stock-market-app. But no streamlet mentioned whereas all others are mentioned. As in, no status changes to MODIFIED/ADDED
• Undeploy and then re-deploy
• All streamlets get deployed successfully

Expected behavior

• All streamlets should deploy correctly in the first go.

Screenshots

• None

Additional context
Add any other context about the problem here.

Describe the bug
Commit 67eb5e0 introduces a requirement on bash >= 4 based on a BASH_VERSINFO env var for running the GKE installer. However, when using other shells, like Zsh on MacOS, it fails to meet the requirements due to BASH_VERSINFO not being set.

To Reproduce
Run ./install-gke.sh <cluster_name> with Zsh (https://ohmyz.sh/)

Expected behavior
Allow the installer to run with multiple shells.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Is your feature request related to a problem? Please describe.
Currently, the GKE installer relies on Helm 2. Since Helm 3, Tiller is no longer required and helm init no longer exists as a CLI command. I've already created a PR to include dependencies required to run the installer in #30

Describe the solution you'd like
Either offer support for both Helm 2 and 3 or migrate to 3 only. I'm willing to work on a PR but I'd first like to know which approach you prefer.

Describe alternatives you've considered
I haven't looked into how Helm is being used by Cloudflow extensively but if it's purely for templating then maybe switching to Kustomize is an option

Describe the bug
If runLocalConfigFile is misconfigured, runLocal just runs anyway.
Instead it should provide error feedback that the runLocalConfigFile could not be found, and not continue to run the application locally.

This issue showed up in the examples. The sensor-data-scala example project specifies runLocalConfigFile incorrectly as:

runLocalConfigFile := Some("resources/local.conf")

which should be:

runLocalConfigFile := Some("src/main/resources/local.conf")

This was not noticed because runLocal runs, even though runLocalConfigFile is set incorrectly.

To Reproduce
The sensor-data-scala defines a runLocalConfigFile, which specifies source-data-mount="/tmp/cloudflow". When you runLocal, you'll see output in the sbt console similar to what is shown below:

[info] Using sandbox configuration from resources/local.conf
---------------------------------- Streamlets ----------------------------------
file-ingress [sensordata.SensorDataFileIngress]
	- mount [source-data-mount] available at [/var/folders/l5/mt77lw_92cxfgx15mk8d78vr0000gn/T/local-cloudflow732248201622748214/source-data-mount]

The source-data-mount should be available at /tmp/cloudflow, not at /var/folders/l5/mt77lw_92cxfgx15mk8d78vr0000gn/T/local-cloudflow732248201622748214/source-data-mount (or similarly created temp directory, which should happen only if you do not specify a value for the mount in the local.conf)

Change the setting for sensor-data-scala example project (reload sbt after), setting:
runLocalConfigFile := Some("src/main/resources/local.conf")

Execute runLocal again. This will show that the mount is now available at /tmp/cloudflow, which proves that the local.conf is used.

Expected behavior
runLocal must stop and provide an error in the sbt console. The error must indicate, that the file that runLocalConfigFile refers to, cannot not be found.

Screenshots
n/a

Additional context
n/a

Migrated from: lightbend/cloudflow-installer#6
Submitted by: https://github.com/dhillonr

While installing Cloudflow in a cluster using ./install-gke.sh their is a part where it installs NFS Server. It installs the server using a helm chart whose path is hardcoded.

Is your feature request related to a specific runtime of cloudflow or applicable for all runtimes?
This applies only to the Akka Streams runtime

Describe the solution you'd like
Upgrade our Alpakka Kafka dependency to the (soon to be released) 2.0 version to benefit from increased Kafka client performance and to enable #96

Describe the bug
Currently in the getting started guide and the examples READMEs we do port-forwarding at the pods directly. This is not a solid approach since if a pod is restarted then its name will change and kubectl will not handle automatically this event.

Expected behavior
Use services created instead.

The cloudflow sbt plugin only deploys to GCP. However, it will be ideal to have support for other cloud providers. EKS seems to be mature enough and could be an initial candidate.

Is your feature request related to a problem? Please describe.
Flink Streamlet implementation does not expose the Flink consumer used internally for creating the required streams.

That means user does not have access to lower APIs such as the one that allows to pass a custom watermark emitter as described here

Is your feature request related to a specific runtime of cloudflow or applicable for all runtimes?
This affects the Flink runtime.

Describe the solution you'd like

Extend the Flink Streamlet API to be able to pass a custom watermark at the source.
The advantage of this solution is that solves the issue without exposing the internals of Cloudflow.
The downside is that this might lead to using wrappers for any other interface that is exposed via Flink and user needs it.

Describe alternatives you've considered
a) expose the consumer directly via API extension or other means. This gives greater flexibility to the user but it may cause the user to mess with the internals of how we use kafka.

b) use a workaround instead, where the user will create his custom watermark and timestamp (https://ci.apache.org/projects/flink/flink-docs-stable/dev/event_timestamps_watermarks.html) emitter.

Additional context
This is a user request as discussed on our channel: https://gitter.im/lightbend/cloudflow

Is your feature request related to a problem? Please describe.

Currently in AkkaStreamlet we have a way to indicate that the streamlet is ready for processing after it starts :

private def readyAfterStart(): Boolean = if (attributes.contains(ServerAttribute)) false else true

private def signalReadyAfterStart(): Unit =
  if (readyAfterStart) context.signalReady

This is currently private. But this feature can be useful when we want to have streamlets that want to test is a streamlet setup/initialization is complete for building the computation graph. Currently this is used only for HttpServerLogic to ensure that the server is started and the ports are bound.

Users cannot use this feature in normal streamlets unless they add the Server trait as a mixin.

Is your feature request related to a specific runtime of cloudflow or applicable for all runtimes?

Initially for Akka, but may be we can think of implementing the same for Flink and Spark.

Describe the solution you'd like

We would like to make the above API public for use in general streamlets.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Describe the bug
Looks like that transitive dependencies are not being resolved properly when using runLocal.

To Reproduce
e.g "lib - depends on (d1,d2)" , "proj - depends on (lib)" won't work.
worked around: "proj - depends on (lib, d1, d2)"

Expected behavior
Transitive dependencies should be resolved

Additional context
Reported by @blublinsky when developing model serving applications that depend on TensorFlow

Describe the bug
On master branch helm charts use the 1.3.0 release tags for installing the cloudflow operator.
That creates bugs as we update master. Right now things fails with:

com.typesafe.config.ConfigException$Missing: application.conf @ jar:file:/app/lib/com.lightbend.cloudflow.cloudflow-operator-71-fa7ef6d-dirty.jar!/application.conf: 2: No configuration setting found for key 'cloudflow.platform.cluster-type'
	at com.typesafe.config.impl.SimpleConfig.findKeyOrNull(SimpleConfig.java:156)
	at com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:174)
	at com.typesafe.config.impl.SimpleConfig.findOrNull(SimpleConfig.java:180)

To Reproduce
Install cloudflow from master.

Expected behavior
Cloudflow operator should run without issues.

Screenshots
N/A

Additional context
This blocks #65

Is your feature request related to a problem? Please describe.

For users in order to build their application image they need first to either build locally the Cloudflow base image (assuming they need to changes) or use a released one.
This is currently done via bash scripts and could be migrated to the sbt-cloudflow plugin to unify the UX.
Goal also should be to add only the required dependencies on the fly based on the application type since now all deps are added in the Dockefile definition. Fixing this would reduce the image size and would remove the dependency on the Spark image when no Spark is used. Adoptjdk should be our base image only.
Downside would be that if the user wants to do any changes he will have to modify the sbt plugin, publish it and use the local version of the plugin. Now its easier to make changes as we just use plain dockerfiles and bash scripts.

Is your feature request related to a specific runtime of cloudflow or applicable for all runtimes?
Some feature requests may be relevant only for specific runtimes. Please describe in details.

It is targeting all runtimes.

Describe the solution you'd like
We should detect application type during app image build phase in the ImagePlugin
and build the minimal image required to runt he app.

Describe alternatives you've considered
N/A
Additional context
N/A

Describe the bug
When running the gke-installer on a new machine with the base configuration, the installer fails with:

./install-gke.sh: line 62: helm: command not found
Error from server (NotFound): deployments.extensions "tiller-deploy" not found

helm should be documented as a required pre-requisite in the cluster installer documentation.
https://cloudflow.io/docs/current/get-started/prepare-development-environment.html

Describe the bug
Not really a bug, but would be good for consistency, to remove use of scalariformat, and use scalafmt instead, everywhere.

Describe the bug
The namespace for the protocol version configmap cloudflow-protocol-version is hardcoded to lightbend.
If the lightbend namespace doesn't exist, the configmap is not created, and the CLI cannot be used.
It should not be hardcoded.

To Reproduce
Install Cloudflow on a namespace different than lighbend (can be changed in install-gke.sh).

Run kubectl cloudflow list.

It fails with

[Error] Cannot find the 'cloudflow-protocol-version' configmap, please make sure that the Cloudflow operator is installed

workaround: create a lightbend namespace, and kill the cloudflow-operator pod to restart it.

Expected behavior
The configmap to be installed and looked for in the current namespace.

Screenshots
N/A
Additional context
N/A

Thank you @skyluc for reporting this.

Is your feature request related to a problem? Please describe.
Upgrading to Akka 2.6.x

Is your feature request related to a specific runtime of cloudflow or applicable for all runtimes?
Akka

Describe the solution you'd like
Upgrade dependencies on Akka to version 2.6.x

It would be great to support Kafka-Streams as an additional stream processor.
This would include a KafkaStreamsStreamlet API to write streamlets using the Kafka-streams API and a runner to execute it in the deployment cluster.

Reported by @samcschneider
Describe the bug
The getting-started guide should specify that logging must be configured and show 'how to' do that to avoid the default Debug level that produces too much output.

Describe the bug
The setup-gke-cluster.adoc page still refers to the cloudflow-installer repository, which we have now inlined into the main cloudflow repo

To Reproduce
Live page: https://cloudflow.io/docs/current/get-started/setup-gke-cluster.html
Source: https://github.com/lightbend/cloudflow-docs/blob/master/docs-source/docs/modules/get-started/pages/setup-gke-cluster.adoc

Expected behavior
All docs should refer to the correct source of the installation scripts

Describe the bug
I was trying to get started with the wind-turbine example, documented here:
When attempting to run runLocal or verifyBlueprint from sbt shell, I received the following error

[IJ]sbt:sensor-data> runLocal
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

Able to workaround by initializing git repository and providing initial commit.

To Reproduce

1. Create skeleton cloudflow project (insure plugin is enabled for project)
2. Do not initialize git repo
3. launch sbt interactively
4. run verifyBlueprint or runLocal

Expected behavior
It might not be unreasonable to require a git repository, given that source generation is part of the framework. However, perhaps as a gap in documentation or in the tutorial it could be added that this is also a requirement.

Additional context
From gitter and @agemooij:
The first issue is caused by a feature of our SBT plugin, which will calculate a deterministicly unique version number for each image it produces. It appears our runLocal feature depends on this verison number while that is not strictly necessary. Feel free to report this as an issue!

Describe the bug
After installing the cluster to GKE, it upgrades itself creating the node pools taking the cluster offline for a few minutes. If you try and run install-gke.sh at this time it fails with the misleading error "It looks like you are not logged into a Kubernetes cluster. Please 'gcloud init' before running the installer." The install script assumes any error returned by kubectl get pods command is related to authentication, which isn't always the case.

Expected behavior
The script should preferably detect that the cluster is not finished initialization or, if that is not possible, the error message should indicate this might be the case and advise the user to verify this and perhaps wait a little bit for the initialization to finish.

Describe the bug
For whatever reason my gcloud local configuration did not have Zone or Region setup although the project did have a default Zone and region. This resulted in the error below being thrown multiple times when trying to setup the GKE Cluster

ERROR: (gcloud.container.clusters.create) One of [--zone, --region] must be supplied: Please specify location.

To Reproduce
Run the commands below to clear local Zone and Region settings

gcloud config unset compute/zone
gcloud config unset compute/region

Then run the GKE Cluster Setup Comand

./create-cluster-gke.sh cloudflow-cluster

Expected behavior
A clear and concise description of what you expected to happen.

I would expect that the create-cluster-gke.sh script would check if the Zone and Region local configs are set. If they are not present the script should indicate what the problem is and print how to fix it.

Additional context
Add any other context about the problem here.

Adding the missing configs involves running the commands below to set the Zone and Region

gcloud config set compute/zone ZONE
gcloud config set compute/region REGION

Describe the bug
In the documentation some links/images are not processed correctly.

To Reproduce
Go to https://cloudflow.io/docs/current/get-started/hello-world-example.html
The link to 'avro schema' and the image pipe.001.jpeg are not displayed.

Expected behavior
To be processed correctly

Screenshots

Additional context

Describe the bug
MacOS default version of bash is still v3 which causes a problem when running the install-gke.sh script.

To Reproduce
Switch to any v3 version of bash and run the install-gke.sh script.

Expected behavior
The documentation should reflect that v4 or higher of bash is required to run the cloudflow installer and if possible the installer should check the bash version

Cloudflow-on-bashv3.2.57(1)-release.txt

I would like to have project templates that can help me to bootstrap my project faster and avoid common pitfalls, like using the correct plugin, project structure, or sbt version.

Is your feature request related to a specific runtime of cloudflow or applicable for all runtimes?
Having one template per runtime would be really handy.

Describe the solution you'd like
Available templates.

Is your feature request related to a problem? Please describe.
As today the installation and deployment of all K8s custom resources required by Cloudflow have no tests. The reason probably is because this was unfortunately done with bash from the beginning.

Is your feature request related to a specific runtime of Cloudflow or applicable for all runtimes?
All runtimes

Describe the solution you'd like
I would like to use a programming language that can be tested so we can warranty that the installation is never broken (this happened two times during the last week).

Describe alternatives you've considered
Perhaps the most natural solution would be terraform since we can easily build reusable modules that could potentially be used for deploying to the 3 major cloud providers: AWS, Azure, and GCP.

Another potential good fit could be https://docs.aws.amazon.com/cdk/latest/guide/getting_started.html. Since we have a lot of custom steps during the installation process (helm init, etc.), having a programming language can give us this flexibility. CDK supports the following languages:

• TypeScript
• Python
• Java
• C#

Additional context
I am willing to help with this task if we pick terraform, or scala/java/python if we move with the CDK route. FWIW, I rather do scala or terraform, but I would be also comfortable doing it in python (unfortunately no golang support yet, aws/aws-cdk#547).

As an alias with cloudflow-akka, but this is a little more low priority because cloudflow-akka can support Java API well .

Describe the bug
If one Streamlet classname is the prefix of another one, the verifyBlueprint step report an error.
Example of classnames which trigger the bug: a.A and a.AB.

To Reproduce

• create 2 streamlets a.A and a.AB in an application.
• reference a.A in the blueprint:

  blueprint {
    streamlets {
      a = a.A
      other = a.Other
    }
    connections {
      a.out = [ other.in ]
    }
  }
  

• Run verifyBlueprint. It fails with the following error:

  [error] (verifyBlueprint) cloudflow.sbt.BlueprintVerificationError: 
  [error] [...]/blueprintAB.conf:
  [error] Inlets (other.in) are not connected.
  [error] ClassName matching `a.A` is ambiguous for streamlet name a.
  [error] 'a.out' does not point to a known streamlet inlet or outlet.
  [error] Total time: 80 s, completed Mar 3, 2020 4:52:26 PM
  

Expected behavior
verifyBlueprint should not fail. The classnames are not ambiguous.

Additional context
Reproduced on 1.3.0 and master.

Is your feature request related to a problem? Please describe.

This is not specifically a new feature.

• The suggestion is to improve upon how we can override config params in FlinkStreamletTestkit and make it uniform with the way we do this in SparkStreamletTestkit and AkkaStreamletTestkit.
• Also related is the issue that the streamletRef is currently hardcoded in the testkit. Will be better to have it as an argument to the constructor of the testkit.

(Attaching some images of the gitter conversation)

Is your feature request related to a specific runtime of cloudflow or applicable for all runtimes?

• Mostly for Flink but Akka and Spark also need change

Describe the solution you'd like

Please refer to the discussion attached.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Describe the bug
When using the Akka streamlet testkit, the streamlet under test is not stopped when the stream completes. This does however happen in normal execution of an AkkaStreamlet (when the stream completes unexpectedly).

To Reproduce
Write an AkkaStreamlet with one inlet, no outlet, read from in, write to Sink.ignore.
Test with a inletFromSource, use a Source(Vector(...)), which should complete the stream once all elements in the vector have passed through the sink.

use AkkaStreamletTestKit method:

  def run[T](streamlet: AkkaStreamlet, ip: List[InletTap[_]], op: List[OutletTap[_]]): StreamletExecution

And await on completed on returned StreamletExecution.

Expected behavior
The streamlet under test should be stopped (the completion future should be completed) when the stream completes.

Additional context
Add any other context about the problem here.

Describe the bug

After verifying blueprint successfully while doing runLocal application not able to read parameters from local.conf

After verifyBlueprint , I ran runLocal I received following error:

Streamlet [cassandra-ingress] failed to launch. Reason: empty config: No configuration setting found for key 'cassandra-ip-address'

To Reproduce
My local.conf is :
cassandra-ingress {
cassandra-ip-address = ###
cassandra-port = 9042
username = ##
password = ###
query = ###
fetchsize = 0
}
prediction-logger{
prediction-uri=##
columns-list= ##
}

In build.sbt

runLocalConfigFile := Some("src/main/resources/local.conf")

And I have placed local.conf in src/main/resources folder.

Expected behavior
Application should be able to read params from local.conf
Screenshots
NA

Additional context
NA

We are mostly using protobuf for now, hope this will come.

Describe the bug
The automated build sometimes fails with the following error:

[info] SparkEgressSpec:
[info] SparkEgress
[info] - should materialize streaming data to sink *** FAILED ***
[info]   List() did not contain element "NAME1" (SparkEgressSpec.scala:85)
[info] SparkJoin3Spec:
[info] SparkJoin3
[info] - should process streaming data
[info] SparkIngressSpec:
[info] SparkIngress
[info] - should produce elements to its outlet
[info] Run completed in 1 minute, 18 seconds.
[info] Total number of tests run: 6
[info] Suites: completed 6, aborted 0
[info] Tests: succeeded 5, failed 1, canceled 0, ignored 0, pending 0
[info] *** 1 TEST FAILED ***
[error] Failed tests:
[error] 	cloudflow.spark.SparkEgressSpec
[error] (cloudflow-spark-tests / Test / test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 240 s, completed Nov 25, 2019 10:22:24 AM
Failure in building of core

To Reproduce
It only shows in the github actions.

Expected behavior
The build should not fail when all tests are passing on a dev machine.

Screenshots
n/a

Additional context
n/a

We have heard from various users that they have made incorrect choices for the RWO and RWM storage classes during installation, leading to performance problems (Kafka on NFS) or even broken installs (choosing a RWO class where a RWM is required).

This is made hard by the fact that unfortunately K8s doesn't seem to have an API for checking the RW capabilities of a storage class (or the underlying provisioner. This seems to only be available at runtime when creating a PVC and it fails (at best) or succeeds when it should have failed (at worst).

The install script is trying to be as clear as possible by using some documented (in a public K8s docs page) provisioner capabilities but clearly this is not enough.

At least the docs and the install script should try to prevent as many problems as possible

cc: @nolangrace @blublinsky @lightbend/cloudflow-team

Describe the bug
Installer is broken, issues introduced when merging #65.

./install.sh: line 75: unexpected EOF while looking for matching `"'
./install.sh: line 76: syntax error: unexpected end of file

I will crate a PR shortly.
To Reproduce
Running the installer on GKE & EKS fails.
Expected behavior
Installer should work.

Screenshots
N/A

Additional context
N/A

Describe the bug
Received following error while deploying example application sensor-data-scala:
[Error] The following volume mount needs to be bound to a Persistence Volume claim using the --volume-mount flag

• file-ingress.source-data-mount

To Reproduce
I followed all the steps mentioned in getting started link for sensor-data-scala example pipeline.
I reached till the point where we create docker image for application using "buildAndPublish"
Next steps was to deploy in cluster using following:
kubectl-cloudflow deploy -u oauth2accesstoken "docker image path" -p "$(gcloud auth print-access-token)"

Expected behavior
Successful deployment of application.

Additional context
For applications requiring volume mounts are we supposed to manually create pvc's in respective namespaces ?

Is your feature request related to a problem? Please describe.
Currently flowWithOffsetContext[T] is accepting CommittableOffset, which prevents one from introduction internal events (for example, timer) and still have input events commitable

Is your feature request related to a specific runtime of cloudflow or applicable for all runtimes?
This is only relevant for Akka Streams

Describe the solution you'd like
I would like flowWithOffsetContext[T] to accept Committable, so that for internal events I can use (v, CommittableOffsetBatch.empty) and I can merge internal and external events

Describe alternatives you've considered
The only alternative currently is to use plainSink, which means that kafka events are not committed and restartability of streamlet does not work

Additional context
None

This is a very selfish feature request. I have been using the Scala library FS2 and have come to absolutely love it. Meanwhile I also see the value of Frameworks like Cloudflow. So here is me hoping the team considers providing support for FS2 in the cloudflow ecosystem.

Describe the bug
This PR merged a change that switched the Flink operator Helm chart from the closed-sourced one to the open source one. But the open source chart is compatible with Helm 3 only (i.e. not installable using Helm 2 binary). The failure of Helm 2 install is due to Helm 2 not recognizing manifests in /crds directory, therefore not creating CRDs in the cluster.

To Reproduce
Using install.sh to install Cloudflow will fail.

Expected behavior
A successful install of the Flink operator with all the right resources.

Additional context
The plan is to update the Flink operator chart to be compatible with both Helm 2 and 3.

I think that should be easy to copy from it?

Describe the bug
A typo in the question of whether or not the user would like to remove PVCs and Kafka topics leads to undesired behaviour if the user tried to type 'yes'.

To Reproduce
Answer the question with something other than 'yes' or 'no'

Expected behavior
Request new input from the user which matches the 2 options

Screenshots

Additional context
Add any other context about the problem here.

Scala Steward helps to stay up to date on dependencies.

Is your feature request related to a problem? Please describe.
For simple queries trigger.continuous allows to produce results continuously, not as mini batch. This allows for faster processing in Sparklets putting them on par with Akka streams and Flink

Is your feature request related to a specific runtime of cloudflow or applicable for all runtimes?
Only relevant for Spark runtime

Describe the solution you'd like
I would like to allow to specify trigger option for Spark query

Describe alternatives you've considered
There are none

Additional context
None

as it is, here should use Duration

Describe the bug
The kubectl-cloudflow binaries cannot be executed on MacOS Catalina. The operating system is blocking them for security reasons.

To Reproduce
On MacOs Catalina, download the latest kubectl-cloudflow, unzip it, (optionally) put it in your path, and execute it.

You will see the following OS-level error:

Expected behavior
The kubectl-cloudflow CLI should work on MacOS Catalina. It will likely have to be code-signed.