king / bravo Goto Github PK

Would it be possible to publish jars for bravo to Maven Central? It would make it much easier to use as a dependency in projects.

Now that Flink 1.7.0 has been released [0], can we upgrade bravo to use Flink 1.7.0? I'm updating my Flink cluster to 1.7 and would love to continue to be able to use bravo

[0] https://flink.apache.org/news/2018/11/30/release-1.7.0.html

Will you support blink 1.5.1 wich use savepoint version3 and is base on flink 1.5.1?
I changed Gradle dependencies form org.apache.flink to com.alibaba.com and changed flink version to 1.5.1 and some java classes.
I read state from savepoint successfully but failed while writing state back.
My changes here : DG-Wangtao@5cf2589

Currently only rocksdb savepoint formats are supported which is a silent assumption not explicitly checked and enforced in many places where we could.

We should extend support for RocksDB incremental checkpoint reading and writing as that is a very common use-case.

I'm interested in trying bravo, but I didn't see any mention of a software license. It would be good to have a LICENSE file in the repository.

Bravo

Right now only reading and writing of states without namespaces are supported which excludes things like windowoperations at the moment

gradlew clean build

FAILURE: Build failed with an exception.

* Where:
Build file '~/bravo/build.gradle' line: 93

* What went wrong:
A problem occurred evaluating root project 'bravo'.
> Could not get unknown property 'nexusUsername' for object of type org.gradle.api.publication.maven.internal.deployer.DefaultGroovyMavenDeployer.

I know this is trivial to fix locally by removing these lines:

After that I get:

> Task :bravo:signArchives FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':bravo:signArchives'.
> Cannot perform signing task ':bravo:signArchives' because it has no configured signatory

which can be fixed by removing:

It would be nice to have some official way to avoid these errors so that there's no need to work on a dirty repo locally. What would you suggest?

It seems that there are significant changes in state handling and api of some of the methods in flink 1.8

Currently the reader writer interfaces for keyed states only work well with Value states but they should be extended to other state types.

Currently the reader writer interfaces for keyed states only work well with Value states but they should be extended to other state types.

I'm using bravo to debug a MapState, and find the deserialized value is wrong while the outkey and the mapkey are right.

The MapState is used to record user login times in a day, so the outvalue values should be mostly 1, but what I get is mostly 358 .

Could you please help me narrow down the problem? Thanks a lot!

// MapState definition in Flink application
MapStateDescriptor<String, Integer> stateDescriptor = new MapStateDescriptor<>(
                "loggedInUsers",
                String.class,
                Integer.class
        );

// relative code in bravo application
DataSet<Tuple3<String, String, Integer>> state = reader.readKeyedStates(
                new MapStateKKVReader<>(stateId, BasicTypeInfo.STRING_TYPE_INFO, BasicTypeInfo.STRING_TYPE_INFO,
                        BasicTypeInfo.INT_TYPE_INFO));

Looks like not catching Throwable #13 wasn't enough yet.

I got this error:

java.lang.IllegalStateException: Cannot read state of a stateless operator.
	at com.king.bravo.reader.OperatorStateReader.lambda$readKeyedStates$6(OperatorStateReader.java:101)
	at java.util.Optional.orElseThrow(Optional.java:290)
	at com.king.bravo.reader.OperatorStateReader.readKeyedStates(OperatorStateReader.java:101)

The actual cause was another Exception, which got silently ignored here:

Any reason to return Optional.empty() here instead of throwing a runtime exception (that wraps the original exception)?

I found the actual exception with debugger:

java.lang.RuntimeException: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 's3'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded.

Yeah, apparently that happens because I had copied the savepoint from its original s3 location to my local machine (is it really so that even savepoint folders can't be moved anywhere from their original location?)