Hi.
I downloaded and opened hazelcast jet samples in my Eclipse (with Windows) and I found the next problems and warnings:

However, when I try to run each samples I find that it runs correctly:

Could you tell me something about the errors, please?
Thank you.

Hi, in hazelcast-jet-code-samples-0.7.2 (hazelcast-jet-code-samples-0.7.2\integration\pcf module), duplicate classes with the same fully-qualified name org.apache.log4j.PropertyConfigurator are included in two different libraries, i.e., log4j:log4j:1.2.12 and org.slf4j:log4j-over-slf4j:1.7.22.

According to "first declaration wins" class loading strategy, only this class in org.slf4j:log4j-over-slf4j:1.7.22 can be loaded, and that in log4j:log4j:1.2.12 will be shadowed.

By further analyzing, your project expects to invoke methods----
(1) <org.apache.log4j.PropertyConfigurator: void doConfigure(java.lang.String,org.apache.log4j.spi.LoggerRepository)>
(2) org.apache.log4j.PropertyConfigurator: void doConfigure(java.net.URL,org.apache.log4j.spi.LoggerRepository)>
(3) org.apache.log4j.PropertyConfigurator: void doConfigure(java.util.Properties,org.apache.log4j.spi.LoggerRepository)>
in log4j:log4j:1.2.12. As they has been shadowed, so that these methods defined in org.slf4j:log4j-over-slf4j:1.7.22 are actually forced to be referenced via the following invocation path (taking <org.apache.log4j.PropertyConfigurator: void doConfigure(java.lang.String,org.apache.log4j.spi.LoggerRepository)> for example):

Path：<CommandController: CommandResponse wordCount(java.lang.String,java.lang.String)> D:\testcase\TestProject\hazelcast-jet-code-samples-0.7.2\integration\pcf\target\classes
<com.hazelcast.jet.JetInstance: com.hazelcast.jet.Job newJob(com.hazelcast.jet.pipeline.Pipeline,com.hazelcast.jet.config.JobConfig)> D:\cEnvironment\repository\com\hazelcast\jet\hazelcast-jet\0.7.2\hazelcast-jet-0.7.2.jar
<com.hazelcast.jet.impl.JetClientInstanceImpl: com.hazelcast.jet.Job newJob(com.hazelcast.jet.core.DAG,com.hazelcast.jet.config.JobConfig)> D:\cEnvironment\repository\com\hazelcast\jet\hazelcast-jet\0.7.2\hazelcast-jet-0.7.2.jar
<com.hazelcast.jet.impl.AbstractJetInstance: long uploadResourcesAndAssignId(com.hazelcast.jet.config.JobConfig)> D:\cEnvironment\repository\com\hazelcast\jet\hazelcast-jet\0.7.2\hazelcast-jet-0.7.2.jar
<org.springframework.aop.interceptor.AsyncExecutionAspectSupport$CompletableFutureDelegate$1: java.lang.Object get()> D:\cEnvironment\repository\org\springframework\spring-aop\4.3.5.RELEASE\spring-aop-4.3.5.RELEASE.jar
<com.hazelcast.executor.impl.RunnableAdapter: java.lang.Object call()> D:\cEnvironment\repository\com\hazelcast\jet\hazelcast-jet\0.7.2\hazelcast-jet-0.7.2.jar
<org.apache.log4j.helpers.FileWatchdog: void run()> D:\cEnvironment\repository\log4j\log4j\1.2.12\log4j-1.2.12.jar
<org.apache.log4j.helpers.FileWatchdog: void checkAndConfigure()> D:\cEnvironment\repository\log4j\log4j\1.2.12\log4j-1.2.12.jar
<org.apache.log4j.PropertyWatchdog: void doOnChange()> D:\cEnvironment\repository\log4j\log4j\1.2.12\log4j-1.2.12.jar
<org.apache.log4j.PropertyConfigurator: void doConfigure(java.lang.String,org.apache.log4j.spi.LoggerRepository)>

Although both of these two conflicting classes contain the referenced methods (with the same signature), they have different implementations. This issue will lead to inconsistent semantic hehavior by changing the control flows and data flows.

Workaround solution:
The easy way to workaround the problem is reversing the declaration order of these two libraries in pom file.
Then, according to "first declaration wins" class loading strategy, class org.apache.log4j.PropertyConfigurator in log4j:log4j:1.2.12 can be loaded (the version that hazelcast-jet-code-samples expects to reference by static analysis).

This fix will not affect other libraries or class, except the above duplicate class.

Dependency tree------

[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ pcf ---
[INFO] com.hazelcast.jet.samples:pcf:jar:0.7.2
[INFO] +- org.springframework.boot:spring-boot-starter-web:jar:1.4.3.RELEASE:compile
[INFO] | +- org.springframework.boot:spring-boot-starter:jar:1.4.3.RELEASE:compile
[INFO] | | +- org.springframework.boot:spring-boot:jar:1.4.3.RELEASE:compile
[INFO] | | +- org.springframework.boot:spring-boot-autoconfigure:jar:1.4.3.RELEASE:compile
[INFO] | | +- org.springframework.boot:spring-boot-starter-logging:jar:1.4.3.RELEASE:compile
[INFO] | | | +- ch.qos.logback:logback-classic:jar:1.1.8:compile
[INFO] | | | | +- ch.qos.logback:logback-core:jar:1.1.8:compile
[INFO] | | | | - org.slf4j:slf4j-api:jar:1.7.22:compile
[INFO] | | | +- org.slf4j:jcl-over-slf4j:jar:1.7.22:compile
[INFO] | | | +- org.slf4j:jul-to-slf4j:jar:1.7.22:compile
[INFO] | | | - org.slf4j:log4j-over-slf4j:jar:1.7.22:compile
[INFO] | | +- org.springframework:spring-core:jar:4.3.5.RELEASE:compile
[INFO] | | - org.yaml:snakeyaml:jar:1.17:runtime
[INFO] | +- org.springframework.boot:spring-boot-starter-tomcat:jar:1.4.3.RELEASE:compile
[INFO] | | +- org.apache.tomcat.embed:tomcat-embed-core:jar:8.5.6:compile
[INFO] | | +- org.apache.tomcat.embed:tomcat-embed-el:jar:8.5.6:compile
[INFO] | | - org.apache.tomcat.embed:tomcat-embed-websocket:jar:8.5.6:compile
[INFO] | +- org.hibernate:hibernate-validator:jar:5.2.4.Final:compile
[INFO] | | +- javax.validation:validation-api:jar:1.1.0.Final:compile
[INFO] | | +- org.jboss.logging:jboss-logging:jar:3.3.0.Final:compile
[INFO] | | - com.fasterxml:classmate:jar:1.3.3:compile
[INFO] | +- com.fasterxml.jackson.core:jackson-databind:jar:2.8.5:compile
[INFO] | | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.8.5:compile
[INFO] | | - com.fasterxml.jackson.core:jackson-core:jar:2.8.5:compile
[INFO] | +- org.springframework:spring-web:jar:4.3.5.RELEASE:compile
[INFO] | | +- org.springframework:spring-aop:jar:4.3.5.RELEASE:compile
[INFO] | | +- org.springframework:spring-beans:jar:4.3.5.RELEASE:compile
[INFO] | | - org.springframework:spring-context:jar:4.3.5.RELEASE:compile
[INFO] | - org.springframework:spring-webmvc:jar:4.3.5.RELEASE:compile
[INFO] | - org.springframework:spring-expression:jar:4.3.5.RELEASE:compile
[INFO] +- com.hazelcast.jet:hazelcast-jet:jar:0.7.2:compile
[INFO] +- log4j:log4j:jar:1.2.12:compile
[INFO] - com.google.code.findbugs:annotations:jar:3.0.0:provided

Thank you very much.
Best,
Coco

please urgent help,about how to Hazelcast jet JDBC continues source get data when new record insert or existing record updated in a database table?

Please add a code sample for custom aggregation

Team,

I am using Pipeline api to read from a list , transform them to uppercase and then write to an a list using Jet.

When i am running this on server experiencing below exception:

Exception in thread "main" com.hazelcast.nio.serialization.HazelcastSerializationException: Failed to serialize 'com.hazelcast.jet.core.DAG'
	at com.hazelcast.internal.serialization.impl.SerializationUtil.handleSerializeException(SerializationUtil.java:75)
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:157)
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:133)
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toData(AbstractSerializationService.java:118)
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toData(AbstractSerializationService.java:106)
	at com.hazelcast.jet.impl.AbstractJobProxy.doSubmitJob(AbstractJobProxy.java:205)
	at com.hazelcast.jet.impl.AbstractJobProxy.<init>(AbstractJobProxy.java:80)
	at com.hazelcast.jet.impl.JobProxy.<init>(JobProxy.java:53)
	at com.hazelcast.jet.impl.JetInstanceImpl.newJob(JetInstanceImpl.java:61)
	at com.hazelcast.jet.JetInstance.newJob(JetInstance.java:72)
	at com.hazelcast.jet.JetInstance.newJob(JetInstance.java:92)
	at com.walmart.jet.hadoop.PipelineSample.main(PipelineSample.java:45)
Caused by: com.hazelcast.nio.serialization.HazelcastSerializationException: Failed to serialize 'com.hazelcast.jet.core.Vertex'
	at com.hazelcast.internal.serialization.impl.SerializationUtil.handleSerializeException(SerializationUtil.java:75)
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.writeObject(AbstractSerializationService.java:254)
	at com.hazelcast.internal.serialization.impl.ByteArrayObjectDataOutput.writeObject(ByteArrayObjectDataOutput.java:375)
	at com.hazelcast.jet.core.DAG.writeData(DAG.java:440)
	at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.write(DataSerializableSerializer.java:244)
	at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.write(DataSerializableSerializer.java:51)
	at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.write(StreamSerializerAdapter.java:43)
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toBytes(AbstractSerializationService.java:154)
	... 10 more
Caused by: com.hazelcast.nio.serialization.HazelcastSerializationException: Failed to serialize 'com.hazelcast.jet.impl.execution.init.CustomClassLoadedObject'
	at com.hazelcast.internal.serialization.impl.SerializationUtil.handleSerializeException(SerializationUtil.java:75)
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.writeObject(AbstractSerializationService.java:254)
	at com.hazelcast.internal.serialization.impl.ByteArrayObjectDataOutput.writeObject(ByteArrayObjectDataOutput.java:375)
	at com.hazelcast.jet.impl.execution.init.CustomClassLoadedObject.write(CustomClassLoadedObject.java:50)
	at com.hazelcast.jet.core.Vertex.writeData(Vertex.java:187)
	at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.write(DataSerializableSerializer.java:244)
	at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.write(DataSerializableSerializer.java:51)
	at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.write(StreamSerializerAdapter.java:43)
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.writeObject(AbstractSerializationService.java:252)
	... 16 more
Caused by: com.hazelcast.nio.serialization.HazelcastSerializationException: There is no suitable serializer for class com.hazelcast.jet.impl.execution.init.CustomClassLoadedObject
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.serializerFor(AbstractSerializationService.java:487)
	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.writeObject(AbstractSerializationService.java:250)

Below is the code:

	 JetConfig cfg = new JetConfig();
        cfg.setInstanceConfig(new InstanceConfig().setCooperativeThreadCount(
                Math.max(1, getRuntime().availableProcessors() / 2)));


        JetInstance jet = Jet.newJetInstance(cfg);
       
        Jet.newJetInstance(cfg);
        IListJet<String> input = jet.<String>getList("input");
        input.addAll(new ArrayList<>( Arrays.asList("test", "test1", "test2")));
        
        
	Pipeline p = Pipeline.create();
	p.drawFrom(Sources.<String>list("input"))
	 .map(String::toUpperCase)
	 .drainTo(Sinks.list("result"));
	
	try {
        System.out.print("\nCounting words from "+"\n");
        long start = nanoTime();
        jet.newJob(p).join();
        
        System.out.println("Done in " + NANOSECONDS.toMillis(nanoTime() - start) + " milliseconds.");
        System.out.println("Output written to " + jet.getList("result").get(1));
    } finally {
        Jet.shutdownAll();
    }

When i run this in my local it is working, but when i run on server it is throwing above exception.
Please suggest me with a solution.

This is what I see (in the master branch):

/me = 😕

at very least the top-level readme.md should explain what I can find in each directory.
"As a developer I have no clue what PCF stands for. I feel confused and lost."

(yes, I know I can click at the pcf directory and there is an extra readme.md. but at that point I am already lost. Don't Make Me Think)

From @jerrinot on November 15, 2017 13:37

It took me a while to figure out how to use the built-in aggregations. We have code-samples for e.g. counting but:

1. There is too much noise. I don't want to deal with Shakespeare, splitting words and lines. All I want to see how to use averaging(). As simple example as possible. With minimum noisy.
2. We do not have any example showing how to use an aggregation which does not use key-based classification. e.g. averaging() (as opposed to counting()`

I am not sure if we need to have this as a code-sample. The current samples tries to create something resembling a real-word use-case. Even a simple junit integration test in the Jet project itself would help me a lot.

Copied from original issue: hazelcast/hazelcast-jet#599

It doesn't exclude ".checkstyle" files, so the repo contains more than it should.

TensorFlow demo requires a trained model to work. One must install Python 3.6 (current is 3.7.2) to train the model. This harms the experience as running Python code isn't specific to Jet.

Save the trained model into GitHub for smoother convenience.

Training the model should still remain as an option.

Trained code is based on the TensorFlow Dataset (https://www.tensorflow.org/datasets/overview). Please, create the NOTICE file in the root of the repository stating that TensorFlow demo contains the model trained from the TensorFlow dataset (Apache 2 copyright notice).

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] hazelcast-jet-code-samples ......................... SUCCESS [ 1.354 s]
[INFO] batch .............................................. SUCCESS [ 0.028 s]
[INFO] access-log-analyzer ................................ SUCCESS [ 1.942 s]
[INFO] co-group ........................................... SUCCESS [ 0.606 s]
[INFO] sample-data ........................................ SUCCESS [ 0.495 s]
[INFO] java.util.stream ................................... SUCCESS [ 1.048 s]
[INFO] hazelcast-connectors ............................... SUCCESS [ 0.494 s]
[INFO] wordcount .......................................... SUCCESS [ 0.397 s]
[INFO] wordcount-hadoop ................................... SUCCESS [ 0.440 s]
[INFO] samples-core-api ................................... SUCCESS [ 0.023 s]
[INFO] core-api-batch ..................................... SUCCESS [ 0.019 s]
[INFO] map-dump ........................................... SUCCESS [ 0.397 s]
[INFO] mapreduce-migration ................................ SUCCESS [ 0.419 s]
[INFO] prime-finder ....................................... SUCCESS [ 0.425 s]
[INFO] tf-idf ............................................. SUCCESS [ 0.393 s]
[INFO] wordcount-core-api ................................. SUCCESS [ 0.366 s]
[INFO] enrichment-core-api ................................ SUCCESS [ 0.448 s]
[INFO] core-api-streaming ................................. SUCCESS [ 0.017 s]
[INFO] access-stream-analyzer ............................. SUCCESS [ 0.373 s]
[INFO] fault-tolerance .................................... SUCCESS [ 0.617 s]
[INFO] session-windows .................................... SUCCESS [ 1.183 s]
[INFO] trade-generator .................................... SUCCESS [ 0.976 s]
[INFO] stock-exchange ..................................... SUCCESS [ 1.016 s]
[INFO] top-n-stocks ....................................... SUCCESS [ 0.975 s]
[INFO] pcf ................................................ SUCCESS [ 2.761 s]
[INFO] refman ............................................. FAILURE [ 1.100 s]
[INFO] streaming .......................................... SKIPPED
[INFO] enrichment ......................................... SKIPPED
[INFO] map-journal-source ................................. SKIPPED
[INFO] kafka-source ....................................... SKIPPED
[INFO] socket-connector ................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 19.027 s
[INFO] Finished at: 2017-11-30T16:58:29-06:00
[INFO] Final Memory: 37M/601M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:2.17:checkstyle (default) on project refman: An error has occurred in Checkstyle report generation.: Failed during checkstyle execution: There are 2 errors reported by Checkstyle 6.11.2 with /hazelcast\examples/checkstyle/checkstyle.xml ruleset. -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

End of life was 5 August 2015

Build works on Mac, but fails on Windows 10,

[INFO] --- maven-checkstyle-plugin:2.17:checkstyle (default) @ map-dump ---
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] hazelcast-jet-code-samples ......................... SUCCESS [  2.078 s]
[INFO] core ............................................... SUCCESS [  0.047 s]
[INFO] map-dump ........................................... FAILURE [  1.844 s]
[INFO] books .............................................. SKIPPED
[INFO] mapreduce-migration ................................ SKIPPED
[INFO] prime-finder ....................................... SKIPPED
[INFO] tf-idf ............................................. SKIPPED
[INFO] wordcount .......................................... SKIPPED
[INFO] java.util.stream ................................... SKIPPED
[INFO] wordcount-j.u.s .................................... SKIPPED
[INFO] simple-operations .................................. SKIPPED
[INFO] hadoop ............................................. SKIPPED
[INFO] wordcount-hadoop ................................... SKIPPED
[INFO] kafka .............................................. SKIPPED
[INFO] consume-kafka ...................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.328 s
[INFO] Finished at: 2017-03-17T15:26:30+00:00
[INFO] Final Memory: 19M/223M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:2.17:checkstyle (default) on project map-dump: Execution default of goal org.apache.maven.plugins:maven-checkstyle-plugin:2.17:checkstyle failed. NullPointerException -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :map-dump

for

Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00)
Java version: 1.8.0_121, vendor: Oracle Corporation
Java home: C:\Program Files\Java\jdk1.8.0_121\jre
Default locale: en_GB, platform encoding: Cp1252
OS name: "windows 10", version: "10.0", arch: "amd64", family: "dos"

README needs updating with all the new projects and reorganization for 0.6

Running the mapWithUpdating "as is " generates the exception described above.

Changing String.valueOf(e.getValue()) to e.getValue()
and
? oldValue +"--even"
: oldValue +"--odd"
to
? oldValue
: oldValue

makes the code running ok but with no parsing and suffixing