no JCublas-10.0.0-linux-x86_64 in java.library.path about flink-mnist HOT 14 CLOSED

Does the jcuda-natives-10.0.0-linux-x86_64.jar also in /flink/lib/?

from flink-mnist.

@KarmaGYZ yes. both are in /flink/lib/

from flink-mnist.

perkz@bfs:/data/perkz/flink/lib$ jar -tf jcublas-natives-10.0.0-linux-x86_64.jar  
META-INF/MANIFEST.MF 
META-INF/
lib/
META-INF/maven/ 
META-INF/maven/org.jcuda/
META-INF/maven/org.jcuda/jcublas-natives/
lib/libJCublas-10.0.0-linux-x86_64.so
lib/libJCublas2-10.0.0-linux-x86_64.so
META-INF/maven/org.jcuda/jcublas-natives/pom.xml
META-INF/maven/org.jcuda/jcublas-natives/pom.properties

perkz@bfs:/data/perkz/flink/lib$ jar -tf jcuda-natives-10.0.0-linux-x86_64.jar  
META-INF/MANIFEST.MF 
META-INF/
lib/
META-INF/maven/
META-INF/maven/org.jcuda/
META-INF/maven/org.jcuda/jcuda-natives/
lib/libJCudaDriver-10.0.0-linux-x86_64.so
lib/libJNvrtc-10.0.0-linux-x86_64.so
lib/libJCudaRuntime-10.0.0-linux-x86_64.so
META-INF/maven/org.jcuda/jcuda-natives/pom.xml
META-INF/maven/org.jcuda/jcuda-natives/pom.properties

from flink-mnist.

@PerkzZheng why does the JCublas initiated at the MNISTClassifier.close? Would you like to share the jm and tm log here?

from flink-mnist.

@KarmaGYZ Sorry for replying so late. I am working on setting up everything again. And it works now, it could be some version problems. Thanks for your time.

Finally, I got results in flink-perkz-taskexecutor-2-bfs.out.

7
2
1
0
4
1
4
9
....

However, --output seems not working, when I set it, I only got an empty directory like 2020-10-29--15

from flink-mnist.

Would you like to share all the log files in the /flink/log/ directory?

from flink-mnist.

Appreciate it.
Flink-logs.zip

from flink-mnist.

As the javaDoc says, the output path is the base path where all the buckets are going to be created as sub-directories. In your logs, it seems the parameter is invalid and the job is actually failed because it could not create "/2020-10-29--15" directory.

from flink-mnist.

It is previous testing. I use valid directory under my current directory, and it still gets a empty directory.
So, the flink-perkz-taskexecutor-2-bfs.out is a valid output ?

from flink-mnist.

Flink-log-1.zip
Sorry for the previous testing. I start over to make sure everything relates to one recent test. Please check it.

from flink-mnist.

The flink-perkz-taskexecutor-2-bfs.out is a valid output.

I could not find anything wrong in your logs. Not sure what happens.

from flink-mnist.

Thanks for taking time. It is great to see that flink enables gpu awareness.

from flink-mnist.

Hi, @PerkzZheng . After a deep investigation, I find it is a valid issue. The root cause is the StreamingFileSink require to open the checkpoint. The latest commit fixes that issue.

from flink-mnist.

Hi, @PerkzZheng . After a deep investigation, I find it is a valid issue. The root cause is the StreamingFileSink require to open the checkpoint. The latest commit fixes that issue.

@KarmaGYZ Thanks for your investigation.

from flink-mnist.