Comments (17)
@jmabuin Please, I need help to solve this problem.
from sparkbwa.
Hi @Asmaa-Ali
SparkBWA creates temporary .sam files in the computing nodes (executors) while the program is executing. Can you connect to one of your nodes and check if this temporary files have some content?
This files are stored in the spark temporary directory or in /tmp/, you can find them by using the find
command
find /tmp/ -name "*.sam"
Also, what is the content of your /Data/HumanBase/ dir? is this content available in all nodes?
from sparkbwa.
I run into the same issue, and my temporary .sam files inside /tmp also have no content.
from sparkbwa.
Can you please check if it works with SparkBWA 0.2 ?
from sparkbwa.
Hi, I have tried it with SparkBWA 0.2 and I run also into the same issue. It works fine and then the produced sam files are empty ! The content of /Data/HumanBase/ dir is ivailable for all my worker nodes. Any suggestions, please ?
from sparkbwa.
Do you have permissions to write in the tmp folder? also, which versions of Hadoop and Spark are you using?
from sparkbwa.
Yes I do have thr ights to write in /tmp.
I am using spark2.0 and hdfs 2.7.3 .
Any ideas welcome !
from sparkbwa.
Have you tried to use yarn-cluster
instead of yarn-client
?
Actually in newer versions of Spark it should be --master yarn --deploy-mode cluster
from sparkbwa.
Same issue here. I try also to run with the local scheduler (no yarn). Could that be the reason for missing/empty sam files? Does all input data need to be through hdfs? It also complains it can't find the index file and have set up proper permission to all locations. Thank you.
Update: Obviously, one really needs a running hadoop cluster so that the code can work on the data in HDFS. Hence the empty sam files in Spark standalone cluster mode. It would be nice if there were an option for running a spark standalone instance. Hadoop can be a real pain under Torque/PBS job schedulers.
from sparkbwa.
I fix. Now SparkBWA-0.2 can run yarn or standalone and output sam file in my local cluster.
from sparkbwa.
@xubo245 Thanks a lot - the fix works. I can confirm that it also runs in Spark standalone mode (no Hadoop FS)
from sparkbwa.
You are welcome.
from sparkbwa.
@xubo245 It is still not working for me in standalone mode.Have you done any more specific changes
from sparkbwa.
Yes, I have temporary changes for standalone, but it is not best solutions...
com.github.sparkbwa.BwaAlignmentBase#copyResults
Configuration conf = new Configuration();
conf.set("fs.default.name","hdfs://Master:9000/");
FileSystem fs = FileSystem.get(new URI("hdfs://Master:9000/"),conf);
The Master should be your cluster hostname.
from sparkbwa.
@xubo245 We are running on non-pdfs environment using GPFS.
How can we make it general fs.The fs is available on every nodes similar to hdfs
from sparkbwa.
#38 Attached the spark run
from sparkbwa.
You should replace HDFS code with GPFS API, but I do not known GPFS...
SparkBWA has many many HDFS code...
from sparkbwa.
Related Issues (20)
- mvn package compilation failing with certificate error HOT 3
- Unable to build with Maven HOT 2
- Index file cannot be located HOT 1
- Error when using Azure blob storage paths for FASTQ files and SAM output file HOT 1
- SparkBWA generates empty sam files HOT 8
- BUILD FAILURE: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.5.0:exec HOT 2
- Wouldn't it be good to add a feature in your program to execute the BWA program directly? HOT 3
- Single-end reads crash HOT 4
- ERROR: java.lang.ClassNotFoundException: com.github.sparkbwa.FASTQRecordGrouper HOT 10
- Problem when using spark-bwa (not linked to the reducer) ends up with bad header HOT 1
- Output is empty .sam file!! HOT 1
- If you run it with Spark-submit, will not the BwaJni class and native BWA run?
- Missing an output location for shuffle 0 HOT 1
- jni_md.h file is not found HOT 4
- Can't run SparkBWA on Amazon EMR Yarn cluster HOT 1
- --master local[n] results in an error HOT 1
- Error while running master as local
- Spark job fails saying methods cannot be called on a stopped SparkContext
- [Question - RDD creation step , BWA-MEM step] WIth alignment input size 280GB
- mvn.128.no/maven2 unreachable HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sparkbwa.