[error] terasort on spark rdma about sparkrdma HOT 3 CLOSED

Hi,
Thanks for quick response. I used following submit scripts, and the problem has been fixed.

./bin/spark-submit --class com.github.ehiggs.spark.terasort.TeraSort --master yarn --deploy-mode cluster /root/spark-terasort-1.0-SNAPSHOT-jar-with-dependencies.jar hdfs:///input hdfs:///output

from sparkrdma.

Hi, seems that you are running SparkRdma in a local mode (that's basically running executor in a separate thread and no network shuffle involved). Can you please provide command that you use to submit spark job. Basically you need to have:

1. RDMA capable network adapter.
2. Run in spark standalone or in spark yarn modes with several executors on different physical machines.

Let me know if you'll have questions.

from sparkrdma.

Hi @li7hui ,

Have you run SparkRDMA in yarn-cluster mode? We have some problems when running the jobs in SparkRDMA because of it always use the wrong IP in another NIC. Do you have multiple network interface cards in the server? Or have you set the LOCAL_SPARK_IP in spark-env.sh?

from sparkrdma.