"Blocks read inconsistent" happened when shuffle read about firestorm HOT 5 CLOSED

When shuffle read, after process all blocks, it will check if all expected blocks are processed. "Blocks read inconsistent" will be thrown when some blocks are lost. It may be caused by writing shuffle data failed.

from firestorm.

Thank you for your reply, Does the LOCALFILE_AND_HDFS mode write data to the HDFS and Shuffle server at the same time?

from firestorm.

@Augus-smile please try release 0.2.0 which has a lot of improvements and bug fix. You can refer readme for detail configuration. In 0.2.0, MEMORY_LOCALFILE_HDFS is introduced for multiple storages. We will publish related doc soon.

from firestorm.

Shuffle server process was killed due to OOM, related server conf : physical memory : 16g, XMX_SIZE: 12g. However, during the Spark task execution, the Shuffle Server process occupies nearly 16g memory. What are the memory consumption components of the Shuffle Server? Is there a parameter to restrict the memory usage of shuffle Server?

from firestorm.

set xmx_size in rss-env.sh
in server.conf:
rss.server.buffer.capacity # memory cache for write and read
rss.server.read.buffer.capacity # memory cache for read
BTW, please left extra 5G memory for shuffle server

from firestorm.