Running Low on Storage when Building Specific Spark Version about spark-ec2 HOT 8 OPEN

Thanks @felixmaximilian for the report. The trouble is that increasing EBS volume size requires AMIs to be rebuilt for all the regions.

One workaround might be to build Spark on the ephemeral disk at /mnt. Could you see if that works and if so we can make a code change for that ?

from spark-ec2.

Building spark in /mnt/ is possible. Do you plan to copy the compiled spark back to the EBS volume? Then its necessary to make sure you don't copy the whole target folder etc. We should build the distribution (make-distribution.sh) in /mnt/ and than uncompress it back to the EBS volume.
What do you think? We could try to jointly find a comfortable solution on Monday.
Have a nice weekend.

from spark-ec2.

Just for the record, I'm running into this issue as well.

@felixmaximilian - Have you made any progress on solving this? I can help you write a patch, if you are interested in writing one.

from spark-ec2.

A colleague created an ami with much more (I guess ebs) space on the main
partition (root). Haven't really tried it again but it should be solved
though.

Fixing this problem within the code of ec2 spark wasn't very much
successful on my side. Tried different things but ended with the problem
that you cannot really do much on the externally mounted mnt2 mnt3 etc
while starting the cluster because they are added and removed during the
process. Didn't really get it why. (Idea was to build it on external
storage and to copy it back to root then).
We can give it another try with combined forces :)

But another question : is 8gb on the root partition really enough if just
the installation files fit there!? What is about the hdfs in the ephemeral
folder? As far as I can remember this is also existing in root which means
we can hardly save anything to the hdfs, right?
It might be worth to resize all the Amis to a bigger partition or have at
least another partition from the very beginning to be able to do stuff
there.
Nicholas Chammas [email protected] schrieb am Do., 5. Nov. 2015 um
20:51:

Just for the record, I'm running into this issue as well.

@felixmaximilian https://github.com/felixmaximilian - Have you made any
progress on solving this? I can help you write a patch, if you are
interested in writing one.

—
Reply to this email directly or view it on GitHub
#17 (comment).

from spark-ec2.

Hmm, anything that requires updating all the spark-ec2 AMIs is a tough sell since that takes a lot of work and the process is not automated.

from spark-ec2.

Yeah just to clear some things - AFAIK to increase root partition size needs an AMI rebuild. However I think we should be able to clone and build Spark on /mnt using make-distribution.sh and then unzip to the root partition.

The HDFS thing is not really an issue -- the HDFS binaries are on /root but it uses /mnt on every machine for storage, so it can use all the ephemeral storage.

from spark-ec2.

I think this issue can be resolved without having to do any work on the AMIs. See this comment.

from spark-ec2.

+1

from spark-ec2.