Comments (8)
Thanks @felixmaximilian for the report. The trouble is that increasing EBS volume size requires AMIs to be rebuilt for all the regions.
One workaround might be to build Spark on the ephemeral disk at /mnt
. Could you see if that works and if so we can make a code change for that ?
from spark-ec2.
Building spark in /mnt/ is possible. Do you plan to copy the compiled spark back to the EBS volume? Then its necessary to make sure you don't copy the whole target folder etc. We should build the distribution (make-distribution.sh) in /mnt/ and than uncompress it back to the EBS volume.
What do you think? We could try to jointly find a comfortable solution on Monday.
Have a nice weekend.
from spark-ec2.
Just for the record, I'm running into this issue as well.
@felixmaximilian - Have you made any progress on solving this? I can help you write a patch, if you are interested in writing one.
from spark-ec2.
A colleague created an ami with much more (I guess ebs) space on the main
partition (root). Haven't really tried it again but it should be solved
though.
Fixing this problem within the code of ec2 spark wasn't very much
successful on my side. Tried different things but ended with the problem
that you cannot really do much on the externally mounted mnt2 mnt3 etc
while starting the cluster because they are added and removed during the
process. Didn't really get it why. (Idea was to build it on external
storage and to copy it back to root then).
We can give it another try with combined forces :)
But another question : is 8gb on the root partition really enough if just
the installation files fit there!? What is about the hdfs in the ephemeral
folder? As far as I can remember this is also existing in root which means
we can hardly save anything to the hdfs, right?
It might be worth to resize all the Amis to a bigger partition or have at
least another partition from the very beginning to be able to do stuff
there.
Nicholas Chammas [email protected] schrieb am Do., 5. Nov. 2015 um
20:51:
Just for the record, I'm running into this issue as well.
@felixmaximilian https://github.com/felixmaximilian - Have you made any
progress on solving this? I can help you write a patch, if you are
interested in writing one.—
Reply to this email directly or view it on GitHub
#17 (comment).
from spark-ec2.
Hmm, anything that requires updating all the spark-ec2 AMIs is a tough sell since that takes a lot of work and the process is not automated.
from spark-ec2.
Yeah just to clear some things - AFAIK to increase root partition size needs an AMI rebuild. However I think we should be able to clone and build Spark on /mnt using make-distribution.sh
and then unzip to the root partition.
The HDFS thing is not really an issue -- the HDFS binaries are on /root but it uses /mnt on every machine for storage, so it can use all the ephemeral storage.
from spark-ec2.
I think this issue can be resolved without having to do any work on the AMIs. See this comment.
from spark-ec2.
+1
from spark-ec2.
Related Issues (20)
- Where to find the spark home?
- Enable python2.7
- Support for S3 V4 HOT 1
- Quick question on setting fs.s3a.endpoint HOT 2
- Can I add an extra disk at /mnt3? HOT 1
- Cluster is created but Spark is not able to installed in ec2 master and slaves HOT 1
- support for spark 2.2.0? HOT 8
- "--spark-version" does not work HOT 1
- tachyon not supported? HOT 1
- Documentation incorrect regarding missing "ec2" directory HOT 3
- VPC/Subnet requirements not documented HOT 2
- branch-2.0 should use scala 2.11 HOT 2
- Getting InvalidClassException when running SparkPi example locally but pointing to master on AWS HOT 2
- salve nodes not started on re-start HOT 2
- ap-northeast-2 seoul region support HOT 4
- Submitting to EC2 cluster HOT 10
- Attach an existing persistant EBS volume to the cluster
- ERROR StandaloneSchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up. HOT 3
- Classpath/dependency resolution of JAR app HOT 1
- Need support for ca-central-1 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spark-ec2.