Comments (8)
Hello,
After looking at the first error log, I believe the reason is running out of memory (Java.OutOfMemoryError
). If possible, you can give more memory. Or you may consider redistribute the memory allocation for driver and executor, e.g., 1GB memory for driver and 3GB for executor.
Please let me know if that helps!
from cloudsuite.
Hello,
After looking at the first error log, I believe the reason is running out of memory (
Java.OutOfMemoryError
). If possible, you can give more memory. Or you may consider redistribute the memory allocation for driver and executor, e.g., 1GB memory for driver and 3GB for executor. Please let me know if that helps!
Thanyk you for reply.
$ docker create --name movielens-data cloudsuite/movielens-dataset
$ docker run -dP --net host --name spark-master cloudsuite/spark:3.3.2 master
$ docker run -dP --net host --volumes-from movielens-data --name spark-worker-01 cloudsuite/spark:3.3.2 worker
spark://NODE_IP:7077
$ docker run --rm --net host --volumes-from movielens-data cloudsuite/in-memory-analytics /data/ml-latest
/data/myratings.csv --driver-memory 1g --executor-memory 3g --master spark://NODE_IP:7077
As you said, driver memory 1GB and executor memory 3GB were given, but unlike when 2GB was given, it appears that the resources are insufficient.
Conversely, even if 3GB of driver and 1GB of executor are given, it will not run.
If there is nothing that can be set, should it be regarded as insufficient physical memory?
from cloudsuite.
Hello,
Thanks for doing the test. Indeed, this means the memory is not enough to run the workload.
There might be another way around: You can try to restrict the number of cores allocated to the container using --cpuset-cpus
. Memory consumption may be reduced when the worker count becomes smaller. The trade-off is that it will take longer time to finish.
from cloudsuite.
Thank you for reply.
Are you saying that the problem is caused by the fact that the actual memory size is small even though there is a lot of data?
There was a problem when running by allocating 2GB, but since the result came out, can it be considered normal?
from cloudsuite.
Yes. It is an implication that the physical memory is not enough. You can explain it as a normal case, but not a representative case.
from cloudsuite.
Yes. It is an implication that the physical memory is not enough. You can explain it as a normal case, but not a representative case.
Thank you for reply.
I understand little bit.
The result came out, but you're saying that it's hard to see it normally because a memory error came out?
As an example, I will show you a picture of the result.
from cloudsuite.
Hello,
Yes. Even though you finally have the result and workload successfully finished, my understanding is that it still cannot represent a real server: This workload is supposed to run on a server with large amount of memory, so you should not see any out-of-memory error during running.
However, it is OK if your ideal case is not a server :)
Best,
from cloudsuite.
Hello,
Yes. Even though you finally have the result and workload successfully finished, my understanding is that it still cannot represent a real server: This workload is supposed to run on a server with large amount of memory, so you should not see any out-of-memory error during running.
However, it is OK if your ideal case is not a server :)
Best,
Thank you for your kind reply. You have been very helpful. Thank you
from cloudsuite.
Related Issues (20)
- HHVM and Django left overs
- Better documentation how to run the workloads
- Add to the configuration parameters of the docker, the number of vcpus that will be allocated to Spark and Hadoop
- Media Streaming Should Have High-Resolution Video (1080p, 4K)
- In-Memory doc
- Data Analytics implementation questions HOT 10
- Media Streaming Multi-node HOT 4
- Ramp up workload intensity for Data-Caching and Data-Serving Benchmarks HOT 1
- media streaming dataset: Issue with running ffmpeg - timestamp out of range HOT 1
- Can I run cloudsuite benchmark on arm64 architecture directly by offical code? HOT 13
- Sysbench Meets Error When Running on ARM64 Platform
- media-streaming dataset error HOT 7
- Investigate CloudSuite web serving workload in QEMU
- Instructions for compiling for RISC-V ISA (Data-caching) HOT 3
- Starting the web-server for the the web-serving benchmark fails as rootless
- data-analytics: Native library missing for arm64 HOT 1
- data-analytics: failure to run benchmark on different hosts without proper DNS resolution of nodes
- data-caching: cache not warmed up?
- Data caching DNS Error HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cloudsuite.