vemonet / setup-spark Goto Github PK
View Code? Open in Web Editor NEW:octocat:✨ Setup Apache Spark in GitHub Action workflows
Home Page: https://github.com/marketplace/actions/setup-apache-spark
License: MIT License
:octocat:✨ Setup Apache Spark in GitHub Action workflows
Home Page: https://github.com/marketplace/actions/setup-apache-spark
License: MIT License
Describe the bug
I'm aware that the action isn't tested for versions below 3, but wondered if you might be able to advise on how I can adjust the configuration to work.
When calling spark though pytest
(pyspark), we get an error stating that spark-submit
is not where it is is expected to be:
FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/work/repo/repo/..//spark/./bin/spark-submit': '/home/runner/work/repo/repo/..//spark/./bin/spark-submit
I expect that this is due to the lower spark version, but have been unable to find how the configuration of later versions differ. Any pointers here would be greatly appreaciated.
Which version of the action are you using?
v1
Environment
GitHub Actions
Spark Versions
- uses: actions/setup-python@v2
with:
python-version: 3.6.8
- uses: actions/setup-java@v1
with:
java-version: '8'
- uses: vemonet/setup-spark@v1
with:
spark-version: '2.4.0'
hadoop-version: '2.1.1'
Run/Repo Url
See run here
Describe the bug
error when install spark in the github action env
Which version of the action are you using?
v1
Environment
If applicable, please specify if you're using a container
Spark Versions
3.0.1
To Reproduce
Steps to reproduce the behavior:
error when install spark env
Describe the bug
Error downloading the Spark binary
Which version of the action are you using?
v1
Environment
GitHub-hosted
Spark Versions
3.1.1
To Reproduce
- uses: vemonet/setup-spark@v1
with:
spark-version: '3.1.1'
hadoop-version: '3.2'
Notes
It seems that the requested file is not right now at the specified location.
https://www.apache.org/dyn/closer.lua/spark/spark-3.1.1/spark-3.1.1-bin-hadoop3.2.tgz?as_json returns
"path_info": "spark/spark-3.1.1/spark-3.1.1-bin-hadoop3.2.tgz",
"preferred": "https://ftp.cixug.es/apache/"
but I think it's wrong, If I go to this location I cann't find this file :(
If I go manually to https://www.apache.org/dyn/closer.lua/spark/spark-3.1.1/spark-3.1.1-bin-hadoop3.2.tgz (no ?to_json query string), a message appears that says "The requested file or directory is not on the mirrors.
The object is in our archive : https://archive.apache.org/dist/spark/spark-3.1.1/spark-3.1.1-bin-hadoop3.2.tgz"
May be a parameter in the GitHub Action to enter manually the url (if user wants to use it) could solve the problem now and for future changes too
Thank you so much
The latest release (v1) does not include multiple newly-solved issues. A new release version should be added that includes the latest changes.
Describe the bug
A clear and concise description of what the bug is.
Run vemonet/setup-spark@v1 started failing since yesterday, not sure tried checking multiple versions but still the same. Would you mind helping to find some resolution here? The download of the jar file is taking too long, which is resulting in an
Error: The operation was cancelled
which is happening during the git-hub build.
Which version of the action are you using?
v1
, main
, other tag?
vemonet/setup-spark@v1
Environment
GitHub-hosted, Self-hosted?
GitHub-hosted
Spark Versions
Please list all of the effected versions of Spark (3.1.2
, etc.)
spark-version: '3.1.2'
hadoop-version: '3.2'
To Reproduce
Steps to reproduce the behavior:
Run/Repo Url
If applicable, and if your repo/run is public, please include a URL so it is easier for us to investigate.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.
Describe the bug
When Github reuses a runner to execute the action, the spark installation fails due a symbolic link error. See here:
Note that when a "new" runner is used, the installation works fine. See how I distinguish between "old" and "new" runners in the To Reproduce section below.
I forked this repository and solved my issue by by changing the flags in line 67 of dist/.js from -s to -sf. As per the man page, this removes existing links before creating a new one.
ln -sf "${installFolder}/spark-${sparkVersion}-bin-hadoop${hadoopVersion}${scalaBit}" ${installFolder}/spark;
If you're on board, I'd create a PR with this change.
Which version of the action are you using?
v1
Environment
Self-hosted, linux
To Reproduce
Steps to reproduce the behavior:
pip3 install pyspark pytest
as a commandRun/Repo Url
My fork: https://github.com/johnnylarner/setup-spark
Describe the bug
The set-output
command is deprecated and will be disabled soon.
Which version of the action are you using?
v1
Environment
GitHub-hosted
Spark Versions
All
Additional context
Warning: The set-output
command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
Describe the bug
When you specify a combination of versions that does not exist, the Action does not fail:
The URL https://archive.apache.org/dist/spark/spark-3.2.1/spark-3.2.1-bin-hadoop3.3.tgz leads to a 404, the correct URL is https://archive.apache.org/dist/spark/spark-3.2.1/spark-3.2.1-bin-hadoop3.2.tgz.
I made this mistake because https://spark.apache.org/downloads.html advertises 3.2.1 as "with Hadoop 3.3 and later".
In any case, the action should fail if Spark cannot be installed instead of silently progressing.
Which version of the action are you using?
v1
Environment
GitHub-hosted
Spark Versions
Please list all of the effected versions of Spark (3.1.2
, etc.)
To Reproduce
- uses: vemonet/setup-spark@v1
with:
spark-version: 3.2.1
hadoop-version: 3.3
Run/Repo Url
https://github.com/anovos/anovos/runs/5326819596?check_suite_focus=true
Screenshots
Additional context
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.