university-of-delaware-it-rci / auto_tmpdir Goto Github PK
View Code? Open in Web Editor NEWSlurm SPANK plugin for automated handling of temporary directories for jobs.
License: BSD 2-Clause "Simplified" License
Slurm SPANK plugin for automated handling of temporary directories for jobs.
License: BSD 2-Clause "Simplified" License
When the auto_tmpdir.so
shared library has been installed, and a /etc/slurm/plugstack.conf
configuration file has been created, there is still one missing step which should be documented in README.md
: The slurmd needs to be restarted!
I propose to add a new section at the end of README.md
:
The plugin is enabled by adding it to your plugstack.conf
file on all nodes as shown above, for example:
required auto_tmpdir.so mount=/tmp mount=/var/tmp
The slurmd
does not pick up the new plugin until it is restarted:
systemctl restart slurmd
With each major version upgrade to Slurm a new build- directory should be created and a new build done therein.
Using the latest "dir-removal-fixup" branch on our slurm 20.11.6 installation, I get this error when running any job:
error: auto_tmpdir::auto_tmpdir_fs_init_with_file: unable to open state file /local/auto_tmpdir_fs-1654803.cache
(errno = 2)
we want to use the dir-removal-fixup branch since we are running into the problem where jobs with the using the --array variable weren't removing their temp files properly.
some more context in the logs:
[2021-06-15T15:44:44.503] [1654803.batch] debug: /var/spool/slurm/d/conf-cache/plugstack.conf: 1: include "/etc/slurm/plugstack.conf.d/*.conf"
[2021-06-15T15:44:44.503] [1654803.batch] debug: spank: opening plugin stack /etc/slurm/plugstack.conf.d/tmpdir.conf
[2021-06-15T15:44:44.503] [1654803.batch] debug: spank: /etc/slurm/plugstack.conf.d/tmpdir.conf:1: Loaded plugin auto_tmpdir.so
[2021-06-15T15:44:44.503] [1654803.batch] debug: SPANK: appending plugin option "no-rm-tmpdir"
[2021-06-15T15:44:44.503] [1654803.batch] debug: auto_tmpdir::__auto_tmpdir_fs_default_state_file: 1654803
[2021-06-15T15:44:44.503] [1654803.batch] debug: auto_tmpdir::__auto_tmpdir_fs_default_state_file: state_dir=/local
[2021-06-15T15:44:44.503] [1654803.batch] error: auto_tmpdir::auto_tmpdir_fs_init_with_file: unable to open state file /local/auto_tmpdir_fs-1654803.cache
(errno = 2)
We have slurm 21.08.8-2 installed as RPM packages. The OS is AlmaLinux 8.6. I installed the distributon's cmake RPM package.
Following the build instructions in README.md I'm trying to make a build but get an error about something called SLURM extensions directory:
[ohni@testlogin build-20220627]$ cmake -DSLURM_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release -DAUTO_TMPDIR_ENABLE_SHARED_TMPDIR=YES -DAUTO_TMPDIR_DEFAULT_SHARED_PREFIX=/scratch ..
CMake Deprecation Warning at CMakeLists.txt:1 (CMAKE_MINIMUM_REQUIRED):
Compatibility with CMake < 2.8.12 will be removed from a future version of
CMake.
Update the VERSION argument <min> value or use a ...<max> suffix to tell
CMake that the project does not need compatibility with older versions.
CMake Error at CMakeLists.txt:27 (MESSAGE):
SLURM extensions directory could not be found
-- Configuring incomplete, errors occurred!
See also "/home/niflheim/ohni/auto_tmpdir/build-20220627/CMakeFiles/CMakeOutput.log".
I wonder what's missing?
The title says it all: The table entry for AUTO_TMPDIR_ENABLE_SHARED_TMPDIR is duplicated with different texts in README.md.
Hi,
would it be possible to add CPackDeb support too?
https://cmake.org/cmake/help/v3.7/module/CPackDeb.html
Best,
Stefan
I'd like to test your auto_tmpdir tool, but I'm unable to get started:
Question: How do I build the auto_tmpdir library? Which are the prerequisite tools?
Thansk,
Ole
I have built the 1.0.2 RPM package, but it comes out as 1.0.1:
$ cmake3 -DSLURM_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release -DAUTO_TMPDIR_DEFAULT_LOCAL_PREFIX=/scratch/slurm- ..
-- The C compiler identification is GNU 4.8.5
-- Check for working C compiler: /bin/cc
-- Check for working C compiler: /bin/cc - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Found SLURM: /usr/lib64/libslurm.so
-- SLURM modules directory: /usr/lib64/slurm
-- SLURM version detected as 21.08.8, 21.08
-- Configuring done
-- Generating done
-- Build files have been written to: /home/niflheim/ohni/auto_tmpdir/builddir
$ make package
Scanning dependencies of target auto_tmpdir
[ 33%] Building C object CMakeFiles/auto_tmpdir.dir/fs-utils.c.o
[ 66%] Building C object CMakeFiles/auto_tmpdir.dir/auto_tmpdir.c.o
[100%] Linking C shared module auto_tmpdir.so
[100%] Built target auto_tmpdir
Run CPack packaging tool...
CPack3: Create package using RPM
CPack3: Install projects
CPack3: - Run preinstall target for: auto_tmpdir
CPack3: - Install project: auto_tmpdir []
CPack3: Create package
CPackRPM: Will use USER specified spec file: /home/niflheim/ohni/auto_tmpdir/builddir/auto_tmpdir.spec
CPack3: - package: /home/niflheim/ohni/auto_tmpdir/builddir/auto_tmpdir-1.0.1-21.08.8.el7.x86_64.rpm generated.
The generated SPEC file starts with:
$ head auto_tmpdir.spec
Buildroot: /home/niflheim/ohni/auto_tmpdir/builddir/_CPack_Packages/Linux/RPM/
Summary: Slurm auto_tmpdir SPANK plugin
Name: auto_tmpdir
Version: 1.0.1
Release: 21.08.8%{?dist}
License: BSD
Group: Unspecified
Prefix: /usr/lib64/slurm
Requires: slurm == 21.08.8
Maybe this line needs to be updated in CMakeLists.txt?
PROJECT (auto_tmpdir VERSION 1.0.1 LANGUAGES C)
When I build auto_tmpdir
as a normal user and want to sudo make install
, errors are found because root doesn't have permission to write files to the user's directory:
[ohni@testlogin build-20220629]$ sudo make install
Built target auto_tmpdir
Install the project...
-- Install configuration: "Release"
-- Installing: /usr/lib64/slurm/auto_tmpdir.so
CMake Error at cmake_install.cmake:77 (file):
file failed to open for writing (Permission denied):
/home/niflheim/ohni/auto_tmpdir/build-20220629/install_manifest.txt
make: *** [Makefile:93: install] Error 1
I fixed this by cloning the Git repo on a local scratch directory and installing from there.
But I wonder if there exists a more elegant method?
Add CPack RPM generator infrastructure to the project's CMake configuration.
Initial implementation tracking in cpack-rpm branch.
In Bug 14344 - job_container/tmpfs jobs fail because of race condition with NFS autofs mounting
we're discussing the issue that NFS automounted home directories do not work when using mount namespaces in the job_container/tmpfs plugin.
Question: Have you tried out the auto_tmpdir plugin together with NFS automounts? I'm hoping that this solution would work for us.
This is a great plugin! I'm testing and found that created directories are not automatically deleted as the documentation states. Also, I am not using the --no-rm-tmpdir flag. This was odd but then I noticed this commit. From my very limited understanding it appears that this has changed the documented behavior. Could you clarify how I can have automatic deletion of created directories?
Thanks so much!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.