university-of-delaware-it-rci / auto_tmpdir Goto Github PK

When the auto_tmpdir.so shared library has been installed, and a /etc/slurm/plugstack.conf configuration file has been created, there is still one missing step which should be documented in README.md: The slurmd needs to be restarted!

I propose to add a new section at the end of README.md:

Enabling the plugin

The plugin is enabled by adding it to your plugstack.conf file on all nodes as shown above, for example:

required    auto_tmpdir.so          mount=/tmp mount=/var/tmp

The slurmd does not pick up the new plugin until it is restarted:

systemctl restart slurmd

With each major version upgrade to Slurm a new build- directory should be created and a new build done therein.

Using the latest "dir-removal-fixup" branch on our slurm 20.11.6 installation, I get this error when running any job:
error: auto_tmpdir::auto_tmpdir_fs_init_with_file: unable to open state file /local/auto_tmpdir_fs-1654803.cache (errno = 2)

we want to use the dir-removal-fixup branch since we are running into the problem where jobs with the using the --array variable weren't removing their temp files properly.

some more context in the logs:
[2021-06-15T15:44:44.503] [1654803.batch] debug: /var/spool/slurm/d/conf-cache/plugstack.conf: 1: include "/etc/slurm/plugstack.conf.d/*.conf"
[2021-06-15T15:44:44.503] [1654803.batch] debug: spank: opening plugin stack /etc/slurm/plugstack.conf.d/tmpdir.conf
[2021-06-15T15:44:44.503] [1654803.batch] debug: spank: /etc/slurm/plugstack.conf.d/tmpdir.conf:1: Loaded plugin auto_tmpdir.so
[2021-06-15T15:44:44.503] [1654803.batch] debug: SPANK: appending plugin option "no-rm-tmpdir"
[2021-06-15T15:44:44.503] [1654803.batch] debug: auto_tmpdir::__auto_tmpdir_fs_default_state_file: 1654803
[2021-06-15T15:44:44.503] [1654803.batch] debug: auto_tmpdir::__auto_tmpdir_fs_default_state_file: state_dir=/local
[2021-06-15T15:44:44.503] [1654803.batch] error: auto_tmpdir::auto_tmpdir_fs_init_with_file: unable to open state file /local/auto_tmpdir_fs-1654803.cache (errno = 2)

We have slurm 21.08.8-2 installed as RPM packages. The OS is AlmaLinux 8.6. I installed the distributon's cmake RPM package.

Following the build instructions in README.md I'm trying to make a build but get an error about something called SLURM extensions directory:

[ohni@testlogin build-20220627]$ cmake -DSLURM_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release  -DAUTO_TMPDIR_ENABLE_SHARED_TMPDIR=YES     -DAUTO_TMPDIR_DEFAULT_SHARED_PREFIX=/scratch ..
CMake Deprecation Warning at CMakeLists.txt:1 (CMAKE_MINIMUM_REQUIRED):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


CMake Error at CMakeLists.txt:27 (MESSAGE):
  SLURM extensions directory could not be found


-- Configuring incomplete, errors occurred!
See also "/home/niflheim/ohni/auto_tmpdir/build-20220627/CMakeFiles/CMakeOutput.log".

I wonder what's missing?

The title says it all: The table entry for AUTO_TMPDIR_ENABLE_SHARED_TMPDIR is duplicated with different texts in README.md.

Hi,

would it be possible to add CPackDeb support too?

https://cmake.org/cmake/help/v3.7/module/CPackDeb.html

Best,
Stefan

I'd like to test your auto_tmpdir tool, but I'm unable to get started:

Question: How do I build the auto_tmpdir library? Which are the prerequisite tools?

Thansk,
Ole

I have built the 1.0.2 RPM package, but it comes out as 1.0.1:

$ cmake3 -DSLURM_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release -DAUTO_TMPDIR_DEFAULT_LOCAL_PREFIX=/scratch/slurm- ..
-- The C compiler identification is GNU 4.8.5
-- Check for working C compiler: /bin/cc
-- Check for working C compiler: /bin/cc - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Found SLURM: /usr/lib64/libslurm.so  
-- SLURM modules directory: /usr/lib64/slurm
-- SLURM version detected as 21.08.8, 21.08
-- Configuring done
-- Generating done
-- Build files have been written to: /home/niflheim/ohni/auto_tmpdir/builddir

$ make package
Scanning dependencies of target auto_tmpdir
[ 33%] Building C object CMakeFiles/auto_tmpdir.dir/fs-utils.c.o
[ 66%] Building C object CMakeFiles/auto_tmpdir.dir/auto_tmpdir.c.o
[100%] Linking C shared module auto_tmpdir.so
[100%] Built target auto_tmpdir
Run CPack packaging tool...
CPack3: Create package using RPM
CPack3: Install projects
CPack3: - Run preinstall target for: auto_tmpdir
CPack3: - Install project: auto_tmpdir []
CPack3: Create package
CPackRPM: Will use USER specified spec file: /home/niflheim/ohni/auto_tmpdir/builddir/auto_tmpdir.spec
CPack3: - package: /home/niflheim/ohni/auto_tmpdir/builddir/auto_tmpdir-1.0.1-21.08.8.el7.x86_64.rpm generated.

The generated SPEC file starts with:

$ head auto_tmpdir.spec
Buildroot:      /home/niflheim/ohni/auto_tmpdir/builddir/_CPack_Packages/Linux/RPM/
Summary:        Slurm auto_tmpdir SPANK plugin
Name:           auto_tmpdir
Version:        1.0.1
Release:        21.08.8%{?dist}
License:        BSD
Group:          Unspecified
Prefix:         /usr/lib64/slurm
Requires:       slurm == 21.08.8

Maybe this line needs to be updated in CMakeLists.txt?

PROJECT (auto_tmpdir VERSION 1.0.1 LANGUAGES C)

When I build auto_tmpdir as a normal user and want to sudo make install, errors are found because root doesn't have permission to write files to the user's directory:

[ohni@testlogin build-20220629]$ sudo make install
Built target auto_tmpdir
Install the project...
-- Install configuration: "Release"
-- Installing: /usr/lib64/slurm/auto_tmpdir.so
CMake Error at cmake_install.cmake:77 (file):
  file failed to open for writing (Permission denied):

    /home/niflheim/ohni/auto_tmpdir/build-20220629/install_manifest.txt


make: *** [Makefile:93: install] Error 1

I fixed this by cloning the Git repo on a local scratch directory and installing from there.
But I wonder if there exists a more elegant method?

Add CPack RPM generator infrastructure to the project's CMake configuration.

Initial implementation tracking in cpack-rpm branch.

In Bug 14344 - job_container/tmpfs jobs fail because of race condition with NFS autofs mounting
we're discussing the issue that NFS automounted home directories do not work when using mount namespaces in the job_container/tmpfs plugin.

Question: Have you tried out the auto_tmpdir plugin together with NFS automounts? I'm hoping that this solution would work for us.

This is a great plugin! I'm testing and found that created directories are not automatically deleted as the documentation states. Also, I am not using the --no-rm-tmpdir flag. This was odd but then I noticed this commit. From my very limited understanding it appears that this has changed the documented behavior. Could you clarify how I can have automatic deletion of created directories?

Thanks so much!