Comments (12)
brianmcmichael
commented on August 31, 2024
The fastest partial solution would probably be to just regex check that the user is at least not trying to access a root file system.
These should do the trick.
/\/nfs\/[0-9]{2}\/\w+\//
/\/nfs\/gpfs\/\w+\//
from ood-myjobs.
brianmcmichael
commented on August 31, 2024
This was added at 73341c5
A user can still templatize a home dir with this regex, so it might be useful to add another w+ to the end to match that they are trying to copy a subfolder of a homedir. Not sure if this would be too restrictive (on GPFS, for example)
from ood-myjobs.
brianmcmichael
commented on August 31, 2024
Added the extra w+ check. I can't think of a situation where we would want to templataize a location where the path looks like a home dir, and now that the system templates are off of the wiag space we don't have to worry about that.
from ood-myjobs.
ericfranz
commented on August 31, 2024
This looks promising:
irb(main):037:0> stdout, stderr, status = Open3.capture3 "timeout 1s du -cbs /nfs/17/efranz/crimson_files/ContainerFillSim/containers/16"
=> ["", "", #<Process::Status: pid 29954 exit 124>]
irb(main):038:0> stdout, stderr, status = Open3.capture3 "timeout 10s du -cbs /nfs/17/efranz/crimson_files/ContainerFillSim/containers/16"
=> ["145849564\t/nfs/17/efranz/crimson_files/ContainerFillSim/containers/16\n145849564\ttotal\n", "", #<Process::Status: pid 29559 exit 0>]
irb(main):039:0> stdout, stderr, status = Open3.capture3 "timeout 10s du -cbs /nfs/17/efranz/process/tmp/p4trunk"
=> ["4096\t/nfs/17/efranz/process/tmp/p4trunk\n4096\ttotal\n", "du: cannot access `/nfs/17/efranz/process/tmp/p4trunk/.svn': Permission denied\ndu: cannot access `/nfs/17/efranz/process/tmp/p4trunk/p4_calico': Permission denied\ndu: cannot access `/nfs/17/efranz/process/tmp/p4trunk/p4': Permission denied\ndu: cannot access `/nfs/17/efranz/process/tmp/p4trunk/README': Permission denied\n", #<Process::Status: pid 29667 exit 1>]
irb(main):040:0>
- we can limit both by timeout (10s for example) and a byte limit (1000MB worth, for example) to prevent people from going crazy
- if some of the files are inaccessible the exitstatus is 1
- if timeout occurs exitstatus is 124
require 'open3'
require 'shellwords'
MAX_JOB_TEMPLATE_SIZE=1024*1024*1024
stdout, stderr, status = Open3.capture3 "timeout 10s du -cbs #{Shellwords.escape(DIR_PATH)}"
if status.success?
size = stdout.split.first
if size.nil?
raise "failed to calcuate directory"
elsif size.to_i > MAX_JOB_TEMPLATE_SIZE
raise "directory too large"
else
# good
end
elsif status.exitstatus == 124
raise "timeout occurred when trying to determine directory size; size must be computable in less than 10 seconds; template is too large or file system is slow - please try again later: #{stderr}"
else
raise "error with status #{status} occurred when trying to determine directory size: #{stderr}"
endfrom ood-myjobs.
ericfranz
commented on August 31, 2024
Of course we would do this: http://guides.rubyonrails.org/active_record_validations.html#performing-custom-validations
from ood-myjobs.
brianmcmichael
commented on August 31, 2024
We might want to make a way to override the max size in case the template is based on a large dataset or set of meshes.
Alternatively, provide instructions on how to access template data from a single source.
from ood-myjobs.
ericfranz
commented on August 31, 2024
It appears to be working:
irb(main):017:0> Filesystem.max_copy_safe_du_timeout_seconds
=> 10
irb(main):018:0> Filesystem.new.validate_path_is_copy_safe "/nfs/17/efranz/ood_dev/osc-jobconstructor"
=> [false, "Timeout occurred when trying to determine directory size. Size must be computable in less than 10 seconds. Either directory has too many files or the file system is currently slow (if so, please try again later)."]
irb(main):019:0> Filesystem.new.validate_path_is_copy_safe "/nfs/17/efranz/ood_dev/osc-jobconstructor/jobs"
=> [true, nil]
irb(main):020:0> Filesystem.new.validate_path_is_copy_safe "/nfs/17/efranz/ood_dev/osc-jobconstructor/app"
=> [true, nil]
irb(main):021:0> Filesystem.new.validate_path_is_copy_safe "/nfs/17/efranz/tmp"
=> [false, "Timeout occurred when trying to determine directory size. Size must be computable in less than 10 seconds. Either directory has too many files or the file system is currently slow (if so, please try again later)."]
irb(main):022:0> Filesystem.new.validate_path_is_copy_safe "/nfs/17/efranz/tmpp"
=> [false, "Error with status pid 18209 exit 1 occurred when trying to determine directory size: du: cannot access `/nfs/17/efranz/tmpp': No such file or directory\n"]
irb(main):023:0> Filesystem.new.validate_path_is_copy_safe "/nfs/17/efranz/tmp"
=> [true, nil]
irb(main):024:0> Filesystem.new.validate_path_is_copy_safe "/nfs/17/efranz/process/tmp/p4trunk"
=> [false, "Error with status pid 18730 exit 1 occurred when trying to determine directory size: du: cannot access `/nfs/17/efranz/process/tmp/p4trunk/.svn': Permission denied\ndu: cannot access `/nfs/17/efranz/process/tmp/p4trunk/p4_calico': Permission denied\ndu: cannot access `/nfs/17/efranz/process/tmp/p4trunk/p4': Permission denied\ndu: cannot access `/nfs/17/efranz/process/tmp/p4trunk/README': Permission denied\n"]
irb(main):025:0> Filesystem.new.validate_path_is_copy_safe "/"
=> [false, "Timeout occurred when trying to determine directory size. Size must be computable in less than 10 seconds. Either directory has too many files or the file system is currently slow (if so, please try again later)."]
irb(main):026:0> Filesystem.new.validate_path_is_copy_safe "/nfs/17/efranz/crimson_files/FanPortal/fans"
=> [false, "The directory is too large to copy. The directory should be less than 1073741824 bytes."]
from ood-myjobs.
ericfranz
commented on August 31, 2024
from ood-myjobs.
brianmcmichael
commented on August 31, 2024
Is 1 GB large enough?
from ood-myjobs.
ericfranz
commented on August 31, 2024
I don't know. @basilgohar @nickjer I set default limits on the directory size that we will let you create a new template from through the web interface. You can see the full pull request here: https://github.com/AweSim-OSC/osc-jobconstructor/pull/81/files
Essentially we validate whether the path is safe to copy from by verifying that the directory's size is not too large and that it doesn't have too many files. The method is validate_path_is_copy_safe
The two limits I created were:
- timeout 10s - if du takes longer than 10 seconds to compute we assume its too large for creating a template
- directory size 1GB
1GB is probably a little low; but right now each user has 500GB and in the future this will be higher so maybe we should up the size a little bit further... but what would that limit be? At what point do we say "this is not what you should be using the job constructor for"? Currently copying 500M file takes about 5 seconds - that will get faster of course. But we probably don't want a copy of a template to take a minute.
from ood-myjobs.
ericfranz
commented on August 31, 2024
If copying 500MB takes 5 seconds, then 3GB is a reasonable limit.
from ood-myjobs.
brianmcmichael
commented on August 31, 2024
This is going to be wildly variable based on file system load. We should re-run that benchmark once the new FS is in place.
from ood-myjobs.
Related Issues (20)
- SECURITY: Fix paperclip vulnerability.
- Update ood_core gem with bugfixes
- Add arbitrary "starting state" when submitting jobs till actual status is returned from the batch server HOT 4
- Address Gemfile security vulnerability
- Suppress "Unable to access log file" warnings HOT 1
- This is a test issue
- Log job submission arguments
- Script Name, when creating a new job from a template, should not be editable
- Update sprockets to address security vulnerability HOT 2
- Support WHITELIST_PATH HOT 2
- Template notes should be rendered using Markdown HOT 1
- The "Edit files" button should say "View files" if the template is read-only HOT 1
- Add support for localizing the help text of the default job options
- Add Placeholder / Updated Help For Job Array Spec
- Logfiles Reporting Statusable.classes Array is empty
- Default "simple sequential job" is reported to have no executions on job script HOT 2
- Datatables does not handle long job names well HOT 1
- Job Composer layout is cramped on a smaller screen HOT 1
- setup-production script fails if DATABASE_PATH does not exist HOT 2
- disk io error HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ood-myjobs.