osc / ood-myjobs Goto Github PK

View Code? Open in Web Editor NEW

0.0 12.0 2.0 788 KB

[MOVED] Job Composer allows creating and submitting batch jobs

Home Page: https://osc.github.io/Open-OnDemand/

License: MIT License

Ruby 42.60% JavaScript 25.42% CoffeeScript 7.77% CSS 6.47% HTML 17.39% Shell 0.36%

ood-myjobs's Introduction

Moved to https://github.com/OSC/ondemand/tree/master/apps/myjobs

Job Composer (renamed from My Jobs)

OOD Rails app for Open OnDemand for creating and managing batch jobs from template directories.

New Install

Installation assumptions: you have an Open OnDemand installation with File Explorer and Shell apps installed and a cluster config added to /etc/ood/config/clusters.d directory.

Start in the build directory for all sys apps, clone and check out the latest version of the myjobs app (make sure the app directory's name is myjobs):
```
scl enable rh-git29 -- git clone https://github.com/OSC/ood-myjobs.git myjobs
cd myjobs
scl enable rh-git29 -- git checkout tags/v2.10.1
```
Install the app for a production environment:
```
RAILS_ENV=production scl enable rh-git29 rh-ruby24 rh-nodejs6 -- bin/setup
```
this will setup a default Open OnDemand install. If you'd like a specific pre-defined portal such as OSC OnDemand you'd specify OOD_SITE and OOD_PORTAL as:
```
OOD_SITE=osc OOD_PORTAL=ondemand RAILS_ENV=production scl enable rh-git29 rh-ruby24 rh-nodejs6 -- bin/setup
```
assuming the corresponding .env.local.$OOD_SITE.$OOD_PORTAL file exists.

(Optional) Add "System" job templates to make available to each user of "My Jobs", i.e.

# the templates directory is hidden in the .gitignore,
# so we can pull a directory of templates from another source
# this is an example of deploying OSC's templates it provides to its users
git clone https://github.com/OSC/osc-myjobs-templates.git templates

Copy the built app directory to the deployment directory, and start the server. i.e.:

sudo mkdir -p /var/www/ood/apps/sys/myjobs
sudo cp -r . /var/www/ood/apps/sys/myjobs

Updating to a New Stable Version

Navigate to the app's build directory and check out the latest version:

cd myjobs # cd to build directory
scl enable rh-git29 -- git fetch
scl enable rh-git29 -- git checkout tags/v2.10.1

Update the app for a production environment:
```
RAILS_ENV=production scl enable rh-git29 rh-ruby24 rh-nodejs6 -- bin/setup
```
You do not need to specify OOD_SITE and OOD_PORTAL if they are defined in the .env.local file.

Copy the built app directory to the deployment directory:

sudo rsync -rlptv --delete . /var/www/ood/apps/sys/myjobs

Usage

"Job Composer" attempts to model a simple but common workflow. When creating a new batch job to run a simulation a user may:

copy the directory of a job they already ran or an example job
edit the files
submit a new job

"Job Composer" implements these steps by providing the user job template directories and the ability to make copies of them.

copy the a directory of a job they already ran or an example job
1. User can create a new job from a "default" template.
  1. A custom default template can be defined at /etc/ood/config/apps/myjobs/templates/default or under the app deployment directory at /var/www/ood/apps/sys/myjobs/templates/default
  2. If no default template is specified, the default is /var/www/ood/apps/sys/myjobs/example_templates/torque
2. user can select a directory to copy from a list of "System" templates the admin copied to /etc/ood/config/apps/myjobs/templates or under the app deployment directory at /var/www/ood/apps/sys/myjobs/templates during installation
3. user can select a directory to copy from a list of "User" templates that the user has copied to $HOME/ondemand/data/sys/myjobs/templates
4. user can select a job directory to copy that they already created through "Job Composer" from $HOME/ondemand/data/sys/myjobs/projects/default
edit the files
1. user can open the copied job directory in the File Explorer and edit files using the File Editor
submit a new job
1. user can use the Job Options form specify which host to submit to, what file is the job script
2. user can use the web interface to submit the job to the batch system
3. after the job is completed, the user can open the directory in the file explorer to view results

Templates

A template consists of a folder and a manifest.yml file.

The folder contains files and scripts related to the job.

The manifest contains additional metadata about a job, such as a name, the default host, the submit script file name, and any notes about the template.

name: A Template Name
host: ruby
script: ruby.sh
notes: Notes about the template, such as content and function.

In the event that a job is created from a template that is missing from the manifest.yml, "Job Composer" will assign the following default values:

name The name of the template folder.
host The cluster id of the first cluster with a valid resource_mgr listed in the OOD cluster config
script The first .sh file appearing in the template folder.
notes The path to the location where a template manifest should be located.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/OSC/ood-myjobs.

License

The gem is available as open source under the terms of the MIT License.

ood-myjobs's People

Contributors

Watchers

Forkers

ddietz89 morganrodgers

ood-myjobs's Issues

Have a separate brand for OSC OnDemand

Related to

Solution:

git clone osc-systemstatus, checkout branch ood_port then do this diff:

git diff 71cf2c0 5720ae0 -- app/assets/stylesheets/application.css.scss app/views/layouts/application.html.erb

diff --git a/app/assets/stylesheets/application.css.scss b/app/assets/stylesheets/application.css.scss
index 50a8055..ffd27f4 100644
--- a/app/assets/stylesheets/application.css.scss
+++ b/app/assets/stylesheets/application.css.scss
@@ -5,10 +5,6 @@
 // Load up bootstrap style sheet
 @import "bootstrap";

-// display header at top of app layout
-body {
-  @extend .awesim-banner-background-img;
-}

 // Your custom css styles below (must be manually added)
 @import "pages";
@@ -124,4 +120,4 @@ pre {

 a:-webkit-any-link{
   text-decoration:none !important;
-}
\ No newline at end of file
+}
diff --git a/app/views/layouts/application.html.erb b/app/views/layouts/application.html.erb
index 8635650..0af5c09 100644
--- a/app/views/layouts/application.html.erb
+++ b/app/views/layouts/application.html.erb
@@ -13,8 +13,6 @@
 </head>
 <body>

-<%= awesim_banner "System Status OSC" %>
-
 <header>
   <!-- navbar  -->
   <nav class="navbar navbar-inverse navbar-static-top" role="navigation">
@@ -27,7 +25,8 @@
           <span class="icon-bar"></span>
           <span class="icon-bar"></span>
         </button>
-        <%= awesim_breadcrumbs "System Status", root_path %>
+
+        <%= ood_breadcrumbs('System Status', root_path) %>
       </div>

       <!-- Collect the nav links, forms, and other content for toggling -->

Add a 'copy template' feature.

On the 'templates' page.

Add a button to copy a template.

Feature request: Re-run a job from the same staged folder.

Currently, a user can copy a job and edit/run the new job.

@nickjer has mentioned that this may be cumbersome because some jobs may have multi-gigabyte inputs.

This could be accomplished with a new route that deletes the has_many :jobs relations, then the job could be resubmitted.

This should probably include an 'are you sure you want to reset this job?' confirmation.

/jobs/new - add links to "add" and "edit" template source

"edit" should be disabled for "My Templates" and "OSC's Templates", enabled otherwise
A new model "TemplateSource" or "Source" is created via #18. Any of these the user has created should be added to the "Source:" dropdown

Minor modifications to index page buttons

Change name of button from "Set template" to "Create Template" and help text: "Create template from selected job"
add "pull-right" class to delete button so its far away from the rest

/jobs/new - add support for displaying metadata from manifest.yml in template directories to support creating Ruby cluster specific templates

A template should be able to provide a manifest.yml file containing information like:

name: Abaqus 6.14
cluster: oak-batch.osc.edu
notes: Uses: module load abaqus/6.14-nimbis

name: Abaqus 6.14
cluster: ruby
notes: Uses: module load abaqus/6.14-nimbis

Then this would appear in the template and the specified cluster would be used, providing support for Ruby templates.

This would be scraped and pulled similar to how the apps list for users is built by the dashboard - by reading the manifest.yml files for each of the templates. At this point a database column for templates could become useful for caching this information which might mean a list of the directories AND timestamps on the manifests that you cache, but caching in the database to speed this up (or providing another datastore for this cache) is a separate issue.

The point of this non-database approach to this information is that the cache never gets "outdated" - we can add new templates and remove old ones from the templates directory and all users of the app are ensured to always have the latest list of templates.

Disable cluster dropdown instead of dropdown options in job details view

I think instead of

<select class="form-control" id="job-details-server-select">
                <option id="oakley" value="oakley" disabled="">Oakley</option>
                <option id="ruby" value="ruby" disabled="">Ruby</option>
          </select>

use

<select class="form-control" id="job-details-server-select" disabled="" >
                <option id="oakley" value="oakley">Oakley</option>
                <option id="ruby" value="ruby">Ruby</option>
          </select>

/jobs - copy job

Create a new job from the selected job.

This copies the selected job's staged dir to a new unique staged dir and adds a new associated record to the database for this new job.

/jobs - stop a running job

This will stop the job from running and set the status to failed (or whatever the default is for osc-machete-rails).

Change "Templates" to "My Templates" and show only user's templates on this page

/myjobs/templates shows all the templates returned from Template.all. Instead it should show only templates from Source.my.templates.

This will make it more useful for the user to manage his own templates if he chooses too. All the information for OSC's templates are available in the "New Job" page so this view doesn't add any value for OSC's templates.

Visual bug when resizing browser window

When I shrink the width of my browser window, the table extends behind one of the elements on the page:

Add 'delete template' button

All templates are dynamically generated. A user can delete a template by deleting the folder that contains it, but it may be useful to have a 'delete' button on the templates page that deletes the folder in-app.

Change job_attrs type to text

The type for this is a string - this is dangerous when using this for a store. text should be used instead, which doesn't have the size limits that string has

https://github.com/AweSim-OSC/osc-jobconstructor/blob/68aadf2022b5f6b6243513d650a6fe46f406c338/db/schema.rb#L29

Consider changing schema to avoid needing database migrations

With osc_machete_rails the jobs schema uses a job_cache text column to store the attributes script, pbsid, and host:

store :job_cache, accessors: [ :script, :pbsid, :host ], coder: JSON

Currently, this is the osc_jobs schema:

  create_table "osc_jobs", force: true do |t|
    t.string   "name"
    t.string   "batch_host"
    t.string   "staged_dir"
    t.datetime "created_at"
    t.datetime "updated_at"
    t.string   "script_name"
  end

If we changed this to use a similar column it would end up something like:

  create_table "osc_jobs", force: true do |t|
    t.text "job_attrs"
    t.datetime "created_at"
    t.datetime "updated_at"
  end

and then we would add this line to the model:

store :job_attrs, accessors: [ :name, :batch_host, :staged_dir, :script_name ], coder: JSON

My understanding is that this change would be minor and would not affect the rest of the app. It would shift the burden of handling "data migrations" to the app itself - so if we add a new column in the future we would need to handle gracefully situations where that value was nil for preexisting jobs; and we would probably want to avoid renaming any "columns".

@basilgohar @nickjer @brianmcmichael should we consider this before OnDemand release? If we did this for the jobs table we would do the same for the templates table. I'd wait of course till we get the functionality working as we want to with the existing schema.

If we don't do this when we deploy a new version of the job constructor with a different schema (if it ever gets that) we would deploy it in a separate location as a separate app, and optionally offer an import button to import the data from the old app (or try to auto-import during the first launch).

Error when creating a new template when path is file.

When creating a new template, if the template path is set to a file path instead of a folder path there will be an error on create.

Allow datatables to save state with cookie.

Datatables has an option to save state. Consider enabling this to save user's sort/show selections.

Version templates with app

We currently configure the location of OSC specific templates to be a directory outside of the app. The temporary location was in WIAG's project space - because it was something that would be easy for us to add to without deploying a new version of the app. This is the code that sets that:

https://github.com/AweSim-OSC/osc-jobconstructor/blob/a76f6f629439a0ca9591c23ed5818f76dcc35d76/app/models/source.rb#L9-L11

def self.osc
  Source.new("OSC's Templates", "/nfs/01/wiag/PZS0645/ood/jobconstructor/templates")
end

These templates should be versioned and probably deployed on local disk along with the app. If we find that we are adding new templates or changing templates often, and it becomes cumbersome that updating or adding new templates requires deploying new app updates to provide these to users, then we could move this directory to a location outside of the app that we could update separately.

That would make deployment simpler for now and have one less thing dependent on GPFS. Perhaps:

def self.osc
  Source.new("OSC's Templates", Rails.root.join('templates').to_s)
end

Edit Job: "Specify Job Script" gets unwieldy when there are many files to choose from

This is related to #56

The specify job script using a radio button works great when there are a few files:

It is a little unwieldy when there are many files:

1. It might be easiest for now to just have this be a select control whose size attribute is set to 10 (or something that looks nice). We can revisit later, maybe when we find a good tree control (that is preferably AJAX powered, etc. so we aren't getting the full recursive list of all the files in the directory on the request) 2. Also, we should omit directories from this list (only show files, whether they be in directories or subdirectories).

/jobs - Open selected job's (staged) directory in cloud commander

New Job: List of templates should include the templates source

Add a column "Source" that will say "OSC's Templates" or "My Templates, depending on where it comes from.

Notice in this list that all of the templates are OSC provided except for the bottom (CFS) which is a template in "My Templates" directory.

With a column specifying which templates are which, I could sort and put all of my custom templates at the top if I wanted to.

A simple way to get this information is to add an attribute to the Template class source and when you instantiate template, add the source as an optional argument. Then when instantiating Template objects in this line https://github.com/AweSim-OSC/osc-jobconstructor/blob/2c7f83338a1978df4279a721d37b2dbe2588559c/app/models/source.rb#L25 you can add the self as the source i.e.

folders.map {|f| Template.new(Pathname.new(path).join(f), self) }

Of course you would use template.source.name for the table.

New Job: remove "Back" and "New Template" buttons from this view

This will be consistent with the other views. All three main pages are accessible from the nav bar at top. My Templates is a more natural place for creating a new template from scratch, versus creating a new template from scratch on the "New Job" page.

Though we might see a user wanting to create a template from another template... but that isn't what happens when you click this button - we might revisit that idea later...

User testing.

This app has not been tested in practice by anyone but me and could benefit by someone who isn't me trying to break it.

Modification to 'script_path' broke script selection.

When selecting the job script from the edit screen, the app throws an error on the script_path call.

Document templating

Templates will rely on a folder in wiag's gpfs location. This needs adequate documentation of

Location
Format
Manfest requirements

/jobs/1/edit - edit job details

This view lets you change only three things about a job:

The optional human readable name of the job
Whether to submit the job to Oakley or Ruby
Which shell script in the job should be the script submitted. This means providing a listing of all of the files in a job's staged directory and letting the user choose the file to act as the job script. This can be done by any number of means besides a radio button (though radio button might be the easiest):

Find a good way to get app root.

I need to come up with a good way to get the app root to use for building links.

This will be an issue that touches other projects in the magnum paradigm so this may be a broader issue for internal.

Basically the problem is that nginx stage is rewriting our url's, so the Rails.root is ~/ood_dev/<app>, but the app root is ~/pun/dev/<app>

I can't really rely on javascript's window.location because that may or may not include trailing slashes, subfolders, or other surprises.

I came up with a solution for a similar problem I was having in osc-fileexplorer, but it is something we may want to consider packaging up for developers.

https://github.com/AweSim-OSC/osc-fileexplorer/blob/12d0a72b89beb67e7d74cfeb6a48653101a40f45/app.js#L13-L20

/sources/new - create a new template source

Either /sources/new or /template-sources/new
A Source or a TemplateSource model should be created with two attributes: Name and Path

/jobs - Edit selected job

Take you to the jobs edit page for this job: #11

No screenshot provided, but should be a simple edit button (i.e. pencil) above next to the other buttons.

Better sizing of templates table.

The templates table uses fixed width columns.

Resize these to better fit content.

Change glypicons for buttons on jobs index page

Use glyphicon-duplicate for the copy button icon
Use glyphicon-edit for the edit button (instead of pencil)

/jobs - delete selected job

This will delete the selected job's database record and the job's staged directory.

New Job page: Add a button to open the selected template in cloud commander

On the new job page I see a list of all the templates. When I select a template, I should be able to click a button to open that template in cloud commander. It might make sense to have that button go above the table or above template notes.

/jobs - create a new template from the selected job

This will copy the staged directory to the user's "my templates" directory so when choosing a template to create a new job from it will appear in the list of templates when the data source is "my templates"

Create a job from a template on the system without copying it to your templates directory first

We may want to support the ability to create a new job from a random directory on the filesystem, that is not recognized as a template in one of the predefined template source directories. It would also be cool if Cloud Commander could have a context menu option when right clicking on directories to create a job from this (or copy this to my templates). Ideally this would be implemented in the app by having a "file picker" control that would let you "choose a template on the file system".

Brian's comment in #55:

I don't think we should require them to templatize a location before they convert it to a job.

/templates/new - Change name of "Path" field in the form to "Template will be created by recursively copying files from:"

This tells the user exactly what we are going to do with the path:

Show static details view of selected job

This should just show a simple recursive view of all the files in the staged directory of the job. The job script should be bolded or somehow indicated that it is the current job. This doesn't need to be done with a radio button - it can literally be any means to signify it, including bolding the text of the filename and using a simple html bullet list for the files.

/jobs/new - add ability to select templates from a list of predefined OSC templates

A template is a directory. This directory is copied recursively to the user's home directory as a job. The directory may contain sub directories and contains a job script.

The basic version of this view assumes:

all templates are submitted to Oakley (no Ruby templates)
the "name' of the template is the name of the template directory
the shell script is assumed to be the first script found that ends in the extension sh OR the script is named main.sh (probably easier)

OSC's templates (subject to change) are stored in this directory: /nfs/gpfs/PZS0645/ood/jobconstructor/templates

when the user selects a new template, the Path field updates to show the path to the selected template, and the static view on the right shows the files in the template and bolds the shell script
the list of templates corresponds to the list of directories in /nfs/gpfs/PZS0645/ood/jobconstructor/templates so this should be true:
```
$ ls -1 /nfs/gpfs/PZS0645/ood/jobconstructor/templates
abaqus
open_foam
```
When the user presses "Use selected template" the location specified by the Path field is recursively copied to the location specified by Template will be copied to field.
The dropdown "Source" lets you switch between OSC's Templates and My Templates. The path to My Templates is stored in the app's $DATAROOT/templates. When you change this dropdown, the list of templates in the list changes. This doesn't need to be AJAX - the view can just have the list of templates for each template source embeded in it.
Open Dir button opens the directory of the selected template in cloud commander

/templates/new - Change title of new template page to "Create Template"

The title of this page is currently "New Template Source" which is incorrect.

/sources/edit - edit or delete a template source

of course this is /sources/edit or /template-sources/edit depending on what model is created

/templates/new - allow to create a template with the path field empty

If I click "My Templates" at the top there is a "Create Template" button. If I click it, it takes me to the "Create Template" page with the Path field empty. If I try to create a template with the Path field empty, I am unable to save - so I am forced to add a directory to copy from.

Instead we should:

Allow submitting with an empty Path. If no path is provided, we will copy from the "Simple Sequential Job" template that submits to Oakley to the new template. If no path is provided and the server specified is Ruby, we should copy from a "Simple Sequential Ruby Job" - which we will need to add (basically the same as the "Simple Sequential Job" template but updated so it runs on Ruby).
If we want to get fancy, we can let the user know which this will be by setting the "placeholder" value of the field to this path - and updating the placeholder field when they change the server they are submitting to.
Add a "help text" block underneath the field explaining to the user what will happen if left empty:

<p class="help-block">* If left blank, a default job script will be added to your new template.</p>

Templates should also use .pbs default

Currently, a template without a manifest defaults to selecting the first .sh file in the folder. Our templates all use the .pbs suffix.

The template default should probably grab the first pbs or sh file, with preference given to the former.

Detect the folder size that a user is attempting to templatize and warn before copy.

If the user is attempting to make a template out of a folder of n size, we should warn them and force them to confirm. Preferably with a note saying how large the folder is.

I just templatized my home directory, but this app would make it pretty easy to start copying the entire file system into their templates folder.

Drop table "templates"

It appears we are not using the table "Templates" anymore, yet it is still defined in the schema. We should drop this table.

https://github.com/AweSim-OSC/osc-jobconstructor/blob/68aadf2022b5f6b6243513d650a6fe46f406c338/db/schema.rb#L32-L39

/jobs - start button

This is the play icon. It goes above with the other buttons. The selected job is submitted when this occurs.

No screenshot added.

When submitting a job, use the "underscore" version of the human readable name provided

This is the logic to determine the name of the job submitted:

If the user provides a human readable name for the job, use the "underscore" version by calling http://api.rubyonrails.org/classes/ActiveSupport/Inflector.html#method-i-parameterize with "_" as the separator
If the user doesn't, use the name of the directory or use what is in the job script itself?