hpc-carpentry / hpc-python Goto Github PK

View Code? Open in Web Editor NEW

57.0 10.0 48.0 5.85 MB

HPC Python lesson materials

Home Page: https://hpc-carpentry.github.io/hpc-python/

License: Other

Makefile 7.97% HTML 0.83% R 6.53% Python 83.26% Shell 0.52% Ruby 0.89%

lesson hpc-carpentry english pre-alpha hpc-python hpc-carpentry-lab

hpc-python's Introduction

HPC Python

This is the HPC Carpentry Python lesson.

This lesson is focused on teaching the basics of high-performance computing (HPC). There are 4 primary components to this lesson. Each component is budgeted half a day's worth of teaching-time, resulting in a two day workshop.

UNIX fundamentals
Working on a cluster
Programming language introduction/review
Introduction to parallel programming

Sections 3 and 4 (programming) will feature two programming languages: Python and Chapel. There are strong arguments for both languages, and instructors will be able to choose which language they wish to teach in.

Topic breakdown and todo list

The lesson outline and rough breakdown of topics by lesson writer is in lesson-outline.md. The topics there will be initially generated by the lesson writer, and then reviewed by the rest of the group once complete.

Lesson writing instructions

This is a fast overview of the Software Carpentry lesson template. This won't cover lesson style or formatting (address that during review?).

For a full guide to the lesson template, see the Software Carpentry example lesson.

Lesson structure

Software Carpentry lessons are generally episodic, with one clear concept for each episode (example). We've got 4 major sections, each section should be broken up into several episodes (perhaps the higher-level bullet points from the lesson outline?).

An episode is just a markdown file that lives under the _episodes folder. Here is a link to a markdown cheatsheet with most markdown syntax. Additionally, the Software Carpentry lesson template uses several extra bits of formatting - see here for a full guide. The most significant change is the addition of a YAML header that adds metadata (key questions, lesson teaching times, etc.) and special syntax for code blocks, exercises, and the like.

Episode names should be prefixed with a number of their section plus the number of their episode within that section. This is important because the Software Carpentry lesson template will auto-post our lessons in the order that they would sort in. As long as your lesson sorts into the correct order, it will appear in the correct order on the website.

Publishing changes to GitHub + the GitHub pages website

The lesson website is viewable at hpc-carpentry.github.io/hpc-python.

The lesson website itself is auto-generated from the gh-pages branch of this repository. GitHub pages will rebuild the website as soon as you push to the GitHub gh-pages branch. Because of this gh-pages is considered the "master" branch.

Previewing changes locally

Obviously having to push to GitHub every time you want to view your changes to the website isn't very convenient. To preview the lesson locally, run make serve. You can then view the website at localhost:4000 in your browser. Pages will be automatically regenerated every time you write to them.

Note that the autogenerated website lives under the _site directory (and doesn't get pushed to GitHub).

This process requires Ruby, Make, and Jekyll. You can find setup instructions here.

Example lessons

A couple links to example SWC workshop lessons for reference:

Example Bash lesson
Example Python lesson
Example R lesson (uses R Markdown files instead of Markdown)

hpc-python's People

Contributors

Stargazers

Watchers

hpc-python's Issues

Broken link in readme

That link points to:
https://github.com/hpc-carpentry/hpc-python/blob/gh-pages/hpc-carpentry.github.io/hpc-novice

Which results in 404 error.

Change words/books example to genomic use case

Hello hpc-carpentry team,

I am an Carpentry Instructor and I'd be quite interested to turn this lesson into an official Data Carpentry lesson ultimately as I see a considerable interest for Snakemake among Life Scientists.

As Snakemake is primarily used in genomics as far as I can tell, I'd like to change the example from words and books to some small sequencing dataset.
An example can be seen here in the official Snakemake read the docs.

What do you think?
Cheers
Marc

Need jump lists (anchors) for section headings

@zkamvar
We need to update style for this lesson so that jump lists (anchors for different sections) becomes automatically available for different levels of headings.

Use standard Gemfile from carpentries/lesson-example

Hi!

I forked this repo with the intent of contributing to it.

When I tried to render the lesson locally, it failed with the following error:

Could not find gem 'mdl' in any of the gem sources listed in your Gemfile.
Run `bundle install` to install missing gems.

After running bundle install, the dependency was installed and I could render the lesson.

However, I propose that the template Gemfile from carpentries/lesson-example be used. This will ensure consistency with other lessons and thus helps to advance the development stage of this lesson. I was able to successfully render the lesson locally using this Gemfile.

I will submit a PR addressing this, but please feel free to reject it if there's a reason to use a custom Gemfile.

Best,
V

I am interested in helping maintain this repository.

setup CI for snakemake lesson?

Based on what I saw here, I think it's a nice treat to add (travis) CI for this repo. This could be integrated once #16 is merged.

I am not sure though if this interacts with Github Pages deployment somehow. Does anybody know?

hpc-python lesson should cover Dask

this and the dask documentation already has some really good examples that could serve as a jumping off point: https://github.com/sdsc/sdsc-summer-institute-2019/tree/master/hpc0_python_hpc

an overview of the "why" of dask: https://notamonadtutorial.com/interview-with-dasks-creator-scale-your-python-from-one-computer-to-a-thousand-b4483376f200

'plotcount.py isles.dat show' doesn't open a window

As mentioned here, the call to

$ python3 plotcount.py isles.dat show

should open a window to produce a plot. On my system:

Python 3.7.5 (default, Oct 17 2019, 12:09:47) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux

Nothing happens.

code owners of this repo?

@tkphd I added you to the list of maintainers of this project for now. This should give you merge permission. We can revoke this permission at any time.

I am not aware of any code owners right now. But the content of this gem related to snakemake is simply too valuable IMHO. I'd like to make some more suggestions and PRs in order to edge out some inconsistencies in the plot of teaching (just taught this material for the first time). For this, I need an active maintainer to accept the PRs.

@jstaf Are you still involved in this project and have the capacities to review/merge PRs?

Make Jupyter notebooks the default teaching tool

This should be done along with providing more detailed setup instructions.

wrap lines at 79 characters

The GitHub diff interface wraps lines at 80 characters. Since we use 100 characters as the max width, this makes for ugly diffs with lots of continued lines. To improve the clarity of these diffs, as well as readability in terminal text editors, we should back away from this edge, and wrap our content at 79 characters.

This is easily done with emacs, but the change would touch almost every file in the repository, so this should be coordinated with other efforts to minimize merge conflicts and the number of branch rebases to be done.

Use --delete-all-output instead of creating a clean rule

To remove all files created by snakemake as output files to start from scratch, you can use

$ snakemake some_target --delete-all-output

https://snakemake.readthedocs.io/en/stable/project_info/faq.html#how-do-i-remove-all-files-created-by-snakemake-i-e-like-make-clean

factor out snakemake part into hpc-workflow

I am bound to teach snakemake at my place. People have expressed high interest, so I just went through the current state of the material. I think it contains a lot of gems and is well written. Thanks to all the authors.

However, I have the feeling that workflows are so important, that they potentially need a repo for their own. I could imagine refactoring the snakemake focussed modules into hpc-carpentry/hpc-workflow. I just wanted to put this idea out there and hope that people comment on this.

code formatting

The Carpentries stylesheet provides magic to format code. These should be applied throughout the hpc-python lessons.

Fork of this lesson in incubator

There is a fork of this lesson in the Carpentries Incubator which focusses purely on Snakemake (and that is indeed the majority of the content in this lesson). Does it make sense to maintain this lesson if others are willing to maintain that independently? Should we put some effort into making the lesson in the incubator more transferable instead?

splitting hpc-python into hpc-python and hpc-workflows

I propose to split the hpc-python in half:

hpc-python: episode 1 to episode 6 remains where it is
new hpc-carpentry repository 'hpc-workflows': episode 11 to episode 18

The introduction to snakemake (the second part of hpc-python) is very well done and would offer more effective real estate to focus contributors attention if in an isolated repository. I feel that it is a bit hidden currently, which I believe is a shame.

tests pass even when mdl is missing

Silent error unveiled by #40.