ncar / ncar-python-tutorial Goto Github PK
View Code? Open in Web Editor NEWNumerical & Scientific Computing with Python Tutorial
Home Page: https://ncar.github.io/ncar-python-tutorial
License: Creative Commons Attribution 4.0 International
Numerical & Scientific Computing with Python Tutorial
Home Page: https://ncar.github.io/ncar-python-tutorial
License: Creative Commons Attribution 4.0 International
Need to check if there already exists an analysis
conda environment. If it already exists, stop execution and tell the user to either (1) rename the environment or (2) remove the environment, and tell them the commands to do so.
Then they have to rerun the configure script.
There is now a skeleton of the 0-30 tutorial in the files z230-pt1.md
and z230-pt2.md
.
There is still missing content in here, though, and someone needs to fill in the blanks.
That is currently missing includes:
-- OHC remove dask mentions
-- Plotting, e.g. remove "tab:red" for colors, fig-size is in inches, add how-to set default values ...
I've been using jupyter notebook on casper successfully... until today. I even did it yesterday.
I've been starting it from jlab-dav.
Today this was the error I got:
Launching notebook server
partition = dav
memory = 8GB
constraint = casper
account =
port = 8888
sbatch: error: You must specify a time limit (-t)
Contact [email protected] for assistance
sbatch: error: Batch job submission failed: Access/permission denied
waiting
Currently, the jlab-machine
launch scripts (using SSH tunneling) ship with the Python tutorial content. However, these scripts are separately useful outside of the tutorial. I believe that these launch scripts should be a stand-alone package that is a dependency for the tutorial.
First, I think that we should pull the launch scripts out into a separate repo and then make this repo installable via pip
. I don't think a conda install is required. We will need a good name for this package.
Second, how we specify the launch-script package as a dependency for the tutorial can be done in a couple of different ways:
pip
installable package, too, and explicitly declare the launch-scripts package as a dependency.Option (2) is the easiest and perhaps the first step to take, regardless.
Need the Pangeo environment/kernel on the JHub updated to include all of the packages that we need for this tutorial.
Also need to tailor this tutorial for this new kernel.
I just updated environments/env-analysis.py to address a bug in MetPy. However, when I update the environment using conda, I get the following message.
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:
- conda-forge/noarch::botocore==1.12.109=py_0
- conda-forge/linux-64::s3transfer==0.2.0=py36_0
- conda-forge/noarch::boto3==1.9.108=py_0
It's been proposed that we break up our tutorials into smaller chunks that are two-hour long segments. Can we develop a curriculum of "seminars" that gets people to the "I'm ready for a hackathon!" stage.
Should be prepared to cover:
dashboard
has changed names.
Is the slurm configuration actually a good one?
How to import modules?
What is in a dataset?
What is the difference between a datset and a dataarray?
conda config --set channel_priority flexible
followed by
./setup/configure --clobber
Hi all, forgive me if this isn't the correct place for this issue, but I am having an issue launching Jupyter notebook in the pangeo environment after the Cheyenne upgrade.
I follow instructions here to launch the notebook, and when I enter:
jupyter lab --no-browser --ip=
hostname --port=8877
I receive the following error:
Traceback (most recent call last):
File "/glade/u/home/doughert/miniconda3/envs/pangeo/lib/python3.5/site-packages/traitlets/traitlets.py", line 528, in get
value = obj._trait_values[self.name]
KeyError: 'runtime_dir'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/glade/u/home/doughert/miniconda3/envs/pangeo/bin/jupyter-lab", line 11, in <module>
sys.exit(main())
File "/glade/u/home/doughert/miniconda3/envs/pangeo/lib/python3.5/site-packages/jupyter_core/application.py", line 266, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/glade/u/home/doughert/miniconda3/envs/pangeo/lib/python3.5/site-packages/traitlets/config/application.py", line 657, in launch_instance
app.initialize(argv)
File "<decorator-gen-7>", line 2, in initialize
File "/glade/u/home/doughert/miniconda3/envs/pangeo/lib/python3.5/site-packages/traitlets/config/application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "/glade/u/home/doughert/miniconda3/envs/pangeo/lib/python3.5/site-packages/notebook/notebookapp.py", line 1627, in initialize
self.init_configurables()
File "/glade/u/home/doughert/miniconda3/envs/pangeo/lib/python3.5/site-packages/notebook/notebookapp.py", line 1317, in init_configurables
connection_dir=self.runtime_dir,
File "/glade/u/home/doughert/miniconda3/envs/pangeo/lib/python3.5/site-packages/traitlets/traitlets.py", line 556, in __get__
return self.get(obj, cls)
File "/glade/u/home/doughert/miniconda3/envs/pangeo/lib/python3.5/site-packages/traitlets/traitlets.py", line 535, in get
value = self._validate(obj, dynamic_default())
File "/glade/u/home/doughert/miniconda3/envs/pangeo/lib/python3.5/site-packages/jupyter_core/application.py", line 99, in _runtime_dir_default
ensure_dir_exists(rd, mode=0o700)
File "/glade/u/home/doughert/miniconda3/envs/pangeo/lib/python3.5/site-packages/jupyter_core/utils/__init__.py", line 13, in ensure_dir_exists
os.makedirs(path, mode=mode)
File "/glade/u/home/doughert/miniconda3/envs/pangeo/lib/python3.5/os.py", line 231, in makedirs
makedirs(head, mode, exist_ok)
File "/glade/u/home/doughert/miniconda3/envs/pangeo/lib/python3.5/os.py", line 241, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/run/user/24367'
I have never had issues with this before, which is why I think this is related to the Cheyenne upgrade. Does anyone have ideas about how to fix this? Thanks in advance.
When I try to run jlab-ch, I get the following (truncated) output:
Launching notebook server
queue = share
account = P93300670
nodes = 1
ncpus = 1
memory = 8GB
walltime = 06:00:00
port = 8888
submitted job: 7186475.chadmin1.ib0.cheyenne.ucar to queue share
waiting for job to runqstat: illegally formed job identifier: 7186475.chadmin1.ib0.cheyenne.ucar
qstat: illegally formed job identifier: 7186475.chadmin1.ib0.cheyenne.ucar
..qstat: illegally formed job identifier: 7186475.chadmin1.ib0.cheyenne.ucar
qstat: illegally formed job identifier: 7186475.chadmin1.ib0.cheyenne.ucar
..qstat: illegally formed job identifier: 7186475.chadmin1.ib0.cheyenne.ucar
It looks like something is going wrong with extracting the job id from the output generated by qsub. It is quite possible that the format of the output from qsub has changed with the updates to cheyenne. That said, the bash for extracting the job id is sjob=${s%.*}
. I don't know bash well enough to know what this is doing, or why it appears to no longer work.
This is a potential headache for users who already have software installed in their base environment. We need a way to make tutorials like this work without needing the user to maintain a special base on their own.
@matt-long It would be good to clarify in the tutorial setup instructions in the readme which host is being used for each step. E.g. do you install conda on your laptop or on cheyenne?
Could we adopt a template for notebooks in this repository? My proposition is to have:
learning objectives
section at the beginning of the notebookGoing Further
section at the end of the notebook. This section could have references to documentation sections that are relevant to what was covered in the notebook and/or references to other notebooks.On the landing page:
"or you are tired of tutorials the leap into advanced third-party packages"
Should "the" be "that"? Or something else?
I found the advanced Xarray plotting tutorial -- archive/old-contents/notebooks/xarray/02-xarray-advanced-plotting.ipynb
but wasn't there a talk on Cartopy and data visualization in general? Is that stored in a different repo?
Create a notebook that introduces Jupyter notebooks!
When I attempt to run jlab-dav, I get
slurm_load_jobs error: Socket timed out on send/recv operation
This seems to be intermittent. execdav
seems to work ok. Is there something we should be doing differently in jlab-dav
?
Can we seriously turn off the set -x
feature within the bash kernel? It really screws up output with output you would expect to not be there.
Let's enumerate topics that deserve formal instruction at the June CGD tutorial.
It would be nice if we can point people to this repo to enable self-guided instruction. I think we have a pretty solid draft of the content we might like.
Can we add a sphinx docs assembly of (some of) the notebooks into a sensible outline?
Considerations:
I think we'd like to include NCAR-specific material, but clearly delineate it as such.
Some of the "workflow" notebooks are more pedagogical than others: we might consider reorganizing. The OHC example, for instance, provides a nice introductory, idealized example, whereas the O2 trends is really an "advanced example." In this sense, the OHC example belongs in Chapter 1, whereas the O2 trends notebook should be in an appendix.
I got a new laptop and after a week of use realized I still needed to install jupyter... I figured I'd go ahead and set up the recommended analysis environment as well. Per the README, I ran ./setup/configure
and things seem to be stalled at
$ ./setup/configure
************** Found an existing Conda installation in: /Users/mlevy/miniconda3/bin/conda **************
***************** Skipping Conda installation... *****************
************** Creating/Updating conda environments (this can take 5-10 min) ***********
It's been 20+ minutes, and while I would not be surprised to learn that conda is doing something behind the screens, a little more information on how to track progress over this period of time would be nice. $ conda list
isn't showing any updates to (base)
yet, but something is happening to my environment:
$ conda list envs
# packages in environment at /Users/mlevy/miniconda3:
#
# Name Version Build Channel
I'll continue to let the script run in the background, hopefully it'll finish eventually...
(Possibly related to #30?)
Same user as in #85 -- he had an old version of anaconda installed, and I ran conda update conda
to bring it up to 4.7.12; when I first ran into #85 I thought the issue was related to his conda installation so we moved ~/anaconda3
-> ~/anaconda3-old
and let setup/configure
install miniconda. After resolving #85 by installing the command line tools, setup
ran successfully but when he runs
$ conda activate python-tutorial
He gets dumped in /
instead of remaining in his current working directory. I wonder if some remnant of the old install of anaconda is causing issues somewhere. Anyone have any thoughts?
(Same thing happens with conda deactivate
, for what its worth)
We've talked about this a lot in the past, but I think we need to come to a conclusion about how to proceed on this. The issue here is that having a 2-3 day tutorial is not enough time to cover both beginner and intermediate topics. One thought for discussion:
Thought: Beginner-level participation looks very different than intermediate- or advanced-level participation. At the intermediate and advanced levels, participants can be expected to participate in a hackathon-like environment. At the beginner level, participants do not have the tools (yet) to do anything like a hackathon. This suggests the following:
All told, this is a curriculum spanning 4-7 days.
Thought: Experience trying to accommodate all levels of experience in a single tutorial does not seem to work as effectively as we would like. Namely, participants who start as beginners rarely are able to participate at the intermediate or advanced level in the same tutorial. So, it seems to me that people need time to develop their knowledge and let the concepts "sink in." This might suggest the following:
This would allow for a 2-day beginner tutorial that might be followed by a week-long tutorial + hackathon about 2 weeks later.
CISL WIP
Staff Notes Daily
Communique
Tweet it
We need to announce the Spring 2020 Tutorial again.
We need to contact registered participants to get confirmation of their registration.
Content should include:
There is a directory called "old-contents" in the archived
directory. What is in here? Is there anything in there that we should keep?
Once everything has been retrieved from the old-contents
, we should delete the archived directory.
Currently, the ssh tunnel scripts in scripts/jlab
require activating a conda environment with jupyter
installed in it prior launching ./scripts/jlab/jlab-machine-name
script. We can improve the user experience by embedding this into the scripts themselves. One solution is to add an option for conda_env_name
to activate prior launching the jupyter server, and adding a conda activate $conda_env_name
step in the scripts themselves.
I have the perfect laptop to test this configure script with! It's brand new with nothing on it!
Currently, with no python3
installed, I get an error when trying to run the script.
Now that we have the Nikola site live, we can get rid of the old Jekyll site which lives in docs/
. However, we need to look through this directory to see if there is anything we should keep, such as the 'self-paced' guide that was previously developed.
Helping a user run setup/configure
on macos 10.14.6, he was getting an error from pip
that basically said xcrun
was providing invalid developer path
. This was fixed by running
$ xcode-select --install
I'm not sure how to check to see if the command line tools are available, and if it really is a requirement I'm not sure why it took until the morning of the tutorial for the issue to appear. (Possibly related to #41 though this was not a new install of the OS)
I ran conda init tcsh and it modified my .tcshrc file, but when I login or source .tcshrc, the mods result in an error:
Illegal variable name. The offending file is:
/gpfs/u/home/tomas/.tcshrc
and the I believe the illegal variable error is coming from this line:
__conda_setup="$('/glade/work/tomas/miniconda3/bin/conda' 'shell.tcsh' 'hook' 2> /dev/null)"
Currently, the OHC notebook is broken because the link to the data doesn't point to a file that exists. We need to fix this ASAP.
CC @jukent
Identify all of parts, create an outline
Branch/PR just for part 1
- pair programming review w @kmpaul
- get xdev team to read/add missing content/create PR
- repeat
Create Nikola md page for part 1
Reach out to Beta testers of part 1
Post and Announcement
Repeat for part 2
References:
I think we should change the tutorial site to a Nikola site. The main reason being that Nikola is python, so getting the environment set up on our laptops to edit and add content is easy. And we have experience with this with the Xdev blog.
We need a setup instructions document. It should probably be placed on the Nikola site:
/site/pages/spring2020/instructions.md
...or *.rst
or similar.
Many people may already have Python setup. Is the configure script robust in this context?
Now that the event is over, we should work to make the repo "stand-alone." We need to remove any event-specific information from the repo, and we need to make the repository a "self-guided" tutorial.
Can we add https://github.com/jwkvam/jupyterlab-vim to post_build_base
?
import xarray as xr
Did that work for you? If not, you do not have xarray installed in your current notebook environment. Check to make sure it reads "Python [conda env: analysis]" in the top right corner of your jupyter notebook screen. If it just says "Python" or something else, then click on the text to change the selection, so that your "analysis" environment is used.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.