snowex-hackweek / website-2021 Goto Github PK

Just noting that some notebooks are not currently rendered with outputs on the website. These were merged pretty hastily and either never fully executed (error in a particular cell), or likely use a package that is missing from the default environment. Depending on how easy the fix is we can either leave them w/o outputs or try to fix.

WARNING: Execution Failed: /home/jovyan/book/tutorials/nsidc-access/nsidc-data-access.ipynb
WARNING: Execution Failed: /home/jovyan/book/tutorials/lidar/ASO_data_tutorial.ipynb
WARNING: Couldn't find cache key for notebook file book/tutorials/lidar/ASO_data_tutorial.ipynb. Outputs will not be inserted.
  Last execution failed with traceback saved in /home/jovyan/book/_build/html/reports/ASO_data_tutorial.log
WARNING: Couldn't find cache key for notebook file book/tutorials/nsidc-access/nsidc-data-access.ipynb. Outputs will not be inserted.
  Last execution failed with traceback saved in /home/jovyan/book/_build/html/reports/nsidc-data-access.log
/home/jovyan/book/tutorials/nsidc-access/nsidc-data-access.ipynb:10011: WARNING: Non-consecutive header level increase; 1 to 3 [myst.header]
/home/jovyan/book/tutorials/nsidc-access/nsidc-data-access.ipynb:330005: WARNING: Non-consecutive header level increase; 2 to 4 [myst.header]

currently markdown links on several pages are not rendered correctly: https://snowex-hackweek.github.io/website/preliminary/jupyterhub.html#how-do-i-access-the-shared-cloud-environment shows up as Just click on [https://snowex.hackweek.io](https://snowex.hackweek.io)!

This is because of incorrectly using myst_substitutions with URLs as described here
https://jupyterbook.org/content/content-blocks.html?highlight=myst%20substitutions#using-substitutions-in-links

There are several ways to accomplish it, but for short URLs just using it verbatim is probably fine

Figure directives are great for generating labels and having things look nice on the website:

```{figure} ../../img/sentinel1_radar_vision.jpg
---
height: 400px
name: sentinel1
---
Artist view of Sentinel-1. [Source](https://www.esa.int/ESA_Multimedia/Images/2014/01/Sentinel-1_radar_vision)

https://snowex-hackweek.github.io/website/tutorials/sar/sentinel1.html

But when actually running the notebook in a jupyterlab environment the figure is not rendered and just restates the directive:

---
height: 400px
name: sentinel1
---
Artist view of Sentinel-1. [Source](https://www.esa.int/ESA_Multimedia/Images/2014/01/Sentinel-1_radar_vision)

Seems the best solution is to use HTML tags as described here https://jupyterbook.org/content/figures.html#raw-html-images cc @mikedurand

If Tutorial notebooks on the website read from S3:// resources that require authentication, we need to inject AWS CLI credentials into the continuous integration system. Otherwise when tutorial notebooks are executed we get authentication errors and can't retrieve data for subsequent computations and output

People may be accustomed to formatting latex-style math equations in jupyter notebook markdown cells by copying and pasting equations from manuscripts

\begin{equation}
z = \frac{vt}{2} \quad
\end{equation}

This doesn't render correctly on the website by default, you have to enable the amsmath myst extension https://jupyterbook.org/content/math.html?highlight=equations#latex-style-math

Alternatively, using the following syntax works for simple equations

$$
z = \frac{vt}{2} \quad
$$ (equation_label)

We want to be able for GitHub Actions to run tutorial notebooks top-to-bottom to ensure that they run. Unfortunately, API-based access to NASA data (via curl and wget) is setup for using a ~/.netrc file rather than reading from environment variables or being able to specify a temporary token. This means we'll have to use a "machine user" with a NASA URS https://urs.earthdata.nasa.gov login and somehow inject those credentials into the continuous integration system.

tutorial documenting how to do this is here:
https://guides.github.com/activities/citable-code/

we want to make sure a release is made at the end of the hackweek so that all content contributors are listed

the preliminary work section of the website currently has a lot of material specific to icesat2 hackweek. just making an issue to keep track and will follow up with a PR to make things either generic or specific to snowex hackweek. https://snowex-hackweek.github.io/website/preliminary/index.html

aws commands in a jupyter notebook cell like this !aws s3 sync s3://snowex-data/tutorial-data/camera-trap/ /tmp/camera-trap currently work on jupyterhub but not when we build the website via CI. Instead of downloading data we get strange output: <botocore.awsrequest.AWSRequest object at 0x7f06fd62ae50>

#72 attempted a fix, but maybe we missed -e AWS_REGION ?

I'd like to demo sliderule-python in my tutorial. Can it be installed on the hub?

UnboundLocalError Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_3204\3970605443.py in <cell line: 2>()
1 # call the tiff convert function, and it will print the information it extracted from the .ann file
----> 2 uavsar_tiff_convert(data_folder)

~\AppData\Local\Temp\ipykernel_3204\4031231228.py in uavsar_tiff_convert(folder)
106 sources.write(re.sub(line[8:13], Lines, line))
107 elif "samples" in line:
--> 108 sources.write(re.sub(line[10:15], Samples, line))
109 elif "LAT" in line:
110 sources.write(re.sub(line[12:15], Latitude, line))

UnboundLocalError: local variable 'Samples' referenced before assignment

project pythia is currently working on a jupyter book that uses read-the-docs to 'preview' website changes before they get merged
ProjectPythia/projectpythia.github.io#17

our CI system is currently a little different. we run the build through docker so that we guarantee to use the exact same environment and package versions that are used on the hackweek jupyterhub.

i think we could use netlify to do website previews https://jupyterbook.org/publish/netlify.html

described in these docs: https://jupyterbook.org/advanced/html.html#advanced-html-outputs

During the hackweek it was very helpful to have tutorials organized by day, but now that the week is done people visiting the website either have to drill down to get to the right tutorial. https://snowex-hackweek.github.io/website/tutorials/index.html

An alternative to this organization is to create flashier landing page with an image and keywords for each tutorial, like we do for our team page: https://snowex-hackweek.github.io/website/team.html

Or a table with links like:

cc @jomey

We currently verify all links work with a GitHub Action workflow
https://github.com/snowex-hackweek/website/blob/main/.github/workflows/qaqc.yaml#L43-L57

Often there are failures having to do with checking certificates, such as (tutorials/geospatial/SNOTEL_query: line 20008) broken https://www.wcc.nrcs.usda.gov/snow/ - HTTPSConnectionPool(host='www.wcc.nrcs.usda.gov', port=443): Max retries exceeded with url: /snow/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1125)')))

Despite this error, the code or link does actually work.

@jomey pointed out that sphinx has the option to disable certificate verification here https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-tls_verify , so this could be set in the jupyterbook configuration file https://jupyterbook.org/customize/config.html

For some reason, although only a few sentences of text only was changed on the last pull request, most of the plots on the UAVSAR tutorial are no longer rendering.

A good point raised by @dshean:

“how to cite,” but would be good to add. It’s a little more challenging with the tutorials as a subdirectory for an event-focused web page vs. a permanent archive, as the citation would be for the entire website repo.

Just attended an excellent presentation by @jules32 and @erinmr, which highlighted running notebooks hosted externally (that way it would be easy for each tutorial to have its own DOI) for this awesome NASA cookbook https://github.com/NASA-Openscapes/earthdata-cloud-cookbook . Wanted to note that alternative here for future events.

https://github.com/snowex-hackweek/website/blob/main/book/norms/CoC.md#expectations

links to [email protected] and the comment box is from IS2 hackweek. Should update these to be generic of have SnowEx 2021 titles.

Don created [email protected] for our admin 'bot' user. Should we also have [email protected] ? cc @aaarendt @lsetiawan

@dshean asked about the possibility of changing the website URL

We didn't put much thought into the website URL, but it's convenient to use GitHub pages hosting, which means the URL is fixed to the organization and repo name:

REPO -> WEBSITE
https://github.com/snowex-hackweek/website --> https://snowex-hackweek.github.io/website

A simpler option is maybe to map a custom domain name, so I think now that https://snowexdata.org exists, @micahjohnson150 or @hpmarshall could link the current website to that URL ? which would be nice because then no changes need to happen here
to #15 (comment)

That said, it's pretty straightforward to change website to whatever, such as jupyterbook, but that would require updating a number of files and causing some confusion for participants.

For future events we can map the organizational page to the jupyterbook, but we thought it might be nice to someday have a flashy landing page that leads to the jupyterbook... https://snowex-hackweek.github.io