Code Monkey home page Code Monkey logo

Comments (16)

andersy005 avatar andersy005 commented on June 19, 2024 1

FYI:

figshare offers 20GB of free storage to account owners. I am thinking of hosting sample data (for users without an account on Cheyenne) under my figshare account if there are no objections

from ncar-python-tutorial.

jukent avatar jukent commented on June 19, 2024

This is the parent directory of the data -- /glade/collections/cmip/CMIP6/CMIP/NASA-GISS/GISS-E2-1-G/historical/r1i1p1f1/Omon/thetao/gn/v20180827

I will take some time this afternoon to dig through the subdirectories to the data used.

**Full path

/glade/collections/cmip/CMIP6/CMIP/NASA-GISS/GISS-E2-1-G/historical/r1i1p1f1/Omon/thetao/gn/v20180827/thetao/thetao_Omon_GISS-E2-1-G_historical_r1i1p1f1_gn_185001-187012.nc

from ncar-python-tutorial.

andersy005 avatar andersy005 commented on June 19, 2024

@jukent, do we need all the 20 years worth of data or can we get a subset from this file?

from ncar-python-tutorial.

jukent avatar jukent commented on June 19, 2024

We can subset it, but we're not nearing that 20GB limit - so is it something we should do anyway?

from ncar-python-tutorial.

kmpaul avatar kmpaul commented on June 19, 2024

What download speeds can we expect from figshare?

from ncar-python-tutorial.

matt-long avatar matt-long commented on June 19, 2024

why not use the CGD ftp location we had setup for exactly this thing?

from ncar-python-tutorial.

kmpaul avatar kmpaul commented on June 19, 2024

Does the CGD ftp server allow anonymous download? Who has permissions to upload?

from ncar-python-tutorial.

andersy005 avatar andersy005 commented on June 19, 2024

why not use the CGD ftp location we had setup for exactly this thing?

@matt-long, I've not looked into this yet. I will give it a try before exploring the figshare option

Does the CGD ftp server allow anonymous download? Who has permissions to upload?

@kmpaul, this needs further exploration

I am going to looking into the CGD's ftp server option this afternoon, and I will let you know how this goes

from ncar-python-tutorial.

andersy005 avatar andersy005 commented on June 19, 2024

@kmpaul

Does the CGD ftp server allow anonymous download?

Yes. As a smoke test you can try downloading this script:

wget ftp://ftp.cgd.ucar.edu/archive/aletheia-data/test.sh

Who has permissions to upload?

As far as I know, Matt and I are among users with upload permissions.

from ncar-python-tutorial.

andersy005 avatar andersy005 commented on June 19, 2024

@matt-long,

I just figured the FTP details out, and I am in favor of using the ftp server for sample data instead of figshare.

from ncar-python-tutorial.

matt-long avatar matt-long commented on June 19, 2024

Good. Regarding @kmpaul question,

Who has permissions to upload?

Ultimately, I think we want to setup some sort of version control and have a means of getting input from other groups, but include a gate-keeper. I don't know what the best solutions in this regard are.

from ncar-python-tutorial.

kmpaul avatar kmpaul commented on June 19, 2024

I think a small ftp service with versioned files might be something Jeff DLB’s division should provide. They do this on a regular basis for things like the RDA.

from ncar-python-tutorial.

jukent avatar jukent commented on June 19, 2024

I limited the data to 5 years, it is now < 600 MB,
The path on Cheyenne is:
/glade/u/home/jkent/tutorial_data/thetao_Omon_historical_GISS-E2-1-G_r1i1p1f1_gn_185001-185512.nc

from ncar-python-tutorial.

andersy005 avatar andersy005 commented on June 19, 2024

This is fixed in #61

*** Found an existing Conda installation in: /opt/conda/bin/conda
*** (23:16:18) Step 1/6: Skipping Conda installation
*** (23:16:18) Step 2/6: Updating `base` environment
*** (23:17:48) Step 3/6: Checking for environment name conflict
*** (23:17:48) Step 4/6: Creating `python-tutorial` environment (this can take several minutes)
*** (23:21:00) Step 5/6: Running post build script for `base` environment
*** (23:22:29) Step 6/6: Running post build script for `python-tutorial` environment (this can take several minutes)
*** (23:25:45) Setup completed successfully.
==> For changes to take effect, close and re-open your current shell. <==
Currently downloading tutorial data

Progress: |--------------------------------------------------| 0.0% /root/project/data/NOAA_NCDC_ERSST_v3b_SST.nc  

Progress: |██████--------------------------------------------| 12.5% /root/project/data/air_temperature.nc  

Progress: |████████████--------------------------------------| 25.0% /root/project/data/co2.nc  

Progress: |███████████████████-------------------------------| 37.5% /root/project/data/moc.nc  

Progress: |█████████████████████████-------------------------| 50.0% /root/project/data/rasm.nc  

Progress: |███████████████████████████████-------------------| 62.5% /root/project/data/sst_indices.csv  

Progress: |██████████████████████████████████████------------| 75.0% /root/project/data/thetao_Omon_historical_GISS-E2-1-G_r1i1p1f1_gn_185001-185512.nc  

Progress: |████████████████████████████████████████████------| 87.5% /root/project/data/woa2013v2-O2-thermocline-ann.nc  

Progress: |██████████████████████████████████████████████████| 100.0%

from ncar-python-tutorial.

andersy005 avatar andersy005 commented on June 19, 2024

Cheyenne/Casper

$ ./setup/configure --download
Creating symlink to existing/local tutorial data directory: /glade/work/abanihi/aletheia-data/tutorial-data
ln -sf /glade/work/abanihi/aletheia-data/tutorial-data/* /glade/work/abanihi/devel/ncar/NCAR-pangeo-tutorial/data/

$ ls -lh data
total 8.0K
lrwxrwxrwx 1 abanihi ncar  66 Sep 12 18:25 air_temperature.nc -> /glade/work/abanihi/aletheia-data/tutorial-data/air_temperature.nc
lrwxrwxrwx 1 abanihi ncar  54 Sep 12 18:25 co2.nc -> /glade/work/abanihi/aletheia-data/tutorial-data/co2.nc
lrwxrwxrwx 1 abanihi ncar  54 Sep 12 18:25 moc.nc -> /glade/work/abanihi/aletheia-data/tutorial-data/moc.nc
lrwxrwxrwx 1 abanihi ncar  74 Sep 12 18:25 NOAA_NCDC_ERSST_v3b_SST.nc -> /glade/work/abanihi/aletheia-data/tutorial-data/NOAA_NCDC_ERSST_v3b_SST.nc
lrwxrwxrwx 1 abanihi ncar  55 Sep 12 18:25 rasm.nc -> /glade/work/abanihi/aletheia-data/tutorial-data/rasm.nc
-rw-r--r-- 1 abanihi ncar   0 Sep  5 12:11 README.md
lrwxrwxrwx 1 abanihi ncar  63 Sep 12 18:25 sst_indices.csv -> /glade/work/abanihi/aletheia-data/tutorial-data/sst_indices.csv
lrwxrwxrwx 1 abanihi ncar 111 Sep 12 18:25 thetao_Omon_historical_GISS-E2-1-G_r1i1p1f1_gn_185001-185512.nc -> /glade/work/abanihi/aletheia-data/tutorial-data/thetao_Omon_historical_GISS-E2-1-G_r1i1p1f1_gn_185001-185512.nc
lrwxrwxrwx 1 abanihi ncar  79 Sep 12 18:25 woa2013v2-O2-thermocline-ann.nc -> /glade/work/abanihi/aletheia-data/tutorial-data/woa2013v2-O2-thermocline-ann.nc

Personal Laptop

$ ./setup/configure --download
Currently downloading tutorial data
Progress: |--------------------------------------------------| 0.0% /Users/abanihi/devel/ncar/NCAR-pangeo-tutorial/data/NOAA_NCDC_ERSST_v3b_SST.nc  
Progress: |██████--------------------------------------------| 12.5% /Users/abanihi/devel/ncar/NCAR-pangeo-tutorial/data/air_temperature.nc  
Progress: |████████████--------------------------------------| 25.0% /Users/abanihi/devel/ncar/NCAR-pangeo-tutorial/data/co2.nc  
Progress: |███████████████████-------------------------------| 37.5% /Users/abanihi/devel/ncar/NCAR-pangeo-tutorial/data/moc.nc  
Progress: |█████████████████████████-------------------------| 50.0% /Users/abanihi/devel/ncar/NCAR-pangeo-tutorial/data/rasm.nc  
Progress: |███████████████████████████████-------------------| 62.5% /Users/abanihi/devel/ncar/NCAR-pangeo-tutorial/data/sst_indices.csv  
Progress: |██████████████████████████████████████------------| 75.0% /Users/abanihi/devel/ncar/NCAR-pangeo-tutorial/data/thetao_Omon_historical_GISS-E2-1-G_r1i1p1f1_gn_185001-185512.nc  
Progress: |████████████████████████████████████████████------| 87.5% /Users/abanihi/devel/ncar/NCAR-pangeo-tutorial/data/woa2013v2-O2-thermocline-ann.nc  
Progress: |██████████████████████████████████████████████████| 100.0% 
$ ls -lh data
total 1308688
-rw-r--r--  1 abanihi  CIT\Domain Users    42M Sep 12 18:27 NOAA_NCDC_ERSST_v3b_SST.nc
-rw-r--r--  1 abanihi  CIT\Domain Users     0B Aug 28 18:22 README.md
-rw-r--r--  1 abanihi  CIT\Domain Users   7.4M Sep 12 18:27 air_temperature.nc
-rw-r--r--  1 abanihi  CIT\Domain Users   487K Sep 12 18:27 co2.nc
-rw-r--r--  1 abanihi  CIT\Domain Users   108K Sep 12 18:27 moc.nc
-rw-r--r--  1 abanihi  CIT\Domain Users    16M Sep 12 18:27 rasm.nc
-rw-r--r--  1 abanihi  CIT\Domain Users    25K Sep 12 18:27 sst_indices.csv
-rw-r--r--  1 abanihi  CIT\Domain Users   570M Sep 12 18:27 thetao_Omon_historical_GISS-E2-1-G_r1i1p1f1_gn_185001-185512.nc
-rw-r--r--  1 abanihi  CIT\Domain Users   268K Sep 12 18:27 woa2013v2-O2-thermocline-ann.nc

Hobart

$ ./setup/configure --download
Creating symlink to existing/local tutorial data directory: /ftp/archive/aletheia-data/tutorial-data
ln -sf /ftp/archive/aletheia-data/tutorial-data/* /scratch/cluster/abanihi/NCAR-pangeo-tutorial/data/
$ ls -lh data
total 12K
lrwxrwxrwx 1 abanihi cgdoce  59 Sep 12 18:28 air_temperature.nc -> /ftp/archive/aletheia-data/tutorial-data/air_temperature.nc
lrwxrwxrwx 1 abanihi cgdoce  47 Sep 12 18:28 co2.nc -> /ftp/archive/aletheia-data/tutorial-data/co2.nc
lrwxrwxrwx 1 abanihi cgdoce  47 Sep 12 18:28 moc.nc -> /ftp/archive/aletheia-data/tutorial-data/moc.nc
lrwxrwxrwx 1 abanihi cgdoce  67 Sep 12 18:28 NOAA_NCDC_ERSST_v3b_SST.nc -> /ftp/archive/aletheia-data/tutorial-data/NOAA_NCDC_ERSST_v3b_SST.nc
lrwxrwxrwx 1 abanihi cgdoce  48 Sep 12 18:28 rasm.nc -> /ftp/archive/aletheia-data/tutorial-data/rasm.nc
-rw-r--r-- 1 abanihi cgdoce   0 Sep 12 16:31 README.md
lrwxrwxrwx 1 abanihi cgdoce  56 Sep 12 18:28 sst_indices.csv -> /ftp/archive/aletheia-data/tutorial-data/sst_indices.csv
lrwxrwxrwx 1 abanihi cgdoce 104 Sep 12 18:28 thetao_Omon_historical_GISS-E2-1-G_r1i1p1f1_gn_185001-185512.nc -> /ftp/archive/aletheia-data/tutorial-data/thetao_Omon_historical_GISS-E2-1-G_r1i1p1f1_gn_185001-185512.nc
lrwxrwxrwx 1 abanihi cgdoce  72 Sep 12 18:28 woa2013v2-O2-thermocline-ann.nc -> /ftp/archive/aletheia-data/tutorial-data/woa2013v2-O2-thermocline-ann.nc

from ncar-python-tutorial.

andersy005 avatar andersy005 commented on June 19, 2024

@kmpaul, @matt-long

Let me know whether this looks good and/or anything is missing.

from ncar-python-tutorial.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.