Comments (10)
commented on September 2, 2024
This is Issue #26 of The Azimuth Backup Project. We have made a start on it, but the server seems not to hold up it's end, or throttles. We are using:
wget -N -c --dns-timeout=10 --connect-timeout=300 --read-timeout=120 --wait=5 --mirror -e robots=off --random-wait --page-requisites --retry-connrefused --prefer-family=IPv4 --tries=40 --timestamping=on --recursive --level=8 --no-remove-listing --follow-ftp -nv --mirror --append-output=daac-ornl-gov-get-data.log --no-check-certificate https://daac.ornl.gov/
And we get things like:
2017-01-20 15:27:42 URL:http://daac.ornl.gov/cgi-bin/show_pubs.pl?ds=1146 [37050] -> "daac.ornl.gov/cgi-bin/show_pubs.pl@ds=1146" [1]
Last-modified header missing -- time-stamps turned off.
2017-01-20 15:27:53 URL:http://daac.ornl.gov/cgi-bin/show_pubs.pl?ds=1080 [38162] -> "daac.ornl.gov/cgi-bin/show_pubs.pl@ds=1080" [1]
Last-modified header missing -- time-stamps turned off.
2017-01-20 15:28:01 URL:http://daac.ornl.gov/cgi-bin/show_pubs.pl?ds=1166 [35130] -> "daac.ornl.gov/cgi-bin/show_pubs.pl@ds=1166" [1]
Last-modified header missing -- time-stamps turned off.
2017-01-20 15:28:11 URL:http://daac.ornl.gov/cgi-bin/show_pubs.pl?ds=817 [35196] -> "daac.ornl.gov/cgi-bin/show_pubs.pl@ds=817" [1]
Last-modified header missing -- time-stamps turned off.
2017-01-20 15:28:24 URL:http://daac.ornl.gov/cgi-bin/dataset_lister_new.pl?p=34 [53907] -> "daac.ornl.gov/cgi-bin/dataset_lister_new.pl@p=34" [1]
WARNING: cannot verify urs.earthdata.nasa.gov's certificate, issued by `/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2014 Entrust, Inc. - for authorized use only/CN=Entrust Certification Authority - L1M':
Unable to locally verify the issuer's authority.
Last-modified header missing -- time-stamps turned off.
2017-01-20 15:28:34 URL:https://urs.earthdata.nasa.gov/oauth/authorize?client_id=QyeRbBJg8YuY_WBh-KBztA&response_type=code&redirect_uri=https%3A%2F%2Fdaac.ornl.gov%2Fdaacdata%2Fdoesntmater&state=aHR0cDovL2RhYWMub3JubC5nb3YvZGFhY2RhdGEvYWJvdmUv [8950] -> "urs.earthdata.nasa.gov/oauth/authorize@client_id=QyeRbBJg8YuY_WBh-KBztA&response_type=code&redirect_uri=https%3A%2F%2Fdaac.ornl.gov%2Fdaacdata%2Fdoesntmater&state=aHR0cDovL2RhYWMub3JubC5nb3YvZGFhY2RhdGEvYWJvdmUv" [1]
http://daac.ornl.gov/ABOVE/guides/%0A%09%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1307:
2017-01-20 15:29:47 ERROR 404: Not Found.
http://daac.ornl.gov/ABOVE/guides/%0A%09%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1362:
2017-01-20 15:30:28 ERROR 404: Not Found.
http://daac.ornl.gov/ABOVE/guides/%0A%09%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1367:
2017-01-20 15:30:38 ERROR 404: Not Found.
http://daac.ornl.gov/ABOVE/guides/%0A%09%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1324:
2017-01-20 15:31:10 ERROR 404: Not Found.
http://daac.ornl.gov/ABOVE/guides/%0A%09%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1355:
2017-01-20 15:31:32 ERROR 404: Not Found.
http://daac.ornl.gov/ABOVE/guides/%0A%09%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1353:
2017-01-20 15:31:44 ERROR 404: Not Found.
http://daac.ornl.gov/ABOVE/guides/%0A%09%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1354:
2017-01-20 15:32:05 ERROR 404: Not Found.
http://daac.ornl.gov/ABOVE/guides/%0A%09%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1356:
2017-01-20 15:32:34 ERROR 404: Not Found.
http://daac.ornl.gov/ABOVE/guides/%0A%09%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1360:
2017-01-20 15:32:49 ERROR 404: Not Found.
http://daac.ornl.gov/ABOVE/guides/%0A%09%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1333:
2017-01-20 15:33:13 ERROR 404: Not Found.
http://daac.ornl.gov/ABOVE/guides/%0A%09%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1361:
2017-01-20 15:33:27 ERROR 404: Not Found.
http://daac.ornl.gov/ABOVE/guides/%0A%09%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1323:
2017-01-20 15:33:40 ERROR 404: Not Found.
http://daac.ornl.gov/ABOVE/guides/%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1265:
2017-01-20 15:33:47 ERROR 404: Not Found.
http://daac.ornl.gov/ABOVE/guides/%0A%09%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20/cgi-bin/dsviewer.pl?ds_id=1265:
2017-01-20 15:34:05 ERROR 404: Not Found.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
Authorization failed.
A restart of the download seems to begin okay. I have attached a zipped copy of a log from this.
daac-ornl-older.log.gz
from datasets.
mejackreed
commented on September 2, 2024
Is there a size estimate here?
from datasets.
commented on September 2, 2024
Our best grab is 285 GB:
- 284,860,969,884 bytes
- 31441 files
- 2415 directories
but I know its incomplete. In the process of moving it and picking it up again, unless someone else can.
from datasets.
commented on September 2, 2024
I'm working on seeing if there is some way we can put a tarball of the dataset some place on a server of The Azimuth Project in a way it could be shared. This is definitely a partial, but wanted to help out if someone else wanted to pick up and continue.
from datasets.
mejackreed
commented on September 2, 2024
A have a wget job running, but I think bandwidth on their ftp may be limited. I'm struggling to get even close to that amount of data so far.
from datasets.
JeremiahCurtis
commented on September 2, 2024
working on the CARVE and CMS databases, but the latter is about 460 GB and is taking a long time
from datasets.
JeremiahCurtis
commented on September 2, 2024
If somebody with a faster connection can pick up CMS data, go for it....my connection to ORNL is abjectly poor
from datasets.
JeremiahCurtis
commented on September 2, 2024
Any idea where we stand on this? If certain EPA pages referring to climate change are being removed, I can't imagine ORNL data being safe much longer. Not trying to cry wolf, just saying.....
My CARVE and CMS downloads died on download manager and so I am reordering them from Earthdata
Also, are there any FTP directories for ORNL data?
from datasets.
commented on September 2, 2024
On Fri, Feb 3, 2017, at 14:55, JeremiahCurtis wrote:
Any idea where we stand on this? If certain EPA pages referring to
climate change are being removed, I can't imagine ORNL data being safe
much longer. Not trying to cry wolf, just saying.....
My CARVE and CMS downloads died on download manager and so I am
reordering them from Earthdata
Also, are there any FTP directories for ORNL data?
— You are receiving this because you commented. Reply to this email
directly, view it on GitHub[1], or mute the thread[2].
Links:
1. #42 (comment)
2. https://github.com/notifications/unsubscribe-auth/AD3HBw7LSYg0qV7iiqnOz16S05L5JXBEks5rY4ZFgaJpZM4LhVBk
from datasets.
JeremiahCurtis
commented on September 2, 2024
new to azimuth, so I apologize for what might be a stupid question; I'm assuming "resolved" means that at least one copy of all ORNL data, including CMS, CARVE, Daymet, etc. exists?
from datasets.
Related Issues (20)
- AMSU-B/MHS Brightness Temperature Climate Data Record (CDR) HOT 4
- EPA Envirofacts
- Large EOSDIS datasets
- E
- https://www.ncei.noaa.gov/data/avhrr-aerosol-optical-thickness/ HOT 2
- https://www.ncei.noaa.gov/data/avhrr-land-normalized-difference-vegetation-index/ HOT 1
- https://www.ncei.noaa.gov/data/avhrr-reflectance-cloud-properties-patmos-extended/
- NOAA GIBBS HOT 30
- EPA Publication List: NSCEP HOT 13
- EPA Transportation and Air Quality (TAQ) HOT 6
- Pacific Marine Environmental Laboratory ftp site HOT 3
- EPA Air Quality Data archives HOT 13
- LODES7 Census database of jobs with home and work addresses HOT 1
- ORNL Fluxnet Data HOT 2
- EPA Environmental Dataset Gateway HOT 1
- IMO Climates & Weather Archives HOT 7
- Detailed GHG data (EPA Envirofacts)
- NGDC Water Column Sonar Data HOT 3
- National Land Cover Dataset (NLCD) HOT 1
- Help I need to know about this Climate thing
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from datasets.