Comments (4)
tiborsimko
commented on September 27, 2024
1
Example: I just launch two commands quickly after each other, and I saw:
$ wget http://localhost/eos/opendata/atlas/OutreachDatasets/2020-01-22/GamGam.zip
--2020-11-19 11:56:39-- http://localhost/eos/opendata/atlas/OutreachDatasets/2020-01-22/GamGam.zip
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2020-11-19 11:56:40 ERROR 403: Forbidden.
$ $ wget http://localhost/eos/opendata/atlas/OutreachDatasets/2020-01-22/GamGam.zip
--2020-11-19 11:56:44-- http://localhost/eos/opendata/atlas/OutreachDatasets/2020-01-22/GamGam.zip
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1594567234 (1.5G) [application/zip]
Saving to: ‘GamGam.zip’The 403 is a rather frequent error by the underlying reverse proxy that we also worked around in #91 via size and checksum.
After we introduce exit status based retry, we could probably remove that check-error-page-size-and-checksum hack.
from cernopendata-client.
tiborsimko
commented on September 27, 2024
1
Example: 206 Partial Content situation occurred several times while downloading one big file via wget:
$ wget http://opendata.cern.ch/eos/opendata/cms/MonteCarlo2011/Summer11LegDR/BuToKstarMuMuV2_EtaPtFilter_7TeV-pythia6-evtgen/AODSIM/PU_S13_START53_LV6-v1/00000/000043B3-0E0B-E411-B2FE-0026189438E9.root
--2020-06-24 14:53:19-- http://opendata.cern.ch/eos/opendata/cms/MonteCarlo2011/Summer11LegDR/BuToKstarMuMuV2_EtaPtFilter_7TeV-pythia6-evtgen/AODSIM/PU_S13_START53_LV6-v1/00000/000043B3-0E0B-E411-B2FE-0026189438E9.root
Resolving opendata.cern.ch (opendata.cern.ch)... 188.184.93.89, 188.185.82.144, 188.184.28.138
Connecting to opendata.cern.ch (opendata.cern.ch)|188.184.93.89|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4011567937 (3.7G) [application/octet-stream]
Saving to: ‘000043B3-0E0B-E411-B2FE-0026189438E9.root.1’
000043B3-0E0B-E411-B2FE-0026189438E9.root.1 26%[=========================================> ] 1.01G 7.82MB/s in 87s
2020-06-24 14:54:46 (11.9 MB/s) - Connection closed at byte 1081769706. Retrying.
--2020-06-24 14:54:47-- (try: 2) http://opendata.cern.ch/eos/opendata/cms/MonteCarlo2011/Summer11LegDR/BuToKstarMuMuV2_EtaPtFilter_7TeV-pythia6-evtgen/AODSIM/PU_S13_START53_LV6-v1/00000/000043B3-0E0B-E411-B2FE-0026189438E9.root
Connecting to opendata.cern.ch (opendata.cern.ch)|188.184.93.89|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 4011567937 (3.7G), 2929798231 (2.7G) remaining [application/octet-stream]
Saving to: ‘000043B3-0E0B-E411-B2FE-0026189438E9.root.1’
000043B3-0E0B-E411-B2FE-0026189438E9.root.1 29%[++++++++++++++++++++++++++++++++++++++++++===> ] 1.09G 11.5MB/s in 6.7s
2020-06-24 14:54:54 (12.6 MB/s) - Connection closed at byte 1169634135. Retrying.
--2020-06-24 14:54:56-- (try: 3) http://opendata.cern.ch/eos/opendata/cms/MonteCarlo2011/Summer11LegDR/BuToKstarMuMuV2_EtaPtFilter_7TeV-pythia6-evtgen/AODSIM/PU_S13_START53_LV6-v1/00000/000043B3-0E0B-E411-B2FE-0026189438E9.root
Connecting to opendata.cern.ch (opendata.cern.ch)|188.184.93.89|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 4011567937 (3.7G), 2841933802 (2.6G) remaining [application/octet-stream]
Saving to: ‘000043B3-0E0B-E411-B2FE-0026189438E9.root.1’
000043B3-0E0B-E411-B2FE-0026189438E9.root.1 56%[++++++++++++++++++++++++++++++++++++++++++++++=========================================> ] 2.09G 15.1MB/s in 88s
2020-06-24 14:56:24 (11.7 MB/s) - Connection closed at byte 2248454684. Retrying.
--2020-06-24 14:56:27-- (try: 4) http://opendata.cern.ch/eos/opendata/cms/MonteCarlo2011/Summer11LegDR/BuToKstarMuMuV2_EtaPtFilter_7TeV-pythia6-evtgen/AODSIM/PU_S13_START53_LV6-v1/00000/000043B3-0E0B-E411-B2FE-0026189438E9.root
Connecting to opendata.cern.ch (opendata.cern.ch)|188.184.93.89|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 4011567937 (3.7G), 1763113253 (1.6G) remaining [application/octet-stream]
Saving to: ‘000043B3-0E0B-E411-B2FE-0026189438E9.root.1’
000043B3-0E0B-E411-B2FE-0026189438E9.root.1 82%[++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++==========================================> ] 3.10G 13.1MB/s in 82s
2020-06-24 14:57:49 (12.6 MB/s) - Connection closed at byte 3328930017. Retrying.
--2020-06-24 14:57:53-- (try: 5) http://opendata.cern.ch/eos/opendata/cms/MonteCarlo2011/Summer11LegDR/BuToKstarMuMuV2_EtaPtFilter_7TeV-pythia6-evtgen/AODSIM/PU_S13_START53_LV6-v1/00000/000043B3-0E0B-E411-B2FE-0026189438E9.root
Connecting to opendata.cern.ch (opendata.cern.ch)|188.184.93.89|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 4011567937 (3.7G), 682637920 (651M) remaining [application/octet-stream]
Saving to: ‘000043B3-0E0B-E411-B2FE-0026189438E9.root.1’
000043B3-0E0B-E411-B2FE-0026189438E9.root.1 85%[+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++===> ] 3.20G 6.88MB/s in 10s
2020-06-24 14:58:04 (9.83 MB/s) - Connection closed at byte 3433695865. Retrying.
--2020-06-24 14:58:09-- (try: 6) http://opendata.cern.ch/eos/opendata/cms/MonteCarlo2011/Summer11LegDR/BuToKstarMuMuV2_EtaPtFilter_7TeV-pythia6-evtgen/AODSIM/PU_S13_START53_LV6-v1/00000/000043B3-0E0B-E411-B2FE-0026189438E9.root
Connecting to opendata.cern.ch (opendata.cern.ch)|188.184.93.89|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 4011567937 (3.7G), 577872072 (551M) remaining [application/octet-stream]
Saving to: ‘000043B3-0E0B-E411-B2FE-0026189438E9.root.1’
000043B3-0E0B-E411-B2FE-0026189438E9.root.1 100%[+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++======================>] 3.74G 14.6MB/s in 36s
2020-06-24 14:58:44 (15.5 MB/s) - ‘000043B3-0E0B-E411-B2FE-0026189438E9.root.1’ saved [4011567937/4011567937]from cernopendata-client.
tiborsimko
commented on September 27, 2024
Example of "native" retry tools:
Haven't looked at what pycurl/xrootd may have built-in. We could always improve our own for _retry in range loop not to check only fro error = ( DOWNLOAD_ERROR_PAGE[..., but turn it into a more complete function catching other exceptional situations.
from cernopendata-client.
ParthS007
commented on September 27, 2024
Musing after researching on the Issue:
There are two things we need to address
- Error handling
- Retry functionality
Every library we use(pycurl/requests/xrootdpyfs) has their own way of handling these two things.
-
pycurl- Error handling:
- SO Answer(Basic error handling)
- Using
pycurl.errorandgetinfofor different HTTP Codes
- Retry:
- Add a retry loop on
WRITEDATA - Using pycurl.TIMEOUT and/or pycurl.CONNECTTIMEOUT or something similar(haven't tested yet)
- Add a retry loop on
- Error handling:
-
requests -
xrootdpyfs- Error handling
- Add general try exception handling
- Display messages according to error
- Retry
- Add retry loop on
xrootdpyfs - Use underlying
xrootdpython client methods directly.
- Add retry loop on
- Error handling
Please let me know if I can break those steps as well!
from cernopendata-client.
Related Issues (20)
- walker: enrich output with parent directories
- Consistency between top-level CLI notion behind "protocol" and low-level Searcher notion behind "protocol"
- cernopendata-client get-metadata --recid 550 --output-value title
- searcher: JSON Decode error for `--verbose`
- CI: Test on multiple enviornments
- file locations: distinguish between EOSPUBLIC and OPENDATA URIs
- download-files: expose HTTP downloader choice (requests vs pycurl) HOT 1
- download-files: success for 5000, failure for 12341
- installation: xrootd troubles HOT 1
- downloader: CLI breaks when record has no files
- Raise exceptions when using the methods outside the CLI
- Download with docker - where do the files go? HOT 2
- Parallelize Downloads
- cli: add support for multiple filter arguments in the get-metadata command
- Verbose mode in get-file-locations HOT 1
- downloader: partial downloads are marked as success:
- cernopendata-client get-file-locations --recid 282 --protocol xrootd --verbose HOT 1
- searcher: separate out get_list_directory() and friends
- downloader: resume interrupted downloads HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cernopendata-client.