igs / portal_client Goto Github PK

Hi,
I am executing the portal_client tool (with my iHMP username and password) to download the files using a manifest file as follows:
portal_client -m hmp_cart_171723e34.tsv --endpoint-priority FASP --user myusername

However, I get an error message as follows:

Skipping file ID 216b55a2f235085126ee6210d619d9fa as none of the URLs ['FASP'] succeeded.
Skipping file ID 1419f08f554e0c93f3b62fe90c092ab8 as none of the URLs ['FASP'] succeeded.
Skipping file ID 1419f08f554e0c93f3b62fe90c092760 as none of the URLs ['FASP'] succeeded.

Not all files (total of 3) were downloaded successfully. Number of failures:
0 -- no valid URL in the manifest file
3 -- URL is present in manifest, but not accessible at the location specified
0 -- MD5 checksum failed for file (file is corrupted or the wrong MD5 is associated)

The content of the manifest file is below:
file_id md5 size urls sample_id
216b55a2f235085126ee6210d619d9fa 1e82180a7680651167619cf407e889d3 686599 fasp://aspera2.ihmpdcc.org/ibd/transcriptome/host/analysis/CSM5FZ1I.tsv 7cfd74d6803ea319683df75644d2ba27
1419f08f554e0c93f3b62fe90c092ab8 23eb8207442bd452f0d77ecc0fdba7f9 24137 fasp://aspera2.ihmpdcc.org/ibd/genome/microbiome/wgs/analysis/hmscp/CSM5MCXD_taxonomic_profile.biom 1419f08f554e0c93f3b62fe90c08f0f2
1419f08f554e0c93f3b62fe90c092760 c761814a69f7a5cec26859c945e6ccde 141691 fasp://aspera2.ihmpdcc.org/ibd/genome/microbiome/wgs/analysis/hmmrc/CSM5MCXD_pathabundance_relab.biom 1419f08f554e0c93f3b62fe90c08f0f2

Can you please check if my command is correct? I also tried using ascp directly but it didn't give me any output either.

Thanks,
Ishwor

When I run the following command: docker run -ti --rm portal_client portal_client -m example_manifest.tsv -d ./ I get the following error:

Traceback (most recent call last): File "/usr/local/bin/portal_client", line 9, in <module> portal_client.main() File "/usr/local/lib/python3.6/dist-packages/portal_client.py", line 191, in main manifest = file_to_manifest(args.manifest) File "/usr/local/lib/python3.6/dist-packages/convert_to_manifest.py", line 22, in file_to_manifest with open(file) as tsv: FileNotFoundError: [Errno 2] No such file or directory: 'example_manifest.tsv'

I am using the portal_client function with as input a manifest file. However, for each URL in the manifest file it gives the following error:

Not all files (total of 5) were downloaded successfully. Number of failures:
5 -- no valid URL in the manifest file

I tried different manifest files and I get the same error.
Could you please help?

The manifest has valid URLs but I get this message during download with portal_client. If I go to the HMP cart and click on the same URLs for any of the files in the manifest, the data downloads without a problem. Here is the manifest file I am using as downloaded from the cart:

hmp_manifest_2cd2905a2.txt

Hello Victor,

I am using the portal_client with a manifest file. This is the command that I ran:

portal_client -m ibdmbd_16s_3ffb7fc0cc.tsv --endpoint-priority FASP,HTTP --user hec1324

I realized that in the manifest file, every url that starts with fasp://aspera.... works and completed download but urls that start with fas://apsera2 do not work. At the end, it says "URL is present in manifest, but not accessible at the location specified."

I have ran debug process with the command: portal_client -m ibdmbd_16s_3ffb7fc0cc.tsv --endpoint-priority FASP,HTTP --user hec1324 --debug and gave me an error:

019-07-11 11:21:55,575 - aspera - DEBUG - In get_ascp_env.
2019-07-11 11:21:55,575 - aspera - INFO - Setting ASPERA_SCP_PASS environment variable.
2019-07-11 11:21:55,577 - aspera - INFO - Beginning transfer.
2019-07-11 11:21:59,159 - aspera - INFO - Invocation of ascp complete. Return code: 1.
2019-07-11 11:21:59,159 - aspera - ERROR - Aspera authentication failure.
2019-07-11 11:21:59,160 - manifest_processor.ManifestProcessor - ERROR - Aspera transfer failed.
2019-07-11 11:21:59,160 - manifest_processor.ManifestProcessor - DEBUG - Returning error
Skipping file ID d39c1941c8f6e8b0f6ead5d7f75419c3 as none of the URLs ['FASP'] succeeded.

In this case, what should I do?
Thank you so much for your help.

I have the same question. I'm trying to download files from the HMP Project Data Portal. I requested a password from hmpdacc.org but when I try to transfer files, I get the error that none of the urls succeeded. Here is my command-line submission and a section of the output when running in debug mode:

portal_client --manifest ~/path-to-file/hmp_cart_4364ccfe46.tsv --destination ~/path-to-file/data --endpoint-priority FASP,HTTP --user jacknicoludis --debug

2019-09-11 12:53:17,267 - aspera - DEBUG - Command: ascp -T -v -l 300M [email protected]:/t2d/genome/microbiome/16s/analysis/hmqcp/HMP2_J45271_1_ST_T0_B0_0120_ZOZOW1T-6027-1_APATM.biom /Users/jacknicoludis/Dropbox/insight/drugmetabolism/HMP-T2D-abundances/data/HMP2_J45271_1_ST_T0_B0_0120_ZOZOW1T-6027-1_APATM.biom.partial
2019-09-11 12:53:17,267 - aspera - DEBUG - In get_ascp_env.
2019-09-11 12:53:17,268 - aspera - INFO - Setting ASPERA_SCP_PASS environment variable.
2019-09-11 12:53:17,273 - aspera - INFO - Beginning transfer.
2019-09-11 12:53:20,621 - aspera - INFO - Invocation of ascp complete. Return code: 1.
2019-09-11 12:53:20,622 - aspera - ERROR - Aspera authentication failure.
2019-09-11 12:53:20,622 - manifest_processor.ManifestProcessor - ERROR - Aspera transfer failed.
2019-09-11 12:53:20,622 - manifest_processor.ManifestProcessor - DEBUG - Returning error
Skipping file ID 76612bd9a41885add4f6b0b768365c93 as none of the URLs ['FASP'] succeeded.

I just requested my account earlier today so it is possible that it hasn't been approved...

Hi @victor73,
I was trying to install the portal client using docker and I was successfully able to do it. However, when I was trying to run the following command, I was getting an error: -
$docker run -v "$PWD:/tmp" -w /tmp -ti --rm portal_client portal_client --manifest=hmp_manifest_new.tsv

Skipping file ID c3da055b4f1c6b91a79d82ec2ebe66f7 as none of the URLs ['HTTPS'] succeeded.
Skipping file ID 1670203039de370df9a35a04373547c6 as none of the URLs ['HTTPS'] succeeded.
Skipping file ID c3da055b4f1c6b91a79d82ec2ea64bc1 as none of the URLs ['HTTPS'] succeeded.
Skipping file ID c3da055b4f1c6b91a79d82ec2ebfbba1 as none of the URLs ['HTTPS'] succeeded.
Skipping file ID c3da055b4f1c6b91a79d82ec2ea7a441 as none of the URLs ['HTTPS'] succeeded.
Skipping file ID 1670203039de370df9a35a0437369ac9 as none of the URLs ['HTTPS'] succeeded.
Skipping file ID c3da055b4f1c6b91a79d82ec2ed1a7ee as none of the URLs ['HTTPS'] succeeded.
Skipping file ID 1670203039de370df9a35a04374975f3 as none of the URLs ['HTTPS'] succeeded.
Skipping file ID c3da055b4f1c6b91a79d82ec2ebb0ad6 as none of the URLs ['HTTPS'] succeeded.

Not all files (total of 9) were downloaded successfully. Number of failures:
0 -- no valid URL in the manifest file
9 -- URL is present in manifest, but not accessible at the location specified
0 -- MD5 checksum failed for file (file is corrupted or the wrong MD5 is associated)

Here is the sample manifest I was using for your reference. Note that I have converted the tsv to txt just to upload it here. But otherwise, I am reading the manifest as .tsv file.
sample_manifest.txt

I need to download some datasets from HMP and I am unable to do it. Can you please help me with this. Thanks in advance.

Regards,
Diya

One manifest file I tried to fetch using portal_client has HTTPS urls that point to a host whose TLS certificate has expired a month ago. Disabling validation is obviously a security risk, but would it be feasible to add this option?

However, ignoring the warning in FireFox and trying to access the url directly also yields a 403 Forbidden and it does not appear that portal_client is transmitting any sort of authentication, so it seems likely that this wouldn't help either.

When downloading a Manifest file with FASP URLs, it asks for the password once (which is nice), but then there is no feedback whether it is actually downloading something and how much is done already. I have to manually check the download folder to see whether it is still downloading or stuck. So I thought it might be nice to add some sort feedback that either says how much data has been downloaded or how many files are done already.

Just an enhancement request... otherwise it works exactly how it should :)

When downloading multiple FASP URLs via the Human Microbiome Project Data Portal manifest file I have to re-enter the password for every URL again. I can't find any way of specifying my password once so that the portal_client can use that for all the FASP URLs.

Am I overseeing something or is this a missing function?

Besides that, it seems to work great :)

I used this command

./portal_client -m hmp_cart_289e823c35.tsv --endpoint-priority FASP,HTTP --user manar --debug

always give this error when I write the password
Unable to determine version.

2019-10-09 00:49:38,166 - root - DEBUG - In validate_cli.
2019-10-09 00:49:38,166 - aspera - DEBUG - In is_ascp_installed.
2019-10-09 00:49:38,166 - aspera - DEBUG - Installed? False
The ASCP binary is not installed or available.
Please install it or adjust yor PATH.
2019-10-09 00:49:38,166 - root - DEBUG - In obtain_password.
Enter your password: (masked)

2019-10-09 00:49:43,989 - root - ERROR - Aborting execution.

I don't know where is the problem , if the problem in ASCP installation
I installed it from
https://gist.github.com/mfansler/71f09c8b6c9a95ec4e759a8ffc488be3#file-install-ascp-sh
by changing its version and put it in the directory of portal_client

Hello victor , I dowloaded portal_client to collect data from hmpdac.This client works at normal example_manifest , s3_manifest or ftp manifest but It doesnt wok at aspera_manifest.tsv .It say no valid url in aspera_manifest.csv.I check in aspera_manifest.csv link and this is correct.How can I solve this problem? Have a idea? thanks for your comment.

This error image link
https://i.hizliresim.com/mMRja8.png

This is aspera_manifest.tsv
id md5 size urls 07c6c17c2b6f11e98849ef9e212bebbd ed8e04692fc34da01ee9c430612f6b58 3363519 fasp://aspera.hmpdacc.org/ptb/genome/microbiome/16s/hm16str/EP843963_K10_MCKD.trimmedseqset.fasta.gz

In addition.I can dowload below link via aspera connect but portal client not work with aspera_manifest.tsv (I entered below link at chrome).Aspera connect asked user id and password and I enter my hmpdacc account .This link is your aspera_manifest.tsv link.

fasp://aspera.hmpdacc.org/ptb/genome/microbiome/16s/hm16str/EP843963_K10_MCKD.trimmedseqset.fasta.gz

Hi,
I was trying to download data from HMP data portal. However, I observed that for some files I am able to download the file using the download button given whereas for some other files I am not able to do so. For example, for the following file if I click on the download button nothing happens and if I copy and paste the https url in the browser then an error comes up saying forbidden request.
https://portal.hmpdacc.org/files/1670203039de370df9a35a043733fb44

Why is it so? Is it some problem with the file or some access issue which I don't have. Can you please guide me with this. Thanks in advance.

Regards,
Diya.

It appears that the servers are down, I can't download any files using the manifest:

No valid URL found in the manifest for file ID ac37ecc9-2dca-4698-bf7e-1dbaccc903ac
2020-05-05 23:18:57,603 - manifest_processor.ManifestProcessor - DEBUG - In _get_prioritized_endpoint.
2020-05-05 23:18:58,109 - boto - ERROR - Caught exception reading instance data
Traceback (most recent call last):
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1319, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1026, in _send_output
self.send(msg)
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 966, in send
self.connect()
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 938, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socket.py", line 728, in create_connection
raise err
File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
socket.timeout: timed out

This would make it easy to install this software to new users.
(xpost to this user installing portal_client for the first time)

Hi there, trying to download these datafiles: https://assets.nemoarchive.org/dat-zkgd0it

I add files to my cart, download the manifest file, installed portal_client but get the error "no valid URL in the manifest file" (full error with --debug below.

Any alternative way to download these data?

Thanks!
Elaine

(base) eg197@dcc-login-02 /hpc/group/guevaralab/sc-data/existing-data/Ma_data $ portal_client --manifest NeMO_manifest_2b107849a.tsv --destination ./chimpanzee/ --debug
2023-05-17 17:03:53,794 - root - DEBUG - In validate_cli.
2023-05-17 17:03:53,794 - root - DEBUG - Creating ManifestProcessor.
2023-05-17 17:03:53,839 - boto - DEBUG - Retrieving credentials from metadata server.
2023-05-17 17:03:54,846 - boto - ERROR - Caught exception reading instance data
Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1319, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1026, in _send_output
self.send(msg)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 966, in send
self.connect()
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 938, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 728, in create_connection
raise err
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/site-packages/boto-2.49.0-py3.7.egg/boto/utils.py", line 217, in retry_url
r = opener.open(req, timeout=timeout)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1347, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1321, in do_open
raise URLError(err)
urllib.error.URLError:
2023-05-17 17:03:54,943 - boto - ERROR - Unable to read instance data, giving up
2023-05-17 17:03:54,943 - convert_to_manifest - DEBUG - In file_to_manifest.
2023-05-17 17:03:54,945 - convert_to_manifest - DEBUG - In tsv_to_manifest.
2023-05-17 17:03:54,947 - root - DEBUG - About to start downloading manifest.
2023-05-17 17:03:54,947 - manifest_processor.ManifestProcessor - DEBUG - In download_manifest.
2023-05-17 17:03:54,947 - manifest_processor.ManifestProcessor - DEBUG - In _get_prioritized_endpoint.
2023-05-17 17:03:55,448 - boto - ERROR - Caught exception reading instance data
Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1319, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1026, in _send_output
self.send(msg)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 966, in send
self.connect()
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 938, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 728, in create_connection
raise err
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/site-packages/boto-2.49.0-py3.7.egg/boto/utils.py", line 217, in retry_url
r = opener.open(req, timeout=timeout)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1347, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1321, in do_open
raise URLError(err)
urllib.error.URLError:
2023-05-17 17:03:55,449 - boto - ERROR - Unable to read instance data, giving up
No valid URL found in the manifest for file ID PTB2169_4_S1_L001-001.fastq.tar
2023-05-17 17:03:55,449 - manifest_processor.ManifestProcessor - DEBUG - In _get_prioritized_endpoint.
2023-05-17 17:03:55,951 - boto - ERROR - Caught exception reading instance data
Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1319, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1026, in _send_output
self.send(msg)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 966, in send
self.connect()
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 938, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 728, in create_connection
raise err
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/site-packages/boto-2.49.0-py3.7.egg/boto/utils.py", line 217, in retry_url
r = opener.open(req, timeout=timeout)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1347, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1321, in do_open
raise URLError(err)
urllib.error.URLError:
2023-05-17 17:03:55,951 - boto - ERROR - Unable to read instance data, giving up
No valid URL found in the manifest for file ID PTB2169_3_S1_L001-001.fastq.tar
2023-05-17 17:03:55,951 - manifest_processor.ManifestProcessor - DEBUG - In _get_prioritized_endpoint.
2023-05-17 17:03:56,453 - boto - ERROR - Caught exception reading instance data
Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1319, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1026, in _send_output
self.send(msg)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 966, in send
self.connect()
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 938, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 728, in create_connection
raise err
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/site-packages/boto-2.49.0-py3.7.egg/boto/utils.py", line 217, in retry_url
r = opener.open(req, timeout=timeout)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1347, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1321, in do_open
raise URLError(err)
urllib.error.URLError:
2023-05-17 17:03:56,453 - boto - ERROR - Unable to read instance data, giving up
No valid URL found in the manifest for file ID PTB1841_4_S1_L001-001.fastq.tar
2023-05-17 17:03:56,453 - manifest_processor.ManifestProcessor - DEBUG - In _get_prioritized_endpoint.
2023-05-17 17:03:56,955 - boto - ERROR - Caught exception reading instance data
Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1319, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1026, in _send_output
self.send(msg)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 966, in send
self.connect()
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 938, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 728, in create_connection
raise err
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/site-packages/boto-2.49.0-py3.7.egg/boto/utils.py", line 217, in retry_url
r = opener.open(req, timeout=timeout)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1347, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1321, in do_open
raise URLError(err)
urllib.error.URLError:
2023-05-17 17:03:56,955 - boto - ERROR - Unable to read instance data, giving up
No valid URL found in the manifest for file ID PTB166_4_S1_L001-001.fastq.tar
2023-05-17 17:03:56,956 - manifest_processor.ManifestProcessor - DEBUG - In _get_prioritized_endpoint.
2023-05-17 17:03:57,457 - boto - ERROR - Caught exception reading instance data
Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1319, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1026, in _send_output
self.send(msg)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 966, in send
self.connect()
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 938, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 728, in create_connection
raise err
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/site-packages/boto-2.49.0-py3.7.egg/boto/utils.py", line 217, in retry_url
r = opener.open(req, timeout=timeout)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1347, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1321, in do_open
raise URLError(err)
urllib.error.URLError:
2023-05-17 17:03:57,458 - boto - ERROR - Unable to read instance data, giving up
No valid URL found in the manifest for file ID PTB1841_1_S1_L001-001.fastq.tar
2023-05-17 17:03:57,458 - manifest_processor.ManifestProcessor - DEBUG - In _get_prioritized_endpoint.
2023-05-17 17:03:57,959 - boto - ERROR - Caught exception reading instance data
Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1319, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1026, in _send_output
self.send(msg)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 966, in send
self.connect()
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 938, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 728, in create_connection
raise err
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/site-packages/boto-2.49.0-py3.7.egg/boto/utils.py", line 217, in retry_url
r = opener.open(req, timeout=timeout)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1347, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1321, in do_open
raise URLError(err)
urllib.error.URLError:
2023-05-17 17:03:57,960 - boto - ERROR - Unable to read instance data, giving up
No valid URL found in the manifest for file ID PTB166_2_S1_L001-001.fastq.tar
2023-05-17 17:03:57,960 - manifest_processor.ManifestProcessor - DEBUG - In _get_prioritized_endpoint.
2023-05-17 17:03:58,462 - boto - ERROR - Caught exception reading instance data
Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1319, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1026, in _send_output
self.send(msg)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 966, in send
self.connect()
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 938, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 728, in create_connection
raise err
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/site-packages/boto-2.49.0-py3.7.egg/boto/utils.py", line 217, in retry_url
r = opener.open(req, timeout=timeout)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1347, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1321, in do_open
raise URLError(err)
urllib.error.URLError:
2023-05-17 17:03:58,462 - boto - ERROR - Unable to read instance data, giving up
No valid URL found in the manifest for file ID PTB1841_2_S1_L001-001.fastq.tar
2023-05-17 17:03:58,462 - manifest_processor.ManifestProcessor - DEBUG - In _get_prioritized_endpoint.
2023-05-17 17:03:58,964 - boto - ERROR - Caught exception reading instance data
Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1319, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1026, in _send_output
self.send(msg)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 966, in send
self.connect()
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 938, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 728, in create_connection
raise err
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/site-packages/boto-2.49.0-py3.7.egg/boto/utils.py", line 217, in retry_url
r = opener.open(req, timeout=timeout)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1347, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1321, in do_open
raise URLError(err)
urllib.error.URLError:
2023-05-17 17:03:58,964 - boto - ERROR - Unable to read instance data, giving up
No valid URL found in the manifest for file ID PTB2169_1_S1_L001-001.fastq.tar
2023-05-17 17:03:58,964 - manifest_processor.ManifestProcessor - DEBUG - In _get_prioritized_endpoint.
2023-05-17 17:03:59,466 - boto - ERROR - Caught exception reading instance data
Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1319, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1026, in _send_output
self.send(msg)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 966, in send
self.connect()
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 938, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 728, in create_connection
raise err
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/site-packages/boto-2.49.0-py3.7.egg/boto/utils.py", line 217, in retry_url
r = opener.open(req, timeout=timeout)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1347, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1321, in do_open
raise URLError(err)
urllib.error.URLError:
2023-05-17 17:03:59,467 - boto - ERROR - Unable to read instance data, giving up
No valid URL found in the manifest for file ID PTB165_4_S1_L001-001.fastq.tar
2023-05-17 17:03:59,467 - manifest_processor.ManifestProcessor - DEBUG - In _get_prioritized_endpoint.
2023-05-17 17:03:59,968 - boto - ERROR - Caught exception reading instance data
Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1319, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1026, in _send_output
self.send(msg)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 966, in send
self.connect()
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 938, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 728, in create_connection
raise err
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/site-packages/boto-2.49.0-py3.7.egg/boto/utils.py", line 217, in retry_url
r = opener.open(req, timeout=timeout)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1347, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1321, in do_open
raise URLError(err)
urllib.error.URLError:
2023-05-17 17:03:59,970 - boto - ERROR - Unable to read instance data, giving up
No valid URL found in the manifest for file ID PTB2169_2_S1_L001-001.fastq.tar
2023-05-17 17:03:59,970 - manifest_processor.ManifestProcessor - DEBUG - In _get_prioritized_endpoint.
2023-05-17 17:04:00,472 - boto - ERROR - Caught exception reading instance data
Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1319, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1026, in _send_output
self.send(msg)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 966, in send
self.connect()
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 938, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 728, in create_connection
raise err
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/site-packages/boto-2.49.0-py3.7.egg/boto/utils.py", line 217, in retry_url
r = opener.open(req, timeout=timeout)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1347, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1321, in do_open
raise URLError(err)
urllib.error.URLError:
2023-05-17 17:04:00,473 - boto - ERROR - Unable to read instance data, giving up
No valid URL found in the manifest for file ID PTB166_3_S1_L001-001.fastq.tar
2023-05-17 17:04:00,473 - manifest_processor.ManifestProcessor - DEBUG - In _get_prioritized_endpoint.
2023-05-17 17:04:00,974 - boto - ERROR - Caught exception reading instance data
Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1319, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1026, in _send_output
self.send(msg)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 966, in send
self.connect()
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 938, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 728, in create_connection
raise err
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/site-packages/boto-2.49.0-py3.7.egg/boto/utils.py", line 217, in retry_url
r = opener.open(req, timeout=timeout)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1347, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1321, in do_open
raise URLError(err)
urllib.error.URLError:
2023-05-17 17:04:00,974 - boto - ERROR - Unable to read instance data, giving up
No valid URL found in the manifest for file ID PTB165_3_S1_L001-001.fastq.tar
2023-05-17 17:04:00,975 - manifest_processor.ManifestProcessor - DEBUG - In _get_prioritized_endpoint.
2023-05-17 17:04:01,476 - boto - ERROR - Caught exception reading instance data
Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1319, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1026, in _send_output
self.send(msg)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 966, in send
self.connect()
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 938, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 728, in create_connection
raise err
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/site-packages/boto-2.49.0-py3.7.egg/boto/utils.py", line 217, in retry_url
r = opener.open(req, timeout=timeout)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1347, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1321, in do_open
raise URLError(err)
urllib.error.URLError:
2023-05-17 17:04:01,477 - boto - ERROR - Unable to read instance data, giving up
No valid URL found in the manifest for file ID PTB165_2_S1_L001-001.fastq.tar
2023-05-17 17:04:01,477 - manifest_processor.ManifestProcessor - DEBUG - In _get_prioritized_endpoint.
2023-05-17 17:04:01,978 - boto - ERROR - Caught exception reading instance data
Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1319, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1026, in _send_output
self.send(msg)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 966, in send
self.connect()
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 938, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 728, in create_connection
raise err
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/site-packages/boto-2.49.0-py3.7.egg/boto/utils.py", line 217, in retry_url
r = opener.open(req, timeout=timeout)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1347, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1321, in do_open
raise URLError(err)
urllib.error.URLError:
2023-05-17 17:04:01,978 - boto - ERROR - Unable to read instance data, giving up
No valid URL found in the manifest for file ID PTB1841_3_S1_L001-001.fastq.tar
2023-05-17 17:04:01,979 - manifest_processor.ManifestProcessor - DEBUG - In _get_prioritized_endpoint.
2023-05-17 17:04:02,480 - boto - ERROR - Caught exception reading instance data
Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1319, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1026, in _send_output
self.send(msg)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 966, in send
self.connect()
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 938, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 728, in create_connection
raise err
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/site-packages/boto-2.49.0-py3.7.egg/boto/utils.py", line 217, in retry_url
r = opener.open(req, timeout=timeout)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1347, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1321, in do_open
raise URLError(err)
urllib.error.URLError:
2023-05-17 17:04:02,481 - boto - ERROR - Unable to read instance data, giving up
No valid URL found in the manifest for file ID PTB166_1_S1_L001-001.fastq.tar
2023-05-17 17:04:02,481 - manifest_processor.ManifestProcessor - DEBUG - In _get_prioritized_endpoint.
2023-05-17 17:04:02,982 - boto - ERROR - Caught exception reading instance data
Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1319, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 1026, in _send_output
self.send(msg)
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 966, in send
self.connect()
File "/hpc/home/eg197/miniconda3/lib/python3.7/http/client.py", line 938, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 728, in create_connection
raise err
File "/hpc/home/eg197/miniconda3/lib/python3.7/socket.py", line 716, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/hpc/home/eg197/miniconda3/lib/python3.7/site-packages/boto-2.49.0-py3.7.egg/boto/utils.py", line 217, in retry_url
r = opener.open(req, timeout=timeout)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 525, in open
response = self._open(req, data)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 543, in _open
'_open', req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1347, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/hpc/home/eg197/miniconda3/lib/python3.7/urllib/request.py", line 1321, in do_open
raise URLError(err)
urllib.error.URLError:
2023-05-17 17:04:02,983 - boto - ERROR - Unable to read instance data, giving up
No valid URL found in the manifest for file ID PTB165_1_S1_L001-001.fastq.tar

Not all files (total of 16) were downloaded successfully. Number of failures:
16 -- no valid URL in the manifest file
0 -- URL is present in manifest, but not accessible at the location specified
0 -- MD5 checksum failed for file (file is corrupted or the wrong MD5 is associated)

I'm interested in using this tool to download data from the HMP DACC
https://portal.hmpdacc.org/

What might my username be? The data is public, so is it possible that a username is not needed?
--user myusername

Thank you for your time!

Hi,

I'm posting this issue as I found in the history other related issues that didn't seem to be completely solved yet.

In my case the issue centered around the failure to download files with file extension ".fsa.gz", however I noticed that each of these files had a complementary ".tar.bz2" file, for which there was no problem to download.

This just to clarify why this skipping of files happens and maybe to adapt the manifest file to avoid errors while downloading with portal_client. I include the manifest file I used and the output of portal_client.
DOWNLOADhmp_manifest_37ac9a7687.txt
hmp_manifest_37ac9a7687.txt

Best,
Rudy

When downloading from a large manifest, all files are dumped in the same directory with no control beyond --destination. When multiple studies and body-sites are included, if would help if the client could save each file in subdirectories that recapitulate part of the URL.

Closest thing I can think of is how wget -r -x -nH --cut-dirs operates, so that the client would end up putting, for example,
http://downloads.hmpdacc.org/data/HM16STR/HMDEMO/SRP002429/stool/not_affected/SRS066677.fsa
locally in
destination-dir/HM16STR/HMDEMO/SRP002429/stool/not_affected/SRS066677.fsa.

Hi all,
First, I tried alot to download the biological observation matrix and this error always appear:
Not all files (total of 2308) were downloaded successfully. Number of failures:
0 -- no valid URL in the manifest file
5 -- URL is present in manifest, but not accessible at the location specified
10 -- MD5 checksum failed for file (file is corrupted or the wrong MD5 is associated)

I tried alot and each time not all files are downloaded just 2209 out of 2308 are downloaded and others not.
By the way, my manifest file like this form:
9dc112963819ad139611820d22966777 2821344a48240f0fb288e6f0f016d7c5 86000 fasp://aspera.ihmpdcc.org/t2d/genome/microbiome/16s/analysis/hmqcp/HMP2_J79630_1_NS_T0_B0_0120_ZRLZ98T-6031_B9GWL.biom 5a950f27980b5d93e4c16da124908ee3

Second, I found that the URLs that are unique in manifest is 2209, this indicates what was happened and why this error appear or not?
Beside the unique sample ids in manifest is nearly 1900, so 1 sample may have more than 1 URL or not ?
Anyone can help???

Hi,
I am executing the portal_client tool (with my iHMP username and password) to download the files using a manifest file as follows:
portal_client -m hmp_cart_e0473170e2.tsv --endpoint-priority FASP --user username

And the content of hmp_cart_e0473170e2.tsv file is below:
file_id md5 size urls sample_id
7cfd74d6803ea319683df7564459a079 ab27b58083c30b884936ee396b0ee223 29585 fasp://aspera.ihmpdcc.org/ibd/genome/microbiome/wgs/analysis/hmscp/MSM9VZMS_taxonomic_profile.biom 7cfd74d6803ea319683df7564459673c
......
However, I also get an error message as follows:
Skipping file ID 8ef4204e3058e7d0f909de1f96016f5a as none of the URLs ['FASP'] succeeded.
Skipping file ID 8ef4204e3058e7d0f909de1f9600e0c7 as none of the URLs ['FASP'] succeeded.
Skipping file ID 8ef4204e3058e7d0f909de1f9600b107 as none of the URLs ['FASP'] succeeded.
Skipping file ID 8ef4204e3058e7d0f909de1f9601599d as none of the URLs ['FASP'] succeeded.
......
Not all files (total of 81) were downloaded successfully. Number of failures:
0 -- no valid URL in the manifest file
81 -- URL is present in manifest, but not accessible at the location specified
0 -- MD5 checksum failed for file (file is corrupted or the wrong MD5 is associated)

Then I added the --debug patameter , and the error information is :
2019-12-09 15:18:19,871 - root - DEBUG - Creating ManifestProcessor.
2019-12-09 15:18:19,895 - boto - DEBUG - Retrieving credentials from metadata server.
2019-12-09 15:18:20,904 - boto - ERROR - Caught exception reading instance data
....
2019-12-09 15:25:43,569 - aspera - ERROR - Aspera authentication failure.
2019-12-09 15:25:43,569 - manifest_processor.ManifestProcessor - ERROR - Aspera transfer failed.
2019-12-09 15:25:43,570 - manifest_processor.ManifestProcessor - DEBUG - Returning error

Can you please check if my command is correct?
Thanks，
yingc

I'm working on AWS EC2.

If I use portal_client -m manifest.tsv the download is ~100x slower than if I use aws s3 cp. This seems to be the case regardless of destination path (s3 bucket, /ebs, or /efs).

Is the "slowness" due to md5 checks? Something else? portal_client seems like a very valuable tool, but if it's 100x slower I'm not sure it pays off.

I get the same results from eu-west-1 and us-east-1

I have been trying to download this for a couple hours now and can't seem to get it to work. I don't have any experience with python so that might be it.
When I try to download it, I get the following error message:

C:\Users\chiar\portal_client-master>pip install portal_client
ERROR: Could not find a version that satisfies the requirement portal_client (from versions: none)
ERROR: No matching distribution found for portal_client

portal_client-master is the folder downloaded from GitHub.

What am I doing wrong?
Thanks for your help

Command "portal_client --help" rais this error... I tried to install this tool on three different Linux machine, (two fedora, and one Ubuntu ) it shows error on all the machines.

Traceback (most recent call last):
File "/home/joshij/anaconda2/envs/jay/bin/portal_client", line 4, in
import('pkg_resources').run_script('portal-client==1.4.4', 'portal_client')
File "/home/joshij/anaconda2/envs/jay/lib/python2.7/site-packages/pkg_resources/init.py", line 661, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/home/joshij/anaconda2/envs/jay/lib/python2.7/site-packages/pkg_resources/init.py", line 1448, in run_script
exec(script_code, namespace, namespace)
File "/home/joshij/anaconda2/envs/jay/lib/python2.7/site-packages/portal_client-1.4.4-py2.7.egg/EGG-INFO/scripts/portal_client", line 6, in
File "build/bdist.linux-x86_64/egg/portal_client.py", line 12, in
File "build/bdist.linux-x86_64/egg/manifest_processor.py", line 11, in
File "/home/joshij/anaconda2/envs/jay/lib/python2.7/site-packages/portal_client-1.4.4-py2.7.egg/portal_http.py", line 120
print("\r{0}".format(status), end="")
^
SyntaxError: invalid syntax

I want to download via FASP protocol and need a username.
Can you please show where to apply for it? Thanks.