Comments (4)
A function like this could help to tackle #130 through a simple gallery example.
For example, supose we have the urls of a large number of files that we want to automatically download just to compute their hashes and then build a registry file for future usage through Pooch.
So, we could create the registry file using the proposed functions like this:
import pooch
urls = [...] # list containing the urls
filenames = [...] # list containing the filenames used to save the downloaded files
with open("registry.txt", "w") as registry:
for fname, url in zip(filenames, urls):
file_path = pooch.retrieve(url, fname=fname)
hash = pooch.file_hash(file_path)
registry.write(f"{name} {hash} {url}\n")
Maybe we could add an optional argument to pooch.retrieve
to avoid printing the hash.
from pooch.
Maybe we could add an optional argument to pooch.retrieve to avoid printing the hash.
No need: https://www.fatiando.org/pooch/latest/usage.html#adjusting-the-logging-level
from pooch.
You're right! We should add the lines for reducing the verbose on the example I proposed.
from pooch.
After #149 we'll also need to refactor out the Pooch.fetch
code that does the temp file creation, sanity checks, cleanup, hash algorithm determination, etc.
from pooch.
Related Issues (20)
- Download from URL with expired SSL certificate HOT 4
- Problem accessing public files on Dropbox HOT 4
- Add support for downloading from AWS S3 butckets HOT 1
- If Unzip/Untar fails because of a non-existent member, calling without members won't work HOT 2
- Feature: allow null hash in registry files HOT 3
- Make sure that any test that uses the network is properly marked HOT 1
- pooch.make_registry does not handle spaces in filenames correctly HOT 1
- `KeyError: 'key'` in ZenodoRepository.download_url() after Zenodo migration HOT 8
- Mention in docs that registry files must use POSIX compatible file names
- Release v1.8.0 HOT 1
- Sorry wrong repo for the issue HOT 1
- fatiando.org is down HOT 2
- Add "git blob hash" as available hash HOT 3
- Add or document support for Dryad via DOIDownloader HOT 1
- Add support for downloading from Azure cloud storage HOT 2
- Release v1.8.1 HOT 2
- Add support for downloading from Google Cloud Storage HOT 1
- Increase timeout on HTTP requests and use a variable HOT 3
- Inconsistent file hashes between operating systems HOT 3
- file locks for concurrent downloads from the same url
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pooch.