Problem accessing public files on Dropbox about pooch HOT 4 CLOSED

Hi @tloredo. Thanks for opening this issue.

What Pooch is doing is downloading the HTML file that Dropbox gives you when you access to https://www.dropbox.com/s/5b9m1pq5qif5obf/lambda-3923-4010-phases-100.h5?dl=0. The Download button is not a regular anchor with a static link, but a dynamic button that triggers the download of the desired file.

After reading Dropbox's docs, you could force it to give you the download link by replacing the trailing dl=0 for dl=1. I just try it out and it downloaded a binary file, which I suppose is the hd5 file you want to fetch.

I'm not sure how wget is able to download the file, even if you pass the dl=0. Maybe it parses the url and uses the one with dl=1 instead.

BTW, if you are going to pass custom urls for every file in the registry, you could use and empty string for your base_url, since it won't be used at any point.

This should work:

import pooch

fetcher = pooch.create(
    path=pooch.os_cache("SOAP2-1Spot"),
    base_url="",
    registry={
    "lambda-3923-4010-phases-100.h5" : None,
    "lambda-3923-6664-phases-4.h5" : None
    },
    # Now specify custom URLs for some of the files in the registry.
    urls={
        "lambda-3923-4010-phases-100.h5" : "https://www.dropbox.com/s/5b9m1pq5qif5obf/lambda-3923-4010-phases-100.h5?dl=1",
        "lambda-3923-6664-phases-4.h5" : "https://www.dropbox.com/s/pyeapovhk4q6az0/lambda-3923-6664-phases-4.h5?dl=1"
    },
)

# These paths end up pointing to files named as indicated, but containing HTML
# corresponding to a Dropbox "can't preview" response:
full_spec_path = fetcher.fetch("lambda-3923-6664-phases-4.h5")
ca_spec_path = fetcher.fetch("lambda-3923-4010-phases-100.h5")

from pooch.

@santisoler, thanks so much for the quick and helpful response (and the tip about the base url string)! dl=1 does solve this problem. I wonder what wget is doing, but in any case this Pooch issue is solved.

from pooch.

Glad to be helpful! 🙂

from pooch.

Just a further followup: Some poking on Stack Exchange, after geting @santisoler's solution, suggests that Dropbox recognizes some user agents and handles requests from them in special ways. See, e.g., linux - how to download dropbox files using wget command? - Super User and curl - User-Agent affects Dropbox shared links download - Stack Overflow. The lesson being that having a Dropbox URL work with some user agents doesn't mean that URL will work for others.

from pooch.