Add option recursive to listdir about fiona HOT 4 CLOSED

@adriantre after some reflection, I'm inclined not to do this. First of all, the workaround isn't terrible, and not adding a new feature to fiona (or rasterio) saves us testing, documentation, &c. But mostly I don't want fiona and rasterio to become cloud storage APIs. GDAL's VSI system has a whole cloud storage API, but I believe Python developers are much better off using their cloud's Python SDK (boto3, for example) or a dedicated cross-cloud library like https://libcloud.apache.org/.

from fiona.

@adriantre I'm interested. Can you tell me more about how you would use this?

from fiona.

Thanks Sean!

I started using TorchGeo and found that there were only some minor blockers for it to be able to read files directly from cloud storage through GDAL VSI. The only blocker is they are pattern-matching files using glob.

root = "/vsigs/my_bucket'
filename_glob = "T*_B02_10m.tif"
pathname = os.path.join(root, "**", filename_glob)
for filepath in glob.iglob(pathname, recursive=True):
    ...

I don't know how easy it would be to mirror the behaviour of glob, allowing wildcards, but it would then be a very easy fix in that repo :P

A similar option is Pathlib.rglob.

Another approach would be to create a fiona.walk which mirrors os.walk.

for root, subdirs, files in os.walk(walk_dir):
    for file in files:
        asb_path_for_file = os.path.join(root, file
        #file is filenames only but os.path.join will magically yield the abs path

I created a PR to TorchGeo with the above workaround. But I think swapping out the usage of glob would be a cleaner solution.

from fiona.

I understand. Cross-cloud is why I was looking to GDAL/Fiona, to have a platform-agnostic solution. libcloud looks promising.

from fiona.