Credentials should be resolved from metadata service, if available, in default case. about gcsfs HOT 6 CLOSED

Have you tried using storage_options={'project': 'myproject', 'token': 'cloud'} in to_textfiles?

from gcsfs.

I haven't tried that specifically, as I can't (in my case) couple storage configuration to the call to to_textfiles. The advantage of dask's approach is that the implementation can, in principle, specify different output targets via the path alone. Accommodating a scenario where I can specify an output file as gs://my-bucket/output_dir/output.*.json in the cloud and ~/output_dir/output.*.json locally, without needing to modify my job code, is relatively important.

I think a small number of tweaks to the credential behavior could enable this. Would you mind if I opened a few tracking issues, with the intention of submitting a PR?

from gcsfs.

@asford , sure we of course welcome suggestions and PRs.
Note that dask does support URLs like s3's 's3://id:secret@bucket/path', so that could be done for gcsfs; but in general I agree that sensible defaults are a good idea too.

from gcsfs.

Would the headless option described above be an acceptable solution? Given #56, I would envision an implementation that (in order of priority).

1. Resolves default credentials via the oauth2client. (Ideally resolving to user credentials have been established by gsutil or the service account.)
2. Resolves credentials from a cached token.
3. Resolves service credentials via internal metadata.
4. Falls back to unauthenticated api access.

from gcsfs.

I'm not qualified to say what a good policy is here (I don't have enough on-the-ground experience here) but just wanted to say that your engagement on this problem (and this repository generally) would be quite welcome.

from gcsfs.

@asford , after conversations, I think I agree with your order of preference, for the default case of token=None. The options for token='cloud', token='anon' (new) and token='file'/{refresh: token} should still be respected as before.
I'll make this a separate PR which will likely conflict with the debug statements in #57

from gcsfs.