add a bucket#getDirectories() helper method about nodejs-storage HOT 9 CLOSED

From @stephenplusplus on September 5, 2017 21:14

The apiResponse pattern is library-wide and could certainly be re-opened for discussion. Many of our API calls are not autopagination methods, and because of this, it's easy to return our simplified response alongside the raw API response. However, the autopagination methods involve multiple raw API responses, which is why we didn't include them; it wouldn't be clear which of many objects were relevant to the user.

Do you think we should return an array of responses?

A couple notes;

For list operations, we assign the relevant section of the raw API response to our custom object that's returned. E.g., for storage.getFiles, each returned File object has a metadata property that includes many details from the API response. I'm pointing this out just to note that we don't discard the entire API response, just the outer shell.

The other way to fix this problem (and ones like it) is to add a new method to get what you're after, e.g. storage.getDirectories()

@callmehiphop @lukesneeringer for thoughts.

from nodejs-storage.

From @esprehn on September 5, 2017 21:4

Looks like this is paginator.run_:

  if (autoPaginate) {
    this.runAsStream_(parsedArguments, originalMethod)
      .on('error', callback)
      .pipe(concat(function(results) {
        callback(null, results);
      }));

It's super confusing and not at all clear that to list the sub-directories you need to do getFiles with delimiter: '/', paginate turned off, and pass a callback. It'd be really nice this API could be made more ergonomic. :)

from nodejs-storage.

From @esprehn on September 5, 2017 21:57

There's two problems here, the first is that the docs are confusing and show examples with apiResponse but never explain that you need to pass autoPaginate: false:
https://googlecloudplatform.github.io/google-cloud-node/#/docs/storage/1.2.0/storage/bucket?method=getFiles

The second is that getting the list of sub-directories is hard and has some confusing surprises. Google Storage has files, prefixes and sometimes prefixes are also files...

ex. I'm seeing:

getFiles({ prefix: 'foo', delimiter: '/', autoPaginate: false }) returns the prefixes 'foo/' and no files.

getFiles({ prefix: 'foo/', delimiter: '/', autoPaginate: false }) returns prefixes 'foo/bar' and 'foo/baz' and a single file with the name 'foo/'.

I assume this is because creating a folder in the console generates a _$folder$ file, while just uploading a file with a prefix doesn't, and getFiles() will return the magic $folder$ files (though the name is just foo/ without the $ stuff), but not the ones that are just prefixes?

from nodejs-storage.

From @stephenplusplus on September 5, 2017 22:2

@frankyn might be able to help with those questions about the Storage API.

from nodejs-storage.

From @esprehn on September 5, 2017 22:7

The docs seem to contradict the results here, I mostly want the GoogleStorage node API to behave similarly to what gsutil does. :)

ex. creating a folder in the console does not create a file with _$folder$ anymore, I just get a file with a / on the end and application/x-www-form-urlencoded;charset=UTF-8 as the Content-Type.

I'm happy to migrate that part of this discussion elsewhere if you tell me where to file those bugs.

Separately a new getDirectories() API would be super useful. Lots of folks all over are trying to figure out how to make the Google Storage API behave like gsutil, and needing to remember to pass delimiter: '/', prefix, autoPaginate, and then potentially filter out yourself (since the prefix can both be a prefix and a file of the same name, but not always) is a lot of ceremony to list some files.

from nodejs-storage.

@esprehn, I think there are 2 things we can do:

1. Remove the documentation for the apiResponse argument, since it doesn't exist by default. The examples showing how to use autoPaginate will point out that the callback will receive the nextQuery and apiResponse arguments.

2. We should add a bucket.getDirectores() method.

I'll get started on fixing the docs, and put a help wanted label to the issue for the bucket.getDirectories() feature request. If anyone from the community is willing to contribute their time for that, please do!

from nodejs-storage.

I think the docs might still be wrong or at least confusing. The docs for GetFilesRequest say about delimiter

Results will contain only objects whose names, aside from the prefix, do not contain delimiter. Objects whose names, aside from the prefix, contain delimiter will have their name truncated after the delimiter, returned in apiResponse.prefixes. Duplicate prefixes are omitted.

Based on this I would expect

bucket.getFilesStream({delimiter: '/'})
    .on('data', function(f) { console.log(f) })

to return objects for each root prefix in the bucket. Say the bucket had these objects

a/foo.txt
b/foo.txt
b/foo/bar.txt

I would expect to see File objects emitted for a/ and b/, but nothing is ever passed to .on('data',...). Yet if you try exactly this say from https://cloud.google.com/storage/docs/json_api/v1/objects/list?apix=true you indeed get just the root prefixes in the bucket. I can find no combination of adding autoPaginate: false or prefix: '' that will work either.

Aside, from the behavior being different, what I really need is to know how to work around this. I need to be able to discover (I can't know them a priori) the root prefixed in a bucket

from nodejs-storage.

It seems to me that the API is saying: we'll give you files that do not contain your delimiter. But, so you know more about the files that we're excluding, we'll give you some accessory information in the response's "prefixes" property.

To get that information, you need the apiResponse argument, which is not supported in stream mode. You can use bucket.getFiles() with autoPaginate: false:

var prefixes = []

function callback(err, files, nextQuery, apiResponse) {
  prefixes = prefixes.concat(apiResponse.prefixes)

  if (nextQuery) {
    bucket.getFiles(nextQuery, callback)
  } else {
    // prefixes = array of prefixes
  }
}

bucket.getFiles({ autoPaginate: false, delimiter: '/' }, callback)

from nodejs-storage.

I sent a PR to accept a directory option to getFiles(): #162

At this point, it's basically just an alias for prefix. I wasn't sure what would be the most useful way to add the concept of a directory, so feel free to comment with suggestions. Thanks!

from nodejs-storage.