Inconsistent pagination about aws-sdk-go-v2 HOT 13 CLOSED

As a workaround the Service API issue you can use aws.StringValue in both files and check against empty string to determine that the end of pagination has been reached.

This will also ensure your code is forward compatible if the MediaPackage service determine they are able to convert the empty string to a null value in the service response. You can also use ListChannelsPages methods to perform the pagination automatically for you.

The V2 SDK also has outstanding #36 to refactor how the v2 SDK will perform pagination.

from aws-sdk-go-v2.

Thanks for pointing this issue out @kaihendry. This looks to be an inconsistency with the Elemental MediaLive and MediaPackage service. I don't think the MediaLive service should not be sending the NextToken field or sending it as JSON null. I can forward this issue internally to the MediaLive service, but it would be helpful if you submit feedback to the service directly via their API reference documentation. There is a Feedback button in the bottom right corner of the page you can submit feedback directly to the service teams on an API.

from aws-sdk-go-v2.

Thanks, I struggled to find a regular "List*Pages" example. I ended up with something like:

err = svc.ListInputsPages(&medialive.ListInputsInput{},
    func(page *medialive.ListInputsOutput, lastPage bool) bool {
        log.Println("Received", len(page.Inputs), "inputs in page")
        Inputs = append(Inputs, page.Inputs...)
        return lastPage
    })

Which worked. But I am not sure if return lastPage is right.

from aws-sdk-go-v2.

That code should work, except you should changed return lastPage to return true. The bool return value of the pagination callback is to tell the SDK whether to continue paging or not. This is apart of the reason we want to refactor pagination in #36.

A design ideas we're considering for refactoring pagination to would be something similar to the following, using a similar pattern to the bufio's scanner.

This isn't available yet in the SDK but I think something like the following is how we'd like V2 pagination to work.

req := svc.ListInputsRequest(&medialive.ListInputsInput{})
pager := req.Paginate()

for pager.Next() {
    page := pager.Page()
    log.Println("Received", len(page.Inputs), "inputs in page")
    Inputs = append(Inputs, page.Inputs...)
}
if err := pager.Err(); err != nil {
   panic("paging failed, " + err.Error())
}

from aws-sdk-go-v2.

Thank you Jason. Sorry, little confused when you say I should write return true instead. I want it to stop paging when I have everything, and that should be when lastPage is true, no?

from aws-sdk-go-v2.

@kaihendry returning "false" from the callback function will instruct the SDK's paginator to stop paginating. The callback function can return this on any page stating that your application no longer wants to continue to paginate the API. With that said, the SDK will also automatically stop paginating if it reaches the last page. The lastPage input parameter is there to let your application know that this will be the last page of the API's data. Does that help clarify?

from aws-sdk-go-v2.

Hm... I'm still confused. Can you give me an example of code which gets all the data from every page? Trouble is, unless I am going crazy, I can't get a true from lastPage.

err = svc.ListInputsPages(&medialive.ListInputsInput{},
    func(page *medialive.ListInputsOutput, lastPage bool) bool {
        log.Println("Received", len(page.Inputs), "inputs in page")
        Inputs = append(Inputs, page.Inputs...)
        log.Println(lastPage)
        return true
    })

from aws-sdk-go-v2.

@kaihendry Here is an example using S3 list objects.

Is your callback function only being called once? I'm curious if this issue is related to the testing failure found in #89.

Are you using Go's dep tool? If not could you try checking out 0b12d6b5 of github.com/jmespath/go-jmespath and see if that corrects the pagination issue.

cd $(go env GOPATH)/src/github.com/jmespath/go-jmespath
git checkout 0b12d6b5

from aws-sdk-go-v2.

@kaihendry Also how many inputs do you expect to have? Any how many are being returned in the ListPages calls? Is it possible that all Inputs have been returned in a single callback?

from aws-sdk-go-v2.

Happy to use dep, perhaps you can show the Gopkg.toml and I'll use it by running dep ensure after?

Tbh I don't think I am hitting multiple pages with this API strangely. When using the aws cli, it definitely needs paging.

from aws-sdk-go-v2.

Thanks for helping us with this issue @kaihendry. I've identified this issue as directly related to MediaLive's API sending a NextToken as an empty string. I created PR #94 as a workaround with this issue. I've reached out to the service team also to address this issue, as it may be preventing other SDKs from paginating the API response correctly.

from aws-sdk-go-v2.

Hope you go to the https://godoc.org/bufio#Scanner approach, since I am still struggling to comprehend return true, aka (continuePaging bool). I also would like a straightforward way of dropping in aws cli style --query, preferably in the Input object.

from aws-sdk-go-v2.

Thanks for the feedback. The idea of the return true is a bit awkward and one reason we are looking at replacing the pagination pattern. If you don't mind please create a new feature request for the query concept. We can discuss more how this concept could be incorporated.

from aws-sdk-go-v2.