Can only see 20 buckets in the CLI with `influx bucket list` about influx-cli HOT 4 CLOSED

This is a similar but not directly related issue, with the API. influxdata/influxdb#20048

from influx-cli.

@danatinflux
Could really use this as a parameter for the netreo migration(https://github.com/influxdata/InfluxDB-Cloud-2-Migration/issues/19) They have 36 customers and 5 buckets for each. I use the customer id in the bucket name. For token and dbrp processing on the scripts, I do a bucket list, grep out the buckets and then process them. Now I get clipped at 20. You should be able to specify something like --size 500

from influx-cli.

A --size or --limit flag could be part of the solution, but not enough to fully address this; the DB backends (both OSS and Cloud) enforce a max-size on list requests to protect themselves from abuse.

To fully address this, the CLI will need to make iterative requests to the buckets API, setting after=<ID-of-last-bucket-from-previous-batch> and limit=<some-page-size>. The iteration would stop once a request returned an empty list. We could expose flags for users to control the process:

• --limit: Limit on total number of buckets to fetch & print
• --page-size: Value to set for limit in each HTTP request
• --offset: Number of buckets to skip on initial request
• --after: ID of bucket to set as after in the initial request

We'd probably want this for all of our list commands, but we see it requested most frequently for buckets so it makes sense to implement it there first as a one-off.

from influx-cli.

@btasker gave me a work around in python:

#!/usr/bin/env python3

import json
import requests
import sys


# https://docs.influxdata.com/influxdb/v2.0/api/#tag/Buckets
INFLUX="https://us-west-2-1.aws.cloud2.influxdata.com"
ORG="xxxx"
TOKEN="xxx"


def reqAPI(path):
    headers = {
        "Authorization": f"Token {TOKEN}"
    }

    r = requests.get(f'{INFLUX}{path}', headers=headers)
    return r.json()


def printBuckets(r):

    for bucket in r['buckets']:
        print(f"{bucket['id']}\t{bucket['name']}")


r = reqAPI('/api/v2/buckets?org={ORG}')

if r:
    while "links" in r and "next" in r['links']:
        printBuckets(r)
        r = reqAPI(r['links']['next'])

printBuckets(r)

from influx-cli.