What problem are you facing?

Provide a way to list the "descendants" of a CompositeResourceClaim instance. That is the XR (CompositeResource) that the XRC generated as well as the MRs (ManagedResources) and XRs or XRCs that it composes and recursively the composed resources of those XRs or XRCs.

What problem are you facing?

We don't yet have a clean way to build pagination in xgql, largely because we're constrained by the Kubernetes API. We likely want to implement cursor based pagination if possible, but it's not a priority to do so yet. For now we just want to make sure we could implement pagination without a breaking API change.

Currently xgql supports a "limit" argument to all or most of its fields that return a connection to an array of types. This argument is not very useful without a cursor or offset.

How could xgql help solve your problem?

xgql should remove "limit" support until we have a workable plan for pagination.

What problem are you facing?

Provide a method of listing claims (instances of a CompositeResourceClaim) that are in an "unhealthy" state. Will likely also be helpful to provide field that allows for finding out if a given claim instance is unhealthy. Need to determine if unhealthy is a boolean and what constitutes an unhealthy state.

What happened?

During CI builds on the main branch the promote step is not running.

Edit: #40 removed support for resolving ProviderConfig references per the details below. I'm repurposing this issue to track adding them back once crossplane/crossplane#2255 is solved.

What problem are you facing?

Per crossplane/crossplane#2255 it's challenging for xgql to resolve a managed resource's spec.providerConfigRef because we need to do a bunch of spelunking to figure out which provider a managed resource belongs to.

How could xgql help solve your problem?

I think the right way to solve this is to have Crossplane give us the information we need. In the meantime, we should consider removing support for resolving a managed resource's provider config from the schema. We could still expose the information as an unresolved reference in the meantime, though doing so would be a little unidiomatic for xgql.

What problem are you facing?

Currently we're building a v0.0.0 tag; I think we're ready to ship v0.1.0.

How could xgql help solve your problem?

Ship v0.1.0!

What problem are you facing?

xgql currently uses a <T>Connection type to represent a connection between two types - e.g. a connection between a Provider and its ProviderRevisions:

type Provider implements KubernetesResource {
  # Irrelevant fields redacted

  revisions(limit: Int, active: Boolean): ProviderRevisionConnection!
}

These connection types are "relay-ish" in that they give us a place to model counts and metadata about the array of types a particular type might be 'connected' to, though they don't currently support cursors and pagination (because it's tricky to build that on the Kubernetes API).

Meanwhile, xgql uses a <T>List type for queries that list a particular type, e.g.:

type Query {
    providers(limit: Int): ProviderList!
    configurations(limit: Int): ConfigurationList!
    compositeResourceDefinitions(limit: Int, dangling: Boolean = false): CompositeResourceDefinitionList!
    compositions(limit: Int, dangling: Boolean = false): CompositionList!
}

The <T>List and <T>Connection types are identical. I notice that other GraphQL APIs (e.g. the GitHub API) just use <T>Connection for queries.

How could xgql help solve your problem?

xgql could just use <T>Connection types.

The ids returned by xgql can be used in web interfaces where the id is in the url. This puts some constraints on length though those are fuzzy and browser/os dependent. It is also not as clean to have a very large/long id in a url from a user experience perspective. It would be nice to keep these ids short.

Hit this issue (slack thread) because the involvedobject (some pod) no longer existed, which would throw in xgql:

could we be more resilient?

What problem are you facing?

xgql makes heavy use of caching. The first time a particular user asks for a particular kind of resource it needs to start and hydrate a new cache for that kind of resource, which it does by listing the resource and then starting a watch. The user's call blocks waiting for that cache to by hydrated, and times out after 5 seconds of waiting.

In some cases the cache layer can detect very quickly that the cache isn't going to start and therefore that the blocking request is going to timeout - e.g. because the caller doesn't have RBAC access to the requested type - but there's unfortunately no mechanism for the cache layer to communicate this to the caller.

How could xgql help solve your problem?

We could modify our cache layer to allow calls to fail fast in cases where it's obvious that the cache isn't going to hydrate - e.g. due to RBAC issues. We're currently using the controller-runtime caching machinery, mostly because it's well proven and was expedient to reuse.

I could see a few options here:

1. We implement our own cache that satisfies the controller-runtime client.Client interface, but that fails fast and surfaces failures up to the caller.
2. We raise an issue against controller-runtime to see whether they'd be interested in this behaviour. I don't recall whether the change would need to happen in controller-runtime or the lower level client-go machinery. This could be a bit of a hard sell because we're not really using the controller-runtime cache the way it was intended to be used.
3. We fall back rapidly when a cache takes too long to start and make an uncached call, which would fail in a more obvious manner if there were RBAC or network connectivity issues.

What happened?

When a CompositeResourceClaims spec.compositionRef.name is not set to a valid composition name queries that reference spec.compoisition fail. One specific case is when the name is "" which it is when a composition has not yet been selected. Similarly the spec.resourceRef can have values that point at an object that does not or no longer exists and also causes spec.resource queries to fail. Both of these cases seem to be non-exceptional. They may be a degenerate or error state but not exceptional. The current behavior fails the whole query which is an exceptional case. The query should not fail but rather return either an NULL (and make these fields nullable) or return a union and define an error case that explains why they are not present. Also, I suspect that. spec.compositionRevisionRef is another instance of this issue.

How can we reproduce it?

Construct a CompositeResourceClaim that has a compositionRef and/or a resourceRef that does not point to something valid.

What environment did it happen in?

dev

What problem are you facing?

I would like to export xgql traces to Google Cloud Trace (stackdriver).

How could Upbound help solve your problem?

Add support for exporting traces to Cloud Trace.
https://cloud.google.com/trace/docs/setup/go-ot

What problem are you facing?

xgql is built around a cache of Kubernetes API clients. The intent is for each client to be scoped to a specific user of the Kubernetes API (or perhaps more generally each "RBAC subject"). This approach ensures clients, which are cache backed, are aligned with the RBAC policies of the underlying API. If each subject has their own client with its own cache we can leave RBAC enforcement up to the API server; if a particular user doesn't have access to a particular resource they won't be able to load it into their client's cache. If we instead shared a cache across multiple subjects we'd need to somehow verify that the subject had RBAC access to read/list/watch a particular type before we allowed them to read it from the shared cache.

Currently we uniquely identify subjects by their Kubernetes bearer token. We assume that requests to our GraphQL query endpoint will be made with an Authorization: Bearer <some-great-bearer-token> HTTP header. We expect this bearer token to be valid for the Kubernetes API, and thus use it to create a (list/watch cache backed) client of the Kubernetes API. Each client automatically takes a watch on any type it is asked to read in order to hydrate its cache. To prevent clients from trying to watch namespaced resources (like secrets) across all namespaces when they only have access to one we optionally allow namespaced clients - i.e. clients that are cached by both bearer token and namespace.

Note that xgql has "a cache of clients", and also that each client is backed by a cache (of Kubernetes resources).

The problem here is that subjects aren't always uniquely identified by a bearer token. Per the Kubernetes authn docs and the *rest.Config we use to create clients, a subject might be uniquely identified by:

• A bearer token (e.g. OIDC, service account, webhook validated, etc).
• Their basic auth username.
• The CN (username) and Os (groups) of their X509 client certificate.
• The content of the Impersonate-* HTTP headers.
• Other, less frequently used methods (auth proxies, plugins).

How could xgql help solve your problem?

xgql could support "passing through" at least the most commonly seen kinds of credentials such as basic auth, client certificates, and impersonation headers rather than assuming everyone will use bearer tokens. It would also need to consider these authentication methods when keying entries in its cache of clients. Perhaps the current cache key (a sha256 hash of a salt, the bearer token and the namespace) could be updated to account for all authentication methods. e.g.

func id(salt []byte, cfg *rest.Config, namespace string) {
	h := sha256.New()
	_, _ = h.Write(salt)
	_, _ = h.Write([]byte(cfg.Username))
	_, _ = h.Write([]byte(cfg.BearerToken))
	_, _ = h.Write(getImpersonationConfig(cfg))
	_, _ = h.Write(getTLSConfig(cfg))
	_, _ = h.Write([]byte(namespace))
	return fmt.Sprintf("%x", h.Sum(nil))
}

What problem are you facing?

Most xgql queries and mutations pertain to Kubernetes resources; i.e. types that satisfy the KubernetesResource interface. This interface exposes an events field that is intended to return events pertaining to a particular resource, but no type currently implements a resolver for this field (only stubs exist).

How could xgql help solve your problem?

xgql could support resolving events for all resources. This should involve listing corev1.Event and filtering the list down to those whose involvedObject is the relevant resource.

We may also want to add a query to fetch events without first needing to fetch the resource (object) with which the events are associated. e.g.:

type Query {
  # Returns all events if id is nil, otherwise returns events involving id.
  events(id: ID, ref: Reference): EventList! 
}

What happened?

{
  kubernetesResource(id: "Hi8wQzg9X0M4XYY8YVZQVQE") {
    metadata {
      name
    }
    ... on CompositeResourceDefinition {
      metadata {
        uid
      }
    }
  }
}

Yields this:

{
  "data": {
    "kubernetesResource": {
      "metadata": {
        "name": "compositeclusters.aws.platformref.crossplane.io"
      },
      "metadata": {
        "uid": "a8dcfa8e-3909-4bf8-968b-64c65ca9b65b"
      }
    }
  }
}

Which is invalid JSON and should yield this:

{
  "data": {
    "kubernetesResource": {
      "metadata": {
        "name": "compositeclusters.aws.platformref.crossplane.io",
        "uid": "a8dcfa8e-3909-4bf8-968b-64c65ca9b65b"
      }
    }
  }
}

This may be related 99designs/gqlgen#1311.

How can we reproduce it?

Make the query above.

What environment did it happen in?

What problem are you facing?

xgql in its current form assumes its consumers are concerned only with a single Crossplane installation. Some consumers of xgql - like Upbound Cloud - are concerned with multiple Crossplane installations. It's currently unclear how xgql can enable consumers like these to potentially select and query from an evolving set of xgql instances, each with different (but backward compatible) iterations of the xgql GraphQL schema.

How could xgql help solve your problem?

Perhaps xgql would make it possible to support schema stitching or Apollo federation? In doing so we need to be careful that we don't compromise the experience for callers who aren't concerned with multiple Crossplane installations.

What problem are you facing?

xgql allows callers to introspect its schema, but it may still be useful for callers to be able to discover what version of xgql is running.

How could xgql help solve your problem?

We could:

• Return the version as part of a well known HTTP header.
• Add a /version endpoint that returned the version.

What problem are you facing?

xgql represents connections between nodes (and the result of some queries) as arrays of nodes. Currently these arrays have a random order that differs from call to call. This could be surprising to API callers, and will likely make API pagination challenging in future.

How could xgql help solve your problem?

xgql could return stably sorted arrays of nodes in its connections. The problem is likely that xgql reads from a controller-runtime cache that (I suspect) is backed by a map. Go randomizes iteration over map entries.

What problem are you facing?

Users of the (as yet unreleased) UXP distribution of Crossplane may wish to create bespoke platform consoles (or CLIs) that are backed by Crossplane. We think xgql could be a compelling API for these users to target, for the various reasons that many folks prefer GraphQL to REST.

How could xgql help solve your problem?

xgql could be released as an open source component of the UXP distribution, the licensing and CLA details of which are still being ironed out.

In the meantime, please be wary of discussing private Upbound initiatives and technologies on this GitHub repo. Ideally we will be able to make this repo public at some point without needing to worry about cleaning up sensitive information.

What problem are you facing?

It's common (perhaps borderline ubiquitous) for GraphQL types to satisfy the Node interface:

interface Node {
  id: ID!
}

In this interface id is a scalar field containing an ID that is intended to be opaque to the client. GraphQL states:

To provide options for GraphQL clients to elegantly handle caching and data refetching, GraphQL servers need to expose object identifiers in a standardized way.

As best I can tell whether this id must be unique within its type or globally unique is undefined. I note that in the GitHub GraphQL API an ID is often (always?) a base64 encoded string which includes type metadata, and thus an ID can be resolved to arbitrary types using an inline fragment.

xgql is an abstraction atop the Kubernetes API - it is constrained by the API shape and query support of the Kubernetes API, and therefore constrained in how it may model IDs. The Kubernetes API uniquely identifies objects by their API version, kind, name, and (where relevant) namespace. In some cases xgql can derive some of this information from the type; e.g. our schema has a type Provider, which we know to have API version pkg.crossplane.io/v1, kind Provider, and no namespace. This means we could uniquely identify a Provider (relative to other Providers) by its name alone. In other cases our GraphQL type corresponds to a class of Kubernetes types. For example a Crossplane "managed resource" never has a namespace, but can be of arbitrary API version and kind. This means to look up a type ManagedResource we need its API version, kind, and name.

If xgql were to consistently expose an ID it would likely make sense to take an approach similar to GitHub's - e.g. derive an ID by encoding each object's API version, kind, namespace, and name in some opaque-seeming fashion. Perhaps base64 encoded JSON. This would reinforce the opaqueness of the ID to callers, while providing xgql with all the information it would need to resolve a Node to a concrete resource (and therefore all the information it needs to look up any kind of Crossplane resource).

This all said, it's not clear to me what the consequences of not supporting the Node interface would be. While clients seem to cache and refetch on the id field by default, it doesn't seem to be mandatory. For example the Apollo GraphQL client supports caching based on other key fields.

interface KubernetesResource {
  apiVersion: String!
  kind: String!
  metadata: ObjectMeta!
  # Irrelevant fields redacted
}

type ObjectMeta {
  name: String!
  namespace: String
  # Irrelevant fields redacted
}

All xgql types that correspond to 'atomic' resources in the Kubernetes API (e.g. the most granular types we can ask the Kubernetes REST API to return - "a provider" rather than "a provider's status") satisfy the above interface. This means all types already expose the four distinct fields that uniquely identify a Kubernetes resource. We'd be encoding, and thus duplicating, this same data in our ID.

How could xgql help solve your problem?

xgql could support the Node interface! Or maybe not? At the very least it seems like supporting Node would:

• Be idiomatic to GraphQL.
• Make life easier for folks using common GraphQL clients (e.g. no need to override the default cache primary keys)
• Allow us to expose a consistent GraphQL ID for things that aren't Kubernetes resources (e.g. things that aren't uniquely identified by API version, kind, namespace, and name) should we ever need to.

What problem are you facing?

#41 adds board support for creating, updating, and deleting Kubernetes resources via xgql. I believe this technically covers all our use cases and is in some cases the best we can really do - e.g. for types like XRs and managed resources where we don't know the full schema of the resource.

There are however some mutations for well known resources - e.g. installing a provider or a configuration - that will be common and where we do know the schema. It would be ideal if we had 'strongly typed'/'structured' mutations for these types.

How could xgql help solve your problem?

xgql could add structured mutations for well-known types, particularly creating and updating configurations, and providers. At some point we may also want to support working with XRDs and Compositions directly.

What happened?

When querying for a list of providers without supplying any authentication token the query appears to block indefinitely (or at least for a minute or so). Based on my understanding of how controller-runtime caches work I would have expected xgql to act as if no providers existed, but return a result fairly quickly.

It seems like perhaps we need to set a timeout somewhere on client requests, though I am curious what is blocking this in the first place.

How can we reproduce it?

Install some Crossplane providers, run xgql without any RBAC access to providers in the pkg.crossplane.io API group, and run the following query:

query {
  providers {
    totalCount
  }
}

What environment did it happen in?

xgql version: A local build of #30

What problem are you facing?

Upbound Cloud is the first consumer of xgql. We intend xgql to be useful to other consumers in future (so it should not be purpose built to serve Upbound Cloud), but supporting Upbound Cloud use cases is a great way to discover the mutations xgql should support.

Upbound Cloud's "XRM Explorer" is a console platform operators can use to create, update, and delete:

• Configurations and providers.
• Composite resource claims.
• Composite resources.
• Managed resources.

xgql does not currently implement any mutations.

How could xgql help solve your problem?

xgql could support the mutations needed by the XRM explorer. In particular I believe creating configurations and claims are a priority. There are some open design questions to solve as part of this implementation, including how create, update, and delete payloads should be modelled (e.g. as raw JSON?), and whether we should model create and update as separate operations (like the underlying Kubernetes API), or expose them as an 'apply'/'upsert' mutation.

The ConditionStatus enum has values like "TRUE" and "FALSE" but xgql is returning "True" and "False" which is invalid.

What problem are you facing?

Upbound Cloud would like to be able to extract metadata about a particular UXP environment (e.g. the versions of running UXP components). The UXP Helm chart will write these to a ConfigMap, and we'd like to be able to query them via xgql.

How could xgql help solve your problem?

xgql could support querying for ConfigMaps. ConfigMaps aren't really related to Crossplane in any way, but they're not too far from Secrets (which we already query) and are generally a good catch-all way to surface metadata about "Crossplane stuff" that doesn't fit anywhere better via GraphQL.

Note that I don't think we should try to transform the value of the ConfigMap to present it as "Crossplane version data"; we should instead let the caller extract what information it needs.

What problem are you facing?

xgql currently has support for metrics and tracing, but there's no simple way to detect whether the xgql pod is in a fundamentally broken state. The most common reason xgql might be completely non-functional would be because it can't reach the API server.

How could xgql help solve your problem?

xgql could add a readiness endpoint (e.g. /ready) that returned 200 OK if some basic sanity checks passed, e.g. that the Kubernetes API server was reachable.

In an effort to make sure that ids are unique across multiple instances of xgql for purposes of advanced schema stitching like what is done in Upbound Cloud it would be nice to be able to add a prefix/salt to the ids. The request could contain a x-id-salt header that would be used to prefix all ids generated during that request.

What problem are you facing?

The only way to get a CustomResourceDefinition is either to get it from the list of all resource definitions (customResourceDefinitions.nodes) or directly by id from kubernetesResource(id: $id). The list of all CRDs can be quite long and if the user is just wanted the CRDs the correspond to the CompositeResourceDefinitions fetching the whole is and expensive way to get them. The CompositeResourceDefinition does offer the apiVersion and kind of composite and claim CRDs via status.controllers.compositeResourceType and status.controllers.compositeResourceClaimType respectively but there is no way to fetch a CRD from that info.

How could Upbound help solve your problem?

If a CompositeResourceDefinition offered fields that returned the CRD for the claim and the composite resource this would be a much smaller list and a more performant way of getting these details. Also, it would be nice to be able to fetch a CRD from kind and apiVersion but this is not strictly needed currently.

What happened?

rm, err := apiutil.NewDynamicRESTMapper(cfg)

Currently xgql uses a "Dynamic REST mapper" to discover what Kubernetes API resources exist. The advantage of this REST mapper is that it will learn about new kinds of custom resource as they are defined; e.g. when their CRDs are created.

An issue with this REST mapper is that it assumes that a request for a kind of resource it doesn't already know about should be handled by calling the discovery API again. Querying for a resource that doesn't actually exist appears to cause calls to hang indefinitely, until they timeout.

How can we reproduce it?

Query for an unsupported kind of resource, e.g.:

query {
  kubernetesResources(
    apiVersion: "does.not.exist/v1"
    kind: "Nope"
  ) {
    totalCount
    nodes {
      id
      metadata {
        name
        uid
      }
    }
  }
}

What environment did it happen in?

xgql version: adb10e6

What problem are you facing?

Currently xgql listens only on unencrypted HTTP. This is a bad practice, particularly because we expect callers to pass xgql their Kubernetes credentials (e.g. bearer tokens and potentially more per #9) so that it may query the Kubernetes API on their behalf.

How could xgql help solve your problem?

xgql should at the very least support serving via HTTPS (e.g. when manually configured with a TLS cert and key).

Ideally it would also magically provision itself TLS certificates when installed, though this might be hard to do. Perhaps using something like https://github.com/kelseyhightower/certificate-init-container or cert-manager? It's likely that using Kubernetes CSRs to get a certificate trusted by anyone who trusts the cluster.

What problem are you facing?

Running make reviewable currently invokes various Go linters to lint xgql's Go code. Ideally we'd lint the GraphQL schema too to enforce good practices, such as requiring descriptions.

How could xgql help solve your problem?

We could configure make reviewable such that it (if necessary) installed and ran a GraphQL schema linter, such as https://github.com/dotansimha/graphql-eslint.

What problem are you facing?

Upbound Cloud is the first consumer of xgql. We intend xgql to be useful to other consumers in future (so it should not be purpose built to serve Upbound Cloud), but supporting Upbound Cloud use cases is a great way to discover the queries xgql should support.

Upbound Cloud's "XRM Explorer" is a console platform operators can use to:

• See which Crossplane providers and configurations are installed.
• List the kinds of managed resource and provider config installed by a provider.
• List the XRDs and Compositions installed by a configuration.
• List any XRDs and Compositions that aren't associated with any configuration.
• List extant managed resources and provider configs of a particular kind.
• List extant XRs and claims of a particular kind.
• List extant Compositions of a particular kind.
• List the resources (either managed or composite) of which an XR is composed.

xgql currently implements two queries that lets a platform operator broadly walk these trees:

type Query {
    providers(limit: Int): ProviderList!
    configurations(limit: Int): ConfigurationList!
}

The providers query can be used to drill down from Provider -> ProviderRevision -> CustomResourceDefinition -> extant defined ManagedResource or ProviderConfig resources (including counts).

The configurations query can be used to drill down from Configuration -> ConfigurationRevision -> CompositeResourceDefinition (or Composition) -> extant defined CompositeResource resources.

How could xgql help solve your problem?

xgql could support resolving the various queries listed above that are not currently implemented. In particular resolving an XR to its composed resources would be a good start.

What problem are you facing?

Currently xgql doesn't have a dedicated type to represent ProviderConfigUsage resources, but it could. Today a ProviderConfigUsage would appear to be a GenericResource.

How could xgql help solve your problem?

Add a ProviderConfigUsage schema type and have PCUs returned as that type, not a GenericResource.

What problem are you facing?

xgql currently implements a handful of resolvers, but they do not have unit tests.

How could xgql help solve your problem?

Add unit tests for the existing resolvers.

What problem are you facing?

From @thephred:

Currently requests going to the proxy subdomain which is handled by the crossplane-router component are not being gripped. Gzipping GraphQL requests has a significant impact on their size.

I actually naively thought the Go HTTP server had compression enabled by default, but it turns out that's not the case.

How could Upbound help solve your problem?

We could enable gzip! Looks like there's a middleware for chi, the HTTP framework that we use.

https://github.com/go-chi/chi/blob/master/middleware/compress.go

What problem are you facing?

Like many Kubernetes clients xgql performs 'discovery' - a process via which it walks the API server to determine what APIs it supports. Discovery makes a request per API group, and there are about 350 API groups when the "big three" Crossplane providers are installed with all of their CRDs. xgql's discovery client uses a rate limiter:

I believe these rate limits are too low (I also can't remember why we have both dcfg values and a rate.NewLimiter). We've been working with upstream Kubernetes to bump most discovery clients to at least 300qps burst, per kubernetes/kubernetes#109141. There's also a compelling school of thought that we should just disable client-side rate limiting altogether and let server-side API Priority and Fairness let clients know when they should back off.

How could Upbound help solve your problem?

I think we should disable rate-limiting on xgql and try out API Priority and Fairness. If I recall correctly this issue is more of a nuisance than a huge blocker on xgql, because it uses a global discovery client (not one per client) and because it only performs discovery once at startup.

It would be nice to have the xgql GraphQL schema published somewhere. Upbound Cloud is publishing schemas to https://studio.apollographql.com which supports "variants" so there could be a master variant and a stable variant for instance. This allows pulling down of the published schema as well as a listing of the changes over time and if they are breaking changes or not.