ipni / specs Goto Github PK

In specific situations such as HTTP providers, there may not be any peer ID or peer ID may be redundant because HTTP provider uses TLS.

Is there a way to relax the need for Peer ID in such cases?

There are no IPNI specifications for discovering addresses given a peer ID even though the IPNI implementation running at cid.contact offers this via the /providers/<peer-id> endpoint.

• Update IPNI specification with HTTP endpoints that allow address discovery given peer ID
• Update OpenAPI specification as needed.

Relates to: ipni/caskadht#10

Add lint for all markdown files similar to the setup used by ipfs/specs

Calculating the height of advertisement chain is cheap and fast for a provider. That information can provide a significant cause in conflict resolution across indexers.

Indexers can also calculate this number while ingesting ads. However

• adding that on provider's side within the signed head payload reduces the need for trusting indexers.
• existing indexers who have not measured height do not need to re-ingest or re-traverse the ad chain to calculate this number.

The concept topic is present in gossipsub announcements which separates publications by network name. But direct HTTP announce has no such primitive defined in message schema.

We most likely want the topic separation at IPNI protocol level to reduce testing noise from non-mainnet announcements.

Specify a HTTP protocol for retrieving advertisements from other indexers via their mirror instance. The specification should provide additional headers to communicate rate limits on retrieval to its users. It may also be only limited to indexers in the same federation (instead of public access)

currently this is the behavior described in https://github.com/ipni/specs/blob/main/IPNI.md#http-1

we probably want to express a specific implicit path we might want as a root for the advertisement chain over http.

This would be a more specific doc targeted at defining what a provider would need to do to be considered an 'http index provider'

The response structure used by IPNI for GET /multihash/<multihash> is optimised for batch find: it includes the multihash that is being looked up as base64 encoded string, along with a nested array of results.

I believe the rationale has been to use the same response model for both batch find, and single find request. For non-streaming responses this is fine; however, for responses streamted using new-line delimited json (i.e. ndjson) it becomes suboptimal:

• in ndjson each response needs to be a JSON object.
• because the results for a single mulithash are nested in an array, to stream the response using the current model one has to repeat the multihash that is being looked up in every streamed response.

This is less than optimum considering the vast majority of traffic received by cid.contact is for explicit multihash lookups.

Consider spec changes for a more optimal response structure for single multihash lookup, such that:

• the original multihash looked up via GET request is not repeated in the response or optionally included.
• the response model is preferably made up of a series of self contained JSON objects optimised for the streaming case, which hopefully becomes the default/preferred response media type when Accept request header permits it.

If all of the advertisements in the oldest segment of an advertisement chain no longer advertise valid content, then it may be reasonable for the provider to have a way to stop serving those advertisements. The reasons for doing so are:

• Reduce storage used for old outdated advertisements
• Prevent indexers from traversing empty end segment of chain

Even if all of the advertisements at the oldest end of the advertisement chain have been deleted by removal ads later in the chain, indexers will still traverse those advertisements looking for non-deleted advertisements. This incurs a chain traversal for each indexer that could be prevented if a chain truncation mechanism were available.

Chain truncation may also be wanted in cases where a lost/corrupt database results in older advertisement data being unavailable and unrecoverable, yet it is not desirable to start an entirely new chain. Indexing will not be able to proceed unless the unavailable end of the chain is understood by indexers to be truncated.

Possible truncation mechanisms:

• Publish a new advertisement with a new TruncateAt field that specifies where to end traversal of the ad chain.
• Return NotFound or new Truncated error when fetching advertisement after truncation point.

Requiring a new advertisement that specifies chain truncation is explicit, so cannot happen due to some error condition as interpreting an error can.