ipni / heyfil Goto Github PK

It would be super useful if we have a list of the largest deal-making SPs that are not covered by IPNI in a given time period so we can focus our outreach/conversion to increase coverage.

heyfil in prod now returns the error

rpc call Filecoin.StateMinerInfo() on https://api.node.glif.io/ status code: 429. could not decode body to rpc response: json: unknown field \"message\"

we probably need an auth key to be able to access the api at the rate that heyfil is running at

Reduce noise in stats by filtering miners by state miner power similar to the approach in boostx stats here. This should reduce the total number of states a miner scanned by heyfil could be in and would produce statistics that are easier to digest by a human.

The current implementation uses the old uncompressed JSON dump from s3. The format by which data is now offered is changed to compressed ZST. Reflect changes in the logic so that we can handle S3 as an alternative source of state market deals metadata.

Compressed data available at: https://marketdeals.s3.amazonaws.com/StateMarketDeals.json.zst

Capture information on which multiaddrs SPs use to expose indexing or markets protocol.

This should also enable translating from the current multiaddr of a boost graphsync/http endpoint back to a sp ID

rather than just doing dealCount - it would be interesting if we could try correlate on contextID since we know filecoin providers will put in their deal ID as the context ID

Originally posted by @willscott in #6 (comment)

The filecoin API client library functionality is consolidated at https://github.com/masih/telefil. Refactor code here to use that repo instead.

As part of offloading functionality from index-observer report FileCoin deal coverage as: the percentage of deals covered by the indexer across FileCoin network.

When a SP supports /fil/retrieval/transports/1.0.0, query it and capture information on the transport protocols that it offers.

This would then expose a set of other peer IDs that a SP might be using which should also be captured.

See:
https://github.com/filecoin-project/boost/blob/main/cmd/boostx/stats_cmd.go#L159-L168

Since Heyfil already reaches out to all SPs and lists their protocol gather stats on which version of Boost they are running.

it would be great to the the list of sp's

/sp

but only the ones in a given status (e.g. with working markets)

would be great to understand the filecoin providers to ipni broken out by:

• software (boost vs venus markets)
• version

Periodically compare the list of OK filecoin provider IDs with the list of total known providers from an indexer and report the number of IDs that are not in the filecoin provider list.