schemas: Add for `FCMaterials` data about eddn HOT 14 CLOSED

We also need to be 100% sure that the CAPI data is there generally, and not only for the owner of the FC.

I'd say it is. My bubble commander doesn't have a carrier and got that data from 3 different carriers (I only tried those 3).

from eddn.

We also need to be 100% sure that the CAPI data is there generally, and not only for the owner of the FC.

from eddn.

As requested.

market.zip

from eddn.

I've had a different idea about Journal vs CAPI.

Rather than wanting to be 99.9% sure of the differences in format, and what data might be in one format and not the other...

... why not just two separate schemas?

Other than PII removals both can then just be "this envelope, plus what's in the file (Journal) or CAPI data".

So, fcmaterials_journal-v1.json and fcmaterials_capi-v1.json.

Especially for the CAPI I have low confidence that Frontier won't suddenly change something about it without pre-notifying us. Keeping it simple, and separate, means that Senders won't need to adjust and Listeners can be as nimble as they feel the need to be.

from eddn.

With not transforming either format to the other we will end up with e.g. EDMC doing

1. Player docks at an FC, CAPI queries made, fcmaterials_capi/1 message sent.
2. Player then disembarks and visits the bartender resulting in a possibly 'duplicate' fcmaterials_journal/1 message (but then no more duplicates unless the journal data has changed).
3. In EDMC's case if the player presses 'Update' after docking, and the CAPI-sourced data isn't changed there would also be no duplicate message.

from eddn.

I agree with the above.

But we can have one schema, just have it have an object with the data in, named either "CAPI" or "Journal".

from eddn.

I agree with the above.

But we can have one schema, just have it have an object with the data in, named either "CAPI" or "Journal".

1. I'd like to hear the preference of the EDDN Listeners.
2. It'll complicate tying the schema down, if that seems the way to go, with having two different formats. Much easier to do that with separate schemas.

For EDMC I'm happy to support two different schemas, it goes through different code paths anyway.

from eddn.

After changing the in-development schema to allow for CAPI data as well a bit of a discussion ensued on Discord as to if we should have a single schema allowing both data sources, or a separate schema for each.

Whilst I was convinced enough to try a single schema, I personally still think separate schemas keep things simpler for both Senders and Listeners. I could easily see some Listeners switching on $schemaRef and expecting to be able to "just go" from that point, rather than needing to check things like "what does data-source say?" or "Is this Items an array or a dict object?".

@clonedArtie appears to be Team One Schema To Rule Them All
@spansh seems to prefer One Schema For Each Source

Any other comments from the likes of @robbyxp1 @AnthorNet @Tkael @themroc5 ?

You can in effect see examples of the data format differences in https://edgalaxydata.space/EDDN/dev/Unknown-2022-08-31.jsonl .

Array (Journal) versus dictionary (CAPI, and then it contains two keys, the value of which can either be an empty array, or a non-empty dict), different cased key names, different form of the (symbolic) name, Journal has Demand/Stock when CAPI only has one (due to that split into two keys).

Listeners are very much going to want to check the data source (be that key or schema name) and then use different code.

No, Senders won't be trying to transform one into the other, that runs into issues if there are changes, whereas each Listener can be much more nimble than waiting for "1. Sender dev codes changes, 2. Sender dev releases new version, 3. Sender users upgrade to new version", which might also call for Listener changes as well.

from eddn.

So,

message": {"CarrierID": "X3F-N5Z", "CarrierName": "WARD'S OLOGIES", "Items":
[{"Demand": 0, "Name": "$compactlibrary_name;", "Price": 10000, "Stock": 0,
"id": 128962598}, {"Demand": 0, "Name": "$aerogel_name;", "Price": 500,
"Stock": 0, "id": 128961524}, {"Demand": 0, "Name": "$accidentlogs_name;",
"Price": 4000, "Stock": 0, "id": 128972282}], "MarketID": 3706433792,
"data-source": "Journal", "event": "FCMaterials", "horizons": true, "odyssey":
true, "timestamp": "2022-08-31T13:43:55Z"}}

Comments: Items is a direct copy of the .json, fine.

"message": {"CarrierID": "X3F-N5Z", "Items": {"purchases": [], "sales":
{"128961524": {"id": 128961524, "name": "aerogel", "price": 500, "stock": 0},
"128962598": {"id": 128962598, "name": "compactlibrary", "price": 10000,
"stock": 0}, "128972282": {"id": 128972282, "name": "accidentlogs", "price":
4000, "stock": 0}}}, "MarketID": 3706433792, "data-source": "CAPI", "event":
"FCMaterials", "timestamp": "2022-08-31T14:54:40Z"}}

Comments: names are different, not a problem, but where is odyssey and horizons flags? And lacking CarrierName but I guess this is CAPI.

But then later you had:

"message": {"CarrierID": "X3F-N5Z", "Items": {"purchases": [], "sales":
{"128961524": {"id": 128961524, "name": "aerogel", "price": 500, "stock": 0},
"128962598": {"id": 128962598, "name": "compactlibrary", "price": 10000,
"stock": 0}, "128972282": {"id": 128972282, "name": "accidentlogs", "price":
4000, "stock": 0}}}, "MarketID": 3706433792, "data-source": "CAPI", "event":
"FCMaterials", "horizons": false, "odyssey": true, "timestamp":
"2022-08-31T15:03:47Z"}}

with them back in.

I'm happy its one schema, and i'm happy there are not complicated name translations at the client end to do.

from eddn.

Yes, as I noted on Discord, I initially failed to actually set the horizons and odyssey flags on my CAPI-sourced messages, but later fixed that and at least one went through in that format.

The current fcmaterials/1 in #195 is the result of me experimenting yesterday with what was necessary and possible with a single schema for both data sources. It should absolutely not be taken as the canon of how the schema(s) will end up on the live service. I didn't update the README for yesterday's work.

I'm most interested in hearing from all the major EDDN Listeners now. If they've not spoken up by about this time tomorrow then I'll make the decision myself, which is currently leaning towards:

1. Two schemas, one for each data source.
2. Lock them down as much as current knowledge allows. Despite my worries about Frontier changing things without warning... I can't recall the last time they actually did so in a way that caused EDDN messages to be rejected by the Gateway. Thus I'd much rather lock things down so we have fewer bad-Sender messages make it to Listeners.
3. No augmentations where they're un-necessary (i.e. not leveraging a CarrierName into the CAPI data).
4. I do prefer keeping the Journal names for keys however. So CAPI will have id -> MarketID, and name -> CarrierID. Yes, the CAPI data has name for what is actually the ID of a Fleet Carrier.

from eddn.

That 24 hours notice is now up. I've started work to use a schema per data source for this. This includes updating some documentation around that for the future.

from eddn.

The PR is now updated for separate schemas, including the _capi README being in first draft form.

These schemas are now up on dev.eddn.edcd.io for testing.

from eddn.

One last round of end-to-end testing just now. Results are in https://edgalaxydata.space/EDDN/dev/Unknown-2022-09-02.jsonl from "gatewayTimestamp": "2022-09-02T15:44:17.357422Z" to "gatewayTimestamp": "2022-09-02T16:01:30.891159Z"`.

This covered:

1. No orders of either type.
2. Things being bought: single item, multiple items different counts, the buy orders having been fulfilled and either still there or now gone.
3. Things being sold: single item, multiple items different counts, having been bought but still listed, no longer listed.

My focus was on the CAPI side of things, but having to be at the bar tender to buy/sell meant I also triggered the Journal side for most of this.

All passed through without issues in the EDMC branch, and no rejections/errors on the dev.eddn.edcd.io side.

So, unless anyone can spot a problem by 12:00 UTC tomorrow, I'll merge this and move towards putting it up on the live service.

from eddn.

The schemas are now live.

from eddn.