ambiretech / adex-supermarket Goto Github PK

This check should be present in:
https://github.com/AdExNetwork/adex-supermarket/blob/cc9a80c40b555662b64973efddd57ad8c5ba5bf3/src/cache/api_client.rs#L43-L55

Adapter changes: TODO add issue

The Ethereum Adapter has this validation here:

https://github.com/AdExNetwork/adex-validator-stack-rust/blob/dev/adapter/src/ethereum.rs#L162-L173

We need to be able to load statically a Configuration file in the binary.

I think there was a nice crate to do the whole deserialiazation into a stuct,

• Loading a configuration from file (dev & prod) in the binary #13
  https://github.com/AdExNetwork/adex-validator-stack-rust/blob/master/primitives/src/config.rs#L9

  • We can later add the option to the binary (clap) #13
  • Log the environment (dev/prod) and configuration #13

• Timeouts:

  • fetching new campaigns from the Market #13
  • updating the cache Campaigns statuses (the batch action) #13
  • Validator Request timeout #13
    If validator is offline or there is connection issue it shouldn't fail the whole Update action of Campaigns statuses

• recency (4 minutes default from https://github.com/AdExNetwork/adex-market/blob/eed1f7f0bfa89fb5e0cf7061f10b8a6492299c5c/cfg.js#L9) #13

• limitedIdentityEarningsLimit from https://github.com/AdExNetwork/adex-market/blob/eed1f7f0bfa89fb5e0cf7061f10b8a6492299c5c/cfg.js#L13 #13

• Fetch campaigns from Market every X secs/mins #13

• Update campaigns statuses (from validators) every X secs/mins #13

Enhancement from AmbireTech/adex-market#101

issues and/or stats: a list of possible reasons why there are no returned units: NO_ACTIVE_CAMPAIGNS, CAMPAIGNS_NOT_SOUND, NO_DEPOSITASSET_CAMPAIGNS, NO_UNITS_FOR_SIZE, NO_UNITS_FOR_TARGETING, NO_UNITS_FOR_ADSLOTRULES, SLOT_NOT_VERIFIED (if acceptedReferrers.length == 0)

Performance

Apart from improving the performance of requests to the Market (see Issue #41 and PR #42) we also need to make REST API requests only if we don't have a cached version of the AdSlot / AdUnits.
This is quite obvious from the experiments I've made with the Supermarket, showing that most of the time when calling /units-for-slot is actually calling a Market API endpoint and waiting for a response. Although we use TCP / HTTP Keep alive feature (for the hyper used in the proxy client and the API client reqwest) it is not enough to improve the actual response time of the Market endpoint itself.

Improvements:

This is why we need to cache (in memory) the AdSlots & AdUnits for specific time period each, i.e. configurable

• Cache AdSlots (from Market)
• Cache AdUnits (from Market)
• Configurable caching times for AdSlot & AdUnit before invalidating old record and fetching them again.

Performance results:

Examples from the performance review of the /units-for-slot endpoint:

Supermarket is just started, no cache or connections exist

mode: Release
Url: http://localhost:3000/units-for-slot/QmSx6jMjm9K1XXgv6AphBqe2wRFn9uu2Y3rn9pTu3DMr8f
Fallback unit: None
HTTP Cache-Control: no-cache
Total response time: 601 ms
Market API endpoints time: 589 ms

Jan 11 12:49:07.369 INFO Fetched AdSlot, AdSlot: QmSx6jMjm9K1XXgv6AphBqe2wRFn9uu2Y3rn9pTu3DMr8f
Jan 11 12:49:07.369 INFO Fetching AdSlot from Market took 307 ms
Jan 11 12:49:07.651 INFO Fetching ALL AdUnits from the Market took 282 ms
Jan 11 12:49:07.652 INFO Fetching Fallback AdUnit from the Market took 0 ms
Jan 11 12:49:07.652 INFO Fetched 366 AdUnits for AdSlot, AdSlot: QmSx6jMjm9K1XXgv6AphBqe2wRFn9uu2Y3rn9pTu3DMr8f
Jan 11 12:49:07.654 INFO Retrieving campaigns from Cache took 0 ms
Jan 11 12:49:07.655 INFO Preparing and Applying targeting rules took 2 ms

Supermarket after a couple of requests on the same route

I suspect this is thanks to Cloudflare caching, the cache itself doesn't stay for long, after > 30 seconds the request time increases again.

mode: Release
Url: http://localhost:3000/units-for-slot/QmSx6jMjm9K1XXgv6AphBqe2wRFn9uu2Y3rn9pTu3DMr8f
Fallback unit: None
HTTP Cache-Control: no-cache
Total response time: 313 ms
Market API endpoints time: 302 ms

Jan 11 12:54:09.933 INFO Fetching AdSlot from Market took 13 ms
Jan 11 12:54:10.221 INFO Fetching ALL AdUnits from the Market took 288 ms
Jan 11 12:54:10.223 INFO Fetching Fallback AdUnit from the Market took 1 ms
Jan 11 12:54:10.223 INFO Fetched 366 AdUnits for AdSlot, AdSlot: QmSx6jMjm9K1XXgv6AphBqe2wRFn9uu2Y3rn9pTu3DMr8f
Jan 11 12:54:10.224 INFO Retrieving campaigns from Cache took 0 ms
Jan 11 12:54:10.226 INFO Preparing and Applying targeting rules took 0 ms

Resolves AmbireTech/adex-market#5

• #6 enum Status
• #6 fn get_status
  • test
• #6 fn is_finalized
  • #10 test
• #7 fn is_date_recent + testing @simzzz
• #7 fn is_initializing + testing @simzzz
• #6 fn is_offline
  • #10 test
• #7 fn is_disconnected + testing @simzzz
• #17 fn is_rejected_state + testing @simzzz
• #17 fn is_unhealthy + testing @simzzz
• #20 fn is_active + testing @simzzz
• #20 fn is_ready + testing @simzzz

The Supermarket is quite fast handling logic, however requesting resources from the market on the /units-for-slot route has a significantly bad performance.
Requesting the AdSlot and AdUnits takes 90% of the processing of the /units-for-slot route, which makes it slower than the Market's /units-for-slot route.

Check these Github actions:

• https://github.com/marketplace/actions/create-a-release
• https://github.com/marketplace/actions/upload-a-release-asset

Make sure that the Market and Supermarket implementations of /units-for-slot are aligned and return the same results.

Maybe we could do the following:

• Start the Market server (with the AdUnits & AdSlots that will be used)
• Setup MockCache with the same and the correct Market server url that was started

TODOs:

• Make sure this comparison is repeatable (maybe add steps, commands, gist, etc to this issue)
• Seed market and supermarket with the same data
• Compare matching with the rules Channels
• Compare non-matching with the rules Channels

https://github.com/AdExNetwork/aips/issues/61

supermarket will retrieve only campaigns where the configured validators are leaders (/campaigns?isLeader=true)

This call should be towards the AdEx Platform once it's moved to the adex-platform repository AmbireTech/adex-platform#585

Since the Supermarket is very dependant on the Market to retrieve the campaigns, it makes the Supermarket work quite obsolete (apart from safety & performance).
This is why we want to make the changes to decouple it from the Market and instead fetch the campaigns from the validators in a configured list .

Changes:

1. PR #24 Configuration list of validators to fetch the campaigns from (instead of the market)
2. PR #24 Fetch the active (channel.valid_until > Now) campaigns (and compute their status)

TODOs:

• Check ChannelId when Channel is fetched from Validator: This is not really required since the get_status will check both validators and if they don't agree a bad status will be returned
  https://github.com/AdExNetwork/adex-validator-stack-rust/blob/dev/adapter/src/ethereum.rs#L162-L173
• Test Cache::update & Cache::fetch_new_campaigns (after initialization)

The flow stays the same:

• ✔️ On start-up initialize the Cache by querying each validator for Channels and compute the Status - repeat every couple of minutes
• ✔️ Compute the latest status by querying the Leader/Follower for latest messages - repeat every couple of seconds

Related to AmbireTech/adex-market#101

What needs to be done:

• a route /units-for-slot/${slotId} , which returns all ad units which match this slot; it will apply publisher earning limits and optionally apply targeting

Steps:

• PR #26 Applying publisher limits. We should check the publisher ID (owner property of the slot) whether he exceeds the publisher limits. This is done by comparing the earnings of the publisher with the limit (defined by the market). If it exceeds it, return an empty array for the slots and the earning limit.

  • #11 (PR #13) The limit is defined in the market cfg.js with the property limitedIdentityEarningsLimit. It is not exposed by the market. We can copy-paste both limitedIdentityEarningsLimit and limitedIdentityEnabled from the market's cfg.js

    • impled in with use of an Option

  • (PR #14) To get the publisher earnings we can use the cached data. (All campaigns including finalized)

    • Cache::get_earnings_for

• PR #14 First we will need to get the information about the Slot, other than the id. Luckily the market has a route for retrieving slot information. We need to call ${market_url}/slots/${slot_id}. This will return the slot object.

• PR #14 Next we need to find ad units which match that slot. First we need to get all units by calling ${market_url}/units/ and then filtering them to have the same type as our slot.

  • #15 (PR #22) NOTE: To optimise we can modify the market route to accept type as a query parameter which would be added to the mongo query.

• PR #26 Applying targeting. Optional but should always be applied unless there is a ?noTargeting query parameter provided. It should find matches between the unit targeting and the slot tags. More details on how to implement after the targeting overhaul

  • PR #14 ?noTargeting https://github.com/AdExNetwork/aips/issues/31 :

    NOTE: the supermarket no longer needs a special mode in which it doesn't apply targeting, since it will always apply targeting, but the rules which depend on sensitive (AdEx profile) data will simply be ignored, and they will be applied on the client side (AdView).

  • PR #26 Apply targeting ( https://github.com/AdExNetwork/aips/issues/31 )
    • Apply maximum N of campaigns you can earn from as a Publisher
      • PR #26 Add configuration for the maximum N(umber) of campaigns
      • PR #26 Limit the returned Campaigns to N

• Return the filtered ad units.

  • issues and/or stats: a list of possible reasons why there are no returned units: NO_ACTIVE_CAMPAIGNS, CAMPAIGNS_NOT_SOUND, NO_DEPOSITASSET_CAMPAIGNS, NO_UNITS_FOR_SIZE, NO_UNITS_FOR_TARGETING, NO_UNITS_FOR_ADSLOTRULES, SLOT_NOT_VERIFIED (if acceptedReferrers.length == 0) Moved to own issue for enhancement #32

TODOs:

• PR #26 Make sure that we are handling the GET params (for depositAsset) the same way as JS in order to have consistency. See AmbireTech/adex-adview-manager#67 (comment) for more details

Make it possible to configure the url of the market when executing the binary.

• Use clap to take the argument passed from the console (e.g. marketUrl: String)
• Use the value when building the proxy request to the market
• Use the value when querying the campaign from the market

Using clap for validator_worker's sentryUrl: https://github.com/AdExNetwork/adex-validator-stack-rust/blob/89a2c1ed7350870c95301c94d350d6c4a73830f2/validator_worker/src/main.rs#L61

More comprehensive details can be found here: AmbireTech/adex-market#101

• Fetch campaigns on start-up
• Query both validators (approve/newstate) #6
• Compute status of campaign #8

Part of: #9

Awaits: AmbireTech/adex-market#110

TODO: https://github.com/AdExNetwork/adex-supermarket/blob/c35a6bdb2424f23417c706c292d19924b42e93de/src/market.rs#L86-L101