Example from the spec repo: https://github.com/LibertyDSNP/spec/blob/main/.github/PULL_REQUEST_TEMPLATE.md

We need to add a Checklist section (example):

# Checklist
- [ ] Chain spec updated
- [ ] Updated the js/rpc code if a custom RPC added/changed 
- [ ] Design doc(s) updated
- [ ] Tests added
- [ ] Benchmarks added
- [ ] Weights updated
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have performed a self-review of my own code
- [ ] [Others?]

Perhaps add a section to the top asking for related design docs?

WW: to check to see if we can make sure we have all those things checked off before merging

DEPENDENT: #28

As a User, I want to know how much my schema is going to cost.

Acceptance Criteria

• RPC: schema_estimator
  • params: same as register_schema
  • Response: Tokens it would cost to register the schema (not counting tips)

As a consumer of the network, I want to be able to

• Retrieve all messages on chain
• for a given schema
• within a block range
• and be able to page the results

Blocked by #27

Description

See "Read" section of Design Doc, implement endpoint with checks specified in the document

Acceptance Criteria

• Expose custom RPC that lets user retrieve the range of messages, given:
  • starting block number
  • ending block number
  • schema id
  • page
  • page size
• tests for same

As a user, I want to create a Message Source Account so I can send messages without holding a token balance.

Description

Creating an MSA account involves generating a u64 id (MSAIdentifier)

The account id should be stored in a Map<MSAIdentifier, MSA>.

The MSA struct can be empty, we will flesh it out in another story.

We will also need an id counter to keep track of the MSAIdentifier.

Note

• This story will involve creating a pallet to handle MSA related operations
• Eventually we may need a struct to track capacity
• Refer to DIP
• Talk to Enddy or Wil

Acceptance Criteria

• Expose an extrinsic that creates an MSA account
  • Params: key, public key of user
• Have a new record inserted into the Storage map
• Storage for MSA identifiers
• Counter for MSA identifiers
• Event for creating MSA MsaCreated
  • should contain MSAIdentifier
  • should contain key that created the MSAIdentifier
• Tests
• Basic Benchmarks

Blocked by #7

Notes:

• Creates a connection from one Static Id to another (directed)
• Sets permissions
  • Limit which message schema types are allowed by the delegation
  • How do we want to structure these permissions, data storage?
  • Other permissions?
• Must be "approved" by both sides
• Should consider and document a flow from the user's wallet

• Talk with Parity and get a version to use for now
• Initialize Parachain with Cumulus
• Initialize initial CI linting check
• Initialize initial CI build & test check
• Post merge: @wilwade will turn on requiring checks to pass before merge

As a user I want to create a new schema.

Check in on #27 for schema + message format, but not blocking

Acceptance Criteria

• I should be able to call register_schema to register a [Thrift|Avro|TBD] schema
• The schema should be assigned unique schema_id (sequential, u16)
• An active node should receive the event SchemaRegistered
• Tests
• Basic Benchmarks
• Generate Spec files

Extrinsic: register_schema
Cost: MRC Token, one time fee
Parameters:

• serialized_schema: The serialized thrift schema

Emits: SchemaRegistered: block_number, schema_id, serialized_schema

Assumptions

• Zero length schemas are malformed and handled by future validation stories (#29 )
• Maximum schema is determined by the maximum size of a block + it just costs more for larger schemas (byte length pricing on extrinsic parameters)

References

• #17

As a user, I want to be able to get the details about a given schema id.

Acceptance Criteria

• RPC: get_schema
  • Might be custom or built in. tbd in implementation
  • param: schema_id
  • reutrns: schema_id, serialized_schema, created_at (block number)
• Tests

As a developer I want to be able to run a local command to update the running chain's wasm.

Acceptance Criteria

• Documentation on how to do this to the Readme
• Add/update scripts to trigger the upgrade make upgrade-forkless-local see scripts/onboard.js for example call methods

Help

Reach out to @enddynayn and/or @shannonwells

Details:

• Infra will need to expose bootnodes related ports
• Define script command to onboard mrc to relay running in infra

Tests

• run tests to register a para_id on infra relay
• update mrc spec to point to para_id from above
• run onboard mrc to relay and ensure successful onboarding

Outcome

• Script that takes in:
  • endpoint (Relay chain)
  • private key (Alice's or Bob's)
  • Result: New chain registered and onboarded.
  • Output: JSON, Alice and/or Bob's private key, paraId

Blocked By #23

As a consumer of messages on the chain, I don't want to ever read a message that doesn't work with the declared schema.

Acceptance Criteria

• On chain messages validated against the provided schema_id, and is rejected if it fails
• Tests
• Update benchmarks

Notes:

• If this benchmarks to "too high" (< 500 per block possible) of a weight, bring it back to the team for discussion.

Is this a duplicate? (Perhaps this should be for the extrinsic for payment by a different account than new MSA owner?)

Description

There should be a payment for invoking the extrinsic that creates an MSA account.

Note

Payers should be able to call the payment extrinsic only after proving that they have enough balance to pay for the invocation.

Acceptance Criteria

• Place a minimum weight on the MSA creation extrinsic
• Calculate base cost of transaction via benchmarking
• Should be in Tokens (Capacity based is future)
• V2? Transaction costs should be covered by the submitter, but the MSA need not be connected to it
• V2? Must have a signature proving the private key holder intends to create an MSA

Description

The network should garbage collect messages from a deleted account.

Acceptance Criteria

• remove all messages relating to an msa account removal

Note

Discussion points:

• can a user decide when garbage collection happens when submitting deletion extrinsic
• processing cost and calculation weight appropriately

Can We:

• Have a method (without std) that checks that the data matches a given schema
  • If so, what is the cost?
• Have a method that deserializes Avro binary serialized messages?
  • If so, what is the cost?

As as user, I should be able to get the list of schemas that are currently available.

As schema_ids are sequential, I just need the maximum schema_id and then use #31 to get the details for each schema id.

Acceptance Criteria

• RPC?: get_last_schema_id
  • params: None
  • Returns: Last registered schema_id
• Tests

Notes:

• This is be an early version without other enhancements

• Token Accounts can stake any amount of tokens to any Message Source Account (MSA)
• Staking will give Capacity at a fixed ratio to the MSA beginning with the next Capacity Epoch.
  • There is NO way to gain more Capacity in the current Capacity Epoch
• Staked tokens may NOT be transferred
• Staked tokens may NOT alter the staking target MSA without first being unstaked
• Staked tokens have several pieces of information:
  • What Account Id staked the tokens (the owner of the tokens)
  • What ratio the tokens were staked at
  • What MSA Id the tokens were staked to
  • What block number tokens were staked at
• Staking tokens costs a standard transaction fee
• Staked tokens are staked until actively unstaked
• Unstaking
  • Triggering the unstaking of tokens costs a standard transaction fee
  • Unstaking will take a number of blocks
  • Users can unstake partial amounts
  • Unstaking will always result in the SAME number of tokens as were staked (less fees)
  • Unstaking takes a set number of blocks
    • Open Possibilities:
      • Unstaking could be gradual (10% of the n blocks to unstake and you have access to 10%)
      • Unstaking could be shorter for those who have staked for longer
      • Unstaking faster could be possible for a fee
      • Unstaking takes less time when you unstake only a portion
        • Unstake only 10%, and access it in 10% of the time
  • Tokens being unstaked are locked
    • Non-transferable
    • Do not provide Capacity
  • Open Questions
    • Can I stake 100% of my holdings?
      • Aka can you stake the existential deposit?
      • Aka can unstaking fees be paid for with unstaked tokens?
    • Should we have value in being staked longer?
• Open Ideas/Questions:
  • Bonus capacity for minimum lock lengths
    • No unstake possible for 2 years
  • Path for altering staking target without having to wait for unstake to complete
    • Users who just want to switch who they are staking to
  • How do staked tokens interact with voting in governance?
    • Or perhaps only staked tokens matter for governance?
    • Does the MSA get the votes or the Token Account?
  • Could staking produce a network reward?

Description

Users should be able revoke public keys from an MSA they are associated with.

Acceptance Criteria

• An extrinsic that deactivates the association between a given public key and an MSA
  • Params: sender origin, a key that has already been associated with the MSA account
  • Params: key, the key that should be removed
  • Payment in Tokens
• Re-adding a revoked key to this MSA should fail
• Adding a revoked key to a different MSA should fail
• Revoking the last remaining key should fail (#44 covers this)
• Submit event to revoke key msa and key information
• validate ownership of key
• Should store the block it was revoked in (Need for listing in #38)

Note

Removing the last key is handled in a different story and part of removal of MSA account.
Revoking all keys means an account is marked deleted or abandoned.
Removing the last key requires a deletion extrinsic to be submitted.
This protects the user from unintentionally removing all keys and
makes it clear to the user that the account is being deleted.

Might need to reference #7

Blocked By https://github.com/Liberty30/polkadot-experiments/issues/53

Notes:

• Starting with the gas version of this that can be converted to capacity later
• On chain messages should have some required schema
• Someway to do extended custom schema
• Message Schemas should be able to be created, but not modified? Perhaps versioned?
• Could schemas have validation rules?
• Must be validated as FROM a specific static id

Notes

• A reference Id for multiple (changing) Substrate Accounts
• Used to reference "users" over key pairs
• Should have a way to add and remove Substrate Accounts
• Needs to retain memory of what accounts used to be approved (Maybe)
• A Substrate Account MUST only map to ONE Static Id

References: https://github.com/LibertyDSNP/meta/blob/main/DESIGN_DOCS.md

Potential Pairs based on Implementation

• @enddynayn
• @saraswatpuneet
• For design doc related questions: @shannonwells / @raebot

Blocked #67

As a consumer of messages on the chain, I don't want to ever read a message that doesn't work with the declared schema.

We need to validate schema when adding a message

Use SchemaValidity struct to store schema validity state.

Acceptance Criteria:

• use schema pallet via loosely coupling
• validate schema in add extrinsic and reject if it fails
• modify tests
• modify benchmarks
• generate new spec

DEPENDENT: #28

As the network, I don't want unused schemas sticking around.

Suggested solution: Schemas have an end block number that can be extended by anyone (different story).

Acceptance Criteria

• When registering a schema, the user can set the schema_block_count (db stored as the block number when the schema deactivates)
• The price per block is a constant
  • Default to 1 token per 5,256,000 blocks over 100,000 blocks (for now)
• If a schema deactivates, no one should be able to send a message with that schema
• get_schema should return the deactivation_block_number

Notes:

• Capacity usage and refill system
• Does NOT include anything about how to get Capacity outside of a sudo call for now (That is [tbd] design doc)
• Should include how to handle the balance adjustment at the block level

Early notes: https://docs.google.com/document/d/1-OKRZuoWmpuFzNcaft5nQ4VCCfsWofBeCIn_a3_j3rU/edit?usp=sharing

Description

Users should be able to create an association between their public key and an MSA account.

Note

MSA accounts can have associations with multiple keys. However, all of those keys should belong to the user that created the MSA.

Acceptance Criteria

• Storage for associations between keys and MSAs (Map?)
• Ability to query for these associations by public key
• Users should not be able to create an association between a key and two MSAs
• Event for creating an association between key and MSA

When using a Mock Object for loosely bounded pallets then benchmarks are not setup correctly and can not execute the mock object from Test context.

We need to find a way to fix this issue.
Example:
If you run benchmarks on messages it will error out since it can not detect the message source account due to the Test mock not being maybe accessible from benchmark?

As a indexer I want to know if an MSA is delegated to another MSA

Acceptance Criteria

• RPC Call check_delegation
• Param: delegator_msa_ids
• Param: provider_msa_id
• Returns Object<delegator_msa_id => boolean>
• Tests

Example PR for a RPC: #96

Acceptance Criteria

• Update the Github action that produces the wasm to save it as an artifact for mainnet (save only 12 weeks)

Related to #63

"Deletion in a Decentralized World"

Basic points:

• Forgetting is hard to do
• Deleting in a centralized world isn't real either, it just pretends. (screenshots, etc..)
• Existence is hard to erase
• Content is easy for one service to erase, but there are lots of caches
• Deletion is always a request
• Most deletions should be respected, but perhaps not all (politicians, public figures, etc...)

TODO: Add link to spec meeting docs where this discussion took place

Notes: How about a blog post on this subject

DEPENDENT: #28

As governance, I should be able to adjust the maximum size of a schema.

Acceptance Criteria

• An extrinsic that only can be called via sudo
• Alters the MAXIMUM_SCHEMA_SIZE
• Only effects future schema, not prior schema
• Schema longer than the set size should be rejected from registration

• Need Details related to which components are tied to what MRC fee like tokens/capacity etc.
• We will need a converter implementation to be able to translate weights to fee to calculate costs etc.

Determine min and max schema size, max schema registrations (all based on Vec size of the model)

• Min Schema size: it's an easier check to bail up front on a minimum size and we should. The SchemaZero byte length is a good starting point.
  • Set MinSchemaModelSizeBytes to 8 bytes across the board.

Discussion

• Max schema size: determined by max weight and also @enddynayn suggests this can be limited before the message even gets put in the mempool by restricting the weight for the extrinsic.
  • Currently 1024. Leave it
• Max Schema registrations: there was some debate about whether there should be a number below whatever the max of a U16 is. If the limit is just U16 max value then it would seem we don't need this config? Perhaps it would be better to leave some wiggle room though.

As a current user of Frequency, I want to be able to remove myself from Frequency so that I can remove my association with Frequency.

Description

Users should be able retire an MSA so that it cannot be used again.

Notes

• Garbage collection of messages will be addressed in a different story. #128
• Fee:No - this is handled by #340
• First attempt: #252

Acceptance Criteria

• The user should be able to call the extrinsic that retires an MSA from the polkadot js ui
• If the delegator does not have a MSA yet, the extrinsic should fail with NoKeyExists
• All delegations should be revoked when calling the RPC to get delegations by MSA
• I should not be able to call this unless there is only one key reference in MsaInfoOf and that is the key I am using to call from.
• An event should be emitted to notify about the retirement Suggested: MSARetired(msa_id)
• The KeyRemoved event should be triggered for the last key
• I should not be able to call this if the MSA is a registered provider
• Benchmarks
• Unit tests
• New transactions associated with the MSA must be rejected
  • FAIL: Add new key
  • FAIL: Add new delegation
  • FAIL: Modify delegation permissions
  • FAIL: Add Message

State Maps for Deletion

• MSA -> u8 (references) deleted MsaInfoOf
• Key -> MSA (Only one will be left) MessageSourceIdOf
• MSA -> Provider deleted ProviderInfoOf (Should have none left)

Work that doesn't need to be done

• ProviderInfoOf should already be emptied
• No new errors, as once MessageSourceIdOf deletion happens NoKeyExists should be triggered already.
• Pays::No is in #340
• Garbage collection of messages will be addressed in a different story. #128

Arch Notes

• Retire MSA extrinsic Signature: retire_msa(origin: OriginFor<T>) -> DispatchResult

Other Notes

• Does not need DispatchResultWithPostInfo as we enforce that all the keys but one and all delegations are revoked.

Blocked by #9, https://github.com/Liberty30/polkadot-experiments/issues/27

Notes:

• Batches messages might need to be handled differently than on chain messages in #9
• Or maybe not?
• The biggest difference is that these reference a batch file
• Do we need to have requirements that it be a parquet file?
• We should have requirements around where it can be stored (IPFS only for now? perhaps this is a different design doc?)
• Need to give the byte-length of the file for capacity transaction reasons as well as limiting super massive files or attacks

If a message schema is faulty or is to be deprecated, how do we do this?

Some thoughts --
Are there extra fields in schema registration storage that show if it's deprecated or invalidated? Is it deleted from schema registration storage? Either way there should almost certainly be an event deposited indicating what has been done to the schema.

If the schema is deleted from storage, then what happens to messages that were in the pool right beforehand which would then fail validation immediately afterward? Similarly what does this mean for message batches that are retrieved and checked for validation historically?

If the schema is not deleted from storage, then almost certainly the schema should be marked as no longer usable for current messages. This means there should also be a block number after which the schema is no longer valid, and this block number should be emitted with the event so that subscribers are alerted.

Perhaps there are multiple end states for a schema, such as:
Invalid -- it was never valid, so any messages using this schema can never be valid. Needed only for registering a schema by mistake or with errors (e.g. it's missing a needed field).
Deprecated -- Still valid but slated to be expired after a certain block number
Unsupported/Expired -- used only for messages that were valid at one time, the message is no longer valid if it was posted after a certain block number

Maybe we just need Valid, Invalid, and Expired? Do we need any other fields?

Related to #12

Motivation

We need to choose a format for storing Messages and Schemas, taking into account

• the need for validation
• efficient storage
• processing cost,
• compatibility with sp_std i.e. Substrate Rust
• need for wide support in multiple languages.
• need to support a single struct for storage of the schema like #28

For more detail see Serialization Concerns section of the Design Doc

Some candidates are Thrift, Protobuf, JSON, Avro, and BSON. So far Thrift is the winner. Unfortunately "When protocol buffers are serialized, the same data can have many different binary serializations. You cannot compare two messages for equality without fully parsing them."

As a user, I want a schema to continue being used longer than the deactivation_block_number.

Acceptance Criteria

• Extrinsic: increase_schema_active_block_count
  • param: block_count (number of blocks to add to the schema end block number)
  • Cost: Token fees + the cost per count as is a constant in #32
• get_schema should return the updated deactivation_block_number

As a user who wants to register a new schema, I need to be able to serialize the schema I want to register.

Acceptance Criteria

• Create a "SchemaZero" that can be used to serialize new schemas
• Add an RPC to fetch "SchemaZero" as Avro JSON Schema definition

Goal

1. Users write MRC Schemas in according to the Avro JSON Schema spec
2. Users serialize their MRC Schemas using SchemaZero
3. Users submit the data portion only of the serialization to register their new MRC Schema

Semi-working SchemaZero

{
  "type": "record",
  "name": "mrc.schema.root",
  "fields": [
    {
      "name": "name",
      "type": "string"
    },
    {
      "name": "type",
      "type": "string"
    },
    {
      "name": "fields",
      "type": {
        "type": "array",
        "items": {
          "type": "record",
          "name": "mrc.schema.child",
          "fields": [
            {
              "name": "name",
              "type": "string"
            },
            {
              "name": "type",
              "type": [
                "string",
                {
                  "type": "record",
                  "name": "mrc.schema.type",
                  "fields": [
                    {
                      "name": "type",
                      "type": "string"
                    },
                    {
                      "name": "items",
                      "type": [
                        "string"
                      ]
                    }
                  ]
                },
                {
                  "type": "array",
                  "items": [
                    "string",
                    "mrc.schema.type"
                  ]
                }
              ]
            },
            {
              "name": "fields",
              "type": [
                "null",
                {
                  "type": "array",
                  "items": "mrc.schema.child"
                }
              ],
              "default": null
            }
          ]
        }
      }
    }
  ]
}

• decouple fee computation in its own pallet to provide room for future updates and adjustments
• Extend transaction payment with base implementation as well as rpc endpoint to compute mrc extrinsic cost
• Provide an endpoint to run schema validation checks #67

Description

There needs to be a record of what public keys are associate with an MSA.

Note

• This will involve a 1:M mapping between MSA ids and public keys.
• Currently there is no storage for directly going from an MSA id to keys.

Acceptance Criteria

• Users should be able to fetch a list of keys given an MSA identifier
  • There should be logic to keep both the map created by #22 and the map above up to date on every insert and delete.
• RPC: get_msa_keys
  • Param: msa_id
  • Returns: list of public keys and when (if) they were deactivated

As a best practice in Substrate we need to replace all Vec used in storage with Boundedvec

A new problem not seen before caused to split this into a new issue.

Acceptance Critera:

• Replace all Vec used in Messages pallet with BoundedVec

NOTE: There is a version of this in a PR from @aramikm, but it is one possible version: #83

As a node operator, so that I minimize my costs, I want to remove old messages from on-chain storage.

Description

See "Cleanup (Retention Policy)" section of Design Doc, implement garbage collection as specified.

Acceptance Criteria

• Implement garbage collection based on retention period
• tests for same
• Benchmarks will probably need to be updated

DEPENDENT: #67

As a user, I should NOT be able to register an malformed schema

Acceptance Criteria

• Schema should "compile/lint/check" in the registration extrinsic

Moved to other stories

• moved to #66 Zero length schemas are malformed
• moved to #66 Length of > [GOVERNANCE VALUE] should be rejected

BLOCKED BY: #64

Goals:

• Jenkis job to deploy/onboard new parachain (using scripts from #64 )
  • Take down the old parachain servers
• Expose rpc, ws ports for the same to connect localy parachain at developers end
• Publish to the team (Jenkins):
  • the Alice and Bob private keys for the parachain
  • rpc endpoint
  • paraId

Test

Future

• Github action/ Jenkins job to deploy relay chain (Future)
• How many validators we would need if devs want to create a local parachain, n+1 n+2 etc. most needed for devnet
• able to trigger job and deploy relay validators (Future)

We talked about various delegation permissions and now it is time to actually have some.

Output: A design doc that covers what Delegation Permissions is, why, and a version of how.

Ideas:

• Account recovery (Likely a different design doc)
• Limited Schema Ids (you can only send schemas 2 and 7 on my behalf)
• Time limited (after n blocks you must request re-authorization)
• Terms of Service? Permissions includes a hash of the ToS. This locks the ToS unless there is a re-delegation.
  • Chain/schema options for ToS instead of app specific

Terms:

• Account/User: The MSA of a user
• Provider: The MSA of the provider/app
• Permission: The generic option for any user
• Grant: The user level action/result. "A user grants a permission to a provider"

Things We are Sure About

• User MUST opt-in to each permission
• Grant modification is a dual opt-in (User and Provider)
• Version 1:
  • Per schema grants by users for Creation
  • Revocation of a delegate ends all grants
  • Revocation of a grant should be "pre-paid" (Not staking it, just paying more for the create so the revoke is "covered")
  • All user account and delegation modification are token OR capacity

Big Open Questions

• Public "Read" permission (DSNP or MRC?)
  • Future
• Do we track "use" of an app separate from write permissions
  • No

Discussion Notes from 2022-05-25

Recording: https://drive.google.com/file/d/1wgsqx5KWTj2_K_NpCk00Ykk53P_GN8gX/view?usp=sharing

• Allowing a site to read my private data

  • Not posting on my behalf, reading my (private) data instead of writing data
  • Key management for reading private data

• Delegation for reading vs writing

• Do we need to have a way for the user to express how their data is used on "3rd party" sites (aka sites you don't have an explicit delegation with)

  • Maybe not MRC, but perhaps at the DSNP layer?
  • DSNP data use policy for each announcement (We already assume friends/followers can see it even on 3rd party sites)

• Do we need extendable/modular permissions or "roles"?

  • Grouping Schemas together somehow?
  • Roles are useful for users, but hide complexity (see Android/iOS permission mess)
  • What if there are 100's of schemas?
  • Could the wallet solve that problem?

• Delegation is a wallet level action

  • Users have a wallet interaction for delegation

• Do schemas have custom permissions?

  • Sub-permissions CRUD? We really only have Create as a permission

• Could service permissions be set at the provider MSA level and the delegation only have exclusions?

  • This is more auto-optin which I think is bad.

• Public "Read" permission (DSNP or MRC?)

  • Robots.txt
  • Could this be on a per service basis?
    • aka I grant the public ability to "read" schema 7 from service A but not schema 7 from service B

• Core delegation model ideas

  • Limited Schema Ids (User allows schema: x, y, z)
    • Want to send a message -> Check delegate -> check schema "permission"
    • Delegation permissions are "stored with" the delegation on-chain
      • This data must be historically immutable and retained
    • Active opt-in

Idea: What if MRC had DAOs to manage roles and schemas and collections of schemas.

We need to use Msa pallet to get Msa account from origin

Acceptance criteria

• Use Msa pallet via loosely coupling
• getting Msa account from origin and use it in message creation (add extrinsic)
• update tests
• update benchmarks
• generate spec

Description

So that I can Authorize multiple accounts to use my MSA, I want to be able to add additional keys to my MSA.
There are multiple use cases for this:

• authorize multiple devices
• authorize someone else to use my MSA

[pending note of why proof of ownership is required]

Acceptance Criteria

• The same key cannot be added to more than one MSA account
• Require proof of ownership of public key.

Extrinsic Interface

• add_key
  • Params: origin, a key that has already been associated with the MSA account
  • Params:
    signedPayload
    • data
      • msa_id
      • nonce
    • key
    • signature

Network consumer can write messages to chain

As a consumer of the network, I want to be able to

• Store a message on chain
• Using a previously registered schema
• And know that it was stored
• And know that I could, and how I would access it

Blocked by #28

Description

See "Write" section of Design Doc

Acceptance Criteria

• Expose an extrinsic that lets user submit the message to be stored
• The message schema_id must exist (validation against the schema is #41)
• The extrinsic call emits event with sufficient info to determine success (this is going to be implemented in on_finalize or on_initialize per schema_Id. We are not going to emit an event per message since that not scalable.)
• tests for same
• benchmarks

Not required

• transaction fee for posting message (depends on capacity and can be parallelized in another story)

Design doc around pruning historical state

Basics

• While people could run archive nodes, we want to make it such that there is limited reason to do so.
• Why? Cost of running nodes. For comparison, Ethereum Archive node is at >10TB while a full node is < 1TB.
• Not forcing, but not required to read state (and any historical state could still be validated)
• Outcomes:
  • We store more information in state than otherwise need to, but total storage is lower.
  • Lower gossip data requests. Effectively full nodes have all the data you could need.
  • Schema Messages don't have to be permanent
• Trust model changes
  • Ethereum you can trust (like most people) the consensus, but you could also do your own validation and run an archive node.
  • MRC still offers that choice, but only from (about) the point in time that you started archiving. If you didn't start early, then you might have to trust that the consensus had the correct state at the point you start archiving.

This is an issue to keep track of ideas around improvements and optimizations that we might want to do, but getting to alpha first is priority.

General

• We should consider removing all the events and just do off-chain websocket event filter registrations.
• Save block space by returning a more accurate weight when hitting errors #812

Schema Related

• We should have a document that helps determine what sort of options a schema should have. We currently have a TTL, and "batch" option. I can imagine that this could feature creep quickly without any guide as to what we want to have happen.
• Should schemas be incremental or be a hash of the schema? Or some version of a hash of the schema? This means larger schema references however which is bad since there is a lot of that.
• What if there were a way to do schema discovery? Say I know a certain public key I can look up schemas published by it using a "name"? #1693

Message Storage Related

• Some messages we only want to store the most recent version of. Could (should) a schema be setup to trigger a TTL for prior messages based on a key? (Example, DSNP: GraphChange with the key being the target of the Graph Change)
• Should we allow a user to target a specific message for complete removal? (This breaks immutability which has issues. Perhaps this is a schema option?)

Message Retrieval Related

• Sometimes I only want messages from a specific user, could something help make that possible? (Perhaps an off-chain worker)
• I want the Custom RPC to validate schema before it passes messages to me so that all messages I get from the custom RPC are known to be valid.

Batch Related

• Could the node retrieve the batch for me via ipfs?
• Could the node store and pin the batch for me via ipfs?

Parallelization

• Separate out the messages into a separate state than the other active state
• Shard the message storage based on schemas
• Process messages in parallel after assigning slots and checking capacity
• Group by origin's MSA and then process submissions in parallel

CI

• Reduce build jobs run times from current 1.5-2h (cache, better hardware, parallelization,...)
• Review how Parity does their builds right now and borrow some good techniques from them like rusty-cachier
• Consider organizing each job around SRP and being as independent as possible for better parallelization (recommend doing it in a diagram first before changing any code).
• Consider moving jobs which don't have to run frequently to a scheduled jobs (ex. cargo audit
• Run a quick benchmark on PRs automatically before kicking off the real ones