jrwashburn / mina-pool-payout Goto Github PK

firerydev reported failures to payout for two addresses using mina explorer datasource. Issue identified 43 transactions that have null coinbase receiver, which can occur when the coinbase is 0 for the block. The null is handled for archived, but not mina explorer data. (Fails at

TypeError: Cannot read property 'publicKey' of null at /Users/jrwashburn/mina/mina-pool-payout/src/core/dataProvider/dataprovider-minaexplorer/block-queries-gql.ts:125:73 at Array.map (<anonymous>) at Object.getBlocks (/Users/jrwashburn/mina/mina-pool-payout/src/core/dataProvider/dataprovider-minaexplorer/block-queries-gql.ts:115:17) at processTicksAndRejections (node:internal/process/task_queues:96:5) at MinaExplorerBlockDataProvider.getBlocks (/Users/jrwashburn/mina/mina-pool-payout/src/core/dataProvider/MinaExplorerBlockDataProvider.ts:13:16) at PaymentBuilder.build (/Users/jrwashburn/mina/mina-pool-payout/src/core/payment/PaymentBuilder.ts:46:32) at PaymentProcessor.run (/Users/jrwashburn/mina/mina-pool-payout/src/core/payment/PaymentProcessor.ts:39:34)

Many pools have specifically negotiated fees for larger delegators. Add support to provide key-specific fees to override the pool default. The override rate would be applied specifically to payouts for that address

Add another configuration file (perhaps .negotiatedFees) to provide public key and override rate in decimal:

B62qkBqSkXgkirtU3n8HJ9YgwHh3vUD6kGJ5ZRkQYGNPeL5xYL2tL1L|0.012573

Anticipate other fee structures / special deals in the future - extracting fee calculation to make it pluggable is a good idea. That would allow forks to do other things like first block is free, or first epoch free for new delegators, or ... etc.

It will be good to add actual mina amount with nanomina

Wonder if there are any plans to implement logic to take into account that MIN_HEIGHT + MAX_HEIGHT might span multiple ledgers and therefore staking/rewards number may vary.

Example:

MIN_HEIGHT = 100
MAX_HEIGHT = 1000

If blocks 100 - 500 belongs to one staking ledger where X amount is delegated to a pool

Let's say delegator increases/decreases their delegation amount which is reflected in the next staking ledger and the transition from the previous staking ledger to the next occurred at block 501. Then rewards calculated for blocks 100-500 would differ from those at 501-1000.

Unless I am mistaking something here

after calculating the transaction total, output the key variables for a ledger offline send to fund the payor wallet

We need to make several assumptions about the preferred runtime environment, balancing ease of operations and security. With online access to a running node and keys, we could fully automate the pipeline for pool payouts including signing and payments. On the opposite end to support fully offline signing, there may be 2-3 steps involved with sneakernet, etc. Middle ground approaches may include private network communications; however, we tend to think those are nearly as vulnerable as fully online access.

We'd like to get feedback from the pool operator community on the proposed process overall, feedback on how you would like to see the dependencies managed (e.g. data sources to support, etc.,) and specifically your current thinking and plans for on-line vs. offline node for transaction generation and signing,

Dependencies that the pool payout algorithm requires include:

1. staking ledger - currently expecting a json file from:
   coda ledger export staking-epoch-ledger

2. postgres mina-archive database - read access to the following tables in an archive db: public_keys, blocks, internal_commands, blocks_internal_commands, user_commands, and blocks_user_commands.

3. storage - write access to some storage option - database or files TBD. The process needs to write out the payout calculations and track the last block processed. We are leaning toward json files for this.

Downstream dependencies:

4. payout signing - after calculating the payouts, transactions need to be generated and signed. We assume this will be an offline operation to support ledger signers, etc. This is probably the most important decision to make re: online vs. offline requirements. This requires access to keys in some form.

5. payout broadcast - after generating and signing transactions, they need to be broadcast to the network. This requires an online node.

Proposed pipeline:

1. Pool operators are expected to provide .json files for the staking ledgers. This would need to be updated for each epoch. To avoid timing issues, support will be added for the current and next staking epoch ledgers. Files should be provided at least every epoch.

2. Payout app execution will output 3 files, the third file (C) would only be created if run with "commit" mode
   A. write payout details in payout_details_[ccyymmddhhmmss][lastBlockHeight].json - this will provide an audit trail for the payout calculations
   B. write payoutJson to payout_transactions[ccyymmddhhmmss]_[lastBlockHeight].json - this would be then copied (sneakernet, admin network?) to the transaction signer node.
   C. write a control file to log last block processed. payout_control.json append-only {payoutDetails: hash, payoutTransactions: hash, lastBlockProcessed: blockHeight}

File C will become a dependency for future runs as the payour will process from lastBlockProcessed heigh to the current chain height. More safe versions may include a signed control file returned from the payout transaction generator so that the payout script can trust the height was processed.

3. Pool operators will (TBD) copy payout_transactions json and payout_control file to transaction gen/signer node if separate.

4. Run transaction gen/sign process (WIP) - optionally sign control file to return to payout calculator.

5. Copy ouptut file(s) from transaction gen/sign to node to broadcast transactions.

3-5 could be automated -- IF you are willing to have an online signer.

The expected result would have been that they would not be generated.

Error: ENOENT: no such file or directory, open '/Users/abc/Desktop/mina-pool-payout/src/core/transaction/data/.paidblocks'

It outputs the same payment twice, maybe it's doing for each block?

This script will payout from blocks 0 to 679
The pool total staking balance is 66000
We won these blocks: 527,484
We are paying out 1451933046127 nanomina in this window.
That is 1451933046127 mina
Our fee is is 72596652306.35 mina
[ { publicKey: 'B62qmPJMvVcEwBSichqB4SrrN3P1ZfiPaVZk5cVSabxQBXYfYgLQPLK',
    total: 1379336393821 } ]
[ { publicKey: 'B62qmPJMvVcEwBSichqB4SrrN3P1ZfiPaVZk5cVSabxQBXYfYgLQPLK',
    total: 1379336393821 } ]

The values by the way compare perfectly now to mine (and mine is calculated via GraphQL and not fee transfers) :)

This script will payout from blocks 0 to 679
The pool total staking balance is: 66000
There are 1 delegates in the pool
We won these blocks: [527, 484]
We are paying out 1451933046127 nanomina in this window.
That is 1451.933046127 mina
Our fee is is 72.596652306 mina
+---------------------------------------------------------+-----------------+----------------+
|                        PublicKey                        | Payout nanomina |  Payout mina   |
+---------------------------------------------------------+-----------------+----------------+
| B62qmPJMvVcEwBSichqB4SrrN3P1ZfiPaVZk5cVSabxQBXYfYgLQPLK |  1379336393821  | 1379.336393821 |
+---------------------------------------------------------+-----------------+----------------+
[{'publicKey': 'B62qmPJMvVcEwBSichqB4SrrN3P1ZfiPaVZk5cVSabxQBXYfYgLQPLK', 'total': 1379336393821}]

With the start of mina distribution, there is a possible increase of stakers with low amounts like 0.1, I saw an issue for calculating these for multiple epoch till they reach a certain amount but looks like it's closed this looks like at least a solution to filter low amounts at this stage.

We need to decide how to get keys for the payout transaction.

.env vars - current approach
Hashicorp vault - (similar risk footprint, but keys not laying there in config files)
Support ledger (not viewed as pracitcal since a) time consuming and b) lots of repetitive user input)
Generate ephemeral keys
??

Alternatively, we could generate a simple mina client send-payment script if we can assume an online node for payment processing - see #6

Allow environmental configuration for output direcotry

current payout results show payout and fee of each delegate.
It will be good to show staking amount of each delegate that can be helpful to check the amount of rewards according to the staking amount

The current payout calculator choices are biased toward locked tokens and unfairly pay a portion of a supercharged coinbase to the locked tokens, without sharing a relative supercharged portion of the "normal" coinbase with any unlocked accounts in the pool.

After considering a range of options, it seems a better approach would be to alter that payout to simply reserve the supercharged coinbase in a separate pool that is only paid out to the unlocked tokens. That would change the payout waterfall to:

1 - straight pro-rata on the normal (720) coinbase based on the account balances delegated.
2 - straight pro-rata of the subset excluding O1 Labs and Mina Foundation for the fees generated.
3 - straight pro-rata of the subset excluding 01 Labs, Mina Foundation, and locked accounts for the supercharged remainder (additional 720).

This will make the supercharged payout lumpy for the unlocked accounts, but will also guarantee the locked tokens will always receive their full share.

Some parameters are not used (e.g. consensus configuration for slots / global slot start) and should be removed.
Simplify and consolidate, reorganize param sections

Problem: The pool will receive many delegations with different amounts of mina tokens. But if the delegated mina amount of a user is too small, then the reward received will be very small as well, maybe some will less than the tx fee.

Solution: Set up a minimum payout threshold, so if the payment is greater than or equal to it, it will be paid, if not, delay it until the next epoch.

Foundation rules require sending discordid in the memo field in case the payout is being sent form an account that is neither of a) the key they delegated to, or b) the coinbase receiver.

Many nodes will use a cold coinbase receiver and payout from a separate hot wallet; they need support to add their discord id in to the memo field of the o1/mina foundation payouts.

Check this conversation for potential updates: https://discord.com/channels/484437221055922177/799597981762453535/822270046663475221

Hello,

I encounter "ERROR: Public Key is invalid at line N." when runnning with a .negotiatedFees file, even if my file does not contain empty lines.
I made a fix here: #94

Pool operators should have their coinbase receiver delegating back to their block producer, and will want to either a) leave their funds on the coinbase-receiver cold wallet, or else b) send them to yet another account. It will not make sense to incur fees to move the pool's share from the cold wallet to the payout wallet, only to move them back again to the same cold wallet.

Operators should be able to either:

1. leave pool shares on the cold wallet - and generate an alternate funding template command reduced by the amount of the pool shares, and do not payout the portion of fees for the pool operator (they will simply be retained where they are); or
2. configure a separate pool operator account where the pool fees/remainder will be sent.

Expected:
signed transactions have a fee that at least meets the minimum

Actual:
signed transactions have a fee of zero

When we moved the fee out to the rollup, we did not revisit which objects are being written out or signed -- we need the ones that have the fee.

When I used the script, I started reporting errors almost after a few transactions were sent. any suggest?

message: `Couldn't send user_command: Error creating user command: {"payload":{"common":{"fee":"0.001","fee_token":"1","fee_payer_pk":"B62qiUYxxxxxxx","nonce":"266","valid_until":"4294967295","memo":"E4YzpZRQxxxxx"},"body":["Payment",{"source_pk":"B62xxxxxxx","receiver_pk":"B62qihyyyyyyy","token_id":"1","amount":"7964041225"}]},"signer":"B62qiUYSjLkQuvgouQ1xxx","signature":["Signature","7mXK2mYsQydaoNzzzzzxxxxxMnYwbUkst3vLoix9Xd"]} Error: Input nonce 266 either different from inferred nonce 265 or below minimum_nonce 264`, path: [ 'sendPayment' ]

Currently, we assume a single staking ledger.

We want to be able to find the correct ledger (and validate it) based on the blocks being processed.

1. Move ledger input directory to a .env config so you can specify the directory to scan for ledger files
2. Scan blocks to determine which epoch staking ledgers are required
3. Find epoch ledger(s) (TBD?)
4. Validate ledgers (hash? TBD?)
5. Process blocks, changing ledger when necessary

See: https://discord.com/channels/484437221055922177/754653322845356103/1126326557670121482

The ledger now looks like this for the timing block. Is it possible not to be Since_genesis and Global_slot_span?

"timing": {
      "initial_minimum_balance": "1000000",
      "cliff_time": [ "Since_genesis", "12000" ],
      "cliff_amount": "500000",
      "vesting_period": [ "Global_slot_span", "6" ],
      "vesting_increment": "150"
    }

Per @jrwashburn comment on PR 23:

Rather than use a SEND_TRANSACTION_FEE environment variable, add a getRecommendedFee function to:

1. Calculate a recommended fee
2. Decide whether the fee is paid by the pool or by the staker.

Once the payor key is funded, the transactions need to be broadcast on the network via the graphql mutation.

See Submitting a Signed Transaction here: https://minaprotocol.com/docs/advanced/operating-for-data

Two things to consider:

1. this need to be a separate command that can be run after the main calculation
2. this command is what should track that we actually made the payout - perhaps mark the files or move them in the short term to a "sent" directory?
3. if we are okay running these on the node and signing with the hot bp key, then we could alternatively just generate mina client send-payment commands to replay -- get rid of the entire signing process, and instead just generate and run a send-payment script after the payor has been funded.

Hi,

I briefly tested the script, and i encountered a problem: when the commission rate is zero, the retained commision rate is the default value (0.05).
I tested with 0, 0.0, even with double quotes, but i'm stuck with the default commission rate.
If i set the commission rate to 0.00000000001, it works as expected, but this solution is not acceptable.

An idea?

determine/validate payment broadcast method and whether they expect mina or nanomina for amount and fee

Not ready yet, but keep an eye on this:
MinaProtocol/mina#9808

Hello,

I just tested the script with a .negotiatedFees file, and i encountered a problem: when the negociated commission rate is zero, the retained commision rate is the default value (COMMISSION_RATE).
If i put 0.00000000000000000000001 as the negociated commission rate in .negotiatedFees, it works well.

Repro:

Re-run an epoch already processed against archive db - as expected no payout is calculated.
npm run payout -- -m=69767 -x=74463

Change data source to Mina Explorer, re-run same range
npm run payout -- -m=69767 -x=74463

Expected: no blocks processed (because the relevant blocks/hashes exist in .paidblocks file)
Actual: payout is calculated and all won blocks are reported / calculated to be paid.

Although payment sends can be automated, there is not guarantee that a payment will clear the mempool and get committed to a block.

We need a mechanism to monitor payments we sent and confirm that they actually made it onto the chain.

1. write control file of payment transactions actually sent
2. new command to query the chain later to see if the payments we sent are on the canonical chain - reports back all okay, or list of payments that are not yet on chain (perhaps output a hash as with running the payout)
3. new command to regenerate the missing payments and send them
4. rinse / repeat - need to know to stop tracking the only transaction that was retried, and start tracking the new retry

The string

"Using password from environment variable CODA_PRIVKEY_PASS"

is at the beginning of the staking-epoch-ledger.json and next-epoch-ledger.json files

Example

Using password from environment variable CODA_PRIVKEY_PASS
[
  {
    "pk": "B62qiy32p8kAKnny8ZFwoMhYpBppM1DWVCqAPBYNcXnsAHhnfAAuXgg",
    "balance": "0.000001",
    "delegate": "B62qiy32p8kAKnny8ZFwoMhYpBppM1DWVCqAPBYNcXnsAHhnfAAuXgg",
    "token": "1",
    "token_permissions": {},
    "receipt_chain_hash":
      "2mzbV7WevxLuchs2dAMY4vQBS6XttnCUF8Hvks4XNBQ5qiSGGBQe",
    "voting_for": "3NK2tkzqqK5spR2sZ7tujjqPksL45M3UUrcA4WhCkeiPtnugyE2x",
.....

It always goes to the latest height despite changing values of k.

i.e. specify K=290 and when the current height is 679 it gives this:

This script will payout from blocks 0 to 679 so not correctly stopping at height 359.

As an aside I renamed this variable to confirmations as some producers might want a value of say 100 confirmations etc... and k has a specific meaning.

Many delegators of locked accounts will want their pool awards paid to a different account, which may also be delegated, but could earn supercharged coinbase.

We need to determine a secure way to establish where funds should be paid out (suggest requiring a signed message from the delegator specifying the new payout address, although that may increase the difficulty for delegators not running any node and with only cold wallets.) Perhaps it would be preferable to the node operator to independently determine the authenticity of a request, and then update their configuration accordingly.

Depending on the solution above, implement a method of redirecting the payout from delegator A to payout address B.

npm run payout -- -m=0 -x=1000

gives the following error

mina@cs3:~/mina-pool-payout$ npm run payout -- -m=0 -x=1000

[email protected] payout /home/mina/mina-pool-payout
ts-node -r dotenv/config src/index.ts "-m=0" "-x=1000"

/home/mina/mina-pool-payout/src/core/send-payments.ts:69
return data.account.inferredNonce ?? 0;
^

SyntaxError: Unexpected token ?
at Module._compile (internal/modules/cjs/loader.js:723:23)
at Module.m._compile (/home/mina/mina-pool-payout/node_modules/ts-node/src/index.ts:1056:23)
at Module._extensions..js (internal/modules/cjs/loader.js:789:10)
at Object.require.extensions.(anonymous function) [as .ts] (/home/mina/mina-pool-payout/node_modules/ts-node/src/index.ts:1059:12)
at Module.load (internal/modules/cjs/loader.js:653:32)
at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
at Function.Module._load (internal/modules/cjs/loader.js:585:3)
at Module.require (internal/modules/cjs/loader.js:692:17)
at require (internal/modules/cjs/helpers.js:25:18)
at Object. (/home/mina/mina-pool-payout/src/index.ts:6:1)

content of debug file is

0 info it worked if it ends with ok
1 verbose cli [ '/usr/bin/node',
1 verbose cli '/usr/bin/npm',
1 verbose cli 'run',
1 verbose cli 'payout',
1 verbose cli '--',
1 verbose cli '-m=0',
1 verbose cli '-x=1000' ]
2 info using [email protected]
3 info using [email protected]
4 verbose run-script [ 'prepayout', 'payout', 'postpayout' ]
5 info lifecycle [email protected]prepayout: [email protected]
6 info lifecycle [email protected]payout: [email protected]
7 verbose lifecycle [email protected]payout: unsafe-perm in lifecycle true
8 verbose lifecycle [email protected]payout: PATH: /usr/share/npm/node_modules/npm-lifecycle/node-gyp-bin:/home/mina/mina-pool-payout/node_modules/.bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
9 verbose lifecycle [email protected]payout: CWD: /home/mina/mina-pool-payout
10 silly lifecycle [email protected]payout: Args: [ '-c',
10 silly lifecycle 'ts-node -r dotenv/config src/index.ts "-m=0" "-x=1000"' ]
11 silly lifecycle [email protected]payout: Returned: code: 1 signal: null
12 info lifecycle [email protected]payout: Failed to exec payout script
13 verbose stack Error: [email protected] payout: ts-node -r dotenv/config src/index.ts "-m=0" "-x=1000"
13 verbose stack Exit status 1
13 verbose stack at EventEmitter. (/usr/share/npm/node_modules/npm-lifecycle/index.js:332:16)
13 verbose stack at EventEmitter.emit (events.js:198:13)
13 verbose stack at ChildProcess. (/usr/share/npm/node_modules/npm-lifecycle/lib/spawn.js:55:14)
13 verbose stack at ChildProcess.emit (events.js:198:13)
13 verbose stack at maybeClose (internal/child_process.js:982:16)
13 verbose stack at Process.ChildProcess._handle.onexit (internal/child_process.js:259:5)
14 verbose pkgid [email protected]
15 verbose cwd /home/mina/mina-pool-payout
16 verbose Linux 5.4.0-73-generic
17 verbose argv "/usr/bin/node" "/usr/bin/npm" "run" "payout" "--" "-m=0" "-x=1000"
18 verbose node v10.19.0
19 verbose npm v6.14.4
20 error code ELIFECYCLE
21 error errno 1
22 error [email protected] payout: ts-node -r dotenv/config src/index.ts "-m=0" "-x=1000"
22 error Exit status 1
23 error Failed at the [email protected] payout script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 1, true ]

I had these files in the path
total 7.9M
drwxrwxr-x 5 mina mina 4.0K Jun 3 14:50 .
drwx------ 9 mina mina 4.0K Jun 3 14:53 ..
-rw-rw-r-- 1 mina mina 948 Jun 3 14:09 .env
drwxrwxr-x 8 mina mina 4.0K Jun 3 13:56 .git
-rw-rw-r-- 1 mina mina 76 Jun 3 13:56 .gitignore
-rw-rw-r-- 1 mina mina 1.1K Jun 3 13:56 LICENSE
-rw-rw-r-- 1 mina mina 9.7K Jun 3 13:56 README.md
-rwxrwxr-x 1 mina mina 479 Jun 3 14:37 export_ledgers.sh
-rw-r--r-- 1 mina mina 2.1M Jun 3 14:38 jwJXdYzAikMHnTsw2kYyS1WQxJrGQsy1FKT5c18eHP2wGANafKf.json
-rw-r--r-- 1 mina mina 1.8M Jun 3 14:38 jxVF5YbC3B5Rk6ibfsL97WaqojfxrgWtEqMJST9pb4X8s3kRD2T.json
-rw-r--r-- 1 mina mina 2.1M Jun 3 14:38 next-epoch-ledger.json
drwxrwxr-x 190 mina mina 32K Jun 3 14:16 node_modules
-rw-rw-r-- 1 mina mina 77K Jun 3 14:16 package-lock.json
-rw-rw-r-- 1 mina mina 898 Jun 3 14:16 package.json
-rw-rw-r-- 1 mina mina 899 Jun 3 13:56 sample.env
drwxrwxr-x 5 mina mina 4.0K Jun 3 13:56 src
-rw-r--r-- 1 mina mina 1.8M Jun 3 14:38 staking-epoch-ledger.json
-rw-rw-r-- 1 mina mina 388 Jun 3 13:56 tsconfig.json

As requested by Nightguy1688#3119 in Discord: https://discord.com/channels/484437221055922177/799597981762453535/866456694540599343

Add logging to show:

Total Coinbase Generated: total block reward
Total User Transaction Fees Generated: fees collected from user transactions
Total Snark Fees Paid: fees paid out for snarks (includes fees from coinbase)
Net Coinbase Received: = Total Coinbase Generated + User Tx Fees - Snark Fees
Total Payouts Sent: sum of payout amounts
Total Payout Transaction Fees: sum of payout transaction send fees
Net MINA to Pool Operator: + Net Coinbase Received - Total Payouts Sent - Total Payout Transaction Fees

Edited based on new O1 rules. This is now mandatory and requires a substantial change to the pooling approach.

• Mina Foundation and O(1) Labs are delegating their stake to block producers; however, they have a restriction on fee percentages of 5%. Several pool providers have higher pool fees, and need to support charging a differentiated fee for "normal" balances vs. foundation and O(1) balances.
• Many pools include delegations from locked accounts, whose owners will not want their share paid back to the locked account, but to another unlocked account to increase the potential for supercharged coinbase.
• Some pool operators also delegate to themselves, and may not want to payout their share, but leave it in the coinbase-receiver wallet.

New Payout Rules

The new payout rules will need 2 classes of "shares" plus the pool operator, with a payout waterfall that implements the following:

Non-Participating shares

• o(1) labs delegators
• mina foundation delegators
• do not participate in supercharged awards
• do not participate in fees
• do not pay fees
• expect 0.95 * share * regular coinbase for each block produced by the pool
  Official rules here
• pay special attention to the requirement to include discord id in payout to foundation/o1 (created issue #35 for that, but it must be released with this.)

Common Pool

• after preferred shares are paid and BP fee on those is deducted
• receives all transaction fees, pays all snark fees, receives coinbase remaining after NPS payout
• calculate remaining weighted share basis including supercharged allocation

Pool Operator

• charges 0.05 on each Non Participating share payout
• charges configured Pool Fee on each Common payout

Other requirements

• Do all math in nanomina, and round down after each division or multiplication to truncate fractional nanomina.
  Note the dust and output at the end as a dust amount?
• Determine whether a coinbase amount is supercharged or not by checking if the WinnerKey is unlocked. If coinbase is supercharged, NPS get 1/2 of that amount in the NPS portion of the calculation.

Future support:

• Just FYI - each staking key should be able to specify a payout-address, which would cause the pool to send their payouts to a different address (see #33 , #34 )

Pool Operation Decisions / Consider for parameterization

• Share NPS portion of super-charged blocks with common pool?
• Charge NPS portion of fees to Common pool or separate for operator to pay from foundation delegation?

Suggested output

Planned Payout:

Confirm Payouts, and if correct and necessary, send funding amounts as instructed below.
Then re-run with this payout with process payout command below to send payouts.

Ledger Funding Command:
mina_ledger_wallet send-payment --offline --network testnet --nonce FUNDERNONCE --fee 0.1 BIP44ACCOUNT FUNDING_FROM_ADDRESS ...

Mina Client Funding Command:
mina client send-payment --amount TOTAL --receiver PAYORKEY --sender TBDYOURFUNDINGACCOUNT --fee 0.1

Process Payout Command:
npm run  ...  --hash aADesdf234sadFASDFsdfdFASDFASDF

load all env constants from main and pass them to sign
remove duplicate params for public key

Transactions can also be broadcast via minaexplorer's public endpoint:
https://docs.minaexplorer.com/rest-api/ref#post-broadcast

Add an implementation that can broadcast here vs. to a daemon graphql endpoint.

I think showing staking balance is good.
Currently, show "index", "publicKey", "amount", "fee", "amountMina", "feeMina"
I suggest "index", "publicKey", "staking balance", "amount", "fee", "amountMina", "feeMina"
As you know "-v" flag show more details but it is not sorted and hard to see for delegators.
But staking amount is changed according to ledger, so.. I'm not sure how to show this challenge.
I think many validator pay rewards in a epoch. yeah.. It's hard thing :)

this output info

------------------- Summary & Totals ------------------
Calculations based on entire pool
Total Coin Base Generated: X
User Transaction Fees Generated (net of snark fees): X
Total Snark Fees Paid From Coinbase: X
Net Coinbase Received: X
Total Amounts Due To Stakers: X
Total Payout Transaction Fees: X
Net MINA to Pool Operator (after send transaction fees): X
------------------- Summary & Totals ------------------

would be very useful to be able to read programmatically along with the payout_transactions_*.json and payout_details_*.json files. specifically to get the commission/pool operator reward and not have to calculate it from the raw data

would you consider implementing this, or accept a PR of same?

Provide factory to get blocks and stakes from alternative data providers.
Initial (current) implementation uses archive db and ledger export from mina daemon.

Alternative implementations suggested:
minaexplorer graphql - https://docs.minaexplorer.com/minaexplorer/graphql-getting-started
Gist from Gareth for basic data requirements: https://gist.github.com/garethtdavies/063da0800109696276c706a04b0ba94f

figment datahub - https://figment.io/datahub-user-guide/

Let's start writing everything to a log. The current file output is good, but really need a more detailed audit trail, especially when we run into issues.

Can you make it more clear a bit. I remember it should be equal in the long run. If it was favouring locked on how it was doing that. Wasn't the existing system does the same. What you outlined here?

Also, I have a proposal to just cut out all the fees generated out instead of just the foundation one. My reasoning is I don't believe these fees never reach any significant amount. When there is some high fee like 500 etc. then we will send it back to the owner which mixes up calculation more. :) From the start, I am just suggesting cutting the fees out of the block when they go significantly towards the coinbase because there is always a possibility of 300-500 snark happening on a block. If there is a significant fee increase where snarks are really ending up to something that we had much more competition on snark marketplace which will drive snark prices up and drop the BP's gainfrom it.

Originally posted by @EmrePiconbello in #58 (comment)

from #30 (comment)

Do we have an option to just sign it without broadcast? I think we can add a variable on the .env file to let users can choose just signing or both signing and broadcasting

Just an idea - as I understand, currently one needs to look for the -m and -x values when executing the payout programme. For BPs who pays once after epoch ends, it would perhaps be more convenient if one can input the epoch number, and the programme queries the GraphQL with the creator key and epoch number, to get a full list of blocks. In which case no need for -m and -x inputs, just a -e would suffice.

@kobigurk identified a bug.

Expected behvaior: non-foundation (Common) shares will still charge a differentiated fee (per .env file config) on the initial NPS Pool shares.
Actual behavior: Common shares are charged the hard-coded 5% fee alongside the NPS shares in the NPS pool

Must implement a differentiated share not only on the common pool, but also on the common share's stake in the NPS pool.