eprbell / dali-rp2 Goto Github PK

According to the documentation the json response for Coinbase should be:

{
"pagination": {
"ending_before": null,
"starting_after": null,
"limit": 25,
"order": "desc",
"previous_uri": null,
"next_uri": null
},
"data": [
{
"id": "58542935-67b5-56e1-a3f9-42686e07fa40",
"name": "My Vault",
"primary": false,
"type": "vault",
"currency": "BTC",
"balance": {
"amount": "4.00000000",
"currency": "BTC"
},
"created_at": "2015-01-31T20:49:02Z",
"updated_at": "2015-01-31T20:49:02Z",
"resource": "account",
"resource_path": "/v2/accounts/58542935-67b5-56e1-a3f9-42686e07fa40",
"ready": true
},
{
"id": "2bbf394c-193b-5b2a-9155-3b4732659ede",
"name": "My Wallet",
"primary": true,
"type": "wallet",
"currency": "BTC",
"balance": {
"amount": "39.59000000",
"currency": "BTC"
},
"created_at": "2015-01-31T20:49:02Z",
"updated_at": "2015-01-31T20:49:02Z",
"resource": "account",
"resource_path": "/v2/accounts/2bbf394c-193b-5b2a-9155-3b4732659ede"
}
]
}

So, shouldn't:

currency: str = account[_CURRENCY][_CODE]

just be:

currency: str = account[_CURRENCY]

Today I learned that BCC and BCH are different symbols for the same coin:

https://bitcointalk.org/index.php?topic=2617659.0

Two different exchanges used this symbol interchangeably. I'm not sure if this is the only case, but might be something we need to handle.

Dali Version: 0.3.25
RP2 Version: 0.9.25

I'm not sure if this is this is an issue with DaLI or with RP2, But Algorand interest is listed in the IN section like the following:

When trying to run with FIFO or LIFO, I get the following error:

2022-03-27 19:56:10,271/rp2/INFO: Country: us
2022-03-27 19:56:10,273/rp2/INFO: Accounting Method: lifo
2022-03-27 19:56:10,290/rp2/INFO: Configuration file: output/crypto_data.config
2022-03-27 19:56:10,290/rp2/INFO: Input file: output/crypto_data.ods
2022-03-27 19:56:10,361/rp2/INFO: Processing ALGO
2022-03-27 19:56:10,389/rp2/ERROR: Fatal exception occurred:
Traceback (most recent call last):
  File "/home/james/.local/lib/python3.10/site-packages/rp2/rp2_main.py", line 93, in _rp2_main_internal
    input_data: InputData = parse_ods(configuration=configuration, asset=asset, input_file_handle=input_file_handle)
  File "/home/james/.local/lib/python3.10/site-packages/rp2/ods_parser.py", line 128, in parse_ods
    _create_and_process_transaction(
  File "/home/james/.local/lib/python3.10/site-packages/rp2/ods_parser.py", line 168, in _create_and_process_transaction
    transaction: AbstractTransaction = _create_transaction(configuration, current_table_type, internal_id, row_values)
  File "/home/james/.local/lib/python3.10/site-packages/rp2/ods_parser.py", line 277, in _create_transaction
    transaction = InTransaction(**argument_pack)
  File "/home/james/.local/lib/python3.10/site-packages/rp2/in_transaction.py", line 61, in __init__
    raise RP2ValueError(f"{self.asset} {type(self).__name__} ({self.timestamp}, id {self.internal_id}): parameter 'spot_price' cannot be 0")
rp2.rp2_error.RP2ValueError: ALGO InTransaction (2021-04-28 12:22:48+00:00, id 4): parameter 'spot_price' cannot be 0
2022-03-27 19:56:10,400/rp2/INFO: Log file: ./log/rp2_2022_03_27_19_56_10_254635.log
2022-03-27 19:56:10,401/rp2/INFO: Generated output directory: output/
2022-03-27 19:56:10,401/rp2/INFO: Done

Some transactions involve two tokens where the spot price of one or both tokens is not provided by the exchange. However, we have enough information to determine the spot prices of both tokens. In a typical transaction, we know the following:

• price ratio of the swap pair -or- quantity of each token
• the price of one of the tokens is either known or can be looked up by the historic crypto service
• the price of the other token is not available in the historic crypto service

In this scenario, we can lookup the historic price of one of the tokens and calculate the historic price of the other token with a simple formula. Are there any plans to add this capability to DaLi?

Since the CCXT library does not have a forex exchange connected to it #48 , we will need to implement another solution to convert between different kinds of fiat. exchangerate.host seems to be a free and easy to implement solution for the time being.

We will need to create a simple converter plugin that makes use of the JSON calls and not require other packages.

I noticed CCXT documents a sample REST response for each of their calls. I propose we do the same for our plugins where needed. For example, I added the following to the plugin I'm working on:

fiat_payments = client.sapiGetFiatPayments(params=({'transactionType':0, 
	'beginTime':(int(startTime.timestamp()) * 1000), 
	'endTime':nowTime}))
# {
#    "code": "000000",
#    "message": "success",
#    "data": [
#    {
#       "orderNo": "353fca443f06466db0c4dc89f94f027a",
#       "sourceAmount": "20.0",  // Fiat trade amount
#       "fiatCurrency": "EUR",   // Fiat token
#       "obtainAmount": "4.462", // Crypto trade amount
#       "cryptoCurrency": "LUNA",  // Crypto token
#       "totalFee": "0.2",     // Trade fee
#       "price": "4.437472", 
#       "status": "Failed",  // Processing, Completed, Failed, Refunded
#       "createTime": 1624529919000,
#       "updateTime": 1624529919000  
#    }
#    ],
#    "total": 1,
#    "success": true
# }

Fiat Payments responds with that json. I think this will make it easier to sort out what variables we need and also document undocumented APIs like what happened with Coinbase plugin. However, it might bloat the code. Any thoughts?

The conversion of ETH to BETH (Beaconed ETH) is only available via csv export and not the API. We will need to import this data via a separate plugin.

Writing a DaLI data loader plugin has high impact on the functionality and usefulness of both DaLI and RP2. Data loader plugins are well-defined, encapsulated modules that translate native exchange REST API-based data into DaLI's standard format. As such they are good first issues for newcomers to the the project.

Before starting, please read the contributing guidelines.

Plugin development is described in the developer documentation, in particular read the Dali Internals and Plugin Development sections, which contain all the information needed to build a new data loader plugin.

CCXT is documented here: https://docs.ccxt.com/en/latest/manual.html. This would be an abstract plugin that is meant to be subclassed by concrete ones.

The Coinbase plugin can be used as an example.

Before submitting a PR for the new plugin, make sure to go through the DaLI Plugin Laundry List.

@houqp I've seen this unit test fail a couple of times, but I cannot make it fail consistently. I've seen 2 failures tonight out of more than 20 test runs. Any other developers seeing this?

tests/test_ods_output_diff.py::TestODSOutputDiff::test_crypto_data_ods PASSED                                                                                                                                                         [ 85%]
tests/test_ods_output_diff.py::TestODSOutputDiff::test_fifo_rp2_full_report_ods PASSED                                                                                                                                                [ 88%]
tests/test_ods_output_diff.py::TestODSOutputDiff::test_fifo_tax_report_us_ods PASSED                                                                                                                                                  [ 90%]
tests/test_ods_output_diff.py::TestODSOutputDiff::test_lifo_rp2_full_report_ods PASSED                                                                                                                                                [ 92%]
tests/test_ods_output_diff.py::TestODSOutputDiff::test_lifo_tax_report_us_ods PASSED                                                                                                                                                  [ 95%]
tests/test_plugin_coinbase.py::TestTrade::test_eth2_stake PASSED                                                                                                                                                                      [ 97%]
tests/test_plugin_coinbase_pro.py::TestSwapFill::test_buy_side FAILED                                                                                                                                                                 [100%]

================================================================================================================= FAILURES =================================================================================================================
________________________________________________________________________________________________________ TestSwapFill.test_buy_side ________________________________________________________________________________________________________
Traceback (most recent call last):
  File "D:\MyData\Documents\Projects\dali-rp2\tests\test_plugin_coinbase_pro.py", line 117, in test_buy_side
    assert out_transaction.asset == "BTC"
AssertionError: assert 'ETH' == 'BTC'
  - BTC
  + ETH
------------------------------------------------------------------------------------------------------------ Captured log call -------------------------------------------------------------------------------------------------------------
DEBUG    Coinbase Pro/tester:coinbase_pro.py:178 Account: {"id": "bbbbbbbb-dddd-4444-8888-000000000000", "currency": "BTC"}
DEBUG    Coinbase Pro/tester:coinbase_pro.py:182 Transaction: {"id": "1111111111", "amount": "-0.0005000000000000", "balance": "0.9995000000000000", "created_at": "2020-12-11T00:20:14.693676Z", "type": "fee", "details": {"order_id": "33333333-bbbb-4444-cccc-aaaaaaaaaaaa", "product_id": "ETH-BTC", "trade_id": "134567890"}}
DEBUG    Coinbase Pro/tester:coinbase_pro.py:200 Redundant fee transaction (skipping): {"id": "1111111111", "amount": "-0.0005000000000000", "balance": "0.9995000000000000", "created_at": "2020-12-11T00:20:14.693676Z", "type": "fee", "details": {"order_id": "33333333-bbbb-4444-cccc-aaaaaaaaaaaa", "product_id": "ETH-BTC", "trade_id": "134567890"}}
DEBUG    Coinbase Pro/tester:coinbase_pro.py:182 Transaction: {"id": "1111111108", "amount": "-0.5000000000000000", "balance": "1.0000000000000000", "created_at": "2020-12-11T00:20:14.693676Z", "type": "match", "details": {"order_id": "33333333-bbbb-4444-cccc-aaaaaaaaaaaa", "product_id": "ETH-BTC", "trade_id": "134567890"}}
DEBUG    Coinbase Pro/tester:coinbase_pro.py:281 Product id: ETH-BTC
DEBUG    Coinbase Pro/tester:coinbase_pro.py:301 Fill: {"created_at": "2020-12-11T00:20:14.693676Z", "trade_id": "134567890", "product_id": "ETH-BTC", "order_id": "33333333-bbbb-4444-cccc-aaaaaaaaaaaa", "liquidity": "M", "price": "0.05000000", "size": "10.00000000", "fee": "0.0005000000000000", "side": "buy", "settled": true, "usd_volume": "25000.000000000000000000000000"}
DEBUG    Coinbase Pro/tester:coinbase_pro.py:178 Account: {"id": "eeeeeeee-4444-5555-aaaa-cccccccccccc", "currency": "ETH"}
DEBUG    Coinbase Pro/tester:coinbase_pro.py:182 Transaction: {"id": "1111111111", "amount": "10.0000000000000000", "balance": "11.0000000000000000", "created_at": "2020-12-11T00:20:14.693676Z", "type": "match", "details": {"order_id": "33333333-bbbb-4444-cccc-aaaaaaaaaaaa", "product_id": "ETH-BTC", "trade_id": "134567890"}}
DEBUG    Coinbase Pro/tester:coinbase_pro.py:281 Product id: ETH-BTC
DEBUG    Coinbase Pro/tester:coinbase_pro.py:301 Fill: {"created_at": "2020-12-11T00:20:14.693676Z", "trade_id": "134567890", "product_id": "ETH-BTC", "order_id": "33333333-bbbb-4444-cccc-aaaaaaaaaaaa", "liquidity": "M", "price": "0.05000000", "size": "10.00000000", "fee": "0.0005000000000000", "side": "buy", "settled": true, "usd_volume": "25000.000000000000000000000000"}
========================================================================================================= short test summary info ==========================================================================================================
FAILED tests/test_plugin_coinbase_pro.py::TestSwapFill::test_buy_side - AssertionError: assert 'ETH' == 'BTC'
======================================================================================================= 1 failed, 41 passed in 9.05s =======================================================================================================

Change price_converter to pair_converter. It appears more than one in the configuration_file.md file.

Users will see this error: ModuleNotFoundError: No module named 'dali.plugin.price_converter'.

In order to process accurate basis costs for crypto assets bought, a more robust pair converter is needed. The CCXT pair converter would allow spot prices to be pulled from any exchange that is supported by CCXT which would support any of the input plugins based on CCXT.

It should also support the conversion of fiat in order to support international users and those who trade in fiat other than what is commonly used in their native country.

First, a big thanks for providing this useful functionality.

I'd like to point out 1) a minor bug, 2) a noteworthy observation, and 3) a suggestion for improvement.

1. First a minor bug. There is a typo in the code. 84600 should be changed to 86400 for daily bars.

2. An observation. When requesting daily bars (86400 seconds), the timestamp of the returned dataframe is unlike the timestamp for other timeframes. This may be an end-of-bar timestamp, not a start-of-bar timestamp. For all other timeframes less than a day, the timestamps are the start of the bar. For example, consider this request:
   actual trade time = "2022-01-01 12:00:30", asset = "ETH-USD"

The resulting dataframes returned by HistoricalData() for each of the granularities [60, 300, 900, 3600, 21600, 86400] looks like this.

ETH-USD 60 from: 2022-01-01-12-00 to: 2022-01-01-12-01
                         low     high     open    close    volume
time
2022-01-01 12:00:00  3693.05  3700.00  3693.05  3699.47  9.021474
2022-01-01 12:01:00  3697.54  3699.77  3699.61  3697.95  5.324811

ETH-USD 300 from: 2022-01-01-12-00 to: 2022-01-01-12-05
                         low     high     open    close      volume
time
2022-01-01 12:00:00  3693.05  3700.00  3693.05  3697.45   45.536298
2022-01-01 12:05:00  3696.49  3705.68  3697.44  3705.06  165.421915

ETH-USD 900 from: 2022-01-01-12-00 to: 2022-01-01-12-15
                         low     high     open    close      volume
time
2022-01-01 12:00:00  3693.05  3708.59  3693.05  3708.01  340.493358
2022-01-01 12:15:00  3698.12  3709.37  3708.02  3700.70  275.891478

ETH-USD 3600 from: 2022-01-01-12-00 to: 2022-01-01-13-00
                         low     high     open    close       volume
time
2022-01-01 12:00:00  3693.05  3719.00  3693.05  3716.94  1220.621166
2022-01-01 13:00:00  3706.87  3725.55  3717.10  3710.46  1443.189983

ETH-USD 21600 from: 2022-01-01-12-00 to: 2022-01-01-18-00
                         low     high     open    close        volume
time
2022-01-01 12:00:00  3693.05  3767.00  3693.05  3762.14  16792.730418
2022-01-01 18:00:00  3736.00  3777.82  3761.82  3765.72  22093.874628

ETH-USD 86400 from: 2022-01-01-12-00 to: 2022-01-02-12-00
               low     high     open    close        volume
time
2022-01-02  3718.2  3855.77  3765.98  3828.87  78466.351135

3. Finally, a suggestion for improvement. As written, the code is returning the highest price during the bar interval, but this may not be representative of the actual price at the time of the transaction. Consider this example (15-minute bars for SHIB on April 5 at 13:45). If you executed a trade during this bar, your actual fill price could have been anywhere in-between the high and low. The highest price in this example is probably not the most accurate choice. Note that we know the time associated with the open and close prices, but we have no way of knowing when the high and low occurred.

I propose the following algorithm.
Determine the timestamp at the start, end, and half-way point of the quotation bar.
Measure the time distance between the actual trade time and each of the quotation time stamps (start, end, middle).
If the actual trade timestamp is nearest the open timestamp, use the open price.
If the actual trade timestamp is nearest the ending timestamp, use the close price.
If the actual trade timestamp is nearest the middle of the quotation bar, use the average of the open and close prices.

This change should smooth-away spurious price spikes and result in more accurate price lookups. However, I want to mindful that you picked the high price for a reason. I'm not sure what that reason was, but maybe you want to keep that behavior. Perhaps you could introduce a new configuration parameter to allow the user to choose the price lookup behavior.

Writing a DaLI data loader plugin has high impact on the functionality and usefulness of both DaLI and RP2. Data loader plugins are well-defined, encapsulated modules that translate native exchange REST API-based data into DaLI's standard format. As such they are good first issues for newcomers to the the project.

Before starting, please read the contributing guidelines.

Plugin development is described in the developer documentation, in particular read the Dali Internals and Plugin Development sections, which contain all the information needed to build a new data loader plugin.

Binance.US REST API is documented here: https://docs.binance.us/#introduction

The Coinbase plugin can be used as an example.

Before submitting a PR for the new plugin, make sure to go through the DaLI Plugin Laundry List.

When running dali to generate my files for rp2 I am getting a fatal error and no output file is generated. The execption I am seeing is:

Internal error: fiat_in_with_fee > fiat_out_no_fee

This is when using the Coinbase API plugin.

I'm about finished with a rough draft of the Binance.com plugin but would like to test it once with real data. How can I run it with my plugin file to see if it pulls the data correctly? I forked the project and pulled it locally, I just can't seem to figure out how to run it. Do I tell python to run the dali_main.py:

python3 dali_main.py -s -o output -p test_ binance_config.ini

When I run this command, it works on it for a moment then exits. The log file is empty.

I know I'm missing something obvious here, but I can't seem to find it.

Version: DaLI 0.3.21
OS: Fedora Linux 35

Looks like inflation_rewards are not currently supported on the Coinbase REST API. Would like to support to have all entries in ods file.

Taken from DEBUG level logs on Coinbase (not Pro) of an Algorand reward:

2022-03-26 08:27:11,229/Coinbase/name/DEBUG: Unsupported transaction type (skipping): inflation_reward
2022-03-26 08:27:11,229/Coinbase/name/DEBUG: Transaction: {"id": "ZZZZZZZZ-ZZZZ-ZZZZ-ZZZZ-ZZZZZZZZZZZZ", "type": "inflation_reward", "status": "completed", "amount": {"amount": "0.000166", "currency": "ALGO"}, "na    tive_amount": {"amount": "0.00", "currency": "USD"}, "description": null, "created_at": "2021-XX-XXT14:55:27Z", "updated_at": "2021-XX-XXT14:55:27Z", "resource": "transaction", "resource_path": "/v2/accounts/135c8    7f3-44a8-5c13-ae9c-14ac0d182a79/transactions/ZZZZZZZZ-ZZZZ-ZZZZ-ZZZZ-ZZZZZZZZZZZZ", "instant_exchange": false, "from": {"id": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX", "resource": "user", "resource_path": "/v2/users    /XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX", "currency": "ALGO"}, "details": {"title": "Algorand reward", "subtitle": "From Coinbase", "header": "Received 0.000166 ALGO ($0.00)", "health": "positive"}, "hide_native_amount": false}

Hi, I noticed in both the coinbase plugin and manual csv input, crypto concurrency swaps are not supported. Is this correct? If so, is there any workaround to model into the report? Do I need to split that into one acquire and one dispose transactions instead?

Calls to APIs can sometimes be slow due to throttling. For example, Kraken limits requests to around 12 / minute. This can make pulling pricing data by API extremely slow (5 seconds per call). Historical data is often available in CSV form for major exchanges. These CSV can be read in by the plugin and cached to greatly increase the speed of pricing trades.

Some of these CSV are available in repositories that could be pulled automatically, but the links to them might be brittle. I propose we set up default download links and if they fail ask the user to report it or update the link themselves, possibly in a .cfg file?

Writing a DaLI data loader plugin has high impact on the functionality and usefulness of both DaLI and RP2. Data loader plugins are well-defined, encapsulated modules that translate native exchange REST API-based data into DaLI's standard format. As such they are good first issues for newcomers to the the project.

Before starting, please read the contributing guidelines.

Plugin development is described in the developer documentation, in particular read the Dali Internals and Plugin Development sections, which contain all the information needed to build a new data loader plugin.

FTX.US REST API is documented here: https://docs.ftx.us/#rest-api

The Coinbase plugin can be used as an example.

Before submitting a PR for the new plugin, make sure to go through the DaLI Plugin Laundry List.

Hello! First a wanted to say thanks for all your work on rp2. I have found it very useful.

Actually over the past few weeks I have been writing my own tool for compiling data into the input format that RP2 takes. Then I saw this, and that youve already started working on this. So I thought I would reach out and at least let you know what I am working on.

So far it looks like you have been targeting US exchanges, however if your a US tax payer, trading on non US CEXs and DEXs can make getting the correct input challenging, since there is a bit of missing data that needs to be reconstructed with historical price data.

Here is at least what my tool does, and I would like to see how you think this could fit into dali-rp2. This is still a WIP and I have 3 exchanges working now, but still have some more work to do before it is shareable. The working name of it was "RP2 Compiler". Here is an section from my README.

Autoinvest transactions can not be retrieved via the REST API and will need to be processed via CSV. They currently have no plans to implement this via API either.

For now, this is low priority. I'm hoping they implement something via the API by the end of the year, so that we don't have to do this twice.

Should we allow for a start time for REST APIs?

For example, Binance will often only give you the most recent transactions by default (e.g. last 30-90 days). You have to explicitly make a request with an earlier start time to get transactions before that. I could start from when the API came online, but it would be useful to limit the requested information to one particular month or tax year.

I'm running into a case where the coin is dead and there's no historical spot price provided by the exchange. The easiest thing to do would be to mark it down to zero or ignore it. what's the best way to do this?

Hi. I'm just getting started with Dali-RP2. I'm trying to use the Coinbase Pro plugin, but I'm getting the following exception. Any guidance on how to resolve? Thanks!

2022-09-09 12:49:38,159/dali/INFO: Country: us
2022-09-09 12:49:38,171/dali/INFO: Initialized input plugin 'dali.plugin.input.rest.coinbase_pro '
2022-09-09 12:49:38,173/dali/INFO: No pair converter plugins found in configuration file: using default pair converters.
2022-09-09 12:49:38,178/dali/INFO: Reading crypto data using plugin 'dali.plugin.input.rest.coinbase_pro '
2022-09-09 12:50:01,429/dali/ERROR: Fatal exception occurred:
Traceback (most recent call last):
File "/home/mint21/.local/lib/python3.10/site-packages/dali/dali_main.py", line 163, in _dali_main_internal
result_list = pool.map(_input_plugin_helper, input_plugin_args_list)
File "/usr/lib/python3.10/multiprocessing/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/lib/python3.10/multiprocessing/pool.py", line 771, in get
raise self._value
File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/home/mint21/.local/lib/python3.10/site-packages/dali/dali_main.py", line 199, in _input_plugin_helper
plugin_transactions = input_plugin.load()
File "/home/mint21/.local/lib/python3.10/site-packages/dali/plugin/input/rest/coinbase_pro.py", line 157, in load
process_account_result_list = pool.map(self._process_account, accounts)
File "/usr/lib/python3.10/multiprocessing/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/lib/python3.10/multiprocessing/pool.py", line 771, in get
raise self._value
File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/home/mint21/.local/lib/python3.10/site-packages/dali/plugin/input/rest/coinbase_pro.py", line 185, in _process_account
self._process_transfer(transaction, currency, intra_transaction_list)
File "/home/mint21/.local/lib/python3.10/site-packages/dali/plugin/input/rest/coinbase_pro.py", line 224, in _process_transfer
transfer_id: str = transaction_details[_TRANSFER_ID]
KeyError: 'transfer_id'
2022-09-09 12:50:01,434/dali/INFO: Log file: ./log/rp2_2022_09_09_12_49_37_655412.log
2022-09-09 12:50:01,434/dali/INFO: Generated output directory: output
2022-09-09 12:50:01,434/dali/INFO: Done

A transfer transaction is recorded in two manual input intra files with matching unique ids and also identical notes. In the DaLi output file the notes appear as one long string with both notes, which now appear to be redundant. Can this be detected programmatically to skip the concatenation when the notes are identical?

Is it correct that the manual CSV plugin expects to receive an "in" file that only contains purchases/acquisitions, and an "out" file that only contains sells/dispositions?

Crypto exchanges generate CSV files with transaction histories that contain a mixture of all transactions, buys and sells. Are there any exchanges that generate one CSV file that contains just purchases and another CSV file that contains just sales? Since there are no exchanges that generate files in this manner, then either a custom script needs to be created to parse the exchange generated CSV files into the files that are ready for DaLI, or manual effort must be done to create the files by hand for DaLI.

Is this correct that a pre-processor is needed for the pre-processor (DaLI)?

When populating unique_id, if it is not a blockchain address, I think we should use <plugin>::<id>, instead of current <id> format just to be safe. While it is very unlikely to happen, there could be a case where two different transactions from two different exchanges end up sharing an identical internal id.

Dali version: 0.3.28

Using Coinbase (not Pro) REST API, and logs are reporting staking rewards unsupported on XTZ transactions.

Below are a couple of examples with masked IDs.

ERROR: Unsupported transaction type (skipping): {"id": "AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAA", "type": "staking_reward", "status": "completed", "amount": {"amount": "0.001642", "currency": "XTZ"}, "native_amount": {"amount": "0.01", "currency": "USD"}, "description": null, "created_at": "2021-XX-XXT19:45:49Z", "updated_at": "2021-XX-XXT19:45:49Z", "resource": "transaction", "resource_path": "/v2/accounts/BBBBBBBB-BBBB-BBBB-BBBB-BBBBBBBBBBBB/transactions/AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAA", "instant_exchange": false, "from": {"id": "CCCCCCCC-CCCC-CCCC-CCCC-CCCCCCCCCCCC", "resource": "user", "resource_path": "/v2/users/CCCCCCCC-CCCC-CCCC-CCCC-CCCCCCCCCCCC", "currency": "XTZ"}, "details": {"title": "Tezos reward", "subtitle": "From Coinbase", "header": "Received 0.001642 XTZ ($0.01)", "health": "positive"}, "hide_native_amount": false}. Please open an issue at https://github.com/eprbell/dali-rp2/issues
ERROR: Unsupported transaction type (skipping): {"id": "DDDDDDDD-DDDD-DDDD-DDDD-DDDDDDDDDDDD", "type": "staking_reward", "status": "completed", "amount": {"amount": "0.001494", "currency": "XTZ"}, "native_amount": {"amount": "0.01", "currency": "USD"}, "description": null, "created_at": "2021-XX-XXT03:38:14Z", "updated_at": "2021-XX-XXT03:38:14Z", "resource": "transaction", "resource_path": "/v2/accounts/BBBBBBBB-BBBB-BBBB-BBBB-BBBBBBBBBBBB/transactions/DDDDDDDD-DDDD-DDDD-DDDD-DDDDDDDDDDDD", "instant_exchange": false, "from": {"id": "CCCCCCCC-CCCC-CCCC-CCCC-CCCCCCCCCCCC", "resource": "user", "resource_path": "/v2/users/CCCCCCCC-CCCC-CCCC-CCCC-CCCCCCCCCCCC", "currency": "XTZ"}, "details": {"title": "Tezos reward", "subtitle": "From Coinbase", "header": "Received 0.001494 XTZ ($0.01)", "health": "positive"}, "hide_native_amount": false}. Please open an issue at https://github.com/eprbell/dali-rp2/issues

Must have missed this from the log files previously, apologies.

Once the CCXT plugin is up and running it would be useful to be able to have some kind of paper trail exist on how the price routing occurred.

For instance, how the CCXT converter plugin found the fiat value of BETH, so that it can be available for the report that is generated in RP2. This would entail listing the market, exchange, and candle where the price was retrieved for each 'hop'. Something like the following:

BETH -> ETH, Binance.com, 5 minute candle starting at Jan 4, 2022 at 12:53 UTC.
ETH -> USDT, Binance.com, 1 minute candle starting at Jan 4, 2022 at 12:53 UTC.
USDT -> USD, Kraken.com, 1 minutes candle starting at Jan 4, 2022 at 12:53 UTC.

This would be useful if a detailed audit did occur and the user would have to explain how they came about the pricing. This hop data would need to be stored in what is returned from get_historic_bar_from_native_source() (I think). Maybe add it to HistoricalBar or an extended class for 'routed' pricing?

Version: 0.3.21
OS: Fedora Linux 35

On Reviewing transactions outputted from DaLI from the coinbase API, conversions do not appear to be entered correctly.

For example I have a conversion of $2.97 worth of ALGO to BTC, there is a IN entry of BUY -$2.97 in the ALGO tab.

I understand that Coinbase will list crypto to crypto trades as transaction type 'trade'. Is the buy side and sell side listed as two different transactions then? Is that why there is a matching process in the Coinbase plugin?

I'm just looking for some clarification because Binance does not list these as separate transactions a crypto to crypto trade comes across as one item.

My guess is that transaction types buy and sell are to and from fiat? and not a stable coin like USDT?

The load() function will need to return a buy and sell transaction in fiat for all crypto to crypto trades correct?

Put another way, when one crypto is traded for the other, we have to imagine a sale of the one crypto into fiat, then that fiat is used to buy the other crypto, and that becomes the basis cost for the purchased crypto. That's my basic understanding of taxes.

Writing a DaLI data loader plugin has high impact on the functionality and usefulness of both DaLI and RP2. Data loader plugins are well-defined, encapsulated modules that translate native exchange REST API-based data into DaLI's standard format. As such they are good first issues for newcomers to the the project.

Before starting, please read the contributing guidelines.

Plugin development is described in the developer documentation, in particular read the Dali Internals and Plugin Development sections, which contain all the information needed to build a new data loader plugin.

Bitstamp REST API is documented here: https://www.bitstamp.net/api/

The Coinbase plugin can be used as an example.

Before submitting a PR for the new plugin, make sure to go through the DaLI Plugin Laundry List.

The Coinbase Pro REST plugin has a few unit tests but more are needed. Extend the existing unit test to test more Coinbase Pro transaction types: deposit, withdrawals, buy, sell, trade, conversion, etc.

I have a one or two exchanges, one being gate.io, which does not allow you to download quant trades as csv or via REST. They do display these trades in html form, however. Is it possible to create HTML plugins?

It would basically be very similar to the CSV plugins, just read in HTML instead of CSV. It would be nice to use the BeautifulSoup package, which is a pretty easy to use html parsing tool instead of Python's html.parser.

The idea would be to download the html of the trades then feed it into dali. Any thoughts?

Writing a DaLI data loader plugin has high impact on the functionality and usefulness of both DaLI and RP2. Data loader plugins are well-defined, encapsulated modules that translate native exchange REST API-based data into DaLI's standard format. As such they are good first issues for newcomers to the the project.

Before starting, please read the contributing guidelines.

Plugin development is described in the developer documentation, in particular read the Dali Internals and Plugin Development sections, which contain all the information needed to build a new data loader plugin.

Kraken REST API is documented here: https://docs.kraken.com/rest/

The Coinbase plugin can be used as an example.

Before submitting a PR for the new plugin, make sure to go through the DaLI Plugin Laundry List.

Writing a DaLI data loader plugin has high impact on the functionality and usefulness of both DaLI and RP2. Data loader plugins are well-defined, encapsulated modules that translate native exchange REST API-based data into DaLI's standard format. As such they are good first issues for newcomers to the the project.

Before starting, please read the contributing guidelines.

Plugin development is described in the developer documentation, in particular read the Dali Internals and Plugin Development sections, which contain all the information needed to build a new data loader plugin.

Gemini REST API is documented here: https://docs.gemini.com/rest-api/

The Coinbase plugin can be used as an example.

Before submitting a PR for the new plugin, make sure to go through the DaLI Plugin Laundry List.

Having a couple issues with black.

1. The newest version of black is behaving differently from older versions. It removes the excess parenthesis here:

$ black --version
black, 22.3.0 (compiled: yes)

                raise RP2ValueError(
                    (
                        f"Invalid converstion {Keyword.INTRA.value}->{Keyword.OUT.value}: "
                        f"{Keyword.CRYPTO_OUT_NO_FEE.value}/{Keyword.CRYPTO_FEE.value} canot be unknown: {transaction}"
                    )
                )

2. There is a compatibility problem with black, click, and pre-commit. Black runs fine by itself from the command line, but pre-commit keeps a separate copy of black and click, and they are not compatible anymore. I think this could be fixed by upgrading the black version inside the pre-commit package cache. However, we will need to re-run black on the entire source tree and commit the resulting changes since the latest version appears to behave a bit differently.

$ black --version
black, 22.3.0 (compiled: yes)

$ black  src/dali/transaction_resolver.py
All done! ✨ 🍰 ✨
1 file left unchanged.

$ pre-commit run --files src/dali/transaction_resolver.py
Check python ast.........................................................Passed
fix UTF-8 byte order marker..............................................Passed
Check builtin type constructor use.......................................Passed
Check for case conflicts.................................................Passed
Check docstring is first.................................................Passed
Check JSON...........................................(no files to check)Skipped
Check for merge conflicts................................................Passed
Check for broken symlinks............................(no files to check)Skipped
Check Toml...........................................(no files to check)Skipped
Check Yaml...........................................(no files to check)Skipped
Debug Statements (Python)................................................Passed
Detect Destroyed Symlinks................................................Passed
Detect Private Key.......................................................Passed
Fix End of Files.........................................................Passed
Mixed line ending........................................................Passed
Pretty format JSON...................................(no files to check)Skipped
Trim Trailing Whitespace.................................................Passed
flake8...................................................................Passed
isort....................................................................Passed
black....................................................................Failed
- hook id: black
- exit code: 1

Traceback (most recent call last):
  File "C:\Users\me\.conda\envs\dali-rp2\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\me\.conda\envs\dali-rp2\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\me\.cache\pre-commit\repoq4eabbla\py_env-default\Scripts\black.EXE\__main__.py", line 7, in <module>
  File "C:\Users\me\.cache\pre-commit\repoq4eabbla\py_env-default\lib\site-packages\black\__init__.py", line 1282, in patched_main
    patch_click()
  File "C:\Users\me\.cache\pre-commit\repoq4eabbla\py_env-default\lib\site-packages\black\__init__.py", line 1268, in patch_click
    from click import _unicodefun  # type: ignore
ImportError: cannot import name '_unicodefun' from 'click' (C:\Users\me\.cache\pre-commit\repoq4eabbla\py_env-default\lib\site-packages\click\__init__.py)

pyupgrade................................................................Passed
blacken-docs.............................................................Passed

Writing a DaLI data loader plugin has high impact on the functionality and usefulness of both DaLI and RP2. Data loader plugins are well-defined, encapsulated modules that translate native exchange REST API-based data into DaLI's standard format. As such they are good first issues for newcomers to the the project.

Before starting, please read the contributing guidelines.

Plugin development is described in the developer documentation, in particular read the Dali Internals and Plugin Development sections, which contain all the information needed to build a new data loader plugin.

Crypto.com REST API is documented here: https://exchange-docs.crypto.com/spot/index.html#introduction

The Coinbase plugin can be used as an example.

Before submitting a PR for the new plugin, make sure to go through the DaLI Plugin Laundry List.

Writing a DaLI data loader plugin has high impact on the functionality and usefulness of both DaLI and RP2. Data loader plugins are well-defined, encapsulated modules that translate native exchange CSV-based data into DaLI's standard format. As such they are good first issues for newcomers to the the project.

Before starting, please read the contributing guidelines.

Plugin development is described in the developer documentation, in particular read the Dali Internals and Plugin Development sections, which contain all the information needed to build a new data loader plugin.

The Trezor plugin can be used as an example.

Before submitting a PR for the new plugin, make sure to go through the DaLI Plugin Laundry List.

Not sure why I did this but I transferred ETH from Coinbase Pro to Coinbase in 2017

INFO: Reading crypto data using plugin 'dali.plugin.input.rest.coinbase <qualifiers>'
ERROR: Unsupported transaction type (skipping):

{
  "id": "b7c4eb75-c701-5533-855f-5af3655fcae9",
  "type": "exchange_withdrawal",
  "status": "completed",
  "amount": {
    "amount": "<REDACTED>",
    "currency": "ETH"
  },
  "native_amount": {
    "amount": "<REDACTED>.00",
    "currency": "USD"
  },
  "description": null,
  "created_at": "2017-12-22T19:31:49Z",
  "updated_at": "2017-12-22T19:31:49Z",
  "resource": "transaction",
  "resource_path": "/v2/accounts/7ced9ea0-8bf6-503b-b623-c012b07e58c8/transactions/b7c4eb75-c701-5533-855f-5af3655fcae9",
  "instant_exchange": false,
  "details": {
    "title": "Transferred Ethereum",
    "subtitle": "From Coinbase Pro",
    "header": "Transferred <REDACTED>.0000 ETH ($<REDACTED>.00)",
    "health": "positive"
  },
  "hide_native_amount": false
}

Maybe this exchange_withdrawal type is similar to a pro_withdrawal intra type ?

My debug log shows a transaction like this (I've redacted a bunch of stuff):

dali/DEBUG: Self-contained transaction: IntraTransaction:
plugin=Coinbase Pro
raw_data={"id": "aaa", "amount": "-100.06", "balance": "0.003", "created_at": "bbb", "type": "transfer", "details": {"transfer_id": "ccc", "transfer_type": "withdraw"}}//{"id": "ddd", "type": "withdraw", "created_at": "eee", "completed_at": "fff", "canceled_at": null, "processed_at": "ggg", "account_id": "hhh", "user_id": "iii", "user_nonce": "jjj", "amount": "100.06", "details": {"fee": "1.06", "subtotal": "99.0", "sent_to_address": "kkk", "coinbase_account_id": "lll", "crypto_transaction_hash": "mmm", "tx_service_transaction_id": "nnn", "coinbase_payment_method_id": ""}, "idem": "ooo"}

For this transaction, DaLi does not have any information from the terminating side of the transfer because I have not provided it yet. The generated rp2 file shows the total amount of the transfer (100.06), but it looks like some of the data elements are not used.
fee = 1.06
subtotal = 99.0

When I eventually provide the data for the terminating side of the transfer, should I expect rp2 will calculate the transfer fee?

Another observation is that the spot price is not filled in. DaLi correctly identified the asset, and I was expecting the spot price to be filled when using the -s option. What am I doing wrong?

Currently we have to manually adjust thread_count for each exchange. However, each exchange specifies its own rate limit.

So I think it would be nice to have some kind of api call limiter. Then we don't have to manually adjust thread_count.

In terms of api call limit, some exchanges limit based on individual endpoint, others limit based on UID (thus, all endpoints combined).

Binance for example,

Each endpoint with IP limits has an independent 12000 per minute limit.
Each endpoint with UID limits has an independent 180000 per minute limit.

https://binance-docs.github.io/apidocs/spot/en/#limits

Coinbase seems to limit only based on individual endpoint

REST API
For Public Endpoints, our rate limit is 3 requests per second, up to 6 requests per second in bursts. For Private Endpoints, our rate limit is 5 requests per second, up to 10 requests per second in bursts.

https://help.coinbase.com/en/pro/other-topics/api/faq-on-api

Kucoin seem to limit only based on UID

REST API

The limit strategy of private endpoints will restrict account by userid. The limit strategy of public endpoints will restrict IP.

https://docs.kucoin.com/#request-rate-limit

Tried to import via Coinbase API, it processes for a minute or two and then errors out. Nothing created in output folder.

Version:
DaLI 0.3.20 (https://pypi.org/project/dali_rp2/)
OS: Fedora

Ini file:

[dali.plugin.input.rest.coinbase]
account_holder = my name
api_key = ***
api_secret = ***

Logs:

2022-03-23 22:30:43,532/dali/INFO: Reading crypto data using plugin 'dali.plugin.input.rest.coinbase'
2022-03-23 22:34:30,411/dali/INFO: Resolving transactions
2022-03-23 22:34:30,411/dali/ERROR: Fatal exception occurred:
Traceback (most recent call last):
File "/home/james/.local/lib/python3.10/site-packages/dali/dali_main.py", line 107, in input_loader
resolved_transactions: List[AbstractTransaction] = resolve_transactions(transactions, dali_configuration, args.read_spot_price_from_web)
File "/home/james/.local/lib/python3.10/site-packages/dali/transaction_resolver.py", line 169, in resolve_transactions
transaction = _apply_transaction_hint(transaction_list[0], global_configuration)
File "/home/james/.local/lib/python3.10/site-packages/dali/transaction_resolver.py", line 211, in _apply_transaction_hint
if transaction.unique_id not in global_configuration[Keyword.TRANSACTION_HINTS.value]:
KeyError: 'transaction_hints'
2022-03-23 22:34:30,412/dali/INFO: Log file: ./log/rp2_2022_03_23_22_30_43_325160.log
2022-03-23 22:34:30,412/dali/INFO: Generated output directory: output/
2022-03-23 22:34:30,412/dali/INFO: Done

I'd like to suggest making the DaLi configuration dict globally available, similar to the way LOGGER is globally available. This will make the configuration available everywhere without passing it around as a function argument. The code could be as simple as this:

# dali_configuration.py
CONFIG: Dict[str, Any] = copy.deepcopy(DEFAULT_CONFIGURATION)

# Everywhere else
from dali.dali_configuration import CONFIG

# Sample code in dali_main
CONFIG[section_name] = _validate_transaction_hints_configuration(ini_config, section_name)

I haven't tried this yet. Any pitfalls I may be overlooking?

Hi,

I've been testing DaLI and have had trouble with including fee-only out-transactions in my out_csv_file

I'm getting this error: rp2.rp2_error.RP2ValueError: Invalid keyword: FEE

I'm using the latest rp2 (0.9.26) and dali-rp2 (0.4.1) python packages.

I've attached my in_csv_file and out_csv_file.

in_binance.csv
out_binance.csv

Kucoin provides sandbox environment for futures trading.
https://sandbox-futures.kucoin.com

Kucoin Futures API Documentation
https://docs.kucoin.com/futures/

The Coinbase REST plugin has a few unit tests but more are needed. Extend the existing unit test to test more Coinbase transaction types: buy, sell, trade, interest, conversion, etc.

Writing a DaLI data loader plugin has high impact on the functionality and usefulness of both DaLI and RP2. Data loader plugins are well-defined, encapsulated modules that translate native exchange CSV-based data into DaLI's standard format. As such they are good first issues for newcomers to the the project.

Before starting, please read the contributing guidelines.

Plugin development is described in the developer documentation, in particular read the Dali Internals and Plugin Development sections, which contain all the information needed to build a new data loader plugin.

The Trezor plugin can be used as an example.

Before submitting a PR for the new plugin, make sure to go through the DaLI Plugin Laundry List.

When I run DaLI with the -s flag using the .csv.manual plugin and any of my input tables are missing spot prices DaLI is throwing the following error:

Exception: Internal error: spot_price is empty

Actually this is an interesting idea: would it make sense to add variants (subclasses) of your CCXT converter that use a fixed exchange (one for Binance, one for Kraken, etc.)? The binance subclass would be exactly equivalent to the pair converter in this PR. Additionally, if we added a CCXT converter subclass for Coinbase, it should be equivalent to the HistoricCrypto plugin. Does that sound correct or am I missing something?

Originally posted by @eprbell in #60 (comment)

Implement subclasses or create a method of prioritizing one exchange's pricing data over another.