snth / numismatic Goto Github PK

See this comment: #56 (comment)

Right now, we have two concepts:

1. Feed - used mainly as a REST endpoint
2. Exchange - used mainly for websockets

Both of these names seem non-optimal:

• Feed is too generic. Anything that can be polled for data can be a feed
• Exchange is just one type of entity that can be used in the crypto-currency eco system (what about wallets and payment services - both of which can have APIs).

Honestly, I am not even sure if there is a better naming, but lets start the debate shall we?

Currently the output_stream writing is handled in cli:collect. The cli module should just be a thin wrapper to expose the API and the actual business logic should be handled elsewhere. There is similar logic in handling the raw_output streams. This should be factored out into common code in a collectors module.

Command "python setup.py egg_info" failed with error code 1 in /private/tmp/pip-install-14pvn2be/toolz/

I have no idea what could it be

In particular I would like to describe how to use the ~/.coinrc file.

What is the modern way of easily adding documentation? I would prefer to write in markdown rather than rst.

The following fails:

$ coin collect --help
Traceback (most recent call last):
  File "/opt/venv-numismatic/bin/coin", line 11, in <module>
    load_entry_point('numismatic', 'console_scripts', 'coin')()
  File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 1085, in invoke
    allow_interspersed_args=False)
  File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 621, in make_context
    self.parse_args(ctx, args)
  File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 880, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
  File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 1404, in handle_parse_result
    self.callback, ctx, self, value)
  File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 78, in invoke_param_callback
    return callback(ctx, param, value)
  File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 809, in show_help
    echo(ctx.get_help(), color=ctx.color)
  File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 496, in get_help
    return self.command.get_help(self)
  File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 830, in get_help
    self.format_help(ctx, formatter)
  File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 845, in format_help
    self.format_options(ctx, formatter)
  File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 859, in format_options
    rv = param.get_help_record(ctx)
  File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 1608, in get_help_record
    rv = [_write_opts(self.opts)]
  File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 1605, in _write_opts
    rv += ' ' + self.make_metavar()
  File "/opt/venv-numismatic/lib/python3.6/site-packages/click/core.py", line 1298, in make_metavar
    metavar = self.type.get_metavar(self)
  File "/opt/venv-numismatic/lib/python3.6/site-packages/click/types.py", line 140, in get_metavar
    return '[%s]' % '|'.join(self.choices)
TypeError: sequence item 0: expected str instance, NoneType found

How about this folder structure (this is just a suggestion, please alter):

• numismatic
  • cli.py
  • lib
    • utils
    • requesters
    • events
  • datafeeds
    • base.py (this is the abc)
    • cryptocompare.py
    • luno.py
    • bravenewcoint.py
      *... etc
  • exchanges
    • base.py
    • luno.py <--- easy to see that luno is used both as a feed and as an exchange
    • bitfinex.py
    • ... etc

This is just a starting point for conversation...

Brave new coin (for example) requires authentication via tokens that are passed as a custom HTTP header. The requests library therefore need the ability to use custom headers

Some exchanges use BTC others use XBT. Similarly for MNR and XMR.

Currently the writes to disk use blocking io operations. This will likely be a problem on high volume feeds.

Change this to use the aiofiles package.

We need to determine what methods are common to every exchange and/or feed. Only those methods should be in the base class. This will take understanding which will come about after we implement a few exchanges/feeds.

The use of symbol isn't consistent across exchanges, e.g. BTCUSD vs BTC-USD vs USDT_BTC. To avoid this I think we should rather record asset and currency separately.

Each symbol may also be better referred to as pair.

These are used together in the same place so can probably be combined or split in a different way.

See #56 (comment)

That should make it more clear to get an overview of what's available for new contributors and help avoid merge conflicts.

The syntax for auto_envvar_variables is obscure to find but I eventually got it as PREFIX_FUNCTION_VARIABLE from values-from-environment-variables

e.g. the function "info" with flag "--assets" expects a value like $ export NUMISMATIC_INFO_ASSETS=ETH

And you do have correct syntax in cli.py, if one wanted to use the auto prefix feature.
coin(auto_envvar_prefix=ENVVAR_PREFIX)

Unfortunately, the feature is restrictive in that it requires the function name in the middle i.e. NUMISMATIC_INFO_ASSETS, when we just want NUMISMATIC_ASSETS for use in info and prices.

Since it doen't work neatly for assets and currencies, then you could remove the feature, provided you don't need it for any other variables (where the function name is the middle is appropriate). That would be a single line code change to just have coin() at the end. And you can keep using your f-string style throughout as envvar=f'{ENVVAR_PREFIX}_ASSETS', which is the most economical in this situation without using the auto_envvars_prefix feature..

Let me know what you think.

The output_stream writer writes every single event. The raw stream does have a batch size but this is of a fixed size. This should use the Stream.timed_window operation but that requires running an event loop in a separate thread.

This issue is for creating that separate thread event loop and then having constant time interval flushes to disk.

A good example is CryptoCompare - it has many methods that are specific to itself (for example, get_historical_hour or get_info. The latter is surprising, but I could not find it on brave new coin.

What we need to do is discuss what specific methods we need (see #35) and move those to the base.

For methods specific, we should add the feed/exchange name as a prefix. For example, cryptocompare_get_info. The trick would come in when we want to run this on the cli. So running coin -f cryptocompare info would run cryptocompare_get_info, and coin -f bravenewcoin xyz would run bravenewcoin_get_xyz