https://www.microprediction.com/stream_dashboard.html?stream=z1~bronx_traffic_absolute_speed~910&horizon=310

As a little starter exercise, I hope someone can generalize the stars.py script at https://github.com/microprediction/microprediction/tree/master/stream_examples_stars to include multiple repositories. This would be a nice way to encourage maintainers of analytics packages to participate!

When using get_transactions it seems to only return a fixed window of results, such as the last n.

If I'm working with many streams I may have more transactions than fit into that window of records. At this time there doesn't seems to be a way to have a cursor through the transaction records such that I can retrieve all of them in a reliable fashion from a fixed point in time.

By not having this I need to rely on the performance trackers per stream which not as granular. If you need me to clarify my use case I'm happy to.

dennis... more details to follow

This is found in one of the READMEs at the root of the repo.
For importing MicroPoll what seems to work is from microprediction.polling import MicroPoll

For create_key it is not clear. I found create_key methods in old_conventions.py but not a standalone function.
Maybe one needs something from the separate muid module (?)

In your blog, you forgot to include pytorch-forecasting which is a new package with already 3372 downloads last month.

I lack permission to add cards to your project as you suggested in the blog.

https://www.microprediction.com/stream_dashboard.html?stream=z1~traffic_absolute_speed~310&horizon=310

Add get_median and a few little conveniences on top of the cdf

See promises made at https://github.com/DrDonDon/push_electricity_2_microprediction/blob/master/README.md

Remove copulas from dependencies as we surely don't need this, do we?

That package insists on an old version of numpy and that kind of thing is a deal-breaker going forward

e.g.

https://github.com/dataiku/dss-plugin-timeseries-forecast/blob/master/python-lib/gluonts_forecasts/model.py

or

https://github.com/awslabs/gluon-ts/blob/master/examples/benchmark_m4.py

If the same data is pushed twice to the same stream resulting in data points like:

8/29 17:17:02 | 16
8/29 17:17:02 | 16

The graph does not deduplicate the time series leading to spikes being shown.

testable clam not working as expected

Weird sorting thing

Numpy version in colab is old and this illustrates a problem with install.

Need to fix the reqs probably to ensure relatively recent numpy

Reproduce:
https://github.com/microprediction/microprediction-notebooks/blob/main/issue_with_colab_older_numpy.ipynb

https://arxiv.org/pdf/1908.05164.pdf

I think that 25 isn't the right delay value to specify here.

https://github.com/microprediction/microprediction/blob/master/docs/glossary.md

In Microreader.get_lagged_times the return value is wrong, it says lagged_values should be lagged_times

   def get_lagged_times(self, name:str, count:int=1000) -> list:
        """ Retrieve lagged times
        :param name:    cop.json   z1~cop.json   z2~cop~qp.json
        :return: [ float ]
        """
        lagged_values, lagged_times = self.get_lagged_values_and_times(name=name, count=count)
        return lagged_values

This seems to have changed, my integration tests caught this.

JSON doesn't allow NaNs.

$ curl http://api.microprediction.org/live/delayed::70::South_Australia_Electricity_Price.json
NaN

$ curl http://api.microprediction.org/live/delayed::310::South_Australia_Electricity_Price.json
NaN

$ curl http://api.microprediction.org/live/delayed::910::South_Australia_Electricity_Price.json
NaN

See also #79

https://www.predictit.org/api/marketdata/markets/7456

e.g.
https://colab.research.google.com/drive/1PAZ6Umk0lWJmcilRUiOg47_hAhftjLGI?usp=sharing

https://www.microprediction.org/stream_dashboard.html?stream=South_Australia_Electricity_Price&horizon=70

https://www.microprediction.com/stream_dashboard.html?stream=three_body_z&horizon=70

Could be caused by not enough actives

Maybe https://github.com/aangelopoulos/conformal-prediction/blob/main/notebooks/weather-time-series-distribution-shift.ipynb

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3572181

don't need values anymore
https://github.com/microprediction/microprediction/blob/master/microprediction/reader.py

Seen this before

building 'accumulation_tree.accumulation_tree' extension
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/accumulation_tree
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/nfs/i1/c/home/peter/virtual-envs/optim/include -I/usr/include/python3.7m -c accumulation_tree/accumulation_tree.c -o build/temp.linux-x86_64-3.7/accumulation_tree/accumulation_tree.o
accumulation_tree/accumulation_tree.c:4:10: fatal error: Python.h: No such file or directory
#include "Python.h"
^~~~~~~~~~
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

Failed building wheel for accumulation-tree
Running setup.py clean for accumulation-tree
Failed to build accumulation-tree
Installing collected packages: contexttimer, muid, contextlib2, schema, pymorton, ordered-set, deepdiff, accumulation-tree, tdigest, microconventions, microprediction
^COperation cancelled by user

Refactor prizes, shrinking of leaderboards (DRY)

Wanker.

:50:01 Submitted to sateb_4.json 310s horizon, and will do so again in 118 seconds.
Nov 23 15:50:04 Submitted to sateb_7.json 70s horizon, and will do so again in 121 seconds.
Nov 23 15:50:05 Traceback (most recent call last):
Nov 23 15:50:05 File "/home/micropredictioncrawlers/crawlers/exhalable_cat.py", line 30, in
Nov 23 15:50:05 crawler.run()
Nov 23 15:50:05 File "/home/micropredictioncrawlers/.local/lib/python3.8/site-packages/microprediction/crawler.py", line 719, in run
Nov 23 15:50:05 self.predict_and_submit(name=name, delay=delay, lagged_times=lagged_times)
Nov 23 15:50:05 File "/home/micropredictioncrawlers/.local/lib/python3.8/site-packages/microprediction/crawler.py", line 482, in predict_and_submit
Nov 23 15:50:05 scenario_values = self.sample(lagged_values=lagged_values, lagged_times=lagged_times, name=name,
Nov 23 15:50:05 File "/home/micropredictioncrawlers/.local/lib/python3.8/site-packages/microprediction/fitcrawler.py", line 118, in sample
Nov 23 15:50:05 state = self.update_state(state=state, lagged_values=lagged_values, lagged_times=lagged_times)
Nov 23 15:50:05 File "/home/micropredictioncrawlers/.local/lib/python3.8/site-packages/microprediction/sequentialcrawler.py", line 59, in update_state
Nov 23 15:50:05 machine.update(value=value, dt=dt)
Nov 23 15:50:05 File "/home/micropredictioncrawlers/.local/lib/python3.8/site-packages/microprediction/univariate/expnormdist.py", line 48, in update
Nov 23 15:50:05 move = self.params['g2'] * math.tanh(self.params['g1'] * dy / self.params['g2'])
Nov 23 15:50:05 TypeError: 'NoneType' object is not subscriptable

(qqq)  ~/Dev/AnacondaProjects/microprediction/ [microprediction] pip install microprediction
Collecting microprediction
Using cached microprediction-0.19.5-py3-none-any.whl (71 kB)
Collecting microconventions>=0.5.0
Using cached microconventions-0.5.4-py3-none-any.whl (16 kB)
Collecting apscheduler
Using cached APScheduler-3.9.1-py2.py3-none-any.whl (59 kB)
Collecting tdigest
Using cached tdigest-0.5.2.2-py3-none-any.whl (9.4 kB)
Collecting copulas
Using cached copulas-0.3.0-py2.py3-none-any.whl (44 kB)
Collecting genson
Using cached genson-1.2.2.tar.gz (34 kB)
Preparing metadata (setup.py) ... done
Collecting hyperopt
Using cached hyperopt-0.2.7-py2.py3-none-any.whl (1.6 MB)
Collecting pytz>=2021.3
Using cached pytz-2021.3-py2.py3-none-any.whl (503 kB)
Collecting contexttimer
Using cached contexttimer-0.3.3.tar.gz (4.9 kB)
Preparing metadata (setup.py) ... done
Collecting requests
Using cached requests-2.27.1-py2.py3-none-any.whl (63 kB)
Collecting pycoingecko
Using cached pycoingecko-2.2.0-py3-none-any.whl (8.3 kB)
Collecting scikit-learn
Using cached scikit_learn-1.0.2-cp310-cp310-macosx_10_13_x86_64.whl (8.0 MB)
Collecting pandas
Using cached pandas-1.4.1-cp310-cp310-macosx_10_9_x86_64.whl (11.5 MB)
Collecting numpy>=1.20.1
Using cached numpy-1.22.3-cp310-cp310-macosx_10_14_x86_64.whl (17.6 MB)
Collecting pathlib
Using cached pathlib-1.0.1.tar.gz (49 kB)
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [1 lines of output]
ERROR: Can not execute setup.py since setuptools is not available in the build environment.
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

https://projects.propublica.org/emergency/state/NY

Probably not updated frequently enough ... but maybe in aggregate

Fix the ordering of streams in master list by:

• latency penalty

https://opendata.transport.nsw.gov.au/node/344/exploreapi