buildwithdata / btc-etf-tracker Goto Github PK
View Code? Open in Web Editor NEWTrack Bitcoin ETFs with high quality data FOR FREE
Home Page: https://twitter.com/BTCETFTracker
Track Bitcoin ETFs with high quality data FOR FREE
Home Page: https://twitter.com/BTCETFTracker
python runner/db/inflows_btc.py -d "2024-04-12"
python runner/db/inflows_btc_bfill.py -d "2024-04-12"
not raising warning message when data are not found
GBTC -> BTC
NEXT STEPS
week TEXT NOT NULL,
change to
week INT NOT NULL,
this is needed otherwise sorting fails like this:
week
1
10
11
...
2
3
4
Add visualization for Hong Kong ETFs
TODO
src/visualization/hk.py
src/runner/visualization/hk.py
OTHERS
src/visualization/us.py
general ? for example current_holdings
this path should be derived from config file
https://github.com/BuildWithData/BTC-ETF-Tracker/blob/bb3f69cb556e29756da29879c017ce1704c57a9a/config.yaml#L3C5-L3C9
Scrapers here run with Chrome only, but would be better to support at least another one to give users more choice
There are other BTC ETF like products outside USA and would be cool to track them
They can all be found here in the etf section, would be great to fill this table:
Ticker | Website | Scraped |
---|---|---|
ETC | https://etc-group.com/products/etc-group-physical-bitcoin | NO |
BITC.SW | https://etp.coinshares.com/physical/bitcoin | NO |
add option --ticker
to runner/db/update_raw.py
to load that for target ETF only:
python ./runner/db/update_raw --ticker BRRR
When scraping FBTC and ARKB we cannot pass this option
options.add_argument('--headless')
if we do it, then scraping fails...
if we wanna put this on a remote server and automate it, I guess we cannot do it... would be possible if we make this option always work
looks like BRRR is faling to extract data from 17-04-2024 bcs the DOM has changed
Scenario: data from 2024-03-28 are missing
Problem: likely the xls file structure has changed
Scenario:
Suppose on day X
there is no trading bcs TradFi is shut down (eg christmas), data are scraped and then data are extracted with ref_date = X -1
On Day X + 1
trading is back, ie on Day X + 2
data are updated and when they are scraped ref_date = X + 1
Problem:
Any consumption table will be missing a row with ref_date= X
Solution:
Add row for ref_date= X
with data from X -1
scraper fails to extract data when day = 24 in ref_date_xls
create heatmap like here
Grayscale has released their mini etf with ticker BTC, we have to add this to the repo
Already added scraper, check branch feature/btc
, next is to write code to extract data and create tables
add scraper to scrape data from here
TODO
src/product/etp/btcw.py
src/runner/scarping/scrape.py
NEXT STEP
Looks like ETFs are being approved by the end of April
resources.md
Let's scrape ETH ETFs
all scrapers can be found on this branch
what's left to do now is to code the extract
and the update_db
methods
let's start by coding the first one, when this is done pls push the code on a new branch called feature/etf_ticker
so we can all test it
if testing is succesfull, we gonna merge it and then code the second method
for some reason sqlite fails to round to 2 decimal digits sometimes, for example:
select * from inflows_btc_bxfill
+------------+-------------------+
| ref_date | TOTAL |
+------------+-------------------+
| 2024-01-11 | 13796.2 |
| 2024-01-12 | 4023.2 |
| 2024-01-15 | |
| 2024-01-16 | -1947.1 |
| 2024-01-17 | 9256.8 |
| 2024-01-18 | -4514.5 |
| 2024-01-19 | -545.900000000002 |
| 2024-01-22 | -2350.3 |
but it seems this is kinda random, for example inflows_btc_bfill
is created with data from inflows_btc
but:
select total from inflows_btc_bfill where ref_date = '2024-03-22'
+---------+
| TOTAL |
+---------+
| -310.93 |
+---------+
select total from inflows_btc where ref_date = '2024-03-22'
+-------------------+
| TOTAL |
+-------------------+
| -310.929999999999 |
+-------------------+
but what is weirder is that rounding is done in both cases:
failing to handle properly database path when specified in config file
Add option to force uploading of data, this would be especially handy for no coders so they do not have to connect to db and run query to:
python runner/db/update_raw --from-date "2024-04-13" --force
implement this in all runners
For developers:
If no coding skills:
create table or graph to aggregate data on a monthly and weekly basis to show the overall trend
wrong logic
hashdex ETF
TODO
src/product/etp/defi.py
src/runner/scarping/scrape.py
NEXT STEP
when user type data more than once, like here:
Path:
root: /Users/bwd/BTC-ETF-Tracker
data: /Users/bwd/BTC-ETF-Tracker/data
db: /Users/bwd/BTC-ETF-Tracker/data_refactor/db
then DATA_PATH
contains multiple paths, instead it should keep the one specified at data
Scraped data are saved with timestamp in their filename, timestamp must be UTC
Holdings are not reported here, the only way we can calculate them is:
holdings ~= AUM / btc_ref_price
Problem:
Scenario: seems like the website has changed and in the landing page a banner pops out asking whether to continue to the USA or EU website
Problem:
Solution: catch this banner and press the link to USA website
Add option to ingest data from a reference target date
python runner/db/update_raw --from-date "yyyy-mm-dd"
this option cannot be passed if option --date
is passed as well and viceversa
add option to drop target tickers when scraping
Hi Team,
Thanks for your great job! I really appreciate it.
This issue is to kindly ask for the update of the scraping data, which starts from Feb 23rd and ends on May 19th.
Many thanks for your help.
Since June when scraping the webpage, which is saved as an html file, relevant data like number of shares are missing from the saved html file.
This is probably bcs they have changed the webpage and the scraping commands are not effective anymore.
This causes the scraper to fail when the extract
method is called as either empty strings or None
values are found
I do not know how to tackle this bug, but we are not using any data from the webpage itself as holdings are found in the csv file.
So let's turn off this for now meanwhile we figure out how to fix this bug
From late July BRRR data are available here https://valkyrie-funds.com/brrr/
Found out about this in early august, so pretty much we have been scraping an empty webpage.
Not a big deal... holdings on the 24th of July were 8877.57 whereas on 2nd of Aug 8876.9,
so we did not miss much
method to extract data from html fails from 2024-05-22 -> DOM has changed
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.