Test by fetching the information from the web (no mocks). This method compromises on using external resources in the tests.
Later to be separated into 1) mocks and 2) live tests.

They have plain data on:

• https://www.boi.org.il/en/Markets/ExchangeRates/Pages/Default.aspx

And XML API is explained on:

• https://www.boi.org.il/en/Markets/Pages/explainxml.aspx

For example:
https://www.boi.org.il/currency.xml?rdate=20221125

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<CURRENCIES>
  <LAST_UPDATE>2022-11-25</LAST_UPDATE>
  <CURRENCY>
    <NAME>Dollar</NAME>
    <UNIT>1</UNIT>
    <CURRENCYCODE>USD</CURRENCYCODE>
    <COUNTRY>USA</COUNTRY>
    <RATE>3.419</RATE>
    <CHANGE>-0.029</CHANGE>
  </CURRENCY>
  <CURRENCY>
    <NAME>Pound</NAME>
    <UNIT>1</UNIT>
    <CURRENCYCODE>GBP</CURRENCYCODE>
    <COUNTRY>Great Britain</COUNTRY>
    <RATE>4.1313</RATE>
    <CHANGE>-0.323</CHANGE>
  </CURRENCY>
  <CURRENCY>
    <NAME>Yen</NAME>
    <UNIT>100</UNIT>
    <CURRENCYCODE>JPY</CURRENCYCODE>
    <COUNTRY>Japan</COUNTRY>
    <RATE>2.4524</RATE>
    <CHANGE>-0.781</CHANGE>
  </CURRENCY>
  <CURRENCY>
    <NAME>Euro</NAME>
    <UNIT>1</UNIT>
    <CURRENCYCODE>EUR</CURRENCYCODE>
    <COUNTRY>EMU</COUNTRY>
    <RATE>3.5558</RATE>
    <CHANGE>-0.056</CHANGE>
  </CURRENCY>
  <CURRENCY>
    <NAME>Dollar</NAME>
    <UNIT>1</UNIT>
    <CURRENCYCODE>AUD</CURRENCYCODE>
    <COUNTRY>Australia</COUNTRY>
    <RATE>2.3033</RATE>
    <CHANGE>-0.152</CHANGE>
  </CURRENCY>
  <CURRENCY>
    <NAME>Dollar</NAME>
    <UNIT>1</UNIT>
    <CURRENCYCODE>CAD</CURRENCYCODE>
    <COUNTRY>Canada</COUNTRY>
    <RATE>2.5587</RATE>
    <CHANGE>-0.105</CHANGE>
  </CURRENCY>
  <CURRENCY>
    <NAME>krone</NAME>
    <UNIT>1</UNIT>
    <CURRENCYCODE>DKK</CURRENCYCODE>
    <COUNTRY>Denmark</COUNTRY>
    <RATE>0.4781</RATE>
    <CHANGE>-0.042</CHANGE>
  </CURRENCY>
  <CURRENCY>
    <NAME>Krone</NAME>
    <UNIT>1</UNIT>
    <CURRENCYCODE>NOK</CURRENCYCODE>
    <COUNTRY>Norway</COUNTRY>
    <RATE>0.3448</RATE>
    <CHANGE>0.378</CHANGE>
  </CURRENCY>
  <CURRENCY>
    <NAME>Rand</NAME>
    <UNIT>1</UNIT>
    <CURRENCYCODE>ZAR</CURRENCYCODE>
    <COUNTRY>South Africa</COUNTRY>
    <RATE>0.1994</RATE>
    <CHANGE>-0.747</CHANGE>
  </CURRENCY>
  <CURRENCY>
    <NAME>Krona</NAME>
    <UNIT>1</UNIT>
    <CURRENCYCODE>SEK</CURRENCYCODE>
    <COUNTRY>Sweden</COUNTRY>
    <RATE>0.3281</RATE>
    <CHANGE>0.336</CHANGE>
  </CURRENCY>
  <CURRENCY>
    <NAME>Franc</NAME>
    <UNIT>1</UNIT>
    <CURRENCYCODE>CHF</CURRENCYCODE>
    <COUNTRY>Switzerland</COUNTRY>
    <RATE>3.6188</RATE>
    <CHANGE>-0.229</CHANGE>
  </CURRENCY>
  <CURRENCY>
    <NAME>Dinar</NAME>
    <UNIT>1</UNIT>
    <CURRENCYCODE>JOD</CURRENCYCODE>
    <COUNTRY>Jordan</COUNTRY>
    <RATE>4.8228</RATE>
    <CHANGE>0.108</CHANGE>
  </CURRENCY>
  <CURRENCY>
    <NAME>Pound</NAME>
    <UNIT>10</UNIT>
    <CURRENCYCODE>LBP</CURRENCYCODE>
    <COUNTRY>Lebanon</COUNTRY>
    <RATE>0.0226</RATE>
    <CHANGE>0</CHANGE>
  </CURRENCY>
  <CURRENCY>
    <NAME>Pound</NAME>
    <UNIT>1</UNIT>
    <CURRENCYCODE>EGP</CURRENCYCODE>
    <COUNTRY>Egypt</COUNTRY>
    <RATE>0.1391</RATE>
    <CHANGE>0</CHANGE>
  </CURRENCY>
</CURRENCIES>

It's a relatively low-priority task as most banks list the official rates (שערים יציגים) themselves, which are the BOI rates.

The release.yml workflow may probably be adapted:

• #76
• #75
• #77
  See: https://github.com/jond01/xil/actions/runs/4645934156/jobs/8222050591#step:5:19

Originally posted by @jond01 in #73 (comment)

PR #12, which solved issue #5, introduced many code duplications between the different modules.
Instead of rewriting the same lines repeatedly, design classes, implement proper inheritance structure, and use it in the bank scrapers.

See:
https://docs.pytest.org/en/7.4.x/example/markers.html

Some of the tests do not consume live data from the web. Therefore, there is no reason for running them without a code change.

• #137
• #138
• #139
• #140

https://hypothesis.readthedocs.io/en/latest/data.html#hypothesis.provisional.urls

This page:
https://www.boi.org.il/en/BankingSupervision/BanksAndBranchLocations/Pages/Default.aspx

Has the list of Israeli banks:

• XLS: https://www.boi.org.il/en/BankingSupervision/BanksAndBranchLocations/Lists/BoiBankBranchesDocs/banking_corporations_en.xls
• XML: https://www.boi.org.il/en/BankingSupervision/BanksAndBranchLocations/Lists/BoiBankBranchesDocs/banking_corporations_en.xml

Both are the same but in different formats.

Add a test that validates the banks list against a constant list - useful to detect changes in Israeli banks activity.

https://github.com/jond01/xil/actions/runs/6708383613

After a quick examination - there are probably two separate issues.

Need to understand if the following assertions are correct:

The raw JSON from a simple scraping is different than the one I see on the web browser.
The scraped one, which is used in XIL, can be obtained with:

import urllib.request

with urllib.request.urlopen("https://dv16ymfyh91nr.cloudfront.net/MarketingRatesReport/MarketingSiteFCYRatesCurrentReport.json") as f:
    print(f.read().decode())

The difference is:

{
    "generatingReportDateTime": "2023-10-31",
    "marketingRecords": [
        {
             "fromCurrency": "EUR",
             "toCurrency": "ILS",
             "representativeExchangeRate": 0.2329,
-            "buyRate": 0.2326,
-            "sellRate": 0.2349
+            "buyRate": 0.2318,
+            "sellRate": 0.2341
         },
         {
             "fromCurrency": "ILS",
             "toCurrency": "EUR",
             "representativeExchangeRate": 4.2934,
-            "buyRate": 4.2573,
-            "sellRate": 4.3001
+            "buyRate": 4.2711,
+            "sellRate": 4.3141
         },
         {
             "fromCurrency": "USD",
             "toCurrency": "ILS",
             "representativeExchangeRate": 0.2466,
-            "buyRate": 0.2467,
-            "sellRate": 0.2492
+            "buyRate": 0.2457,
+            "sellRate": 0.2482
         },
         {
             "fromCurrency": "ILS",
             "toCurrency": "USD",
             "representativeExchangeRate": 4.0550,
-            "buyRate": 4.0135,
-            "sellRate": 4.0538
+            "buyRate": 4.0288,
+            "sellRate": 4.0693
         }
     ]
 }

Need to understand where does this difference data come from.

Following #35 (which solved #29), the pre-commit job added to the total time of the "tests" workflow.
The step that added the most time is:

To save time and resources, cache the environment of the hooks:

• https://pre-commit.com/#managing-ci-caches
• https://pre-commit.com/#github-actions-example

The cache location (on ubuntu-latest) is: /home/runner/.cache/pre-commit/.

xil/__init__.py should contain a __version__ attribute.
It has to match the tool.poetry.version key in pyproject.toml (and ideally with the recent Git tag):

https://peps.python.org/pep-0396/

There are several ways to solve the single source of truth issues with Poetry.
Some options are:

• https://github.com/mtkennerly/poetry-dynamic-versioning

In some other build systems, this problem is solved:

• Hatch: https://hatch.pypa.io/latest/version/
• Flit: https://flit.pypa.io/en/stable/#usage
• PDM: https://pdm.fming.dev/latest/reference/pep621/#package-version
• https://packaging.python.org/en/latest/guides/single-sourcing-package-version/#single-sourcing-the-package-version

E.g., https://www.bankhapoalim.co.il/robots.txt.

Make sure scraping complies with the terms on these pages.
Use: https://docs.python.org/3/library/urllib.robotparser.html.

The BOI updated their website, and the link:

The relevant pages are:

• HE: https://www.boi.org.il/%D7%94%D7%AA%D7%A4%D7%A7%D7%99%D7%93%D7%99%D7%9D-%D7%94%D7%9B%D7%9C%D7%9B%D7%9C%D7%99%D7%99%D7%9D/statistics/%D7%91%D7%A0%D7%A7%D7%99%D7%9D/%D7%94%D7%91%D7%A0%D7%A7%D7%99%D7%9D-%D7%95%D7%A1%D7%A0%D7%99%D7%A4%D7%99%D7%94%D7%9D/%D7%94%D7%91%D7%A0%D7%A7%D7%99%D7%9D-%D7%95%D7%A1%D7%A0%D7%99%D7%A4%D7%99%D7%94%D7%9D/#mainContent
  No appropriate data candidate.
• EN: https://www.boi.org.il/en/economic-roles/statistics/
  Looks like the banks data is missing.

The relevant data as was downloaded from the currently down URL on Nov 25, 22:
banking_corporations_en.xls

(Only the relevant columns are shown above.)

Consider adding a skip or removing the test.

Isort may be completely replaced with Ruff by enabling its rules:
https://beta.ruff.rs/docs/rules/#isort-i

Update the configuration accordingly ([tool.isort] -> [tool.ruff.isort]).

https://beta.ruff.rs/docs/faq/#how-does-ruffs-import-sorting-compare-to-isort

It takes ~11 sec. See this run:
https://github.com/jond01/xil/actions/runs/3863788278/jobs/6586187770#step:3:1

All the other steps in the tests workflow take significantly shorter time - between 0 and 2 sec (with caching).

Use actions/cache to cache this step (conditional on Poetry's version).

https://coverage.readthedocs.io/

The data model completely changed - from an embedded table in HTML to a JSON endpoint:
https://www.discountbank.co.il/api/ExchangeRates/GetExchangeRates?bankCode=0011

The xil.discount module needs an update.

Fetch the data from the web (URLs) asynchronously.
The AIOHTTP package may be helpful for this matter.

Union bank exchange rates URL now redirects to Mizrahi Tefahot exchange rates URL:

https://www.unionbank.co.il/Igud/שוק-ההון-ומטח/מטבע-חוץ/שערי-מטח
⬇️
https://www.mizrahi-tefahot.co.il/brokerage/foreignexchange/

The request with Union bank URL is followed by a permanent redirect response (code 301).

Remove this module from the XIL project.

• List all the banks + data in the README
• CONTRIBUTING file: install Python, Poetry, pre-commit install

For some reason, they added it:
https://github.com/jond01/xil/actions/runs/7801669339/job/21277343405#step:6:51

It's a bit strange, as it's not formally a different bank than Bank Leumi.

https://github.com/regebro/pyroma#version-control-integration

May be achieved with https://github.com/marketplace/pre-commit-ci.

pandas-dev/pandas#52620 resolved pandas-dev/pandas#49944.

See:
https://pandas.pydata.org/docs/dev/whatsnew/v2.1.0.html#other-enhancements

• pandas.read_html() now supports the storage_options keyword when used with a URL, allowing users to add headers the outbound HTTP request (GH 49944)

Use a GitHub workflow triggered when tagging a new version (e.g., 0.2.0).
Ensure the version in the pyproject.toml file is updated:

Looks like there's some data on the website now:

https://www.onezerobank.com/company/information/4514

However, it's quite partial and not updated.

Edit:
This page presents the daily data:
https://www.onezerobank.com/currencies/
Source:
https://dv16ymfyh91nr.cloudfront.net/MarketingRatesReport/MarketingSiteFCYRatesCurrentReport.json

Webpage:

Disable network connection when the "live" marker isn't selected:

pytest . -m "not live" --disable-network

Implementation reference:
https://github.com/best-doctor/pytest_network

A few of the APIs are blocked outside of Israel.
It should be documented and accompanied by an informative error message.

Use GitHub workflows for each pull request and push to the main branch.

They are created in the dist/ folder by poetry build.

https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts#uploading-build-and-test-artifacts

For pip and github-actions updates.

https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file

After a few requests from the same IP, the following URL is blocked:
https://www.bankhapoalim.co.il/he/coin-rates
With the following uninformative error:

urllib.error.HTTPError: HTTP Error 302: The HTTP server returned a redirect error that would lead to an infinite loop.
The last 30x error message was:
Found

Try with:

import pandas as pd

pd.read_json("https://www.bankhapoalim.co.il/he/coin-rates")

Run it a few times.

After a quick research, I found the following cookies are required ("functional" cookies): incap_ses_***_*******.
One can get them from a web-browser.
For example, the following currently works for me:

pd.read_json(
    "https://www.bankhapoalim.co.il/he/coin-rates",
    storage_options={
        "Cookie": " ".join(
            """\
incap_ses_1255_2405596=a/GzCf3fvnceYM2cLahqEUXj6mUAAAAAtfv8l97VCsPJsew65Qa8XA==;
incap_ses_1255_2405249=DKE/HuRYLSgUMtWcLahqEf/t6mUAAAAArYjjw97q1DiiXLULf6cJJw==;
incap_ses_1255_2405640=wc+ZOPCR9VzEeB+dLahqEfF762UAAAAAk7Vee7dBohFS/K9LTiy3wg==
            """.splitlines()
        )
    },
)

Only the last one is necessary.

We need to find a way to overcome it or allow users to insert their cookie conveniently.

This will make the pre-commit job faster (mostly mypy run).

• Mypy:
  Set the MYPY_CACHE_DIR env var:
  https://mypy.readthedocs.io/en/stable/config_file.html#confval-cache_dir
  See also:
  https://mypy.readthedocs.io/en/stable/additional_features.html#using-a-remote-cache-to-speed-up-mypy-runs
  https://github.com/actions/cache
• Black:
  BLACK_CACHE_DIR / XDG_CACHE_HOME
  https://black.readthedocs.io/en/stable/contributing/reference/reference_functions.html#black.cache.get_cache_dir
  https://black.readthedocs.io/en/stable/usage_and_configuration/file_collection_and_discovery.html
• Ruff:
  RUFF_CACHE_DIR
  https://beta.ruff.rs/docs/settings/#cache-dir

Note - caching is not supported in:

• Pylint:
  https://pylint.readthedocs.io/en/latest/faq.html#where-is-the-persistent-data-stored-to-compare-between-successive-runs
  https://github.com/pylint-dev/pylint/projects/3#card-12710131
• isort:
  PyCQA/isort#1837

Probably an MIT license.

• Replace:
  

  xil/pyproject.toml

  Line 5 in c62c1c8

  # TODO: decide on license

• Add a license file

See:
https://docs.pypi.org/trusted-publishers/

This mechanism replaces the API token which is used to upload to PyPI (and Test-PyPI):

Need to see what it's worth and whether it's supported by the build and publish system (currently Poetry).

Allow combining all the data together.
Agree on "must-have" columns and rows.