Fetching `2023-02-07-ppd-mp.pkl.gz` still fails with UnicodeDecodeError about matbench-discovery HOT 6 CLOSED

My bad for not handling pickle files separately in load_train_test(). Maybe we should rename the function into load() since it's clearly morphing into more than just a training and test set loader. @pbenner Curious to hear your opinion!

from matbench-discovery.

Yes sounds good! Also fetch_process_wbm_dataset.py could be fully integrated and called when first running load()

from matbench-discovery.

This is the error I get using the new branch:

Traceback (most recent call last):
  File "/home/pbenner/Source/tmp/matbench-discovery/data/wbm/fetch_process_wbm_dataset.py", line 322, in <module>
    assert sum(no_id_mask := df_summary.index.isna()) == 6, f"{sum(no_id_mask)=}"
AssertionError: sum(no_id_mask)=0

from matbench-discovery.

Are you using pandas v1.x.x? I just changed the code from v1 to v2 compat. I'll downwards pin pandas in pyproject.toml to avoid this in the future.

from matbench-discovery.

Indeed, I had pandas 1.5, trying to check with pandas 2.0. Meanwhile, I think 2023-02-07-mp-elemental-reference-entries.json.gz was modified:

python data/wbm/fetch_process_wbm_dataset.py
Loading 'wbm_summary' from cached file at '/home/pbenner/.cache/matbench-discovery/1.0.0/wbm/2022-10-19-wbm-summary.csv'
Warning: '/home/pbenner/.cache/matbench-discovery/1.0.0/mp/2023-02-07-mp-elemental-reference-entries.json.gz' associated with key='mp_elemental_ref_entries' does not exist. Would you like to download it now using matbench_discovery.data.load_train_test('mp_elemental_ref_entries'). This will cache the file for future use. [y/n] y
Downloading 'mp_elemental_ref_entries' from https://figshare.com/ndownloader/files/40344445

variable dump:
file='mp/2023-02-07-mp-elemental-reference-entries.json.gz',
url='https://figshare.com/ndownloader/files/40344445',
reader=<function read_json at 0x7f9898a875b0>,
kwargs={'compression': 'gzip'}
Traceback (most recent call last):
File "/home/pbenner/Source/tmp/matbench-discovery/data/wbm/fetch_process_wbm_dataset.py", line 24, in
from matbench_discovery.energy import get_e_form_per_atom
File "/home/pbenner/Source/tmp/matbench-discovery/matbench_discovery/energy.py", line 66, in
pd.read_json(DATA_FILES.mp_elemental_ref_entries, typ="series")
File "/home/pbenner/Source/tmp/matbench-discovery/matbench_discovery/data.py", line 217, in getattribute
self._on_not_found(key, msg)
File "/home/pbenner/Source/tmp/matbench-discovery/matbench_discovery/data.py", line 239, in _on_not_found
load_train_test(key) # download and cache data file
File "/home/pbenner/Source/tmp/matbench-discovery/matbench_discovery/data.py", line 111, in load_train_test
df = reader(url, **kwargs)
File "/home/pbenner/.local/opt/anaconda3/envs/crysfeat/lib/python3.10/site-packages/pandas/io/json/_json.py", line 760, in read_json
json_reader = JsonReader(
File "/home/pbenner/.local/opt/anaconda3/envs/crysfeat/lib/python3.10/site-packages/pandas/io/json/_json.py", line 862, in init
self.data = self._preprocess_data(data)
File "/home/pbenner/.local/opt/anaconda3/envs/crysfeat/lib/python3.10/site-packages/pandas/io/json/_json.py", line 874, in _preprocess_data
data = data.read()
File "/home/pbenner/.local/opt/anaconda3/envs/crysfeat/lib/python3.10/gzip.py", line 301, in read
return self._buffer.read(size)
File "/home/pbenner/.local/opt/anaconda3/envs/crysfeat/lib/python3.10/_compression.py", line 118, in readall
while data := self.read(sys.maxsize):
File "/home/pbenner/.local/opt/anaconda3/envs/crysfeat/lib/python3.10/gzip.py", line 488, in read
if not self._read_gzip_header():
File "/home/pbenner/.local/opt/anaconda3/envs/crysfeat/lib/python3.10/gzip.py", line 436, in _read_gzip_header
raise BadGzipFile('Not a gzipped file (%r)' % magic)
gzip.BadGzipFile: Not a gzipped file (b'{\n')

from matbench-discovery.

Yeah, I was in the process of updating the Figshare files but then got carried away. That error will be fixed before I merge #26.

from matbench-discovery.