mwalmsley / galaxy-datasets Goto Github PK

Should not need to install torch to use as a simple downloader.

Refactor out torch dependencies as optional extra.
Refactor imports

Wondering how to include the code in here into say a zoobot installation?

It looks like it's referenced in the Zoobot pytorch install, https://github.com/mwalmsley/zoobot/blob/f42f434614668f6cd30e48f9bcdd50ae36ec8eb4/zoobot/pytorch/training/train_with_pytorch_lightning.py#L10

I can't find it on PyPi - so do we manually install this somehow for the zoobot install?

import os

_expected_internal_urls_loc = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'shared/internal_urls.py')
INTERNAL_URLS_EXIST = os.path.isfile(_expected_internal_urls_loc)

I could not find the internal_urls.py so that I'm unable to download gz_rings, gz_cosmic_dawn and gz_hubble_euclidized. Is it available?

albumentations works directly on numpy images
Loading with PIL gives a PIL.Image which is then converted
It would be better to load with CV2 which gives a numpy array directly

See https://albumentations.ai/docs/examples/migrating_from_torchvision_to_albumentations/

install_requires=[...]

for installing without zoobot

There's some dependencies that are out of date.

I have included the -dr12 and -dr8 columns in the dr5 dataset. Everything still works, but it's a bit confusing for users. I should remove them.

In candels dataset, the root is joined two times:

first in download_utils.py (14):

self.image_dir = os.path.join(self.root, image_dirname)

then in candels.py (44):

catalog['file_loc'] = catalog.apply(lambda x: os.path.join(root, downloader.image_dir, x['filename']), axis=1) 

We should check if that's just for this dataset.

I am currently working on a research topic involving Anomaly detection in galaxy morphology using Outlier exposure. Ideally I need to devise a data setup where I'd like to detect rings and mergers as anomaly. Under the pretext that these morphologies are rather rare compared to others.

I understand the rings dataset is currently internal but I noticed in a separate issue that you may be wiling to share it. I'd like to chat about how we can go about this if possible.

Hi Mike,

There is a mirror of the jpg version of the GZ DESI dataset available on HuggingFace here: https://huggingface.co/datasets/Smith42/galaxies.

Happy to make a pull request to add it in, if this feature would be welcome.

Cheers,
Mike

Copy setup.py style from Zoobot

Linked to recent features like #15 can we cut a release for this?

Should the file_loc be overwritten in the catalogues to show the local path when the GZ2 data are downloaded?

from galaxy_datasets import gz2

catalog, label_cols = gz2(
    root='/Users/ascaife/SRC/MANGA/gz2',
    train=True,
    download=True
)

leaves the paths in file_loc as /home/walml/repos/galaxy-datasets/roots/gz2/... etc