kaiko-ai / eva Goto Github PK

A lot of libraries define models though functions. However, the yaml passing method, due to type verification, can not support them.

Thus, we would like to have a class wrapper, which calls the functions, build and initialises the model in the class.

We need functions to load image files into numpy arrays or torch tensors.

We would like to have a central logger to be used across the library. A great choice would be loguru.

Add EVA library logo

See comment here

We need a dataset class that loads pre-calculated embedding files for offline evaluations where inference of the FM is performed beforehand.

We can potentially speed up the CI process with the following updates;

• Run quality and test checks in parallel
• Install python deps only once and run all the checks
• pytest run only the changed file

We would like to add support of a CI manager, for managing multiple Python test environments. For that, we recommend nox.

nox is a command-line tool that automates testing in multiple Python environments, which uses a standard Python file for configuration. Nox will automatically create a virtualenv with the appropriate interpreter, install the specified dependencies, and run the commands in order.

After #64 and #79, we should abstract all the common methods to a parent class which will give more guidance and make it easier to add more data.

Would be good to add a module / utils file that implements some of the split methods we use, such that we can share that logic among the different dataset classes.

Examples:

• ordered splits
• random splits
  • std
  • balanced / stratified

Also, should be able to choose between train/val/test or train/test.

In order to minimise the code, configs and increase the reliability, we would like to add some common image transforms

Implement workflow to run eva with precalculated embeddings / in "offline" mode.

Add the TotalSegmentator dataclass which will be the first task for the radiology workflow

Dataset link: https://zenodo.org/records/10047263

In order for eva to support multiple modalities efficiently, meaning to be able to share core code but not need to install the package dependencies for all modalities, we would like to any modality as an optional group of the library.

We will start with the vision one, called eva-vision.

As docs directory would possibly be a the default directory where the code documentation would be stored (automatic), we should move to a new place, like assets

In order to allow environmental variables in yaml files, we can use the omega parser in CLI.

Add some default metrics regarding multi-class classification tasks

addresses #47 (comment)

To speed up the CI, we can enable the cache options that the setup-pdm workflow provides.

Use python version 3.10.13 as a default version

For downstream evaluation tasks, one of the most popular ones is a simple linear classification head on a foundation model.

We should have native support of this downstream task.

According to PDM official docs, its better to move the development only dependencies to tool.pdm.dev-dependencies group in pyproject.toml, so that they don't appear in the distribution's metadata.

Bandit is a tool designed to find common security issues in Python code.

The .pdm-python stores the Python path used by the current project and doesn't need to be shared.

We would like to have an abstract lightning model module which would act as the base class of all the models

We need to design a base class to support vision tasks such as classification & segmentation. Note that the benchmark datasets can come in a variety of different format, so we need to decide if the dataset classes should use the original raw data or if we transform to a standardised format.

The data related core modules will compose the base of all the data related modules. These will be the dataset, the dataloader and the datamodule.

teardown is another method which will be called from the DataModule if it is implemented.

Lets add it an document its usage.

Core metrics module will be responsible for all the metrics calculation and grouping

In general we are following Google Style Python Docstrings (here an example) which suggest to use Defaults to to document parameters that have a default value. However, the default value is visible in the function arguments, so there is a notion of duplication.

Thus we will have an exception to the Docstrings style, were we omit the Defaults to document.

BACH will be the first dataset we add to eva. We should design a dataset class that implements the prepare_data and setup method from VisionDataset.

For evaluating slide level tasks we need implementations of MIL networks such as ABMIL

Lets introduce a style guide document, to highlight our codings and documentation style, in order to me known and consistent across the library

The issue suggest to expose some core and frequently used classes in eva public API to make easier ti use.

For example:

# instead of this
from eva.data import datamodules
from eva.models import modules

data = datamodules.DataModule()
model = modules.HeadModule()

# to do the following
import eva

model = eva.DataModule()
model = eva.HeadModule()

The considered package managers are:

• PDM
• setuptools

Here is a good overview comparing the most famous python packaging tools.

Make sure structure of readme's for different datasets (e.g. bach, patch camelyon) are consistent.

When we import or use the library, we would like to trigger some configurations automatically in order to have a better user experience.

This will be to automatically be default (but can be configured) to enable MPS fallback for Apple M chip users

PatchCamelyon is one of the most famous vision medical datasets, that is used widely by the research community to benchmark oncology foundational models.

OpenSSF provides a blueprint for following best practices with open source code. Many popular open-source projects (e.g. TensorFlow) obtained their badge

• during development we want to keep following their guidelines (criteria)
• once eva is published, we will apply to obtain the badge

It would handy to configure an automerge label, which automatically merges PR's that are approved to main.

To make it easier to interact with the library and its various components, we would like to have a common interface which encapsulates all the core part and tasks.

Apart from the python interface, this also will be used for the cli interface

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

This repository currently has no open or pending branches.

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Adding sys args to the nox test function would allow us to run commands as nox -s test -- tests/test_fake.py:

@nox.session(python=PYTHON_VERSIONS, tags=["test"])
def test(session: nox.Session) -> None:
    """Runs the unit tests of the source code."""
    args = session.posargs or LOCATIONS
    session.run("pdm", "install", "--group", "dev", external=True)
    session.run("pdm", "run", "pytest", "--cov", *args)

However, doing just that gives the following error when running nox -s test:

noxfile.py E                                                                                                                                              [100%]

============================================================================ ERRORS =============================================================================
____________________________________________________________________ ERROR at setup of test _____________________________________________________________________
file /Users/nkaenzig/workspace/eva-worktrees/eva-2/noxfile.py, line 61
  @nox.session(python=PYTHON_VERSIONS, tags=["test"])
  def test(session: nox.Session) -> None:
E       fixture 'session' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, monkeypatch, no_cover, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

First example of downstream task using the PatchCamelyon dataset.

Set the formatting tools line-length to 100.

For a variety of evaluation scenarios (e.g. calculating metrics) we need to access additional metadata fields such as the slide-id, patient-id or the medical center. Currently our data classes only return image, target tuples.

Two options:

1. Tuples with an optional third value for metadata

class TUPLE_INPUT_BATCH(NamedTuple):
    """The tuple input batch data scheme."""

    data: torch.Tensor
    """The data batch."""

    targets: torch.Tensor | None = None
    """The target batch."""

    metadata: Dict[str, Any] | None = None
    """The associated metadata."""

3. Batch dictionaries

class DatasetSample(TypedDict):
    data: Union[torch.Tensor, List[torch.Tensor]]
    """The image batch, depending on the transforms used the input will be different."""

    target: torch.Tensor | None
    """ "The target batch."""

    metadata: Dict[str, Any] | None
    """The associated metadata."""

However, there are drawbacks to using dictionaries as outlined here.