Code Monkey home page Code Monkey logo

Comments (16)

alexriedel1 avatar alexriedel1 commented on September 10, 2024 1

The default category for the MVTec datamodule is bottle. This is what you train on in your API script. In your CLI command you train on the category transistor. These two categories might have a different number of training images which could lead to your difference in training time. However this might not be the only reason why CLI training is faster.

Can you tell whats the difference in training time?

from anomalib.

ashwinvaidya17 avatar ashwinvaidya17 commented on September 10, 2024 1

From the first glance I can see quite a big difference due to process spawn when using the API. But I am not sure what is causing it yet. I'll set up my windows machine and test on it.

from anomalib.

pirnerjonas avatar pirnerjonas commented on September 10, 2024

Thanks @alexriedel1! I tested again and made sure both use category bottle by running CLI version like:

anomalib train --model EfficientAd --data anomalib.data.MVTec --data.category bottle --data.train_batch_size 1

After training for 15 minutes API version was at epoch 8 and CLI version was at epoch 34 on my local machine.

from anomalib.

samet-akcay avatar samet-akcay commented on September 10, 2024

Thanks for reporting this @pirnerjonas. This is indeed quite peculiar

from anomalib.

ashwinvaidya17 avatar ashwinvaidya17 commented on September 10, 2024

So far I am not able to recreate this issue. API seems to be slightly faster, and I don't see anything out of the ordinary with cProfile
Here are the results of my experiments

Epochs API (seconds) CLI (seconds)
20 277.02 299.57
30 409.68 426.49

There are a few differences between our environments

OS: Ubuntu 20.04.6 LTS
GPU: RTX 3090
pytorch-lightning: 2.2.0.post0
torch: 2.1.2+cu118
torchaudio: 2.2.1+cu118
torchmetrics: 1.4.0.post0
torchvision: 0.16.2+cu118

In case you are interested, these are the testing snippets.

API

import cProfile
import io
import pstats

from anomalib.data import MVTec
from anomalib.engine import Engine
from anomalib.models import EfficientAd


def main():
    # Initialize the datamodule, model and engine
    datamodule = MVTec(train_batch_size=1)
    model = EfficientAd()
    engine = Engine(max_epochs=30)

    # Train the model
    engine.fit(datamodule=datamodule, model=model)


if __name__ == "__main__":
    stream = io.StringIO()
    pr = cProfile.Profile()
    pr.enable()
    main()
    pr.disable()
    ps = pstats.Stats(pr, stream=stream).sort_stats("cumulative")
    ps.print_stats()
    with open("api_profile_total_time.txt", "w") as f:
        f.write(stream.getvalue())

Note

The API uses fit rather than train to match the snippet you shared. Though replacing this with train makes no difference.

CLI

import cProfile
import io
import pstats

from anomalib.cli.cli import AnomalibCLI, configure_logger


def main():
    configure_logger()
    AnomalibCLI(
        [
            "train",
            "--model",
            "EfficientAd",
            "--data",
            "anomalib.data.MVTec",
            "--data.category",
            "bottle",
            "--data.train_batch_size",
            "1",
            "--trainer.max_epochs",
            "30",
        ],
    )


if __name__ == "__main__":
    stream = io.StringIO()
    pr = cProfile.Profile()
    pr.enable()
    main()
    pr.disable()
    ps = pstats.Stats(pr, stream=stream).strip_dirs().sort_stats("cumulative")
    ps.print_stats()
    with open("cli_profile_total_time.txt", "w") as f:
        f.write(stream.getvalue())

from anomalib.

mohandeeprdori avatar mohandeeprdori commented on September 10, 2024

after my training was done.. anomalib was supposed to create a model.ckpt file under one of the folders in results folder. But i see only a couple of config.yaml files. Is this how it works?. Also while inferencing how do we get scores for each prediction of an image. Please help me out.

from anomalib.

pirnerjonas avatar pirnerjonas commented on September 10, 2024

@ashwinvaidya17 which python version did you use?

from anomalib.

ashwinvaidya17 avatar ashwinvaidya17 commented on September 10, 2024

@pirnerjonas Python 3.10.13

from anomalib.

ashwinvaidya17 avatar ashwinvaidya17 commented on September 10, 2024

@mohandeeprdori is your query related to this issue? I would recommend posting it as a separate question on the discussions page.

from anomalib.

mohandeeprdori avatar mohandeeprdori commented on September 10, 2024

@ashwinvaidya17 You can move this to a different discussion. Thanks for answering!

from anomalib.

pirnerjonas avatar pirnerjonas commented on September 10, 2024

I again tested this time with Python 3.11.9 and with your cli and api scripts you provided @ashwinvaidya17

I only changed number of epochs to 5

API

         25406833 function calls (24380311 primitive calls) in 4379.312 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)

CLI

         38038005 function calls (36798119 primitive calls) in 783.505 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000  783.782  783.782 cli.py:8(main)

Maybe you can make sense of these cProfile reports:

api_profile_total_time.txt
cli_profile_total_time.txt

Environment Settings

[tool.poetry]
name = "bug"
version = "0.1.0"
description = ""
authors = ["jonas pirner <[email protected]>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.10"
anomalib = {extras = ["core"], version = "^1.1.0"}
numpy = "<2"
scikit-learn = "^1.5.1"

[tool.poetry.group.gpu.dependencies]
torch = {version = "^2.2.0", source = "pytorch-gpu-src"}
torchvision = {version = "^0.18", source = "pytorch-gpu-src"}

[[tool.poetry.source]]
name = "pytorch-gpu-src"
url = "https://download.pytorch.org/whl/cu121"
priority = "explicit"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

aiohappyeyeballs       2.3.4        Happy Eyeballs for asyncio
aiohttp                3.10.0       Async http client/server framew...
aiosignal              1.3.1        aiosignal: a list of registered...
anomalib               1.1.0        anomalib - Anomaly Detection Li...
antlr4-python3-runtime 4.9.3        ANTLR 4.9.3 runtime for Python 3.7
attrs                  23.2.0       Classes Without Boilerplate
av                     12.3.0       Pythonic bindings for FFmpeg's ...
certifi                2024.7.4     Python package for providing Mo...
charset-normalizer     3.3.2        The Real First Universal Charse...
colorama               0.4.6        Cross-platform colored terminal...
contourpy              1.2.1        Python library for calculating ...
cycler                 0.12.1       Composable style cycles
docstring-parser       0.16         Parse Python docstrings in reST...
einops                 0.8.0        A new flavour of deep learning ...
filelock               3.15.4       A platform independent file lock.
fonttools              4.53.1       Tools to manipulate font files
freia                  0.2          Framework for Easily Invertible...
frozenlist             1.4.1        A list-like structure which imp...
fsspec                 2024.6.1     File-system specification
ftfy                   6.2.0        Fixes mojibake and other proble...
huggingface-hub        0.24.5       Client library to download and ...
idna                   3.7          Internationalized Domain Names ...
imageio                2.34.2       Library for reading and writing...
imgaug                 0.4.0        Image augmentation library for ...
importlib-resources    6.4.0        Read resources from Python pack...
intel-openmp           2021.4.0     Intel OpenMP* Runtime Library
jinja2                 3.1.4        A very fast and expressive temp...
joblib                 1.4.2        Lightweight pipelining with Pyt...
jsonargparse           4.32.0       Implement minimal boilerplate C...
kiwisolver             1.4.5        A fast implementation of the Ca...
kornia                 0.7.3        Open Source Differentiable Comp...
kornia-rs              0.1.5        Low level implementations for c...
lazy-loader            0.4          Makes it easy to load subpackag...
lightning              2.3.3        The Deep Learning framework to ...
lightning-utilities    0.11.6       Lightning toolbox for across th...
markdown-it-py         3.0.0        Python port of markdown-it. Mar...
markupsafe             2.1.5        Safely add untrusted strings to...
matplotlib             3.9.1        Python plotting package
mdurl                  0.1.2        Markdown URL utilities
mkl                    2021.4.0     Intel® oneAPI Math Kernel Library
mpmath                 1.3.0        Python library for arbitrary-pr...
multidict              6.0.5        multidict implementation
networkx               3.3          Python package for creating and...
numpy                  1.26.4       Fundamental package for array c...
omegaconf              2.3.0        A flexible configuration library
open-clip-torch        2.26.1       Open reproduction of consastive...
opencv-python          4.10.0.84    Wrapper package for OpenCV pyth...
packaging              24.1         Core utilities for Python packages
pandas                 2.2.2        Powerful data structures for da...
pillow                 10.4.0       Python Imaging Library (Fork)
pygments               2.18.0       Pygments is a syntax highlighti...
pyparsing              3.1.2        pyparsing module - Classes and ...
python-dateutil        2.9.0.post0  Extensions to the standard Pyth...
pytorch-lightning      2.3.3        PyTorch Lightning is the lightw...
pytz                   2024.1       World timezone definitions, mod...
pyyaml                 6.0.1        YAML parser and emitter for Python
regex                  2024.7.24    Alternative regular expression ...
requests               2.32.3       Python HTTP for Humans.
rich                   13.7.1       Render rich text, tables, progr...
rich-argparse          1.5.2        Rich help formatters for argpar...
safetensors            0.4.3        
scikit-image           0.24.0       Image processing in Python
scikit-learn           1.5.1        A set of python modules for mac...
scipy                  1.14.0       Fundamental algorithms for scie...
setuptools             72.1.0       Easily download, build, install...
shapely                2.0.5        Manipulation and analysis of ge...
six                    1.16.0       Python 2 and 3 compatibility ut...
sympy                  1.13.1       Computer algebra system (CAS) i...
tbb                    2021.13.0    Intel® oneAPI Threading Buildin...
threadpoolctl          3.5.0        threadpoolctl
tifffile               2024.7.24    Read and write TIFF files
timm                   1.0.3        PyTorch Image Models
torch                  2.3.1+cu121  Tensors and Dynamic neural netw...  
torchmetrics           1.4.0.post0  PyTorch native Metrics
torchvision            0.18.1+cu121 image and video datasets and mo...  
tqdm                   4.66.4       Fast, Extensible Progress Meter     
typeshed-client        2.7.0        A library for accessing stubs i...  
typing-extensions      4.12.2       Backported and Experimental Typ...  
tzdata                 2024.1       Provider of IANA time zone data
urllib3                2.2.2        HTTP library with thread-safe c...  
wcwidth                0.2.13       Measures the displayed width of...  
yarl                   1.9.4

from anomalib.

pirnerjonas avatar pirnerjonas commented on September 10, 2024

Seems to be a windows related problem... just created the same env under ubuntu via WSL and I get much faster training times:

CLI

         33759240 function calls (32844378 primitive calls) in 212.192 seconds

   Ordered by: cumulative time

API

         21262576 function calls (20548194 primitive calls) in 176.971 seconds

   Ordered by: cumulative time

from anomalib.

ashwinvaidya17 avatar ashwinvaidya17 commented on September 10, 2024

I have a Quadro RTX 3000 on my Windows machine, it so took me ages.
I don't see any significant difference due to spawn_win32. But I am not sure if this is a fair comparison.

CLI: 13919.792 seconds
API: 14072.421 seconds

from anomalib.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.