Code Monkey home page Code Monkey logo

coshnet's Introduction

Authors: Manny Ko, Ujjawal K. Panchal, Héctor Andrade Loarca, Andres Mendez-Vazquez

CoShNet Architecture

Architecture

CoShNet is a fully complex hybrid neural network. We use the CoShREM (now call SymFD) http://www.math.uni-bremen.de/cda/software.html signal transform to produce a stable embedding. The network operates entirely in $\mathbb{C}$ domain to take advantage of unique properties of CVNNs.

Architecture Brief
  1. Input is any $32\times32 \in \mathbb{R}$ image.
  2. Input is CoShREM transformed to produce a $32\times32\times20 \in \mathbb{C}$ output.
  3. CoShREM output is convolved with the $2$ cplx-conv layers.

    Each cplx-conv layer is composted of := $\mathbb{C}$-Conv + $\mathbb{C}$-ReLU + $\mathbb{C}$-AvgPool2d.

  4. The response is flattened and passed through $2$ cplx-linear layers.

    Each cplx-linear layer is composted of := $\mathbb{C}$-linear layer + $\mathbb{C}$-ReLU.

  5. The $\mathbb{R}$, $\mathbb{I}$ components are stacked together (see shape) and passed through $1$ final $\mathbb{R}$-linear layer.

Setup

Python 3.8.x and newer are supported:

Automated Setup
  1. Create a virtualenv at the root of the repo: python -m venv venv4coshnet
  2. Activate venv4coshnet:
    • Windows: venv4coshnet\Scripts\activate
    • Linux/MacOS: source venv4coshnet/bin/activate
  3. Run setup.py:
    • with CUDA: python setup.py
    • without CUDA: python setup.py --no-cuda
    • use --no-venv to disable venv check (e.g. inside a docker)
Manual Setup
Docker Setup
  • Build image: docker build -t coshnet-docker:latest . (Some systems might require running this in `sudo` mode.)

Contents

Contents List
  1. code/: Contains all code essential to run experiments in this repo.
  2. libs/: Contains all custom-made and collected libs and modules we use for our experiments. (Installed automatically in setup.txt)
  3. data/: Folder where datasets are present. Created automatically when running for first time.
  4. setup.txt: Steps for setting up repo.
  5. requirements.txt: requirements file.
  6. changelog.md: all changes relevant to releases, branch prs, or any other general notes needed for maintenance.

How to Run?

Running in Local cd code/. Following are the possible cases:
  1. Running our models: run: python test_fashion.py --help to see several arguments you are allowed to tune. (Default run (10k20E) gets 89.2% on RTX 2080 Super). The default will use the 10k test set of Fashion to train for 20 epochs, and the 60k training set to test.
  2. Running resnet(18|50): run: python test_resnet.py --help to see several arguments you are allowed to set. (Default run (RN18, 10k20E) gets 88.3% on RTX 2080 Super).
Note: This code (shown in test_fashion.py,test_resnet.py) will not run in (jupyter|google colab) notebook(s). This is because our code defaults to using `asyncio` for batch generation for speed. Hence, if you absolutely have to run in a notebook, please create your own batch generation code.
Running in Docker
  • Run Image: docker run coshnet-docker:latest (Some systems might require running this in `sudo` mode.)
Note: The above is a brief demo for running our codebase in a docker. If you want to do something specific, e.g. deliver an API endpoint through a docker, you will have to edit the Dockerfile

Some Results

Model Epochs Parameters Size Ratio Top-1 Accuracy (60k) Top-1 Accuracy (10k)
ResNet-18 100 11.18M 229 91.8% 88.3%
ResNet-50 100 23.53M 481 90.7% 87.8%
CoShNet(base) 20 1.37M 28 92.2% 89.2%
CoShNet (tiny) 20 49.99K 1 91.6% 88.0%

Note: 60k = train on train-set (60k observations), test on test-set (10k observations). 10k = vice-versa. K or k = 1000, M = Million.

Cite

@misc{coshnet2022,
  doi = {10.48550/ARXIV.2208.06882},
  url = {https://arxiv.org/abs/2208.06882},
  author = {Ko, Manny and Panchal, Ujjawal K. and Andrade-Loarca, Héctor and Mendez-Vazquez, Andres},
  keywords = {Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {CoShNet: A Hybird Complex Valued Neural Network using Shearlets},
  publisher = {arXiv},
  year = {2022},
  copyright = {Creative Commons Attribution Non Commercial Share Alike 4.0 International}
}

License

coshnet's People

Contributors

angelld23 avatar mannykao avatar ujjawal-k-panchal avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

coshnet's Issues

mnist download command downloading fashion.

Even when we pass --dataset mnist to test_fashion.py, the initial download downloads the Fashion MNIST dataset and downloads it within MNIST directory. This is because, load_fashion() function is not passing kFashion parameter to getdb() function. This is causing kFashion to default to true and for the URL fetched for downloading the dataset to be Fashion MNIST url even if the user wished for MNIST url.

Problem with shear levels

I'm visualizing the shearlets generated and either I'm doing something wrong or I'm not sure that shearing is being applied as intended.

grid_imag
grid_real

From debugging, it seems the shearing itself does work, but it's shearing the shearlets in the wrong direction: horizontal shearlets sheared horizontally, and vertical shearlets sheared vertically makes it a no-op basically.

Have you checked this works in the current repository?

Porting to RGB images and beyond.

I saw a link to your paper on Twitter and have been digging into the code! I noticed there are placeholders for RGB and Lab support in various places already, but it doesn't seem complete.

What would be necessary to support 3 input channels and beyond? Also, what if the images are more than 32x32 — do you recommend using a patch-based representation like ViT?

(I guess this is a likely future path for the research, so if you're working on it that's very interesting!)

Question about reconstructions

How accurate should the reconstructions be? I'm trying out shearrec2D after the sheardec2D and the results aren't as good as I would have expected — even bumping up the number of different shearlets.

Are other filters needed in combination with shearlets to get good reconstructions?

Thanks!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.