Code Monkey home page Code Monkey logo

end2you's Introduction

End2You - The Imperial Toolkit for Multimodal Profiling

We introduce End2You the Imperial toolkit for multimodal profiling. This repository provides easy-to-use scripts to train and evaluate either unimodal or multimodal models in an end-to-end manner for either regression or classification output. The input to the model can be of 1D (e.g. audio, eeg, heart rate etc.), 2D (e.g. spectrogram, image etc), or 3D (e.g. video etc). The main blocks of the unimodal and multimodal models are (i) a Convolutional Neural Network (CNN) that extracts spatial features from the raw data, and (ii) a recurrent neural network (RNN) that captures the temporal information in the data. The models can be combined in any desirable way, and the user can also define new models and combine with the existing implementations.

The End2You workflow (shown below) is comprised of the data generator, that transforms the raw file format to hdf5, the data provider that feeds the data to the models, and finally, the prediction. We provide a number of audio/visual/multimodal models (see tutorials or cli). One can easily use its own models to train or use our pre-train models on their dataset.

alt text

Citing

If you use End2you or any code from End2you in your research work, you are kindly asked to acknowledge the use of End2You in your publications.

Tzirakis, P.,Zafeiriou, S., & Schuller, B. (2017). End2You -- The Imperial Toolkit for Multimodal Profiling by End-to-End Learning. arXiv preprint arXiv:1802.01115.

@article{tzirakis2018end2you,
  title={End2You--The Imperial Toolkit for Multimodal Profiling by End-to-End Learning},
  author={Tzirakis, Panagiotis and Zafeiriou, Stefanos and Schuller, Bjorn W},
  journal={arXiv preprint arXiv:1802.01115},
  year={2018}
}

Challenges

End2You has been used with great success, providing strong baselines in the following challenges:

Installation

Conda Installation

We highly recommended to use conda as your Python distribution. Once downloading and installing conda, this project can be installed by using the conda_setup.yml file as follows:

$ conda env create -f conda_setup.yml

You can now activate the environment and use End2You.

$ conda activate end2you

Pip Installation

Another way to install End2You is via pip as shown below:

$ pip install git+https://github.com/end2you/end2you.git

Dependencies

Below are listed the required modules to run the code.

  • Python >= 3.7
  • NumPy >= 1.19.2
  • Pytorch >= 1.7 (see Installation section for installing this module)
  • MoviePy >= 1.0.3
  • sklearn >= 0.23.2
  • h5py >= 2.10.0
  • facenet-pytorch >= 2.5

end2you's People

Contributors

tzirakis avatar end2you avatar rosastamp avatar mauricege avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.