Code Monkey home page Code Monkey logo

nevolve's People

Contributors

katieb1 avatar shreeramsenthi avatar terciopelo avatar

Watchers

avatar avatar

nevolve's Issues

To-do before February 22nd

  • Write up project background in README.md
  • Get started on cedar
  • NSight tutorial + load on cluster
  • Get acquainted with GPUs (see slide deck from Feb 12th presentation for tutorial links)
  • Complete a training run with the single demographic scenario data through the hyperparameter tuning script and CNN script
  • Get acquainted with multithreading in PyTorch for reading and writing data (pytorch.multiprocessing, pre-fetching, and queuing)
  • Write data to hdf5 format so it is batch-ready (demographic scenarios should be normalized and randomized across batches, and batches should be 100MB) - make sure to set random seed
  • Things to think about:
  • Use GPU when writing to hdf5 files
  • When validation testing, accuracy metric should be weighted by class size (e.g. if you have 50 samples from a demographic scenario and then CNN gets 3 correct, then accuracy should take into account the relative sample size)

TO DO before March 1st

  • Profile conversion script and generate flame graph
  • Create generation script to run simulations (with ms) and create metadata file
  • Generate more training data
  • Figure out way to randomize simulations so we get a random collection of chunks / batch
  • Change labels to categorical in ms
  • Get RAPIDS / PyTorch singularity image set up on respective clusters
  • Work on parallelizing hyperparameter tuning script
  • Create instructions file for chunking / data loader
  • Create instructions file for setting up RAPIDS / PyTorch singularity image

General To Do

  • Profile multi-GPU pandas with 100K dataset (in progress)
  • Profile multi-GPU cudf with 100K dataset
  • Test if Matt's virtual environment with rapids and pytorch will work
  • Build singularity with Matt's git repo
  • Hyperparameter tuning job submission script
  • General research into DDP
  • Train multi-GPU with or without pandas on 1K dataset and make sure nn is learning
  • Run more epochs + 100K dataset through nn training to see if accuracy improves
  • Find baseline hyperparameter values for tuning (perhaps using weights and biases nvidia module wandb)
  • Try network w/ and w/o position data to see if accuracy improves
  • Look into binary formats - e.g. hdf5
  • Visualization code

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.