edwardlib / observations Goto Github PK

Hi Dustin,
I have written a python script to generate observations ready python files for over 1100 datasets available in R and related packages. The project page is https://github.com/Arvinds-ds/datasets_r2py. Could you kindly verify/comment on the following

1. Whether the generated python source files conform to edward/observationsrequirements, the files are automatically generated in 'observations/rdata/', you can look at these files. If you have changes, kindly let me know the modifications to the templated python files init_template.py, template.py or test_template.py
2. If we were to use the files, how will it be structured in edward.I currently generated files in observations/rdata folder with tests in observations/rdata/tests
   Let me know.

In file Util.py at line 129 (version 0.1.4):

file_size = int(response.headers.get('content-range').split('/')[1])

I get this error:

AttributeError: 'NoneType' object has no attribute 'split'

references

• https://archive.ics.uci.edu/ml/index.php
• https://archive.ics.uci.edu/ml/datasets/diabetes
  • https://github.com/scikit-learn/scikit-learn/blob/ef5cb84a/sklearn/datasets/base.py#L552
• http://archive.ics.uci.edu/ml/datasets/optical+recognition+of+handwritten+digits
  • https://github.com/scikit-learn/scikit-learn/blob/ef5cb84a/sklearn/datasets/base.py#L472
• https://artax.karlin.mff.cuni.cz/r-help/library/mixOmics/html/linnerud.html
  • https://github.com/scikit-learn/scikit-learn/blob/ef5cb84a/sklearn/datasets/base.py#L600
• https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)
  • https://github.com/scikit-learn/scikit-learn/blob/ef5cb84a/sklearn/datasets/base.py#L390

#49

http://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html

• http://www-personal.umich.edu/~mejn/netdata/
• http://snap.stanford.edu/data/links.html
• https://networkdata.ics.uci.edu/index.html

references

• https://github.com/hardmaru/write-rnn-tensorflow
• http://www.fki.inf.unibe.ch/databases/iam-on-line-handwriting-database/data-format

references

• http://scikit-learn.org/stable/datasets/index.html
• fetch_olivetti_faces
• fetch_20newsgroups
• fetch_lfw_people
  • fetch_lfw_pairs
• fetch_covtype
• fetch_rcv1

All functions are silent whether it be loading, preprocessing, or saving the data. Currently, the only step that prints to stdout is downloading. These other steps can somtimes be very expensive, such as the preprocessing in small32_imagenet.py.

We should consider adding stdout messages to the loading, preprocessing, and saving steps—depending on how long it may take. And we should establish a standard that applies across all data sets.

From @Arvinds-ds (#21 (comment)):

the build is failing for observations master repo due to random Connection errors (probably due to the volume of data hitting the user's github repo during travis testing). Is there a way to host these reliably. I can change the URLs in the master file and regenerate the files

https://travis-ci.org/edwardlib/observations/jobs/278431971

references

• https://github.com/mila-udem/fuel/blob/master/fuel/datasets/imagenet.py

Change line 67 from
with open(extracted_filepath, 'w') as f:
to
with open(extracted_filepath, 'wb') as f:

for compatibility with Python3

https://deepmind.com/research/open-source/open-source-datasets/

For data sets like multi-MNIST and small ImageNet, we preprocess the data and cache by writing to disk so that future calls can load it into memory. More generally, we need to save and load data when its function requires preprocessing and the data fits in memory to be loaded.

We should decide on a specific option such as pickle, np.savez, or hdf5.

Searching open( in the repo shows that we use Python's open() function sometimes with the 'rb' arg, sometimes no arg, sometimes with w. We should make its usage consistent.

Following discussion in #19.

• blizzard
• timit (https://catalog.ldc.upenn.edu/ldc93s1)
• speech commands (https://research.googleblog.com/2017/08/launching-speech-commands-dataset.html)

We're currently in (pre-)alpha. Writing the 4-page paper will (a) announce the library for an official public release; (b) formalize design principles; (c) formalize design details such as how we handle various data domains, data structures, and data sizes; (d) provide statistics on Observations' collection of data sets.

All contributors are authors.

A major goal of Observations is to support all standard ML data sets. We should make sure we cover all those listed by deeplearning.net (url).

• 256x256 imagenet
  • https://github.com/mila-udem/fuel/blob/master/fuel/datasets/imagenet.py
• omniglot
• deepmind data sets
  • https://deepmind.com/research/open-source/open-source-datasets/
• nyu depth v2
  • http://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html

I am heavily using tf.contrib.data datasets api for image based tasks. With observations for images (LSUN/celebA) etc being no more than an downloader for these datasets, would it be worthwhile to return a tensorflow dataset something along the lines of

lsun_bedroom_x_train  = lsun('~/data',category='bedroom', set='training', 
                                                   batch_size=32, shuffle=True)
training_data = lsun_bedroom_x_train.make_one_shot_iterator()
.....
for i in range(inference.n_iter):
   x_batch = training_data.get_next()
   inference.update(....{x_ph: x_batch)

• mscoco
• flickr30k
• moviebook
• https://nlp.stanford.edu/blog/a-new-multi-turn-multi-domain-task-oriented-dialogue-dataset

Remove CelebA's manual download function.

All functions check for the default filename in path. This doesn't allow the user to load from a renamed file. Enable an optional filename argument.

There must be some care when the filename is a group of files or a directory. I don't know how to handle the arbitrary case.

• on TOC, add sidebar, then observations link with description in TOC linking to all other functions
• references
  • use [@] and @
  • store bibtex in observations/ repo
  • have edward link to it in pandoc command inside compile.sh
• add link in README.md

Use hash to verify if content exists and is correct.

references

• https://peptone.io/dspp
• https://github.com/PeptoneInc/dspp-keras

The r subfolder does not get installed in site-packages

Testing each loading function requires downloading data from the url(s) and verifying the returned data objects. How do we test without having to download many files for each Travis build? Is storing every file on a Travis server feasible? (no)

• imdb (see https://github.com/fchollet/keras/tree/master/keras/datasets)
• reuters (see https://github.com/fchollet/keras/tree/master/keras/datasets)
• bookcorpus
• 1b word benchmark (https://github.com/rafaljozefowicz/lm)
  • https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/data_generators/lm1b.py
  • https://github.com/amirbar/rnn.wgan
• https://archive.ics.uci.edu/ml/machine-learning-databases/20newsgroups-mld/
• amazon reviews
• yahoo answers
• yelp15,etc. reviews
• WMT14,15,etc. (https://github.com/tensorflow/models/tree/master/tutorials/rnn)

It will only extract the files. Basically, extend the function to include url=None.