Data source that uses a local csv file about zipline HOT 15 CLOSED

It would probably save a lot of space if it could use some standard format hdf5 file

from zipline.

Agreed, hdf5 would provide better performance with regards to space (and I'm assuming I/O as well).

I'll open a separate issue, since CSV is important to support, since it's a sort of a lingua franca of data export/import.

from zipline.

Something like this will work. https://gist.github.com/MichaelWS/5886984

from zipline.

Looks like a good start @MichaelWS, is it ok, when time permits, if I base a CSV source off of your linked gist, with attribution to you?

from zipline.

Started a branch here, https://github.com/quantopian/zipline/tree/csv-data-source

Waiting on having a unit test with a small example CSV file before merging that in.
(In the zone of "I'll try to get this done, soon", but accepting patches.)

from zipline.

try this script to create a csv and a hdf5 data source : https://gist.github.com/MichaelWS/dbb8dcf15b0ec3d479e8

from zipline.

I issued a pull request to fix this issue.
#244

from zipline.

I think I can rework this code to handle CSI data as well. does the SID always have to be an int? CSI use an UID for their symbols that are strings "S000XXXX" is the rough format.

from zipline.

michael,
I made SID a string because I was using strings as well. Let me know if you need any help with it. What do the csi files look like?

If they are in a directory of csv's, you can use data_source_tables_gen.py to generate a pytables file that will be pretty quick.

from zipline.

A sid can be a string, e.g. when used with the Yahoo! data source
the ticker symbol string is used as the identifier.

from zipline.

CSI files can be whatever format you want, thats good and bad of it, I can make them per symbol or daily or whatever, its quite flexible the output, shame the app is windows only however, hence my need to handle this all via windoze....

the pytables/HDF5 sounds my favourite option, I think I'll set it up to use a config file of sorts you can place in the loader folder to signify the columns and they also output a rather handy info file that links the CSI GUID to each symbol and has all the stuff like exchange, currency, text name, ticker symbol etc.

So using a String for the SID will help, I might just make it use the ticker symbol as that would be more user friendly but I'm trying to base things off of CSI portfolios of stocks (these can be predefined like russel 3000 or nasdaq 1500 etc) as I want to momentum trade a universe not single stock pairs etc so from what I can understand I have a bit of work to do or wait for some features to be released downstream from quantopian.

Thanks for all your help, you guys are ultra helpful with stuff!

from zipline.

I'm curious, since some applications work better than others with it,
but is it be possible to run CSI under WINE, http://www.winehq.org/ ?

• Eddie

Michael [email protected] writes:

CSI files can be whatever format you want, thats good and bad of it, I can make
them per symbol or daily or whatever, its quite flexible the output, shame the
app is windows only however, hence my need to handle this all via windoze....

the pytables/HDF5 sounds my favourite option, I think I'll set it up to use a
config file of sorts you can place in the loader folder to signify the columns
and they also output a rather handy info file that links the CSI GUID to each
symbol and has all the stuff like exchange, currency, text name, ticker symbol
etc.

So using a String for the SID will help, I might just make it use the ticker
symbol as that would be more user friendly but I'm trying to base things off of
CSI portfolios of stocks (these can be predefined like russel 3000 or nasdaq
1500 etc) as I want to momentum trade a universe not single stock pairs etc so
from what I can understand I have a bit of work to do or wait for some features
to be released downstream from quantopian.

Thanks for all your help, you guys are ultra helpful with stuff!

—
Reply to this email directly or view it on GitHub.

from zipline.

I haven't made the push to try it, will put it on the list of todo's once I
get the rest of the stuff working on windows, one step at a time!

regards

Michael Bennett

On 27 March 2014 13:53, Eddie Hebert [email protected] wrote:

I'm curious, since some applications work better than others with it,
but is it be possible to run CSI under WINE, http://www.winehq.org/ ?

• Eddie

Michael [email protected] writes:

CSI files can be whatever format you want, thats good and bad of it, I
can make
them per symbol or daily or whatever, its quite flexible the output,
shame the
app is windows only however, hence my need to handle this all via
windoze....

the pytables/HDF5 sounds my favourite option, I think I'll set it up to
use a
config file of sorts you can place in the loader folder to signify the
columns
and they also output a rather handy info file that links the CSI GUID to
each
symbol and has all the stuff like exchange, currency, text name, ticker
symbol
etc.

So using a String for the SID will help, I might just make it use the
ticker
symbol as that would be more user friendly but I'm trying to base things
off of
CSI portfolios of stocks (these can be predefined like russel 3000 or
nasdaq
1500 etc) as I want to momentum trade a universe not single stock pairs
etc so
from what I can understand I have a bit of work to do or wait for some
features
to be released downstream from quantopian.

Thanks for all your help, you guys are ultra helpful with stuff!

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/108#issuecomment-38804635
.

from zipline.

Michael, you should be able to do this as you intend. I wrote a little script that is in the utils for this called data_source_tables_gen.py. if you match the headers everything will work cleanly. Otherwise, we can put in an option to pass new headers.

@ehebert how would you want people to pass in columns for this?

Also, I use a lot of this type of stuff in virtual boxes and copy back to linux. It makes development much easier for me.

from zipline.

@MichaelWS one solution might be a parameter which is a dictionary that maps the headers to the expected headers, e.g. custom_header_mapper={'my-foo-price': 'price'}

Also, since the PR has pulled in going to close this ticket, and let's keep track of enhancements to the source in new tickets.

from zipline.