queryverse / statfiles.jl Goto Github PK

Package is usefull , Thanks !
When

data= load("data.dta")

how to make clear data (Julia Array e.g {Any} ) from data? I have big file and Data Frames and other are slowly :/
I hope is posible to fast and easy to do!
Paul

using ReadStat
file = download("http://www.stata-press.com/data/r15/fullauto.dta",
                "data/ologit.dta")
data = read_dta(file)
using StatFiles, DataFrames
output = load(file) |> DataFrame

If you take a look at data you will see that categorical variables have a mapping to labels given by val_labels_keys and val_label_dict. Without taking into account that nuance, the default behavior specified here yields the values instead of the labels (e.g., rep77 gives [1, 2, 3, 4, 5] instead of ["Poor", "Fair", "Average", "Good", "Excellent"]). It might be the case for other file formats, but this is confirmed for Stata's dta.

Rough first steps:

1. Add function stubs for the new functions we need from the C library to this file. Find the function names and signature from https://github.com/WizardMac/ReadStat.
2. Just get one example of writing some file to work.

Is there a way to automatically change the type of all float variables to Float64?

Due to a bug, one cannot fit linear models with Float32 in GLM.jl?

see JuliaStats/GLM.jl#260

I am trying to read stata panel datasets of Russian households and invividuals.
They can be downloaded freely after simple registration.
The houeseholds dataset is ~1gb and individuals is ~4gb. Machine has 16 gb.

Even 1gb file take too long to load (have waited > 15 minutes without any results).

using StatFiles
a = load("USER_RLMS-HSE_HH_1994_2017nt_eng.dta")

Presumably we only need to register the file type in FileIO, and for that we need to find out whether there is some magic byte sequence for that file type.