kshedden / datareader Goto Github PK

Does this library only read stata.dta files? Is it possible to write them?

Thanks!

Getting two types of errors reading in .sas7bdat files:

• Unexpected non-zero end_of_first_byte
• 32 Character byte unknown

Any ideas on what's going on and how to handle?

Hello!

I just came across this package. I'm aware that Go is generally very fast. Do you happen to know how the read speeds for Stata files with this package compare to Stata or Python? Is the reading multithreaded?

Also, "simple column-oriented data container" caught my eye. I'm especially curious if this data structure is similar to one that can be written by the parquet-go package. Since Parquet is a column-oriented file format, I'm guessing that reading Stata files with your package and writing it with parquet-go could be much faster than my current code to do that in Python.

https://github.com/xiaodaigh/sas7bdat-resources/blob/master/README.md

Can this package write to .sas7bdat files?

Seek(,0) happens on every read so first rows of file are repeatedly re-read on each call to Read()

(This is probably true for other file types, but I haven't tested them.)

In sas7bdat.go:818, the code assumes that any non-nil error should void the read, but that is not true for io.EOF. The golang io package says explicitly:
"""
// When Read encounters an error or end-of-file condition after
// successfully reading n > 0 bytes, it returns the number of
// bytes read. It may return the (non-nil) error from the same call
// or return the error (and n == 0) from a subsequent call.
// An instance of this general case is that a Reader returning
// a non-zero number of bytes at the end of the input stream may
// return either err == EOF or err == nil.
"""

Thus, if a File interface chooses to return EOF immediately on the last, positive-byte-sized read rather than waiting to return EOF with a 0-byte read (as the s3 filesystem I am using does), then this breaks the SAS7BDAT reader.

I believe a simple fix would be to change the line (sas7bdat.go:818) to:

	if err != nil && err != io.EOF {

Would that work?

Column labels within sas files are lost with the Go reader.

I often use the stattocsv command line utility to work with data using the GNU utilities and I noticed that some of my columns with addresses, e.g., "50 E. Main St" only came out as "50".

I then wrote my own frontend using your library to spit out as CSV and it didn't have that issue.

Debugging some, I found it was the TrimStrings bool value causing the problem but I didn't debug further.

haven::read_sas is able to correctly parse "numeric" columns into integer or boolean columns. It's still not clear to me how haven::read_sas becomes aware of the underlying type, though if I had to guess it is related to how many bytes are stored for each column value.

It would be nice if datareader were able to similarly infer type and distinguish ints and bools from floats.

I am reading a sas7bdat file that represents a table with ~15K rows. The reader gives the correct column formats, names, and labels. For the data itself, the first 18 rows are incorrect in that columns have incorrect offsets while the rest is correct.