pabloem / beam_utils Goto Github PK

Thank you for providing the package.

The following import is failing:

from apache_beam.io import fileio

You are only using fileio for the CompressionType.

The correct import would be:

from apache_beam.io.filesystem import CompressionTypes

(are you still actively developing this package? I'd be happy to add more struff from my local beam_utils if you do)

Hi, and thanks very much for this library.

I noticed that when trying to feed the CsvFileSource a gzip file, I got this error:

TypeError: argument 1 must be an iterator

After some digging I found out that csv.reader() expects an iterator, which, unfortunately, is not what self.open_file() returns (see FileBasedSource).

This test exposes the problem, and this addition to sources.py fixed my problem.

I had some trouble using the JSONLinesFileSource; it was hanging when trying to process a large file and processing smaller files unpredictably.

After having a look at the source, I realised that JSONLinesFileSource is designed to process streaming JSON where all of the objects are on one line, using a buffer. JSONLines is a format where each object is on a separate line (http://jsonlines.org).

It might be a good idea to rename the Source in order to avoid this confusion?

Have you run into the problem where the csv reader will not read more than one record? Any csv I try only reads one line and converts it into a dict.

https://stackoverflow.com/questions/59861800/unable-to-use-csvfilesource-from-beam-utils-sources-in-apache-beam-program-on-cl

Hi,

thanks a lot for writing this little library. It really helped me handling CSV Files.

But since the delimiter in my files wasn't the default ",", I had problems getting the library to work.
It seems that in CsvFileSource you included an option for different CSV delimiters, but the options are actually never used.

reader = csv.reader(self._file)

should be:

reader = csv.reader(self._file, delimiter=self.delimiter)

Thanks for your help and best regards,
Tobias

Hi,

Do you have any sample code of how to use this utility ?

Many thanks