Carriage return and newline characters in dump about docs-api HOT 5 CLOSED

Duplicate of Podcastindex-org/database#5 – basically the export contains both \r and \r\n as newlines. Trying to process the data, I also get UTF8 errors like illegal byte sequence:

wget https://archive.org/download/podcastindex_dump/podcasts_2020-10-02.csv

sed -e 's/"",/,/g'  podcasts_2020-10-02.csv > podcasts_2020-10-02_fixed.csv
> sed: RE error: illegal byte sequence

iconv -f UTF-8 podcasts_2020-10-02.csv > /dev/null                         
> iconv: podcasts_2020-10-02.csv:172436:66: cannot convert

from docs-api.

I would gladly dump the table as a SQL file with mysqldump, but it contains the itunesOwnerEmail column, which I do not want to distribute. I can’t see a clean way to dump everything but that column without jumping through a bunch of temporary table hoops, which is time consuming on a table that large. Maybe I’ll just bite the bullet and do it anyway.

from docs-api.

There is indeed not really a way around doing a dump that way without temporary tables. But perhaps if you don't want to distribute that column, it should be in another place.

from docs-api.

One could write an own exporter tool, which correctly writes the defined export format. I personally would prefer JSON or maybe even CSV. But we can also write a simple shell script with fixes the current export, e.g. by replacing all /r/n with /n and then all /n with /r/n again. But then we would still have the unicode errors...

from docs-api.

Karsten in the mastadon shot me a python export tool he uses. He said it should preserve the charset and do well. I'm going to give that a shot this next time.

from docs-api.