Should line_delim be a parameter? about readr HOT 4 CLOSED

being able to set the line delimiter is handy when parsing FASTA files efficiently in bioinformatics.

It is can also be used to deal with output from some common unix tools (like the -print0 option of find for example).

It is not a huge need most of the time, but I personally miss it when it is absent. (The lack of a line delimiter option in python's readline makes it about 3x slower than perl for parsing a FASTA of the human genome) [1].

from readr.

Just to be clear, with FASTA, you'd want to use > as a line delimiter? (And then you ignore new lines?)

from readr.

Just to be clear, with FASTA, you'd want to use > as a line delimiter? (And then you ignore new lines?)

With the added complexity that the first line for each record is the header, so that would take an additional step. Doing so lets you do 24 read calls vs ~60 million if you are reading with newline as the delimiter, which is where you lose most of the performance.

FASTA files are probably better read by a custom function anyway (of which there are a number), it was just an example of where I have appreciated the option.

I actually think parsing null delimited strings i.e. find -print0 a more persuasive argument for having the option, it is very handy when you are dealing with filenames containing embedded whitespace. However playing around with it a bit this morning it looks like R errors with Error: embedded nul in string: '\0' if you try to construct a string with a null byte so that use case does not even seem possible. :(

from readr.

I think it makes sense to keep it as on - it's not terribly hard to add, and it fits in nicely with the system of parsing specifications that I'm developing.

from readr.