stackoverflow when doing diagonal concat of 17k CSVs about polars HOT 2 CLOSED

@stinodego the csv is a read herring. This is a stackoverflow in streaming engine. The sink parquet shares the append union which currently is implemented recursively per pipeline. 17K in this case.

Even if this is not possible, I expect a clear error not a seg fault. (I thought it was theoretically impossible to get a seg fault with Rust.)

No it is not. And a stackoverflow also isn't handled gracefully that's why you don't get a nice error.

from polars.

Ah ok, that makes sense.

My workaround for now is to process the files in batches (17k CSV to 17 parquet in 17 batches of 1000, then another step to combine 17 parquet files into 1 parquet file.). But for different datasets the limit is lower than 17k.

from polars.