pl.Concat of empty LazyFrames fails about polars HOT 4 OPEN

The first two cases is expected behavior, the third case is a bug. @coastalwhite could you check that third case out? For some reason concatenating with an empty LazyFrame screws up the order in sink_parquet.

from polars.

The first two cases were very unexpected to me. Can you explain them or point me to a resource that explains them? I'll elaborate on my thoughts too:

Regarding the first case, I'm not comparing df1 and df2. I'm comparing df2 and df2.collect().lazy(). To me it seems that if you collect a dataframe and make it lazy again, you're left with the original lazyframe. It should be equivalent to the identity operation. If there's an error, it should happen when you call .collect()on the LazyFrame, not when you compare it to the original lazyframe. I don't think it's unlikely in a big chain of computations on a lazyframe that someone's gonna call.collect()somewhere and then.lazy()again somewhere else, and I don't think they're gonna expect their frame to be different because of it. Also, my intuition of what the correct behavior is has been guided byDataFrame`s, so when I saw that

df0 = pl.DataFrame()
df1 = pl.DataFrame({"a": []})
df2 = pl.concat([df0, df1])

yielded a concatenated DataFrame no problem, I figured there was a problem in first example case. I'd still argue that if my first example is expected, then this snippet here is unexpected.

Regarding the second case, I again think that writing a LazyFrame and the reading it again should be the identity operation. It's essentially the same argument as for the first case. Here's an alternate formulation of the second example where I concatenate DataFrames and then make it lazy, and there's no problem here:

df0 = pl.DataFrame()
df1 = pl.DataFrame({"a": [1, 2, 3]})
df2 = pl.concat([df0, df1]).lazy()

df2.sink_parquet(file)
df3 = pl.scan_parquet(file)

assert_frame_equal(df2, df3)

I'd argue that either both this and example 2 should fail, or both should succeed.

I have now clarified what I see as the issue. If this clears it up and cases 1 and/or 2 indeed are issues, then I apologize for not adding this to the original issue description. I simply didn't think to.

from polars.

I am looking into the 3rd case. I will discuss the other cases with @orlp and come back.

from polars.

Should these not produce the same result?

>>> pl.concat([df0, df1]).collect_schema()
# Schema()

>>> pl.concat([df1, df0]).collect_schema()
# Schema([('a', Null)])

from polars.