Comments (12)
minimal reproducable code:
import polars as pl
df_silver = pl.DataFrame(
{
"description": ["", "hello", "hi", ""],
}
)
df_gold = df_silver.select(
[
pl.when(pl.col("description") == "")
.then(pl.lit(None))
.otherwise(
pl.concat_list(
pl.struct(
pl.col("description").alias("translation"),
)
)
)
.alias("description"),
]
)
print(df_gold)
df_gold.write_parquet(f"test.parquet")
df_gold_read = pl.read_parquet(f"test.parquet")
print(df_gold_read)
shape: (4, 1)
┌─────────────────┐
│ description │
│ --- │
│ list[struct[1]] │
╞═════════════════╡
│ null │
│ [{"hello"}] │
│ [{"hi"}] │
│ null │
└─────────────────┘
shape: (4, 1)
┌─────────────────┐
│ description │
│ --- │
│ list[struct[1]] │
╞═════════════════╡
│ null │
│ [{""}] │
│ [{"hello"}] │
│ null │
└─────────────────┘
I left away the None value and check for None, because this does not seem to influence the test. Also remove the locale field.
from polars.
So it seems to end up being a problem with write_parquet
specifically?
If we switch to sink_parquet
- it does not replicate the bug:
df_gold.lazy().sink_parquet(f"test.parquet")
pl.read_parquet(f"test.parquet")
# shape: (4, 1)
# ┌─────────────────┐
# │ description_new │
# │ --- │
# │ list[str] │
# ╞═════════════════╡
# │ null │
# │ ["hello"] │
# │ ["hi"] │
# │ null │
# └─────────────────┘
from polars.
The lazy().sink_parquet
does not seem to fix the issue. The bug is extemely specific as to when it occurs, but when it does, it occurs consistently.
Take the following code:
import polars as pl
df_silver = pl.DataFrame(
{
"description": [
"Hello",
"Hello",
"Hello",
"Hello",
"Hello",
None,
"Hello",
None,
None,
"Hello",
None,
"Hello",
"Hello",
"Hello",
"Hello",
"",
"Hello",
"Hello",
"Hello",
"Hello",
],
}
)
df_gold = df_silver.select(
[
pl.when(pl.col("description").is_null())
.then(pl.lit(None))
.when(pl.col("description") == "")
.then(pl.lit(None))
.otherwise(
pl.concat_list(
pl.col("description").alias("translation"),
)
)
.alias("description_new"),
]
)
print(df_gold[10:])
df_gold.lazy().sink_parquet(f"test.parquet")
df_gold_read = pl.read_parquet(f"test.parquet")
print(df_gold_read[10:])
shape: (10, 1)
┌─────────────────┐
│ description_new │
│ --- │
│ list[str] │
╞═════════════════╡
│ null │
│ ["Hello"] │
│ ["Hello"] │
│ ["Hello"] │
│ ["Hello"] │
│ null │
│ ["Hello"] │
│ ["Hello"] │
│ ["Hello"] │
│ ["Hello"] │
└─────────────────┘
shape: (10, 1)
┌─────────────────┐
│ description_new │
│ --- │
│ list[str] │
╞═════════════════╡
│ null │
│ ["Hello"] │
│ ["Hello"] │
│ ["Hello"] │
│ ["Hello"] │
│ null │
│ [""] │
│ ["Hello"] │
│ ["Hello"] │
│ ["Hello"] │
└─────────────────┘
when the ""
value is placed one row higher during the creation of the dataframe:
import polars as pl
df_silver = pl.DataFrame(
{
"description": [
"Hello",
"Hello",
"Hello",
"Hello",
"Hello",
None,
"Hello",
None,
None,
"Hello",
None,
"Hello",
"Hello",
"Hello",
"",
"Hello",
"Hello",
"Hello",
"Hello",
"Hello",
],
}
)
df_gold = df_silver.select(
[
pl.when(pl.col("description").is_null())
.then(pl.lit(None))
.when(pl.col("description") == "")
.then(pl.lit(None))
.otherwise(
pl.concat_list(
pl.col("description").alias("translation"),
)
)
.alias("description_new"),
]
)
print(df_gold[10:])
df_gold.lazy().sink_parquet(f"test.parquet")
df_gold_read = pl.read_parquet(f"test.parquet")
print(df_gold_read[10:])
shape: (10, 1)
┌─────────────────┐
│ description_new │
│ --- │
│ list[str] │
╞═════════════════╡
│ null │
│ ["Hello"] │
│ ["Hello"] │
│ ["Hello"] │
│ null │
│ ["Hello"] │
│ ["Hello"] │
│ ["Hello"] │
│ ["Hello"] │
│ ["Hello"] │
└─────────────────┘
shape: (10, 1)
┌─────────────────┐
│ description_new │
│ --- │
│ list[str] │
╞═════════════════╡
│ null │
│ ["Hello"] │
│ ["Hello"] │
│ ["Hello"] │
│ null │
│ ["Hello"] │
│ ["Hello"] │
│ ["Hello"] │
│ ["Hello"] │
│ ["Hello"] │
└─────────────────┘
There is no longer a bug. lazy().sink_parquet
seems to alter the behaviour of the bug, but the bug is still very much there. Again when reading the parquet I can see the wrong values too, so the bug is not because of the pl.read_parquet
from polars.
Trying to find out when this changed:
pl.__version__
df_gold[10:].equals(df_gold_read[10:])
It seems it happened after 0.20.15
'0.20.16'
False
'0.20.15'
True
Looking at the release notes for 0.20.16
:
https://github.com/pola-rs/polars/releases/tag/py-0.20.16
- add new when-then-otherwise kernels #15089
As the bug seems to depend on when-then, this may be a starting point for further investigation.
from polars.
Can you show an example of the problem?
e.g.
df = pl.DataFrame({
"description": ["", None, "hi", ""],
"locale": ["a", "b", "c", "d"],
})
df.select(
pl.when(pl.col("description").is_null()).then(pl.lit(None))
.when(pl.col("description") == "")
.then(pl.lit(None))
.otherwise(
pl.concat_list(
pl.struct(
pl.lit("en").alias("locale"),
pl.col("description").alias("translation"),
)
)
)
.alias("description")
)
shape: (4, 1)
┌─────────────────┐
│ description │
│ --- │
│ list[struct[2]] │
╞═════════════════╡
│ null │
│ null │
│ [{"en","hi"}] │
│ null │
└─────────────────┘
from polars.
I have been investigating further. The issue seems to be more complex.
df_silver = pl.read_parquet(f"data/silver/locations.parquet")
df_gold = df_silver.select(
[
pl.when(pl.col("description").is_null())
.then(pl.lit(None))
.when(pl.col("description") == "")
.then(pl.lit(None))
.otherwise(
pl.concat_list(
pl.struct(
pl.lit("en").alias("locale"),
pl.col("description").alias("translation"),
)
)
)
.alias("description"),
]
)
# printing the the range of rows where the issue occurs
print(df_gold[60:70])
shape: (10, 1)
┌─────────────────────────────────┐
│ description │
│ --- │
│ list[struct[2]] │
╞═════════════════════════════════╡
│ null │
│ [{"en","Parking working hours … │
│ null │
│ [{"en","Parking working hours … │
│ [{"en","Parking working hours … │
│ [{"en","Parking working hours … │
│ [{"en","Parking working hours … │
│ null │
│ [{"en","Parking working hours … │
│ [{"en","Parking working hours … │
└─────────────────────────────────┘
As you can see, no issue.
If I then do
df_gold.write_parquet(f"data/gold/locations.parquet")
df_gold_read = pl.read_parquet(f"data/gold/locations.parquet")
print(df_gold_read[60:70])
shape: (10, 1)
┌─────────────────────────────────┐
│ description │
│ --- │
│ list[struct[2]] │
╞═════════════════════════════════╡
│ null │
│ [{"en","Parking working hours … │
│ null │
│ [{"en","Parking working hours … │
│ [{"en","Parking working hours … │
│ [{"en","Parking working hours … │
│ [{"en","Parking working hours … │
│ null │
│ [{"en",""}] │
│ [{"en","Parking working hours … │
└─────────────────────────────────┘
I see the empty translation field again, just like I saw when exploring the parquet file in data wrangler.
Even stranger is when i do this:
df_gold.write_json(f"data/gold/eldrive_locations.csv")
df_gold_read = pl.read_json(f"data/gold/eldrive_locations.csv")
print(df_gold_read[60:70])
shape: (10, 1)
┌─────────────────────────────────┐
│ description │
│ --- │
│ list[struct[2]] │
╞═════════════════════════════════╡
│ null │
│ [{"en",null}] │
│ null │
│ [{"en","Parking working hours … │
│ [{"en",null}] │
│ [{"en",null}] │
│ [{"en","Parking working hours … │
│ null │
│ [{"en",null}] │
│ [{"en","Parking working hours … │
└─────────────────────────────────┘
To show the complete values in the dataframe I ran:
for row in df_gold[60:70].iter_rows(named=True):
print(row["description"])
None
[{'locale': 'en', 'translation': 'Parking working hours and tariffs as per location terms'}]
None
[{'locale': 'en', 'translation': 'Parking working hours and tariffs as per location terms.'}]
[{'locale': 'en', 'translation': 'Parking working hours and tariffs as per location terms'}]
[{'locale': 'en', 'translation': 'Parking working hours and tariffs as per location terms'}]
[{'locale': 'en', 'translation': 'Parking working hours and tariffs as per location terms'}]
None
[{'locale': 'en', 'translation': 'Parking working hours and tariffs as per location terms'}]
[{'locale': 'en', 'translation': 'Parking working hours and tariffs as per location terms'}]
from polars.
Can you update to latest Polars (1.2.1) and confirm it still occurs?
from polars.
Sorry, I thought that I already was on the latest version. The ouput of my tests are identical with the new version.
pl.show_versions()
--------Version info---------
Polars: 1.2.1
Index type: UInt32
Platform: Linux-6.8.0-38-generic-x86_64-with-glibc2.39
Python: 3.12.3 (main, Apr 10 2024, 05:33:47) [GCC 13.2.0]
----Optional dependencies----
adbc_driver_manager: <not installed>
cloudpickle: <not installed>
connectorx: <not installed>
deltalake: <not installed>
fastexcel: <not installed>
fsspec: <not installed>
gevent: <not installed>
great_tables: <not installed>
hvplot: <not installed>
matplotlib: <not installed>
nest_asyncio: 1.6.0
numpy: 2.0.0
openpyxl: <not installed>
pandas: 2.2.2
pyarrow: 17.0.0
pydantic: <not installed>
pyiceberg: <not installed>
sqlalchemy: <not installed>
torch: <not installed>
xlsx2csv: <not installed>
xlsxwriter: <not installed>
I also tried using the dataframe you used in your example. Saving as json and parquet give the same results here. printing the read dataframe:
shape: (4, 1)
┌─────────────────┐
│ description │
│ --- │
│ list[struct[2]] │
╞═════════════════╡
│ null │
│ null │
│ [{"en",""}] │
│ null │
└─────────────────┘
from polars.
And have you got a minimal repro? I don't see any way to reproduce your query? Ideally on syntetic data in memory, otherwise from the file. Cut out everything that isn't involved.
from polars.
The struct also does not seem to influence the bug
import polars as pl
df_silver = pl.DataFrame({"description": ["", "hello", "hi", ""],})
df_gold = df_silver.select(
[
pl.when(pl.col("description") == "")
.then(pl.lit(None))
.otherwise(
pl.concat_list(
pl.col("description").alias("translation"),
)
)
.alias("description_new"),
]
)
print(df_gold)
df_gold.write_parquet(f"test.parquet")
df_gold_read = pl.read_parquet(f"test.parquet")
print(df_gold_read)
shape: (4, 1)
┌─────────────────┐
│ description_new │
│ --- │
│ list[str] │
╞═════════════════╡
│ null │
│ ["hello"] │
│ ["hi"] │
│ null │
└─────────────────┘
shape: (4, 1)
┌─────────────────┐
│ description_new │
│ --- │
│ list[str] │
╞═════════════════╡
│ null │
│ [""] │
│ ["hello"] │
│ null │
└─────────────────┘
from polars.
This indeed fixes the issue. Am I right to think this should be considered as a major issue, as it negatively affects the data quality?
Also I tried further testing and it does have something to do with the creation of the df_gold dataframe (so the .select part). When I create a dataframe similar to df_gold directly like this:
df_silver = pl.DataFrame({"description": [None, ["hello"], ["hi"], None]})
print(df_silver)
df_silver.write_parquet(f"test.parquet")
df_silver_read = pl.read_parquet(f"test.parquet")
print(df_silver_read)
┌─────────────┐
│ description │
│ --- │
│ list[str] │
╞═════════════╡
│ null │
│ ["hello"] │
│ ["hi"] │
│ null │
└─────────────┘
shape: (4, 1)
┌─────────────┐
│ description │
│ --- │
│ list[str] │
╞═════════════╡
│ null │
│ ["hello"] │
│ ["hi"] │
│ null │
└─────────────┘
Both the hello and hi appear correctly in the output.
from polars.
import polars as pl df_silver = pl.DataFrame({"description": ["", "hello", "hi", ""],}) df_gold = df_silver.select( [ pl.when(pl.col("description") == "") .then(pl.lit(None)) .otherwise( pl.concat_list( pl.col("description").alias("translation"), ) ) .alias("description_new"), ] ) print(df_gold) df_gold.write_parquet(f"test.parquet") df_gold_read = pl.read_parquet(f"test.parquet") print(df_gold_read)
@coastalwhite can you take a look?
from polars.
Related Issues (20)
- add an `ignore_nulls` option to `json_encode` for the `pl.Struct` column
- Change all columns one time HOT 5
- map_elements replaces all array elements with nulls HOT 1
- Let arr.reshape use Expr as inputs instead of just python tuple
- Import and Export Schema Objects to JSON
- error[E0599]: no variant or associated item named `Struct` found for enum `polars_core::datatypes::DataType` in the current scope HOT 2
- `with_columns(dict)` fail silently if key exists as column HOT 5
- Polars write database - Rest API call failing with AWS Lambda trigger HOT 1
- Change `read_csv` and `read_ipc` to use object store instead of fsspec
- `.over()` fails with `.top_k_by` HOT 3
- `join_nulls` in "asof" join HOT 3
- exception on numpy slicing literal column with object column
- Scanning cloud paths with percentages '%' fails
- Make pl.Enum(...) return type rather than instance HOT 2
- Elementwise check on `join` expressions is too restrictive HOT 3
- Built-in datasets and a function to load them HOT 1
- Python test workflows may fail due to failure to download `torch` dependency HOT 1
- `scan_parquet` does not optimise `slice` or `tail` operations HOT 3
- Apply function to rows of dictionaries in `map_rows`
- De-duplicate decompression in CSV/NDJSON scans
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from polars.