Comments (13)
Hi @ztsweet, in order to reproduce the error, can you let us know the database source and dataframe destination that you are using? And also what data types are included in your query result?
from connector-x.
@wangxiaoying I have edited the question again
from connector-x.
@ztsweet Thanks for the details. Can you also check what is the data type of ORDER_ID in MySQL?
from connector-x.
@wangxiaoying in mysql the type is ``ORDER_ID bigint(19) DEFAULT NULL
from connector-x.
@ztsweet It seems like we have a bug when parsing NULL values in MySQL source. Will fix this issue it in our next release (0.2.1).
from connector-x.
@wangxiaoying thanks
from connector-x.
Hi @ztsweet, we just released the connectorx-0.2.1a1 which has fixed the NULL value bug in mysql. You can try it by using pip install -U connectorx==0.2.1a1
.
If all tests go well, we plan to release the 0.2.1 version in early October. Please feel free to have a try and let us know if you found any issue.
from connector-x.
Are you using Python or Rust?
Python.
What version of polars are you using?
polars==0.10.27
connectorx==0.2.2
What operating system are you using polars on?
CentOS Linux release 7.9.2009 (Core)
Describe your bug.
read_sql
returns following exceptions if any column contains missing values, i.e. None
Note: No error occurs if all rows have values
Errors
import polars as pl
pl.read_sql("select column1 from Table", <url_to_database>)
Replicated the issue using cx.read_sql
. Same errors
If column1 TYPE = MEDIUMTEXT
RuntimeError: Invalid argument error: column types must match schema types, expected Time64(Nanosecond) but found LargeBinary at column index 0
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-67-2867bd04669a> in <module>
----> 1 df_pl = pl.read_sql("""
2 select Replaced_Query_Bill
3 from Event_Master limit 10""", cart)#.to_pandas()
4 # Follows_Waterfall, Condition, Event_Limit, Slug, Conversation, Campaign_Key
5 # Arms, Is_MAB_Driven, Is_Good_Time, Is_On_Demand
~/ipynbs/.venv/lib/python3.9/site-packages/polars/io.py in read_sql(sql, connection_uri, partition_on, partition_range, partition_num)
802 """
803 if _WITH_CX:
--> 804 tbl = cx.read_sql(
805 conn=connection_uri,
806 query=sql,
~/ipynbs/.venv/lib/python3.9/site-packages/connectorx/__init__.py in read_sql(conn, query, return_type, protocol, partition_on, partition_range, partition_num, index_col)
133 raise ValueError("You need to install pyarrow first")
134
--> 135 result = _read_sql(
136 conn,
137 "arrow",
RuntimeError: Invalid argument error: column types must match schema types, expected Time64(Nanosecond) but found LargeBinary at column index 0
If column1 TYPE = DATETIME
PanicException: Could not retrieve chrono::naive::datetime::NaiveDateTime from Value
---------------------------------------------------------------------------
PanicException Traceback (most recent call last)
<ipython-input-101-17df85ddf35d> in <module>
----> 1 cart.read_sql('select * from Event_Master limit 10')
<ipython-input-97-7fb8f6c351ab> in read_sql(self, query)
25 def read_sql(self, query):
26 try:
---> 27 result = pl.read_sql(query, self.__url)
28 except Exception as e:
29 print('Polar failed trying pandas')
~/ipynbs/.venv/lib/python3.9/site-packages/polars/io.py in read_sql(sql, connection_uri, partition_on, partition_range, partition_num)
802 """
803 if _WITH_CX:
--> 804 tbl = cx.read_sql(
805 conn=connection_uri,
806 query=sql,
~/ipynbs/.venv/lib/python3.9/site-packages/connectorx/__init__.py in read_sql(conn, query, return_type, protocol, partition_on, partition_range, partition_num, index_col)
133 raise ValueError("You need to install pyarrow first")
134
--> 135 result = _read_sql(
136 conn,
137 "arrow",
PanicException: Could not retrieve chrono::naive::datetime::NaiveDateTime from Value
What is the expected behavior?
Read empty values in dataframe just like pandas
from connector-x.
Hi @hamzamohdzubair thanks for reporting the issue. It looks like the null data does not match with the type we fetched. May I ask which database you are using? Also what is the type for column1
defined in your database?
from connector-x.
@wangxiaoying updated my previous comment
Database = mysql Ver 15.1 Distrib 10.5.9-MariaDB
The first error comes when TYPE = MEDIUMTEXT
second error comes when TYPE = DATETIME
from connector-x.
@wangxiaoying Also, why am i unable to catch the PanicException
in a try except
clause. I am trying to do this:
read a dataframe using polars, if it fails read it using pandas, but i am unable to catch the PanicException
from connector-x.
@hamzamohdzubair , thanks for the info. It is weird since we have the mysql test case when DATETIME
is null. I also test on MEDIUMTEXT
and it also passes. We use the official docker image of mysql in our environment. We haven't test on mariadb yet, I will try to set up and test on it to see whether I can reproduce the error.
As for the PanicException, I think the panic happens in the underlying mysql driver. I will also take a look at it if I can reproduce this error.
from connector-x.
Hi @hamzamohdzubair, thanks for waiting.
For the first error on MEDIUMTEXT
, turns out it is a bug in the type mapping in arrow. It has been fixed here: 46bf99a. We will release an alpha version to this bug fix. Will let you know once it's done.
For the second error onDATETIME
, I cannot reproduce the error using mysql / mariadb docker image when output pandas
/ arrow
. Is it possible if you could provide an example SQL script for constructing the table (create table and insert values to the table) that can cause the error?
UPDATE: 0.2.3-alpha.3
is released! Please feel free to have a try.
from connector-x.
Related Issues (20)
- Release connectorx 0.3.2 on crates.io HOT 3
- Cut release for manylinux 2_17
- Elasticsearch support
- rust connector-x 0.3.2 not working with polars 0.33 HOT 3
- Passing <ServerName\InstanceName>, as the host value, to the MSSQL connection string doesn't work. HOT 1
- Unable to read SQL Table with _date array: thread '<unnamed>' panicked at 'not implemented: _date',
- Several critical vulnerabilities in connector-x dependencies HOT 2
- Any Query hangs forever on Postgresql in AWS RDS Aurora
- Connecting to Oracle error with oracledb HOT 2
- cannot send arbitrary number of queries in a single call to `read_sql` HOT 1
- PanicException: called `Result::unwrap()` Error { kind: ConfigParse, cause: Some("unexpected EOF") } HOT 1
- parse error: invalid port number with Postgres on Python 0.3.1 HOT 1
- Release support 0.3.2 for manylinux_2_17 Required for AWS Glue Python shell HOT 1
- Unable to get time part from date datatype column from oracle DB
- Cannot connect to MSSQL: `Timed out in bb8` HOT 4
- Issue on page /freq_questions.htmModuleNotFoundError: No module named 'connectorx.connectorx_python'l
- Cassandra or scylladb support.
- Releasing GIL in read_sql
- Python read_sql() crash: Too many open files when used with multiprocessing HOT 1
- special characters on password gives db error HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from connector-x.