Code Monkey home page Code Monkey logo

pysqldf's Introduction

pysqldf

Build Status PyPI Version PyPI Monthly Downloads PyPI License

pysqldf allows you to query pandas DataFrames using SQL syntax. It works similarly to sqldf in R. pysqldf seeks to provide a more familiar way of manipulating and cleaning data for people new to Python or pandas.

Installation

$ pip install pysqldf

Basics

The main class in pysqldf is SQLDF. SQLDF accepts 1 enviroment variable sets or more parametrs in constructor.

  • an set of session/environment variables (dictionary of valiables, locals() or globals())
  • temporary file type
  • user defined functions
  • user defined aggregate functions

pysqldf uses SQLite syntax. Any convertable data to pandas DataFrames will be automatically detected by pysqldf. You can query them as you would any regular SQL table.

$ python
>>> from pysqldf import SQLDF, load_meat, load_births
>>> sqldf = SQLDF(globals())
>>> meat = load_meat()
>>> births = load_births()
>>> print sqldf.execute("SELECT * FROM meat LIMIT 10;").head()
                  date  beef  veal  pork  lamb_and_mutton broilers other_chicken turkey
0  1944-01-01 00:00:00   751    85  1280               89     None          None   None
1  1944-02-01 00:00:00   713    77  1169               72     None          None   None
2  1944-03-01 00:00:00   741    90  1128               75     None          None   None
3  1944-04-01 00:00:00   650    89   978               66     None          None   None
4  1944-05-01 00:00:00   681   106  1029               78     None          None   None

>>> q = "SELECT m.date, m.beef, b.births FROM meat m INNER JOIN births b ON m.date = b.date;"
>>> print sqldf.execute(q).head()
                    date    beef  births
403  2012-07-01 00:00:00  2200.8  368450
404  2012-08-01 00:00:00  2367.5  359554
405  2012-09-01 00:00:00  2016.0  361922
406  2012-10-01 00:00:00  2343.7  347625
407  2012-11-01 00:00:00  2206.6  320195

>>> q = "SELECT strftime('%Y', date) AS year, SUM(beef) AS beef_total FROM meat GROUP BY year;"
>>> print sqldf.execute(q).head()
   year  beef_total
0  1944        8801
1  1945        9936
2  1946        9010
3  1947       10096
4  1948        8766

user defined functions and user defined aggregate functions also supported.

$ python
>>> from pysqldf import SQLDF, load_iris
>>> import math
>>> import numpy
>>> ceil = lambda x: math.ceil(x)
>>> udfs = { "ceil": lambda x: math.ceil(x) }
>>> udafs = { "variance": lambda values: numpy.var(values) }
>>> # or you can also define aggregation function as class
>>> # class variance(object):
... #     def __init__(self):
... #         self.a = []
... #     def step(self, x):
... #         self.a.append(x)
... #     def finalize(self):
... #         return numpy.var(self.a)
...
>>> # udafs={ "variance": variance }
>>> iris = load_iris()
>>> sqldf = SQLDF(globals(), udfs=udfs, udafs=udafs)
>>> sqldf.execute("""
    SELECT
        ceil(sepal_length) AS sepal_length,
        ceil(sepal_width) AS sepal_width,
        ceil(petal_length) AS petal_length,
        ceil(petal_width) AS petal_width,
        species
    FROM iris;
    """).head()
   sepal_length  sepal_width  petal_length  petal_width      species
0             6            4             2            1  Iris-setosa
1             5            3             2            1  Iris-setosa
2             5            4             2            1  Iris-setosa
3             5            4             2            1  Iris-setosa
4             5            4             2            1  Iris-setosa
>>> sqldf.execute("SELECT species, variance(sepal_width) AS var FROM iris GROUP BY species;")
           species       var
0      Iris-setosa  0.142276
1  Iris-versicolor  0.096500
2   Iris-virginica  0.101924

Documents

SQLDF(env, inmemory=True, udfs={}, udafs={})

env: variable mapping dictionary of sql executed enviroment. key is sql variable name and value is your program variable. locals() or globals() is used for simple assign.

inmemory: sqlite db option.

udfs: dictionary of user defined functions. dictionary key is function name, dictionary value is function. see sqlite3 document

udafs: dictionary of user defined aggregate functions. dictionary key is function name, dictionary value is aggregate function or class. If value is function, function gets one argument that is list of column values and it should return aggregated a value. Another case(value is class), see sqlite3 document.

load_iris(), load_meat(), load_births()

load example DataFrame data.

pysqldf's People

Contributors

airtoxin avatar glamp avatar hernamesbarbara avatar colindickson avatar brentpayne avatar xmnlab avatar tomaskazemekas avatar stonebig avatar

Stargazers

Tomohiro avatar Dev Aggarwal avatar  avatar  avatar  avatar Tom Wong avatar Paulo Haddad avatar Hung-Wei Wu avatar gasyoh avatar  avatar WenSui Liu avatar Takeshi Mizumoto avatar Arata Tanaka avatar sam avatar Takuya Kitazawa avatar Hiroki Kiyohara avatar makoto tsuyuki avatar takayuki83 avatar  avatar

Watchers

James Cloos avatar  avatar  avatar  avatar

pysqldf's Issues

how to use at jupyter notebook

Hi,

When I try to use at Jupyter Notebook , I got below error.
Could you give me some hint?

from pysqldf import SQLDF, load_meat, load_births

meat = load_meat()
births = load_births()

sqldf = SQLDF(globals())
sqldf.execute('select * from meat limit 10').head()


OperationalError Traceback (most recent call last)
in ()
1 sqldf = SQLDF(globals())
----> 2 sqldf.execute('select * from meat limit 10').head()

C:\Miniconda2\envs\py27\lib\site-packages\pysqldf\sqldf.pyc in execute(self, query)
74 result = None
75 finally:
---> 76 self._del_table(tables)
77 return result
78

C:\Miniconda2\envs\py27\lib\site-packages\pysqldf\sqldf.pyc in _del_table(self, tablenames)
115 def _del_table(self, tablenames):
116 for tablename in tablenames:
--> 117 self.conn.execute("drop table " + tablename)
118 self.conn.commit()
119

OperationalError: no such table: meat

sqlite3.OperationalError: no such table: meat

hi, i think pysqldf is more powerful library than pandasql.
According to the author's example:
from pysqldf import SQLDF, load_meat, load_births
sqldf = SQLDF(globals())
meat = load_meat()
births = load_births()
print sqldf.execute("SELECT * FROM meat LIMIT 10;").head()
i find the hint that sqlite3.OperationalError: no such table: meat
who can help me?
pandas 1.2.4 python 3

dtype

don't change column dtype

# in _ensure_data_frame
        for col in df:
            if df[col].dtype==np.int64:
                df[col] = df[col].astype(np.float)
            elif isinstance(df[col].get(0), pd.tslib.Timestamp):
                df[col] = df[col].apply(lambda x: str(x))

Compatibility with pandas 0.23.3

I'm using pandas 0.23.3 and pysqldf is not compatible - the flavor argument for to_sql has been removed, so the call fails.

I've edited the code in _write_table of pysqldf/sqldf.py to read
to_sql(df, name=tablename, con=self.conn, dtype=dtype)
instead of
to_sql(df, name=tablename, con=self.conn, flavor="sqlite", dtype=dtype)
which corrects the problem for this version of pandas. I guess a try/except would fix it generically?

Support for a different sql backend (e.g. mysql)

Hi - I'd consider working on this as an enhancement : but would appreciate some insight/ advice on it. It is my belief there were some PR for that on the original yhat repo but I can not find it now.

lmk. stephen boesch

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.