Code Monkey home page Code Monkey logo

jenks's Introduction

Fast python library for jenks natural breaks

The history and intent of the Jenks natural breaks algorithm is well covered by

There are even a few python implementations:

However, it has been noted that the python implementations are tediously slow. There are two obvious reasons for this...

  1. All the data is stored in python lists rather than optimized data structures. The fact that the variables are named "matrices" is perhaps some sort of practical joke given how bad lists are for matrix/array like data structures. Numpy arrays are your friend and I can't imagine doing any numeric computation in python without them.

  2. There is a lot of looping. Like exponential-time looping. The algorithm makes this somewhat inevitable. Python sucks at iterating over simple math, exploding the runtime very quickly. Cython, through it's variable typing, allows us to write the algorithm in python-like syntax, compile it to a shared library that can be imported as a python module and run at near-C speeds.

So I set forth to make good use of my son's afternoon nap time and port the existing python implementation (which is, in turn based on the wonderfully documented javascript implementation) to cython with numpy arrays.

Here's the benchmark against the jenks2.py implementation:

In [1]: from jenks2 import jenks

In [2]: %timeit jenks(data, 5)
1 loops, best of 3: 8.16 s per loop

In [3]: from jenks import jenks

In [4]: %timeit jenks(data, 5)
10 loops, best of 3: 69.2 ms per loop

Yep that's 118X faster for just a little bit of static typing and using arrays instead of lists. It even makes the logic easier to read (for those of us who work with matrices/arrays often).

The only cost is that you need Cython and a C compiler to get it working.

sudo apt-get install build-essential cython python-numpy
pip install -e "git+https://github.com/perrygeo/jenks.git#egg=jenks"

And then test it

In [1]: import json

In [2]: from jenks import jenks

In [3]: # data should be 1-dimensional array, python list or iterable 

In [4]: data = json.load(open('test.json')) 

In [5]: print jenks(data, 5)
[0.002810962, 2.0935481, 4.2054954, 6.1781483, 8.0917587, 9.997983]

jenks's People

Contributors

greggg230 avatar perrygeo avatar sgillies avatar taleinat avatar trjordan avatar valearna avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

jenks's Issues

Include variance_combinations in the output of jenks.jenks()

It seems to me that we should include variance_combinations in the output of jenks.jenks() instead of discarding the data. This would allow one to compute the goodness of variance fit, for example.

I suggest:

(...)
lower_class_limits, variance_combinations = jenks_matrices(data, n_classes)
(...)
return {'breaks':kclass, 'variance_combinations':variance_combinations}

Not working for Windows 10?

Hi,

I tried to install it in Windows 10 and it gives the following message:


Obtaining jenks from git+https://github.com/perrygeo/jenks.git#egg=jenks
  Cloning https://github.com/perrygeo/jenks.git to c:\users\irene\src\jenks
Requirement already satisfied: Numpy in c:\users\irene\appdata\local\conda\conda\envs\envpy34\lib\site-packages (from jenks)
Installing collected packages: jenks
  Running setup.py develop for jenks
    Complete output from command C:\Users\irene\AppData\Local\conda\conda\envs\envpy34\python.exe -c "import setuptools, tokenize;__file__='C:\\Users\\irene\\src\\jenks\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps:
    running develop
    running egg_info
    writing requirements to jenks.egg-info\requires.txt
    writing dependency_links to jenks.egg-info\dependency_links.txt
    writing top-level names to jenks.egg-info\top_level.txt
    writing jenks.egg-info\PKG-INFO
    warning: manifest_maker: standard file '-c' not found

    reading manifest file 'jenks.egg-info\SOURCES.txt'
    writing manifest file 'jenks.egg-info\SOURCES.txt'
    running build_ext
    cythoning jenks.pyx to jenks.c
    building 'jenks' extension
    **error: Microsoft Visual C++ 10.0 is required. Get it with "Microsoft Windows SDK 7.1":** www.microsoft.com/download/details.aspx?id=8279

Naturally, after downloading 20GB of Microsoft stuff (many SDK, Visual Studio, and other things I didnt ask for...) it keeps giving the same message. I also tried with MinGW, but keeps giving this message. Is there any workaround for this? Do you know why is this error happening?

Thanks!

Off by 1 error in jenks.pyx

Line 97 should be k = data.shape[0] instead of k = data.shape[0] - 1 (remove the -1).

Simple test: clustering this data set [1, 2, 3, 100] currently yields [1, 2, 100] but should yield [1, 3, 100]. Basically the large outlier 100 should be isolated into its own cluster.

Note which algorithm

I think it is important to note that this is a version of Jenks which does not provide an optimal solution, i.e. not Fisher-Jenks, not optimal Jenks.

import jenks
import stats

from hypothesis import example, given, assume
from hypothesis import strategies as st


@st.composite
def list_choose_clusters(draw, max_size=None, unique=False):
    xs = draw(st.lists
            ( st.integers(min_value=-10**3, max_value=10**3)
            , min_size=3, max_size=max_size, unique=unique
            ))
    k = draw(st.integers(min_value=2, max_value=len(xs)-1))
    return (xs, k)

@given(list_choose_clusters(max_size=15, unique=True))
def test_jenks_breaks(t):
    tf = lambda cs: sum(stats.variance(c, stats.mean(c)) for c in cs)
    v1 = tf(stats.jenks(*t)[1])
    v2 = tf(stats.break_list(t[0], jenks.jenks(*t)))
    assert v1 == v2
$ pytest perrygeo-tests.py
...
Falsifying example: test_jenks_breaks(
    t=([0, 1, 3, -2], 2),
)

An optimal solution is [[-2], [0, 1, 3]] with a sum variance of 1.55, not [[0, -2], [1, 3]] with a sum variance of 2.

It seems like all Python ports are derived from the same algorithm, so see mthh/jenkspy#22 for more information including documentations/links, sources and full tests.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.