Create a collection of histograms in a file. Utility function to merge multiple files into single file.
Provides a way to distribute processing and histogram creation, store histograms and merge.

I implemented rudimentary versions with json and google protobuf. Tensorflow stores summary data and histograms in protocol buffers, so I gave that a try. The protobuf have versioning capability, which is just one potential benefit. See my fork and the protobuf tag.

Happy to work on this some more with your input.

One thing that would help is to make 'name' required, such that the Summary protocol buffer map <string, Histogram> takes as key the histogram name. This is important for the merging. Possibly merging can be implemented without converting back into the histogram, so we only deserialize, sum and write out new summary.

https://www.researchgate.net/publication/229760716_Averaged_shifted_histogram

This, however, needs something else than matplotlib.

See wiki: https://en.wikipedia.org/wiki/Histogram

Does not install because of missing README.md

If you try to merge bins:

from physt import h2
from scipy.stats import multivariate_normal
hist = h2(*multivariate_normal.rvs((0,0), size=100_000).T, bins=100)
hist.merge_bins(2)

You get a warning from numpy:

/home/schreihf/.local/lib/python3.7/site-packages/physt/histogram_base.py:572: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  new_frequencies[new_index] += old_frequencies[old_index]
/home/schreihf/.local/lib/python3.7/site-packages/physt/histogram_base.py:573: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  new_errors2[new_index] += old_errors2[old_index]

This:

h2(data, range=((min1, min2), None))

is not understood correctly (i.e. no limits for second axis) but as a shared argument. The problem is in np.asarray, binnings.py:766 probably.

If you have a large dataset with a small number of values (such as consisting only of integers 1-10) then it would be nice to have the bin x-axis labels at the center under the respective bin instead of at the bin edges.

I recognise this case is more of a 'histogram as bar plot' kind of thing, but it is a use-case I have often.

• physt.compat.root.from_root(TH1F...)
• physt.compat.root.to_root(HistogramBase)

When creating histogram from huge data, temporarily huge amount of memory is allocated, though no copy should be created.

Suspects:

• dropna ???
• weights

Support dtypes.

To do:

• Support for creation in Histogram1D
• Support for creation in HistogramND & calculate frequencies
• Loss-less arithmetics with constants
• Check fill, fill_n
• Addition of different types
• Support retyping (dtype=...)
• More thorough testing
• Document it properly

Giving twice the correct value (at least in the notebook)

• bin overlaps
• bin continuousness
• non-negativeness

?

Inspired by http://stackoverflow.com/questions/18091694/monte-carlo-simulation-with-python-building-a-histogram-on-the-fly?rq=1 - enable to have histogram with fixed-width bins that grows when underflow/overflow is encountered.

A sparse variant can also exist.

It's features look quite interesting, perhaps more than bokeh's...

Would be nice to be able to plot these in maps.

However, this, together with #31, establishes the necessity to keep axes semantics (for axes groups). Not trivial...

Based on #5, only more general. And has to be compatible with histogramdd

Perhaps base it on https://github.com/pydata/sparse

Binning as a class

• bins as property
• adaptation to new values (start with fixed width)

Makes it much easier to work with functions that calculate angles in this range like math.atan2

to behave like pandas Series/DataFrame

Requirements:

Possible axes:

1. month (cyclic)
2. year (continuous)
3. year+month (continuous)
4. day (continuous)
5. hour (cyclic)
6. weekday (cyclic)
7. week+weekday (continuous)
8. year + week (continuous)
   ...

Properties:

1. It must be possible to have a float quantity on other axes

Use case:

I have the following data:

2010-03-10, 4.3 deg
2010-03-11, -0.3 deg
...
2017-03-13, 3.4 deg
2017-03-14, 6.2 deg

I need a simple way to construct the histogram temperature vs. month of the year, ideally just calling h2...

Compatibility:

• must work with date, datetime objects
• must work with pandas internal and nat values

Use https://github.com/cupy/cupy as computing backend? Given the API similarity to numpy, it should be trivial.

Investigate if numba can lead to any speed up.

Only an idea yet, but it would be nice to be able to understand the distribution also in the console...

And while doing so, extract as much shared code as possible.

Should be easy

Actually, ~10 times slower than np.histogram. It has its reasons but (being more general) but in the simple case, we should not add any overhead.

Suggestion: when parameters allow, fall back to np variant.

It would probably be complicated as all calculation is based on numpy.

Interesting idea presented here:

http://stackoverflow.com/questions/38252879/how-to-plot-a-hanging-rootogram-in-python

• Polar (2D)
• Cylindrical (3D)
• Spherical (3D)
• Projection polar -> radial
• Projection polar -> azimuthal
• Projection cylindrical -> polar
• Projection cylindrical -> rhoz
• projection cylindrical -> phiz
• projection cylindrical -> radial
• Projection spherical -> directional
• projection spherical -> radial
  ...
• add clear interpretation and reasonable call arguments for the facade functions

In general, some mapping of values?

• Analogous data model to Histogram1D
• refactor HistogramBase class -> common behaviour of 1D and 2D
• revisit binning schemas
• histogram2D facade function to be compatible with numpy one
• plotting
• arithmetic operations
• documentation
• stats

See the special_histograms notebook, the spherical and cylindrical are broken.

Hi I found a project on rebinning histogram at https://github.com/jhykes/rebin and I opened an issue (jhykes/rebin#5) on that project page asking about integrating his code to this project.
I hope you will appreciate it.

by a constant, by a map, "intelligently"...

See e.g. http://stackoverflow.com/questions/38180617/pandas-merge-bins

This feature becomes available as of Python 3.7.

Asked at PyData Berlin. A must...

There is no (intuitive) way, that I can find, to add a colorbar to a polar histogram

Something similar to what altair does.

If this were to be supported, we would need to have support for two axes being semantically entangled.

Strange values of std for reasonable distribution in the presence of weights.

In this degenerate case, xlim for a bar matplotlib plot is set to (0, 1):

import matplotlib.pyplot as plt
from physt.histogram1d import Histogram1D
h = Histogram1D([0, 10], [1])
h.plot()
plt.show()

However, this works:

import matplotlib.pyplot as plt
plt.bar([0], [1], [10], align="edge")
plt.show()

When runnin plot() function I get the error below even though matplotlib is installed.
Also the algorithm is pretty slow when running on something bigger than toy example.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/physt/plotting/__init__.py", line 137, in __call__
    return plot(self.histogram, kind=kind, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/physt/plotting/__init__.py", line 91, in plot
    backend_name, backend = _get_backend(backend)
  File "/usr/local/lib/python2.7/dist-packages/physt/plotting/__init__.py", line 70, in _get_backend
    raise RuntimeError("No plotting backend available. Please, install matplotlib (preferred) or bokeh (limited).")
RuntimeError: No plotting backend available. Please, install matplotlib (preferred) or bokeh (limited).