arize-ai / phoenix Goto Github PK
View Code? Open in Web Editor NEWAI Observability & Evaluation
Home Page: https://docs.arize.com/phoenix
License: Other
AI Observability & Evaluation
Home Page: https://docs.arize.com/phoenix
License: Other
https://github.com/switowski/ipython-reverser as a simple example
Make the js bundles gitignored and find a way to make it part of the assets when pip install
is called. Right now it fails to be mounted in the python package if it is gitignored.
As a developer, I want to be able compose together a metric calculation and 1 or 2 datasets such that the metric is calculated for a specified interval over time at a granularity of my choosing(or auto).
metricAggregate = Calculate(df, metrics=[EuclideanDiscance]) // Retrieves the metric at an aggregate over the entire dataframe
metricOverTime = TimeSeries(dv, metrics=[EuclideanDistance], granularity="hourly") // Retrieves the metric over time at the granularity interval specified
psi = Calculate([primary_df, baseline_df], psi)
Returns
print(psi) # 2.73
psiOverTime = CalculateOverTime([primary_df, baseline_df], psi, granularity="1hour")
Returns
print(psiOverTime) # [{ timestamp: "12-12-2022", v: 1.27 }, { timestamp: "12-12-2022", v: 1.27 }]
vector columns are not parsed in CSV appropriately and are not type safe.
Acceptance Criteria
cast
and make the column retrieval methods off of Dataset
type safe.Mimic Tensorboard and start up a local port using magics or other https://ipython.readthedocs.io/en/stable/interactive/magics.html
Reference: https://www.tensorflow.org/tensorboard/tensorboard_in_notebooks
Not sure how to avoid
(notebook) ➜ phoenix git:(47-lasso-select) ✗ (⎈|dev:arize-dev)pip install .
Processing /Users/mikeldking/work/phoenix
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing metadata (pyproject.toml) ... done
Collecting pandas
Downloading pandas-1.5.2-cp310-cp310-macosx_11_0_arm64.whl (10.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.8/10.8 MB 18.7 MB/s eta 0:00:00
Collecting umap-learn
Using cached umap_learn-0.5.3-py3-none-any.whl
Collecting numpy
Using cached numpy-1.23.5-cp310-cp310-macosx_11_0_arm64.whl (13.4 MB)
Collecting hdbscan
Using cached hdbscan-0.8.29-cp310-cp310-macosx_12_0_arm64.whl
Collecting tables
Using cached tables-3.7.0.tar.gz (8.2 MB)
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [12 lines of output]
/var/folders/1s/4vdv59n15b1ghg42frdd8f480000gn/T/H5close646cmm8d.c:2:5: error: implicit declaration of function 'H5close' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
H5close();
^
1 error generated.
cpuinfo failed, assuming no CPU features: No module named 'cpuinfo'
* Using Python 3.10.3 (main, Apr 14 2022, 13:44:37) [Clang 13.1.6 (clang-1316.0.21.2.3)]
* Found cython 0.29.32
* USE_PKGCONFIG: True
.. ERROR:: Could not find a local HDF5 installation.
You may need to explicitly state where your local HDF5 headers and
library can be found by setting the ``HDF5_DIR`` environment
variable or by using the ``--hdf5`` command-line option.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Context here #57 (comment)
Usually we just want to know the distance between two embedding "clouds" so we can determine if there is embedding drift.
Document any install issues - pip install is failing for M1
Context here
implement
EuclideanDistance(primary: Dataset, baseline: Dataset))
Metrics
If we have a server running to host a UI and an API, we need a way to sync and commit state back to the server and vice versa.
Possibilities:
From formats:
The current (commented) github action that checks for docstring coverage needs to:
interrogate
locallyIdeally, we would want to read a table and get in every cell of the embedding vector column an array of floats. This is not the case when reading data from a csv file. The whole embedding is read as a string '[1.31,-0.46,...,-.108]'
instead of [1.31,-0.46,...,-.108]
. Other file formats, like hdf5, conserve better the data structure and don't have this problem.
This happens essentially with any field that expects an iterable inside a cell, i.e., it also happens with the list of token arrays.
Currently it only works for a default of 3D
bottoming out a universal binning strategy based on quantiles
Currently this is how it works: https://docs.google.com/document/d/1MUpJw_DEoHLDRcHO_kkFza5LN03zNEp1/edit
Try to perform a fs read from the browser.
Follow up to #57
Context #57 (comment)
Users should have some control over the number of points they want from each dataset.
In addition, separating the primary and reference with a fixed points per dataset like here will error
primary_dataset_points = construct_dataset_points(
projections[:points_per_dataset], sampled_primary_dataset, embedding_feature
)
reference_dataset_points = construct_dataset_points(
projections[points_per_dataset:], sampled_reference_dataset, embedding_feature
)
Context here
The current implementation would fail if the user reads their dataframe from csv and passes the dataframe directly to our Dataset
object.
Implement parquet parsing -> Dataset object
Context here #57 (comment)
Implement PSI
Add a arize_toolbox.dataset
module that keeps track of inference records in a performant and normalized format. Must support features, actuals, embeddings.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.