jonasrauber / eagerpy Goto Github PK
View Code? Open in Web Editor NEWPyTorch, TensorFlow, JAX and NumPy — all of them natively using the same code
Home Page: https://eagerpy.jonasrauber.de
License: MIT License
PyTorch, TensorFlow, JAX and NumPy — all of them natively using the same code
Home Page: https://eagerpy.jonasrauber.de
License: MIT License
When one of the two conditions is a float64 dtype and the second a number (int, float), where()
method raises an error:
RuntimeError: expected scalar type float but found double
will this lib support tensorflow tensor work with cuda.jit (from numba).
I'm trying to differentiate through predict_fn
provided by https://github.com/google/neural-tangents. This is doable with jax.grad
, but not with eagerpy.value_and_grad
.
In order to simplify the writting of universal functions it could be great to have a decorator function which hide the technical part of the code (convertion of input and output of the wrapped function/method).
For example, the code:
def my_universal_function(a, b, c):
# Convert all inputs to EagerPy tensors
a, b, c = ep.astensors(a, b, c)
# performs some computations
result = (a + b * c).square()
# and return a native tensor
return result.raw
would become:
@eager_function
def my_universal_function(a, b, c):
return (a + b * c).square()
In addition, we could add the feature that if the input tensors are already eagerpy tensors, then no convertion to raw format should done on the output tensors.
I wrote a prototype of such a decorator function. It should not work on any type of arguments and so its usage would require that the wrapped function has a rather "simple" signature (with args and kwargs constituted of tensors or nested containers with tensors on leaves: dict, list, tuple or namedtuple like containers).
Would you consider to have this feature in eagerpy?
i'm trying to modify foolbox like following,
from typing import TypeVar, Any
from abc import ABC, abstractmethod
import torch
import eagerpy as ep
from foolbox.criteria import Criterion
class TargetedMisclassificationML(Criterion):
"""Considers those perturbed inputs adversarial whose predicted class
matches the target classes. Multi-Label
Args:
target_classes: Tensor with target classes ``(batch,)``.
"""
def __init__(self, target_classes: Any):
super().__init__()
self.target_classes: ep.Tensor = ep.astensor(target_classes)
def __repr__(self) -> str:
return f"{self.__class__.__name__}({self.target_classes!r})"
def __call__(self, perturbed: T, outputs: T) -> T:
outputs_, restore_type = ep.astensor_(outputs)
del perturbed, outputs
print(type(outputs_), type(outputs))
# classes = outputs_.argmax(axis=-1)
classes = torch.tensor(torch.round(torch.sigmoid(outputs_)).detach().numpy().tolist()) # ERR
assert classes.shape == self.target_classes.shape
is_adv = classes == self.target_classes
return restore_type(is_adv)
will u support sigmoid function in the future?
and how to implement sigmoid with eagerpy now?
I noticed that the method index_update
seems very slow for tensorflow backend when used in compiled code (a function wrapped with tf.function
).
It could be worth to see how we could enhance the current implementation.
Hello,
I notice that pytorch and tensorflow both have a topk
function, which can be faster than a full sort if k is small. With numpy, it can be emulated using (arg)partition. I think it would be convenient if eagerpy provided a wrapper for this, I have 2 unrelated pieces of code where I would be likely to use it.
(of course it is disappointing that tensorflow does not seem to support the argument dim
, but doing some transposition around the call is easy enough)
First, congrats on such a great project!
Basically the title. PyTorch and Tensorflow both support cross entropy in the ND case, and your implementation in Numpy would work for the multi-dimensional case too. However, in every function, there is an assert that the logits are 2D array. I propose to remove those asserts 😄
I'm considering packaging EagerPy for Debian. I think that effort may encounter some hurdles due to missing copyright notices and unclear licensing. As far as I can tell (please forgive me if I've overlooked something), the license is only specified in setup.py
, and even there just given as "MIT License". It would be of great help if you could add a LICENSE
fle where you clearly state your copyright over the code and the full licensing terms (for example the full terms of the MIT license).
It would be even more helpful if a copyright notice were added to each source file, but I understand if this is too much of a hassle.
I think it would be good to have an eagerpy version of the nonzero()
and flatnonzero()
functions. I guess these can be imitated with a combination of arange
and indexing, but it would be more convenient to have a simple drop-in replacement for these two common functions.
Hi! Thanks for the awesome project -- a unified tensor interface is something that will help a lot of projects :)
I wanted to ask about the current/future scope of eagerpy
, specifically on the inclusion of probability distributions like in torch.distributions
, or tfp.distributions
? It seems like a more substantial project, and so may be more of a stand-alone effort, but I think this would be a great asset for people who want to keep fully agnostic to frameworks.
As an example, a project I work with (pyhf
) has it's own implementation of Tensor
that aims for essentially what eagerpy
does, but we also use it for probability distributions too within the module. This is done by wrapping around existing implementations and adding the extra math where needed -- my first impression with eagerpy
is that this may be better handled if there were stand-alone implementations of distributions using only eagerpy
tensors, but I'm not sure what the more practical option is.
Eager (hehe) to hear your thoughts!
eagerpy seems like a great project, and after an upcoming deadline we might try porting out pytorch code to it.
Reading here about converting native pytorch and tf GPU tensors to eager tensors, could you please implement a ep.totensor method, creating a pytorch or tf GPU tensor?
We currently have to use a mix of pytorch and tensorflow models to evaluate our code :(
And yes, I can get them all running on the same GPU. I just want to avoid shuttling pytorch tensors to CPU so I can evaluate them in TF models.
Is there an equivalent of np.diag
in eagerpy
? If not, what would be the proper way to create a diagonal matrix from a vector?
Let's consider a simple compiled function in tensorflow.
import tensorflow as tf
a = tf.random.normal(shape=(2, 10))
b = tf.random.normal(shape=(10, 3))
@tf.function
def tf_compiled_func(a, b):
c = tf.matmul(a, b)
return c
tf_compiled_func(a, b)
This bunch of code works.
However, its "universal" version :
@tf.function
def compiled_universal_func(a, b):
a, b = ep.astensors(a, b)
c = a.matmul(b)
return c.raw
a = tf.random.normal(shape=(2, 10))
b = tf.random.normal(shape=(10, 3))
compiled_universal_func(a, b)
does not work and raises the error:
.../lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
975 except Exception as e: # pylint:disable=broad-except
976 if hasattr(e, "ag_error_metadata"):
--> 977 raise e.ag_error_metadata.to_exception(e)
978 else:
979 raise
AttributeError: in user code:
<ipython-input-8-6edbe80953ee>:4 compiled_universal_func *
c = a.matmul(b)
.../lib/python3.7/site-packages/eagerpy/tensor/tensorflow.py:499 matmul *
if self.ndim != 2 or other.ndim != 2:
.../lib/python3.7/site-packages/eagerpy/tensor/base.py:115 ndim
return cast(int, self.raw.ndim)
AttributeError: 'Tensor' object has no attribute 'ndim'
(but it works if we comment the @tf.function
)
Let's notice that the equivalent thing with jax seems to work:
import jax
from jax import jit
@jit
def compiled_universal_func(a, b):
a, b = ep.astensors(a, b)
c = a.matmul(b)
return c.raw
seed = 1701
key = jax.random.PRNGKey(seed)
a = jax.random.normal(shape=(2, 10), key=key)
b = jax.random.normal(shape=(10, 3), key=key)
compiled_universal_func(a, b)
Is it a problem with the integration of eagerpy with tensorflow ?
Would you consider to add or accept a contribution to add and implementation of slogdet
function (see for ex the numpy implementation)?
Converting between things like float32/64/int32/64 would be a really nice addition!
TensorFlow's tensor_scatter_nd_update
expects the values to be of the same type as the tensor. Currently, this leads to TensorFlowTensor.index_update
failing since it uses tf.fill
, which produces int32/ float32 tensors, independent of the tensor data type itself.
Reproducible example:
import eagerpy as ep
import tensorflow as tf
x_int32 = ep.astensor(tf.range(4, dtype=tf.int32))
x_int64 = ep.astensor(tf.range(4, dtype=tf.int64))
indices = (ep.astensor(tf.constant([0, 2])),)
x_int32.index_update(indices, 0) # this works
x_int32.index_update(indices, 0.0) # this fails
x_int64.index_update(indices, 0) # this fails
x_int64.index_update(indices, 0.0) # this fails
Similar for float32/ float64 tensors.
Solution:
Cast values to the dtype of the raw tensor.
It could be great to have support for the @ operator.
Given two tensors:
a = ep.astensor(np.random.normal(size=(2, 10)))
b = ep.astensor(np.random.normal(size=(10, 3)))
The expression:
c = a.matmul(b)
could be written as:
c = a @ b
I'm getting this error in /eagerpy/astensor.py", line 56, in astensor:
raise ValueError(f"Unknown type: {type(x)}")
ValueError: Unknown type: <class 'tuple'>
How to resolve the issue.
Feature Request
Would is be possible to support python scalars?
eg.
num_float = ep.astensor(1.0)
num_int = ep.astensor(42)
Why?
This would be helpful for implementing generic functions, for example:
def get_kernel_size(sigma=1.0, trunc=4.0):
sigma, trunc = ep.astensors(sigma, trunc)
radius = (sigma * trunc + 0.5).astype(int)
return (2 * radius + 1).raw
This could be called with default values as well as tensors.
t_sigma = torch.abs(torch.randn(10))
n_trunc = np.abs(np.random.randn(10))
get_kernel_size(sigma=1.0, trunc=4.0)
get_kernel_size(sigma=t_sigma, trunc=4.0)
get_kernel_size(sigma=1.0, trunc=n_trunc)
Hello,
with a pytorch tensor t, I can call t.norm(p, dim)
. This gives a similar result to eagerpy's lp
, but makes a huge difference when it comes to the gradient. Pytorch has this feature where deriving sqrt in 0 gives infinity (mathematically sensible), which often yields a gradient of NaN. However, special functions like norm
are handled specially, similarly to abs
, and return a suitable subgradient (0).
Could you please make l2/lp/... call norm
for the pytorch backend?
Hi, jonasrauber
Thanks for your works!
I wonder if it will add support for SparseTensor (Tensorflow or Pytorch), like this:
import numpy as np
import tensorflow as tf
import eagerpy as ep
x = np.array([[0, 1, 0],
[0, 0, 1],
[1, 0, 1]])
# Tensorflow SparseTensor
sp_x = tf.sparse.from_dense(x)
# eagerpy
# is it implemented?
sp_x = ep.assptensor(x)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.