suned / pfun Goto Github PK
View Code? Open in Web Editor NEWFunctional, composable, asynchronous, type-safe Python.
Home Page: https://pfun.dev/
License: MIT License
Functional, composable, asynchronous, type-safe Python.
Home Page: https://pfun.dev/
License: MIT License
The current implementation of the type analyzer for Curry
types and curry
calls only adds the uncurried function type as the type of __call__
. This means that the following call produces a false negative:
f: Curry[Callable[[int, int], int]]
g: Callable[[Callable[[int], Callable[[int], int]]], int]
g(f)
As a first solution I suggest to add an overloaded signature to Curry.__call__
that is curried in the required arguments only
It might be useful for users of pfun
if they could reuse the testing strategies implemented for the pfun
test suite. Suggested solution is to add an optional test
group that would install hypothesis
with pip install pfun[test]
, and then provide the hypothesis strategies in pfun.hypothesis_strategies
E.g
def f(a: int) -> str:
pass
def g(a: float) -> int:
pass
def h(a: object) -> float:
pass
compose(f, g, *[h]) # Error: too many arguments for compose
First off, cool library and nice docs and examples ๐
However, I am having problems making it work with mypy. It might be something with my setup or my lack of knowledge about mypy, -in that case the outcome might be some more value to add to the readme for other potential users :).
I am simply trying to run mypy on a file mytest.py
which contains following lines
from pfun.io import get_line, put_line, IO
def hello(foo: str) ->IO[None]:
return put_line('hello ' + foo)
hello('sir').run()
I get following error output when I try to run mypy on it
$ mypy mytest.py
mytest.py:1: error: Cannot find module named 'pfun.io'
mytest.py:1: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 1 source file)
I have a mypy.ini
file in the same directory as mytest.py
, and it contains:
[mypy]
plugins = pfun.mypy_plugin
Besides that, I am running NixOs and I am using a python virtualenv which contains following packages:
$ pip freeze
appdirs==1.4.3
astor==0.8.0
beautifulsoup4==4.8.1
bs4==0.0.1
certifi==2019.9.11
chardet==3.0.4
colorama==0.4.1
funcparserlib==0.3.6
hy==0.17.0+100.gf8d3826
idna==2.8
mypy==0.740
mypy-extensions==0.4.3
pfun==0.5.1
pythonping==1.0.4
requests==2.22.0
rply==0.7.7
soupsieve==1.9.4
toolz==0.10.0
typed-ast==1.4.0
typing-extensions==3.7.4.1
urllib3==1.25.6
youtube-dl==2019.7.16
and python version
$ python --version
Python 3.7.3
I have tried various things after reading a bit in the mypy docs, but nothing really helped.
At first I suspected that mypy wasn't picking up on the mypy.ini
file, but then I tried adding ignore_missing_imports = True
and then it reported success for mytest.py
(because it was happily ignoring pfun). I have also verified that there is indeed a file mypy_plugin.py
inside pfun
in the site-packages
of my virtualenv.
Is there something obvious that I am missing?.. Or could the problem be something with my setup? (for example some environment variable pointing to something that doesn't exist in NixOs...)
This could also be used in an eventual effect rotation "god monad" ala ZIO
from pfun.reader import Reader, ask, with_effect
from typing import TypeVar, Generator, Any
A = TypeVar('A')
R = TypeVar('R')
Readers = Generator[Reader[R, Any], Any, A]
@with_effect
def f() -> Readers[str, int]:
env = yield ask()
reveal_type(env) # should be str
With all likelihood caused by python/mypy#7735
There are currently multiple ways to produce RecursionError
when working with IO
.
Some IO classes hold a "continuation" function to pass the result to, once the IO action is run (such as ReadFile
). When composing a large number of ReadFiles
with and_then
calling run
will unfold this large stack of recursive calls to the lambda created in and_then
and cause a RecursionError
:
from pfun.io import read_file, sequence
action = sequence([read_file('test.txt') for _ in range(5000)])
action.run() # RecursionError
Other instances hold references to other IO
actions (such as Put
). Composing a large number of these instances will cause a RecursionError
while building the structure because it calls and_then
recursively.
from pfun.io import put_line, sequence
sequence([put_line('test') for _ in range(5000)]) # RecursionError
Ideally both types of calls should never cause a RecursionError
. This should be achievable by trampolining appropriately (see pfun.trampoline
and e.g pfun.reader
for a usage example). Ideally this should be solved without changing the api.
See https://github.com/suned/pfun/actions/runs/715062592
Maybe the solution is simply to fetch the gh-pages
branch first?
Use mypy plugin to enable static checking of case classes (see pfun/case.py
for a start).
Things that can be checked:
t
must correspond to match type, must be case subclasses, must all have the same base case)when
and then
are of the correct typeOther things that plugin must (should) do:
CaseClass
calls to dataclass to enable __init__
generation (maybe possible to use mypy dataclass
plugin)Things to check for:
from typing import TypeVar, Generic
from pfun import Immutable
A = TypeVar('A')
class C(Immutable, Generic[A]):
a: str
b: int
c: A
c = C('', 1, '') # inferred type is C[str]
c.clone(d='') # type error: Class "C" does not have attribute "d"
c.clone(a=1) # type error: Expected "str" for argument "a" got "int"
c.clone(c=1) # inferred type should be: C[int]
The lens
mypy plugin hook currently doesn't work with generic types, eg in:
from typing import List
[0] | lens(List)[0] << 1 # error: Unsupported operand types for << ("Lens[List[_T], _T]" and "int")
One possible solution approach might be to add the type variable definitions and types of the generic type to the signature of __call__
, __ror__
, and __lshift__
of the Lens
instance.
Run my code without problem since I'm not using proccess
Fail with message:
[Errno 38] Function not implemented: OSError
Add option to disable ProccessPool or ThreadPool when isn't required
For anyone with the same problem
from asyncio import get_event_loop
from concurrent.futures import ThreadPoolExecutor
from contextlib import AsyncExitStack
from pfun.effect import RuntimeEnv, CSuccess
async def call(self, r, max_threads=None):
stack = AsyncExitStack()
thread_executor = ThreadPoolExecutor(max_workers=max_threads)
async with stack:
stack.enter_context(thread_executor)
env = RuntimeEnv(r, stack, None, thread_executor)
effect = await self.do(env)
if isinstance(effect, CSuccess):
return effect.result
if isinstance(effect.reason, Exception):
raise effect.reason
raise RuntimeError(effect.reason)
def run(effect, r):
result = call(effect, r)
return get_event_loop().run_until_complete(result)
Then change
main(event).run(env)
by run(main(event), env)
Isn't a problem of pfun that it fails, but Proccess Pool isn't required either
Minimal example:
from typing import TypeVar, Any
from pfun.effect import Try, combine, catch
A = TypeVar("A")
def test(*effects: Try[Exception,A]) -> Try[Exception,None]:
return combine(*effects)(lambda *a: None)
Blows up with a mypy internal error:
It doesn't matter if I move that lambda to a function.
(Probably a mypy issue, as I tried with the latest dev version of mypy (0.920dev) and didn't get the error. But note I had to disable the pfun mypy_plugin on the dev version, got a different error.)
Currently the Curry
type expects an argument of Callable
, which means that you have to write
Curry[Callable[[int], int]]
It would be nicer to simply write
Curry[[int, int], int]
Its unclear whether there is any meaning in using an argument list or whether to just use Curry[int, int, int]
. One argument is to make it congruent with Callable
Also it might make sense to be able to write Curry[..., int]
for cases where you can't give the exact arguments but you know the return type. Calling a value of that type would have the return type Curry[..., int] | int
.
From the doc string:
List(range(3)).append(3)
this generates a TypeError because append only takes an Iterable. Am I missing something?
The experience of pfun
using Pycharm's own type-checker is currently somewhat subpar to using mypy (because of the pfun
mypy plugin). I suspect that it may be possible to provide an equally good experience by modifying the type checking of code that uses pfun
by hooking into the pycharm inspections. (I haven't looked closely into it though).
Is it a Python environment (like cpython)? Is it a linter (like mypy / pylint / flake8)? Is it a learning project?
Who should use it? In which context is it useful? Why was it created? Are there similar projects?
from pfun import Immutable
class C(Immutable):
pass
class B(Immutable):
c: C
error:
Traceback (most recent call last):
File "mypy/semanal.py", line 4626, in accept
File "mypy/nodes.py", line 927, in accept
File "mypy/semanal.py", line 1022, in visit_class_def
File "mypy/semanal.py", line 1093, in analyze_class
File "mypy/semanal.py", line 1102, in analyze_class_body_common
File "mypy/semanal.py", line 1162, in apply_class_plugin_hooks
File "/Users/sune/pfun/pfun/mypy_plugin.py", line 207, in _immutable_hook
transformer.transform()
File "mypy/plugins/dataclasses.py", line 90, in transform
File "mypy/semanal.py", line 4360, in defer
AssertionError: Must not defer during final iteration
Something inspired by this http://www.valuedlessons.com/2008/01/monads-in-python-with-nice-syntax.html
The main issues involve typing it correctly. Consider:
from typing import Generator, Any, TypeVar
from pfun.maybe import Maybe
from pfun.monad import Monad
A = TypeVar('A')
M = TypeVar('M', bound=Monad)
def do(f):
...
Do = Generator[M, Any, A] # This type could of course be part of pfun
@do
def f() -> Do[Maybe, int]:
a_str = yield produce_a_str() # we somehow have to tell mypy that even though
# the return type of 'produce_a_str' is Maybe[str], 'a_str' is actually 'str'
an_int = yield produce_an_int(a_str)
return an_int
E.g
from typing import TypeVar
from pfun import compose
A = TypeVar('A')
def f(a: A) -> A:
pass
def g(a: int) -> str:
pass
compose(f, g) # error: Cannot infer argument 1 of compose
I think this is a current limitation of the mypy inference, since this also does not work:
R1 = TypeVar('R1')
R2 = TypeVar('R2')
R3 = TypeVar('R3')
def compose(f: Callable[[R2], R1], g: Callable[[R3], R2]) -> Callable[[R3], R1]:
pass
A = TypeVar('A')
def f(a: A) -> A:
pass
def g(a: int) -> str:
pass
compose(f, g) # error: Cannot infer type of argument 1 of compose
Minimal example:
from pfun.maybe import Maybe, Just, Nothing
def test_and_then() -> Maybe[int]:
nope: Maybe[int] = Nothing()
return nope.and_then(lambda x: Just(x + 1))
def test_map() -> Maybe[int]:
nope: Maybe[int] = Nothing()
return nope.map(lambda x: x + 1)
The first one results in:
Cannot infer type argument 1 of "and_then" of "Nothing"
The second:
Returning Any from function declared to return int
The second one seems to be because it is picking up the signature of the ABC: Maybe_.map.
I am using the pfun.mypy_plugin
.
Thanks for this very promising library!
from typing import Any
from pfun.effect import Effect, success, failure
e: Effect[Any, ZeroDivisionError, int]
reveal_type(e.recover(lambda e: success(0) if 1 > 0 else failure('')))
Traceback (most recent call last):
File "mypy/checkexpr.py", line 3749, in accept
File "mypy/nodes.py", line 1544, in accept
File "mypy/checkexpr.py", line 263, in visit_call_expr
File "mypy/checkexpr.py", line 340, in visit_call_expr_inner
File "mypy/checkexpr.py", line 817, in check_call_expr_with_callee_type
File "mypy/checkexpr.py", line 876, in check_call
File "mypy/checkexpr.py", line 984, in check_callable_call
File "mypy/checkexpr.py", line 725, in apply_function_plugin
File "/Users/sune/pfun/pfun/mypy_plugin.py", line 362, in _effect_recover_hook
r2 = e2.args[0]
IndexError: list index out of range
Currently, curry instances returned from decorating methods with curry
doesn't eliminate the self
argument, which issues a type error when trying to call a curried method:
from pfun import curry
class C:
@curry
def f(x: int, y: int) -> int: ...
reveal_type(C().f) # note: Revealed type is 'pfun.functions.Curry[def (self: test_types.C, x: builtins.int, y: builtins.int) -> builtins.int]'
C().f(1) # error: Argument 1 to "__call__" of "Curry" has incompatible type "int"; expected "C"
The main issue in fixing this is finding a way to distinguish this call to curry with a call with an instance that has a __call__
method in e.g:
class C:
def __call__(self, x: int, y: int) -> int: ...
curry(C())
i'm sure it's a typo, should be 'rb'
from pfun import curry2
def f(a: str, b: int) -> str: pass
curry2(f) # inferred type: (any) -> (any) -> str
Creating a long chain of and_then
calls to the same monad can in theory overflow the stack. The common solution to this problem is trampolines.
I'm not sure how big a of a problem this will be in practice, since I think combining imperative style with functional style can in many cases alleviate the problem (see e.g the implementation of util.sequence_
.)
Edit:
It will be a big problem.
E.g
from pfun.maybe import with_effect, Maybe
from typing import Generator, Any, TypeVar
A = TypeVar('A')
Maybes = Generator[Maybe[Any], Any, A]
def g() -> Maybe[int]:
...
@with_effect
def f() -> Maybes[int]:
a = yield g()
reveal_type(a) # should be int
Trampoline
is used extensively to build stack safe, recursive structures, e.g in reader
and state
. When interpreting these structures, a significant amount of time is spent in the main loop of Trampoline.run
. Consider this:
In [13]: from pfun.reader import value, sequence
In [14]: r = sequence([value(v) for v in range(100000)])
In [15]: %prun r.run(None)
9799985 function calls in 18.876 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
100000 7.918 0.000 8.201 0.000 monad.py:31(<lambda>)
1 5.780 5.780 19.344 19.344 trampoline.py:51(run)
1599995 1.396 0.000 1.616 0.000 typing.py:806(__new__)
599996 0.557 0.000 1.350 0.000 trampoline.py:32(and_then)
200000 0.406 0.000 1.588 0.000 reader.py:45(<lambda>)
799994 0.402 0.000 0.402 0.000 <string>:1(__init__)
399996 0.261 0.000 11.689 0.000 trampoline.py:85(_handle_cont)
600001 0.226 0.000 0.472 0.000 {built-in method builtins.isinstance}
599996 0.221 0.000 12.953 0.000 trampoline.py:105(_resume)
1599995 0.220 0.000 0.220 0.000 {built-in method __new__ of type object at 0x107597830}
199998 0.184 0.000 0.462 0.000 trampoline.py:108(and_then)
200000 0.180 0.000 0.459 0.000 reader.py:46(<lambda>)
200000 0.151 0.000 0.588 0.000 reader.py:44(<lambda>)
200001 0.143 0.000 0.376 0.000 reader.py:95(<lambda>)
600001 0.139 0.000 0.611 0.000 trampoline.py:28(_is_done)
199998 0.134 0.000 0.975 0.000 trampoline.py:113(<lambda>)
600000 0.128 0.000 0.128 0.000 {built-in method _abc._abc_instancecheck}
600000 0.119 0.000 0.246 0.000 abc.py:137(__instancecheck__)
100000 0.108 0.000 0.283 0.000 reader.py:84(value)
100000 0.079 0.000 0.204 0.000 reader.py:26(and_then)
200000 0.068 0.000 1.043 0.000 trampoline.py:74(_handle_cont)
100000 0.058 0.000 0.262 0.000 monad.py:30(<lambda>)
1 0.000 0.000 19.344 19.344 <string>:1(<module>)
3 0.000 0.000 0.000 0.000 future.py:47(__del__)
1 0.000 0.000 19.344 19.344 {built-in method builtins.exec}
1 0.000 0.000 19.344 19.344 reader.py:67(run)
1 0.000 0.000 0.000 0.000 typing.py:245(inner)
1 0.000 0.000 0.000 0.000 typing.py:890(cast)
4 0.000 0.000 0.000 0.000 trampoline.py:89(_resume)
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
Most of the execution time is spent in Trampoline.run
. Optimising this loop is therefore a reasonable investment since it will speed up every usage of trampolines.
A mappable
type is something that implemets __map__(self, f)
.
A chainable
type is something that implements __chain__(self, f)
.
A type T
that implements both can be used in do
expressions as
fullname = do user_id('sad') as uid, first_name(uid) as first, last_name(uid) as last in f'{first} {last}'
Which will translate to
fullname = user_id('sad').__chain__(lambda uid: first_name(uid).__chain__(lambda first: last_name(uid).__map__(lambda last: f'{first} {last}')))
The test suite takes so long to run that I probably wouldn't run it very frequently. I'm not sure if this is because of tox or if it's the tests themselves, I haven't looked into it in detail but it makes writing tests quite tedious.
E.g this should be ok:
from pfun.maybe import Just
def f():
return Just(1)
Just(None).and_then(f)
I think the only way to provide correct typing for this is to handle it using the mypy plugin.
At the moment, the mypy plugin supports only one call signature for curried function
from pfun import curry
@curry
def f(a: int, b: int, c: int) -> int:
pass
reveal_type(f) # int -> int -> int -> int
This means that calling f
as e.g f(1, 1)
does not type correctly, although this works fine at runtime.
The mypy plugin should be extended to support all eligible function signatures for curry
The documentation still has a few phrases that are left from a previous version of the Effect
api. While not strictly incorrect, they can be made more clear.
Moreover, the pytkdocs
wheel used to collect api documentation doesn't collect accurate api documentation in all cases for the cython code. A cython mkdocstrings
handler should be implemented that collects accurate information.
from typing import Callable, TypeVar, Union
from pfun import curry
A = TypeVar('A')
B = TypeVar('B')
@curry
def f(v: B, g: Callable[[], A]) -> Union[B, A]:
pass
f(1)(lambda: '') # Argument 1 has incompatible type "Callable[[], str]"
Probably something I'm doing wrong (ie not informing types arguments of Either), but...
mypy MethodContext.default_return_type.args
may return a tuple
instead of a list
.
Raising ''TypeError: 'tuple' object does not support item assignment" for code above
return_type = context.default_return_type
return_type_args: return_type.args
...
return_type_args[0]: _combine_environments(r1, r2)
We could expect that if isn't a list dev made something wrong - like I did - but this and similar assignments may cause unintended side effects ๐ค
This is caused by python/mypy#5485
Similar to https://github.com/vlasovskikh/funcparserlib (or other monadic parsing libraries) but with a focus on static type checking.
from pfun.reader import Reader
from typing import TypeVar
A = TypeVar('A')
Action = Reader[str, A]
reveal_type(Action(lambda _: True)) # Revealed type is 'pfun.reader.Reader[builtins.str*, Any]'
This is probably due to a bug in mypy. At least the following also fails:
from typing import TypeVar, Generic, Callable
A = TypeVar('A')
B = TypeVar('B')
class C(Generic[A, B]):
def __init__(self, f: Callable[[A], B]):
pass
C = TypeVar('C')
Alias = C[str, C]
reveal_type(Alias(lambda _: True)) # Revealed type is 'C[builtins.str*, Any]'
I'm totally new of functional programming, I'm trying to use a reader monad to get rid of passing a lot of function parameter in my code, but I don't understand how to use it properly. Here a very simple use case.
def process(item, c):
# how to get Connection?
print(item, c)
def process_all() -> Reader[Connection, str]:
def _(c):
for item in range(5):
process(item, c)
return value("result")
return ask().and_then(_)
def main():
connection = Connection(URI='testuri')
result = process_all().run(connection)
print(result)
how can I get Connection
in the process
function without explicitly passing it? If I must pass it then I don't understand the advantage to directly pass Connection
through functions without using a reader monad. What I'm missing?
Thanks.
In this example I just want to transform Right
to Data
and Left
to Error
@dataclass
class Error:
msg: str
@dataclass
class Data:
data: str
def i_can_fail2(s: str) -> Either[str, str]:
# do something
return Right(f"{s}!")
def i_can_fail1(s: str) -> Either[str, str]:
if s == 'illegal value':
return Left("illegal value")
return Right('Ok')
def func1() -> Either[Error, Data]:
# what about a map but only for Left value?
return i_can_fail1('illegal value').and_then(i_can_fail2).map(lambda x: Data(data=x))
I can do it with few lines of code, but I want to known if there is a proper way to do this.
Thanks
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.