Comments (6)
Thanks a lot @AndreasBackx for the report, that was really helpful :)
The release info for 0.5.0 should, if possible, be updated highlighting this backwards incompatible change in 0.5.0-0.5.2.
Done, let me know if the workaround mentioned here works for you: release v0.5.0.
You might want to pull those versions from pypi, though that decision is up to you.
Hmm maybe, I'll look into that once the problem is fixed.
The ideal solution is that we could change the signature back to: [...]
I agree, here's an attempt at fixing it: #96
The problem with this PR is that it breaks type inference for the zip
and merge
operators. I suspect this due to mypy not handling too well the mix of generic callback protocols with variadic arguments. Another explanation would be that I just got it wrong.
At that point I'm also considering the solution of not using meta-programming (i.e the operator
decorator) for those 3 operators and simply unroll the logic manually by writing the dedicated operator classes.
Let me know if you have any opinion on that matter.
from aiostream.
Thank you for getting back!
Done, let me know if the workaround mentioned here works for you: release v0.5.0.
Thanks, that's good for now. The pulling versions and possibly issuing a 1.x.x tag can be reviewed once there is a fix.
The problem with this PR is that it breaks type inference for the zip and merge operators. I suspect this due to mypy not handling too well the mix of generic callback protocols with variadic arguments. Another explanation would be that I just got it wrong.
You seem to run mypy 1.6.1 that is shipping with Microsoft's VS Code extension (as that's the only way I get the same single error as you). If you update to 1.9 or the recently released 1.10, you'll see many more errors. 😅 Most seem to be related to the Never
typing change. It might be worth looking into that generally as my assumption is that some were there before the PR.
At that point I'm also considering the solution of not using meta-programming (i.e the operator decorator) for those 3 operators and simply unroll the logic manually by writing the dedicated operator classes.
The only thing I can say on the matter now, is that I find the operator code very fragile-looking and hard to read. There are a lot of things being done manually in there that imo should be avoided. Though, I'm not here to criticise how you decided to implement aiostream.
from aiostream.
You seem to run mypy 1.6.1 that is shipping with Microsoft's VS Code extension (as that's the only way I get the same single error as you). If you update to 1.9 or the recently released 1.10, you'll see many more errors. 😅 Most seem to be related to the Never typing change. It might be worth looking into that generally as my assumption is that some were there before the PR.
Oh right, the pre-commit configuration was running mypy 1.6.1.
I opened an issue about the mypy 1.7 incompatibility here: #105
Sadly there's not much I can do for the moment, but the good news is that we're still compatible with the latest pyright (which is used by pylance). By the way, pyright is now checked in the pre-commit configuration: #99
The only thing I can say on the matter now, is that I find the operator code very fragile-looking and hard to read. There are a lot of things being done manually in there that imo should be avoided.
You're right, I tried improve the readability of the operator code in #106. I think it's much better now, although we still have to rely on meta-programming to remove boiler plate code from the operator declaration (without it, you would need a class declaration with about 10 to 20 lines of code for each operator).
Also I came up with a better way to implement the sources_operator
: #108
It does not have the issue I mentioned earlier about breaking type inference. The only downside is that it adds a Any
to the pipe method signature:
Lines 289 to 292 in ba773f9
This Any
should be P.args
but there's no proper way to type this correctly at the moment.
Anyway I'm pretty happy with it now, so feel free to leave a review if you want to. Otherwise I'll go ahead and merge it, and then release aiostream 0.6.0 :)
from aiostream.
I tried to spend some time reviewing your code and trying to figure out whether we can get rid of the inspect
uses in many locations. Also figuring out whether relying on typing without the need to do so much code generation is possible. I however realise that this is going to cost a few hours to actually understand, which I unfortunately do not have. I would try and see if this is possible, you might already know whether or not it is.
There might be some startup time performance to be gained due to no longer needing to do a large amount of meta programming, and doing this for each and every decorator invocation. Instead reusing a few functions?
Have you ever done any measurements on the import time of aiostream
? I know this might not be something that'll be much of an issue for you, though we've seen import time when Python projects in monorepos get out of control.
Regardless, if the user facing side is well-typed, I think that's the priority alongside with hopefully making 0.6.x backwards-compatible with <0.5.0. I am no big aiostream
user so cannot really tell from seeing the PRs.
I'd love to help you out with some typing, though currently I'd need to spend too much time trying to understand the meta programming and I unfortunately don't have that time outside of work. Please let me know if there are any other questions you might have though.
from aiostream.
I tried to spend some time reviewing your code and trying to figure out whether we can get rid of the inspect uses in many locations.
The use for inspect
in this code somewhat optional. However, it has two important uses:
- Allow for inspection tools to report the right signature for the exposed methods (instead of a generic
*args/**kwargs
signature). This is useful for IPython introspection (e.g. try runstream.take?
in an IPython console) and documentation auto-generation (using sphinxautofunction
). - Detecting invalid patterns at declaration time (e.g decorating a method, or having the first argument being or not being variadic depending on the operator kind). IMO detecting those errors as soon as possible really helps with debugging.
Also figuring out whether relying on typing without the need to do so much code generation is possible.
Some tool will actually rely on typing rather than introspection, as it is the case for pylance. For instance, when opening a parenthesis as in stream.take(...
, pylance will use the type of stream.take
, and solve for the param spec P
to properly display the correct prototype with the docstring of the function that solves P
.
My point is that some tools use introspection and some use typing, so it's good to properly support both.
I however realise that this is going to cost a few hours to actually understand, which I unfortunately do not have.
Sure, I understand that there's a limit to the time you're willing to put into this, no problem at all :)
There might be some startup time performance to be gained due to no longer needing to do a large amount of meta programming, and doing this for each and every decorator invocation. Instead reusing a few functions?
In my opinion, the amount of meta-programming is not too important here. Each operator function is merely transformed in a callable singleton with a couple of extra methods for convenience. For instance, here's what pipable_operator
looks like once you get rid of all the instrospection and extra checking features:
def pipable_operator(
func: Callable[Concatenate[AsyncIterable[X], P], AsyncIterator[T]],
) -> PipableOperator[X, P, T]:
class PipableOperatorImplementation:
original = staticmethod(func)
@staticmethod
def raw(
arg: AsyncIterable[X], /, *args: P.args, **kwargs: P.kwargs
) -> AsyncIterator[T]:
assert_async_iterable(arg)
return func(arg, *args, **kwargs)
def __call__(
self, arg: AsyncIterable[X], /, *args: P.args, **kwargs: P.kwargs
) -> Stream[T]:
factory = functools.partial(self.raw, arg, *args, **kwargs)
return Stream(factory)
@staticmethod
def pipe(
*args: P.args,
**kwargs: P.kwargs,
) -> Callable[[AsyncIterable[X]], Stream[T]]:
return lambda source: operator_instance(source, *args, **kwargs)
return PipableOperatorImplementation()
Have you ever done any measurements on the import time of aiostream? I know this might not be something that'll be much of an issue for you, though we've seen import time when Python projects in monorepos get out of control.
Good point, I've never tested that before. Although in this case, there doesn't seem to be any major overhead:
$ python -m timeit -n 1 -r 1 "import aiostream.stream"
1 loop, best of 1: 28.9 msec per loop
$ python -m timeit -n 1 -r 1 "import asyncio"
1 loop, best of 1: 23.2 msec per loop
Regardless, if the user facing side is well-typed, I think that's the priority alongside with hopefully making 0.6.x backwards-compatible with <0.5.0.
Agreed! Thank you for your input, I'll go ahead and merge #108 then :)
from aiostream.
Fixed in v0.6.0 🎉
Thanks again for the report :)
from aiostream.
Related Issues (20)
- Incompatibility with mypy 1.7 and later
- Docstring and prototype not properly shown with pylance HOT 1
- New operators
- No easy to see installation instructions in the docs or README HOT 1
- merge causes anext(): asynchronous generator is already running Exception HOT 2
- Cancelling pending tasks in same TaskGroup causes RuntimeError HOT 3
- UnboundLocalError when trying to reuse a stream HOT 1
- iterating from a synchronous iterator blocks the event loop HOT 6
- add support python3.10 HOT 1
- Please upload a wheel release to pypi HOT 2
- `stream.list` implementation is x10 slower than using plain Python built-in functionality (list comprehension) HOT 2
- Idea: Timeouts on chunks HOT 4
- action: support task_limit HOT 5
- Regarding `update_pipe_module` HOT 3
- Aiostream fails to import with a TypeError HOT 1
- CI test pipeline doesn't run in Python 3.12: ModuleNotFoundError: No module named 'setuptools' HOT 6
- License change HOT 3
- asyncio.Event for graceful/early termination HOT 7
- How to handle stream splitting HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aiostream.