Consider overloading the "pipe" operator about cppitertools HOT 24 CLOSED

Quick update: I've got something I'm pretty happy with for the simplest iterables such as cycle, still working on something for seq | filter(predicate) that generalizes well. I might need a bunch of enable_ifs to get this to work well.

from cppitertools.

I'll have to try some stuff to see if this is possible. I know there's some odd aspects to rangev3 that keep it from working with regular for loops, and those same limitations would probably hurt comparability here as well.
Tbh I'm not really a fan of the PIPE style, I prefer functional over streams but you're certainly right that it seems the community is going that way

from cppitertools.

I know there's some odd aspects to rangev3 that keep it from working with regular for loops

That has nothing to do with the pipable functions. This is because the iterators share two different types. Other libraries that utilize pipable function do not have issues with the standard range-for.

Tbh I'm not really a fan of the PIPE style, I prefer functional over streams but you're certainly right that out seems the community is going that way

Once pipable function are provided alternatively, reverse function composition can be used instead of using the | operator. So instead of writing this:

auto r = numbers | filter(...) | sliced(...) | reversed;

It could be written like this using flow which implements reverse function composition:

auto r = flow(filter(...), sliced(...), reversed)(numbers);

Just some thoughts.

from cppitertools.

I know it's about the iterator types but because itertools uses the underlying iterators internally, I don't know what places may depend on then being the same type.

The simplest way to do the pipes I can think of would be returning something that acts like a curried function, then have the pipe call through to the regular itertools function, but this is going to need some experimentation

from cppitertools.

I know it's about the iterator types but because itertools uses the underlying iterators internally, I don't know what places may depend on then being the same type.

Using different iterator types for ranges is a breaking change for C++ as algorithms always assume the same type. I don't think the OP is suggesting this kind of change, just that several functions could be piped together. Either way, move-only iterators are also non-conforming(an Iterator requires CopyConstructible and CopyAssignable) and could break on algorithms or range-for loops since they require standard iterators.

The simplest way to do the pipes I can think of would be returning something that acts like a curried function, then have the pipe call through to the regular itertools function, but this is going to need some experimentation

Thats exactly how you would implement it. Ideally, the best way is to a build function adaptor, so all you have to write everytime is a simple function object. This blog post here shows a basic implementation in C++14. However, it curries from the front not the end, which is needed for cppitertools. Also, it doesn't support both pipable and non-pipable calls(ie f(x, y) and x | f(y)). For that, though you can use conditional overloading, which this blog post here shows how to implements. So it can be combined together to get both behaviour like this:

template<class F>
using pipable_plus = conditional<F, pipable<F>>;


struct filter_fn
{
    template<class Predicate, class Containter>
    auto operator()(Predicate&& p, Container&& c) const
    {
        return ...;
    }
};

const constexpr pipable_plus<filter_fn> filter = {};

That is just to give an idea of what would be needed to implement pipable functions. I hope it helps.

from cppitertools.

Sorry for not responding on this, it's been a crazy month. In writing this library I've read through the iterator requirements section of the standard too many times. I'd considered digging into range-v3 to see if I could find the reason why begin() and end() return different types, but the author has always seemed like a smart guy from what I've seen so I trust there's a good reason.

The pipeable functions blog post is a helpful read. I have the advantage of being able to do this intrusively, so I'm currently messing with something with CRTP. We'll see how it turns out. I'm sure I'll go through a few iterations of overhauls before I find something I'm happy with.

from cppitertools.

This is some great news!

As for the reasons why to have begin() and end() return different types, this might be useful:

http://ericniebler.com/2014/02/16/delimited-ranges/

Thanks!
ciao!

from cppitertools.

Everything that takes one argument (just a sequence) is handled (cycle, reversed, permutations, etc.). Everything taking an optional callable either before or after a single sequence is handled (takewhile, filter, groupby, accumulate, etc). Working on supporting second mandatory arguments (chunked, sliding_window, combinations) and what can be done to best support the weird ones like imap and slice that don't fit in as well with anyone. Writing a lot of template templates.

Nothing is pushed yet, I should be able to push at least partial support by the end of the week, and then start getting all the bug reports. This still won't fit range-v3 right away because of the different types for end.

from cppitertools.

Everything that takes one argument (just a sequence) is handled (cycle, reversed, permutations, etc.). Everything taking an optional callable either before or after a single sequence is handled (takewhile, filter, groupby, accumulate, etc). Working on supporting second mandatory arguments (chunked, sliding_window, combinations) and what can be done to best support the weird ones like imap and slice that don't fit in as well with anyone. Writing a lot of template templates.

Hmm, I don't understand why the mandatory arguments require extra work. Usually, the optional arguments are difficult, but they usually require constraining the templates for it to work. I have implemented something like this but it has never required template template parameters, but perhaps I am missing something.

Using the pipable_plus I showed, it is pretty easy for mandatory arguments, but optional arguments will require constraining the template due to how the partial evaluation. So for example, writing this for filter won't work:

struct filter_fn
{
    template<class Range, class Predicate>
    auto operator()(Range&& c, Predicate&& p) const;

    template<class Range>
    auto operator()(Range&& c) const;
};

const constexpr pipable_plus<filter_fn> filter = {};

Calling r | filter(pred) won't work because filter(pred) will try to call the overload that requires a range. So to fix it, you will need to constrain the template(actually its always a good idea to constrain templates). There isn't an easy way to detect the Predicate, however, its easy to detect the range, so it can be written like this:

#define REQUIRES(...) class=typename std::enable_if<(__VA_ARGS__)>::type

struct filter_fn
{
    template<class Range, class Predicate, REQUIRES(is_range<Range>())>
    auto operator()(Range&& c, Predicate&& p) const;

    template<class Range,  REQUIRES(is_range<Range>())>
    auto operator()(Range&& c) const;
};

const constexpr pipable_plus<filter_fn> filter = {};

So now when filter(pred) is called, it will return the partially evaluated function that can be piped(due to defining pipable_plus<F> as conditional<F, pipable<F>> ). A simple way to detect if a type is range is to check if begin and end can be called on the type:

// Separate namespace to detect begin and end using ADL lookup
namespace adl {

using std::begin;
using std::end;

struct check_range 
{
    template<class T>
    auto operator()(T&& x) -> decltype(begin(x), end(x), std::true_type{}) const;
};

}

struct always_false
{
    template<class... Ts>
    std::false_type operator()(Ts&&...) const;
};

template<class T>
struct is_range
: decltype(conditional<adl::check_range, always_false>()(std::declval<T>()))
{}

This should also work for cases like slice.

Now you have some functions that take the range at the beginning and some functions that take the range at the end. So most likely you will need a pipable_front and a pipable_back function adaptor to pipe the parameter into the front or the back of the function parameters. Of course, I think it would be better to pick one way and have all the functions be consistent with that although this could be a breaking change and may differ from the python signatures.

This still won't fit range-v3 right away because of the different types for end.

I think the main feature request of the OP was for pipable functions not compatibility for ranges.

Hopefully all this helps.

from cppitertools.

This approach is close to what I started with and I'll likely have to make more specific functions for the odd tools.
What I noticed was that most of the tools fell into a few different patterns, and the functions I was writing were mostly rewriting the constructors, so atm I'm generating the function types: instead of having an actual Accumulate function or callable type (for example) It looks like this:

template <typename <typename, typename> class IterTool, typename DefaultArg>
class SomeGeneratorType {...};

template <typename Seq, typename Func> class Accumulator;
using AccumulateFn = SomeGeneratorType<Accumulator, std::plus<>>; // plus as default
constexpr AccumulateFn accumulate{};

So in the above example, filter_fn itself would also be generated. I took on the optional arguments first because I knew those would be harder, and wanted to see how well this approach worked.

After the initial completion I'm going to try to condense the multiple handlers, right now I have one for sequence first, and one for sequence second and those can probably be merged with some flip in the middle. Once it's functioning you're welcome to take a crack at it, I'm adding pipes to the tests as I go along.

from cppitertools.

I'm not gonna have time to work on this for a couple of days, but I've pushed what I have to a development branch

the following do not support pipe yet
chain.from_iterable
combinations
combinations_with_replacement
slice
chunked
sliding_window

the following won't support pipe unless someone can explain to me how it would work, make sense, and yield some advantage:
chain
compress
zip
zip_longest
product
range
count

Everything else should work. If you feel like playing around with it just let me know if any problems come up

from cppitertools.

I'd suggest a "curry-like" semantics, so that:

second_range | zip(first_range)

is the same as:

zip(first_range, second_range)

Ideally, there could be also a right-curried version of zip.

from cppitertools.

I'm sure I could support it, but is that really better? If I had a more general partial I could see it making sense, but zip is variadic so it's tougher to reason about. the pipe for that doesn't strike me as being terribly intuitive

from cppitertools.

I'd say the most compelling advantage is to prevent zip from breaking the pipeline. In its current implementation, zip cannot in fact be adopted in any sequential computation of the form:

auto final_range = step1 | step2 | step3;

Secondly, the proposed semantics is aligned with how zip behaves in functional languages such as Haskell. Indeed, given the signature:

zip :: [a] -> [b] -> [(a, b)]

We have that zip xs returns a unary function. I don't know if there's a more intuitive way.

from cppitertools.

The currying doesn't seem obvious to me, especially since a bunch of these bind what would be their second argument. Also as zip is variadic while haskell's takes two arguments it would have to work differently. Each step would need to be pipeable and iterable, which is weird. I'm not sure.

Anyway, I pushed everything else that I can easily make sense of how to do. The only ones left that I'm going to definitely do are slice (not done yet because it has a different pattern than the rest), and chain.from_iterable (only because I forgot about it until just now. I'll likely do repeat as well.

from cppitertools.

In the meantime if anyone has any interesting usecases they want to throw into tests/test_mixed.cpp I'll gladly review any pull requests.

from cppitertools.

Everything that makes sense to be to be piped is done and pushed to the supportpipe branch. I have tests for everything in their individual files under tests/ but still want to add more to test_mixed. I'd also have to write some more docs about it. Here's the list of everything it should be working for:

• accumulate
• chain.from_iterable
• chunked
• combinations
• combinations_with_replacement
• cycle
• dropwhile
• enumerate
• filter
• filterfalse
• groupby
• imap
• permutations
• powerset
• reversed
• slice
• sliding_window
• sorted
• starmap
• takewhile
• unique_everseen
• unique_justseen

from cppitertools.

You did an amazing job! Thank you Ryan!

ATM, I'm not working on any C++ project, but I will test it for sure as soon as I can. When do you plan to push it in the master branch?

from cppitertools.

So I've been pretty bad at keeping up with this project the last few months, sorry everyone.
I merged supportpipe into master, if things go wrong I hope I'll hear about it

from cppitertools.

@iliocatallo I just pushed a bunch of commits to work with iterables that return different types for begin and end. This seems to allow accepting types from rangev3, for example

    auto ns = ranges::view::ints(5) | ranges::view::take(10) | iter::enumerate;

However, rangev3 does concept checking that means it won't accept itertools

     auto ns = ranges::view::ints(5) | iter::enumerate| ranges::view::take(10) 

Fails with:

error: static assertion failed: You can't pipe an rvalue container into a view. First, save the container into a named variable, and then pipe it to the view.

If you have end up mixing these and see any failures that look like my fault, let me know!

from cppitertools.

Hi Ryan,

thank you for the update. Unfortunately, I'm in Javaland lately and it might take a while until my next significant C++ project. Anyhow this is some great news. On a more technical side, have you already investigated which requirements a type should satisfy to model range-v3 Pipeable concept? Do you think itertools containers might be able meet such requirements?

from cppitertools.

from cppitertools.

So the real answer would be updating the pipeable concept in rangev3, wouldn't it?

from cppitertools.

from cppitertools.