sfackler / streaming-iterator Goto Github PK

The design is this:

pub trait StreamingIterator<'s, ImplicitBound: sealed::Sealed = sealed::ImplicitBounds<'s, Self>> {
    type Item;
    fn next(&'s mut self) -> Option<Self::Item>;
}

mod sealed {
    pub trait Sealed {}
    pub struct ImplicitBounds<'a, T: ?Sized>(&'a T); // T: 'a
    impl<T: ?Sized> Sealed for ImplicitBounds<'_, T> {}
}

Here is a playground of it being used to implement WindowsMut. Note that it is working when neither the iterator nor the iterator's item are 'static.

Note that I have only a vague idea of how this actually works, I'm not an expert on rustc. But the general idea is that the hidden type parameter on the trait effectively gives a where Self: 's bound to the trait, but in a special way that works with HRTBs.

I find it strange that map and filter_map both create a StreamingIterator rather than an Iterator, because their mapped items are not tied to the lifetime of the source. You have to use cloned or owned after mapping to get an iterator by value.

Map and FilterMap could be plain Iterators that directly return the mapped value. One could still use convert to get back to streaming if really wanted. Also cloned and owned could essentially be just map(Clone::clone) and map(ToOwned::to_owned). Of course, this would be a breaking change.

Alternatively, there could be new methods like map_deref, just like map but producing an Iterator.

Would it be possible to add support for enumerate()? That's one of the things I miss from the standard iterators.

Is it possible to use fallible items such as StreamingIterator<Item = Result<Page, String>> ? If yes, how do we convert the errors?

It seems that if we use something like item.as_ref()? the error is borrowed and will have a lifetime, which makes it a bit more difficult to e.g. use in external erros such as External(Box<dyn Error + ...>)?

I'm not sure why wasn't fn get_mut(&mut self) -> Option<&mut Self::Item>; added already, but it would be great to have it if possible.

The readme and crates.io point to docs.rs/streaming_iterator but the crate is published as streaming-iterator with a hyphen.

Sometimes when using rust-postgres-binary-copy, which takes values as a streaming iterator, I want to use a slice of borrowed ToSql values. I might not own these values, so I can't give ownership to a streaming iterator in the normal way.

However, since a streaming iterator yields references to the iterator's item anyway, it should be possible to use these APIs without having to provide ownership to the iterator. I've been working around this by implementing the StreamingIterator trait on top of an existing Iterator of references:

use streaming_iterator::StreamingIterator;

pub struct FakeStreamingIterator<'a, I, T>
    where I: Iterator<Item=&'a &'a T>, T: 'a + ?Sized {

    inner: Box<I>,
    current: Option<&'a &'a T>,
}

impl<'a, I, T> FakeStreamingIterator<'a, I, T>
    where I: Iterator<Item=&'a &'a T>, T: ?Sized {

    fn new(inner: I) -> FakeStreamingIterator<'a, I, T> {
        FakeStreamingIterator { inner: Box::new(inner), current: None }
    }
}

impl<'a, I, T> StreamingIterator for FakeStreamingIterator<'a, I, T>
    where I: Iterator<Item=&'a &'a T>, T: ?Sized {

    type Item = T;

    fn advance(&mut self) { self.current = self.inner.next(); }
    fn get(&self) -> Option<&T> { self.current.map(|x| *x) }
}

pub fn convert<'a, I, T>(iterator: I) -> FakeStreamingIterator<'a, I, T>
    where I: Iterator<Item=&'a &'a T>, T: ?Sized {

    FakeStreamingIterator::new(iterator)
}

This then allows me to construct a FakeStreamingIterator from a slice of references, and treat it as a normal StreamingIterator. However, it's a bit of a hack, so I'm curious what the officially supported way to handle this case is — maybe something like my FakeStreamingIterator should be included to support this use case?

We would love to use StreamingIterator, but we are in a situation in which we need a lifetime on the item. We are implementing our own version similarly to this:

pub trait StreamingIterator {
    type Item<'b>
    where
        Self: 'b;

    fn get(&self) -> Option<Self::Item<'_>>;
}

That is, we return actually the item, rather than a reference to the item, and the item has a lifetime that we can use to return references inside the item.

It there any way StreamingIterator can be adapted to this case?

The documentation for is_done() says, "Checks if get() will return None". However, is_done() can be implemented to return true even when get() returns Some. Is the documentation overly specific or is it undefined behavior to implement is_done() such that it can return true when get() returns Some?

P.S. this is relevant to me because I have a use-case where I need to get the final value from a streaming iterator. With the current streaming-iterator API, this requires get() to return Some when is_done() returns true.