Comments (27)
@biapar @n9 I released an alpha version of v2 if you want to try it out.
There is a complete rewrite of the internal query logic, the change feed API an other things.
https://github.com/matteobortolazzo/couchdb-net/blob/master/V2.md
from couchdb-net.
Hi, not yet.
What's the best implementation for this?
from couchdb-net.
Polling? SignalR? MQTT?
from couchdb-net.
@matteobortolazzo Not sure what you mean exactly, I would say:
- reading via feed-continuous
- api
IAsyncEnumerable
withCancellationToken
- callback with
CancellationToken
- [maybe IObservable/IObserver]
from couchdb-net.
Yes, my question was about what's be most effective way for developers. I was thinking of simple events but I will check IAsyncEnumerable. I want to work on this feature from this week onward
from couchdb-net.
FYI I am pretty sure I will use IAsyncEnumerable with CancellationToken
from couchdb-net.
@biapar @n9 I implemented an early code for this in the changes-feed branch . Can you provide me some feedback if you don't mind please?
The are 2 new methods for the database:
// Polling
Task<ChangesFeedResponse<TSource>> GetChangesAsync(ChangesFeedOptions options, ChangesFeedFilter filter)
// Continuous feed
IAsyncEnumerable<ChangesFeedResponseResult<TSource>> GetContinuousChangesAsync(ChangesFeedOptions options, ChangesFeedFilter filter, CancellationToken cancellationToken)
To use the continuous feed:
using var cancelSource = new CancellationTokenSource();
await foreach (var feedEvent in rebels.GetContinuousChangesAsync(option, filter, cancelSource.Token))
{
if (/* Something */)
cancelSource .Cancel();
}
from couchdb-net.
@matteobortolazzo I have some issues with cancellation token (seems not working properly). I will investigate more.
from couchdb-net.
It seems to me that cancellation token is not working because it is not used during reading (~ waiting for new change):
You might need to use ReadAsync
, as ReadLineAsync
is not (yet?) cancellable.
Btw. What situation handles the following condition?
couchdb-net/src/CouchDB.Driver/CouchDatabase.cs
Lines 512 to 515 in 24b851e
from couchdb-net.
I got what you mean, in my test it was working because I cancel the token after receiving a specific document change. But if it waits in the readline it breaks I suppose. I will add tests for sure.
About that piece of code, I copied that somewhere. If not needed I'll remove it
from couchdb-net.
@n9 Can you test again with the latest update please?
from couchdb-net.
@matteobortolazzo
Great that you removed blocking calls (by AsyncContext.Run
).
Cancellation of GetContinuousChangesAsync
is still not working. It's blocked on EndOfStream
. Do not use it.
More thoughts:
Do not allocate buffer in cycle (~ do not use ReadCharAsync
). Allocate one buffer in ReadLineAsync
instead. (You might use MemoryPool<T>.Shared.Rent()
for allocation.)
More thoughts for v2:
I would not recommend to throw exception in dispose. (In case, the Logout will fail.)
from couchdb-net.
@n9 Can you send me a test example? Because it's working for me so probably I am not considering all cases.
About logout, do you want a try/catch on logout?
from couchdb-net.
ad example
I am using simple LINQPad code:
await foreach (var doc in db.GetContinuousChangesAsync(new ChangesFeedOptions
{
IncludeDocs = true
}, null, this.QueryCancelToken))
{
doc.Dump();
}
this.QueryCancelToken
is cancellation token. I cancel cancellation token after all changes are received.
ad Logout
As calling Logout
in Dispose
is not critical, I would add try/catch into Dispose
method around Logout
call. (You might use .NET logging to log Logout
exception in Dispose
.)
from couchdb-net.
I agree that ReadCharAsync is not the solution. However if I create E2E tests with a real DB the cancellation token works as expected. I never used LINQPad but could it be related to it?
About the logout, I agree. What do you think about a "ThrowExceptionOnLogoutFail" option?
from couchdb-net.
ad example
However if I create E2E tests with a real DB the cancellation token works as expected.
For now, let's ignore the issue.
But just do not use reader.EndOfStream
. It is doing blocking read call. (It is not asynchronous.)
Someone might uses your lib in desktop or mobile app and s/he will not expect that asynchronous function will do a blocking call and might get issues with blocking of main thread.
And the issue with cancellation will disappear two:)
I never used LINQPad but could it be related to it?
LINQPad is just IDE.
Issue appears in situation when the function is cancelled after all existing changes are received and there are no new changes. Because in this situation reader.EndOfStream
is waiting. (It internally calls blocking reader.Read()
that is waiting for next input, but no input is coming; see the comment there.)
ad Logout
What do you think about a "ThrowExceptionOnLogoutFail" option?
What the option will do? Will it suppress all logout errors? (I would suppress only errors during dispose not in other cases.)
from couchdb-net.
@n9 this should work https://github.com/matteobortolazzo/couchdb-net/blob/v2/src/CouchDB.Driver/Extensions/StreamReaderExtensions.cs
from couchdb-net.
I have following questions:
EndOfStream handling
if (charRead == 0)
{
continue;
}
Why this? How this solve EndOfStream?
Multibyte characters
var str = Encoding.UTF8.GetString(owner.Memory.Span[..charRead]);
What if buffer ends in the middle of multibyte character?
Hint: Use Decoder
instead.
Line handling
IEnumerable<string> filteredList = lines
.Where(line => line.Length > 0 && line != currentRemainder);
Let's say the currentRemainder
is "abc". You are going to remove all lines that equals to "abc".
from couchdb-net.
EndOfStream
It is true if you cancel the task while ReadAsync is waiting. If ReadAsync is cancelled while reading isMessageComplete
will be false and it will ignore the line.
Multibyte characters
You are right. Sorry I am a noob with bytes and stream. How should I use the Decoder? GetCharCount and GetChars?
But, are they going to be multibyte chars? Because the content of the document is not returned. Just rev, id and seq.
Also, is it possible that HTTP handle this char thing on its own?
Line handling
How can a remainder be equal to a full row? By definition it cannot be possible (in CouchDB)
I tested this implementation cancelling the token after receiving X messages and with > 4k bytes to read, and it works as expected.
from couchdb-net.
Context
I was reviewing this function according its name. Later, you might use it in another context. Spec might change. And you may forget in future that you hacked it for one particular HTTP response. (I know it is currently in ChangesFeed namespace.)
EndOfStream
It is true when the stream has ended. What if CouchDB is going to stop (e.g. to update). Current version might reset the connection, later version might just finish the stream.
In that case, it seems to me that it might start looping.
Multibyte characters
Same as Context. Moreover, are document IDs limited to ASCII?
Create the Decoder
outside the loop. (Decoder
will keep the first part of char.)
For GetCharCount
and GetChars
, check the example from official docs.
Line handling
Same as Context. Moreover, why do you need to compare all lines, why just not ignore the last line in the loop (as you do for the first line) if isMessageComplete
is false?
from couchdb-net.
Implement a generic ReadLineAsync requires a lot of tests and work.
For multibyte I agree that the ID may be not ASCII (I just tested). So I will try to implement it with the Decoder.
About line handling, I didn't want to enumerate the list to know the number of elements. But yes, I will change it.
And with the end of stream, what do you suggest? Return string.empty and check for it where I use the readline?
btw thanks for the feedback
from couchdb-net.
About line handling, I didn't want to enumerate the list to know the number of elements. But yes, I will change it.
- You do not need to enumerate list. You can use
.Length
in case of array (or.Count
in case of list).
And with the end of stream, what do you suggest? Return string.empty and check for it where I use the readline?
- I suggest
yield return prevRemainder
ifprevRemainder
is not empty. Thenbreak
(oryield break
). No check is needed at call site.
from couchdb-net.
@n9 The Decoder is smart! Let me know what you think about the latest changes
from couchdb-net.
Cancellation works for me.
I have added minor comments to your commit.
PS:
In case you would like to tune performance, you could reduce allocations using StringSegment
and StringTokenizer
. Or better simply do not decode bytes to chars: just split raw bytes and use Utf8JsonReader
to directly parse json from byte lines (ReadOnlySequence
).
But I think that current implementation based on pools is not much worse. Always, it is good to remember that premature optimization is the root of all evil (Donald Knuth).
from couchdb-net.
Thanks for the comments! Let me know how it looks now.
About Utf8JsonReader I will check later
from couchdb-net.
LGTM
from couchdb-net.
Great, I close this for now. I'll optimize it if needed
from couchdb-net.
Related Issues (20)
- Non-nullable Tables in Context HOT 4
- Exception when running a query. HOT 2
- CouchDB.NET.DependencyInjection HOT 1
- Support retrieving available revisions of a document HOT 9
- Bug with querying overlaping string arrays HOT 6
- Use design documents with CouchDB.Net HOT 1
- Database Splitting + FirstOrDefault() + CouchContext.MyDatabase.Where doesn't apply predicate
- Database Splitting + Convert --> Crash : Exception thrown: Call failed with status code 400 (Bad Request) HOT 2
- Database Splitting : create selector on type (SplitDiscriminator)
- Support raw/un-typed documents? HOT 1
- Replication : add "create_target" field HOT 1
- "Cannot add or update a document without an ID." System.InvalidOperationException HOT 2
- Performance implications / Gotchas ? HOT 1
- Add Replication state HOT 2
- Change feed not including attachments as per specification HOT 1
- How to deserialize inherited documents? HOT 3
- Add Partitioned Database support
- Unexpected behaviour of GetContinuousChangesAsync HOT 3
- Please release v3.5.0 NuGet HOT 2
- Get all databases on server HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from couchdb-net.