Netty Resource Leak observed with sorted queries leading to OutOfDirectMemoryError about azure-cosmosdb-java HOT 11 CLOSED

Tried with Netty 4.1.31.Final and issue still persist so looks like some issue with way ordered queries are executed

from azure-cosmosdb-java.

thanks @chetanmeh for reporting this.

@mbhaskar could you please take a look?

from azure-cosmosdb-java.

@chetanmeh Thanks for the PR . We could reproduce the above issue as per the testcase provided. We will get back to you after further analysis

from azure-cosmosdb-java.

@chetanmeh, you did a complete diagnosis, thank you. We are looking at this.

from azure-cosmosdb-java.

@chetanmeh could you please try this and let us know if it resolves the leak issue:
https://github.com/Azure/azure-cosmosdb-java/blob/leak-tmp-fix/HOWTO.md

from azure-cosmosdb-java.

@chetanmeh once you confirm the fix addressed the issue we can do the release to maven.
Could you confirm please?

from azure-cosmosdb-java.

@moderakh The change does seem to fix the issue for us and no leak is observed. For my understanding can you confirm the fix approach?

From the change it appears we are now relying on content subscription timeout to discard the response content if not subscribed to within 1 ms from time of receiving the response? If yes would this cause issue in slow setups (vm env) where genuine cases may see responses being discarded?

Some observations on HTTP calls made

Sorted

For fetching 200 top records total 9 HTTP calls were made (thus fetching 900 records) for our setup having 5 partitions and probably only response from 6-7 were consumed. So unconsumed response from the 2 calls was adding up to leak. With this fix ByteBuf for those responses would be now released.

UnSorted

For unsorted case only 4 calls were made and all response was consumed

from azure-cosmosdb-java.

@simplynaveen20 will respond.

from azure-cosmosdb-java.

@chetanmeh This content subscription timeout is the default value of RXNetty HttpClientResponse , we tested different scenario and its working fine , please let us know if you find any issue or odd behavior.

from azure-cosmosdb-java.

@simplynaveen20 Yup its part of rxnetty default. My worry was more about the timeout being low like 1 ms which may cause issue on boxes with high CPU contention. For example in one the usage I see it being set to 5 secs

https://github.com/Netflix/rx-aws-java-sdk/blob/0cc14ddc3d9bdb299f5b5769d53cc83507a22d79/rx-aws-java-sdk-core/src/main/java/com/amazonaws/http/AmazonRxNettyHttpClient.java#L409-L413

I was trying to see if the unconsumed response can possibly be addressed (irrespective of timeout) by implementing onUnsubscribe in AsyncOnSubscribe created in Paginator where I believe we have access to last invoked request for a partition. But got lost in layers!

Anyways for now in our testing we do not see any issue. Would keep a watch and let you know if we observe any related issue

from azure-cosmosdb-java.

Not observed this issue since updating to latest CosmosDB SDK 2.3.0. Resolving it as closed

from azure-cosmosdb-java.