[discuss] Return a request reference instead of an opaque record about hackney HOT 4 CLOSED

Are memory overhead and performance degradation due to message passing significant? If not, advantages are too good to ignore them.

from hackney.

I would like to call out for caution here for the way the GC handles big data. Since it will propably mean sending big binaries around it can lead to keeping a lot of things in memory - I faced exactly that issue [2] when sending chunks read from hackney over messages to another process.

Worst case this could mean that the entire streamed body stays in memory until the hackney streaming process dies (and GC's obviouslly).

@dieswaytoofast has an excelent blog post [1] on that issue (that saved my the behind when I ran into this issue) and I highly recommend reading before this change.

I would opt for keeping both around, let people choose what's right for them like other network sockets work with either async or non async giving different ways to handle reading from the socket. That gives the advantage of staying in line with gen_tcp/udp at least from the philosophy.

[1] http://dieswaytoofast.blogspot.de/2012/12/erlang-binaries-and-garbage-collection.html
[2] https://github.com/project-fifo/sniffle/blob/dev/apps/sniffle/src/sniffle_img_vnode.erl#L345

from hackney.

@kxepal I am trying to see about the real impact, I forgot to add the GC one, thx @Licenser for the link :)

from hackney.

Adding 2 links around this topic to keep a trace somewhere:

http://jlouisramblings.blogspot.fr/2013/10/embrace-copying.html from @jlouis

and

https://blog.heroku.com/archives/2013/11/7/logplex-down-the-rabbit-hole from @ferd

from hackney.