HTTP body extraction does not handle content encodings about tcpflow HOT 7 CLOSED

It didn't used to do that. Hm...

from tcpflow.

Yes, it seems that the newly committed code doesn't unzip.

from tcpflow.

I didn't see unzipping in the code I stripped out. The entire HTTP scanner as of scan_http.cpp @ 2852a1 is:

if(sp.sbuf.memcmp(reinterpret_cast<const uint8_t *>("HTTP/1.1 "),0,9)==0){
    /* Looks like a HTTP response. Split it at the \r\n\r\n into two sbufs and save each */
    ssize_t body_start = sp.sbuf.find("\r\n\r\n",0);
    if(body_start==-1) return;  // no body to be found

    tcpdemux *d = tcpdemux::getInstance();

    std::stringstream xml_head;
    d->write_to_file(xml_head,sp.sbuf.pos0.path+"-HTTP",sbuf_t(sp.sbuf,0,body_start-2));

    sbuf_t sbuf_body(sp.sbuf,body_start,sp.sbuf.bufsize - body_start);
    std::stringstream xml_body;
    d->write_to_file(xml_body,sp.sbuf.pos0.path+"-HTTPBODY",sbuf_body);

    /* Need to do something with the XML */
    /* Need to handle the gzip */
}

Where did that logic live before?

from tcpflow.

Oh, it's in the #if 0 later in the file. I see.

Edit: it looks like process_gzip() is #if 0'd in tcpdemux.cpp as of the current master.

from tcpflow.

It was there before I did the upgrade to the bulk_extractor plug-in system. I probably forgot to re-enable it. The code works, though. We should put it back in. Do you want to handle that?

Overall, do you like the plug-in system?

from tcpflow.

I'll take a stab at handling gzip/deflate. I think I'll do it a little differently -- write an HTTPBODY file in all cases, but if it's a compressed Content-Encoding that we have the ability to decompress, then decompress while writing to the output file instead of writing two output files. That seems most appropriate to me; the output should correspond to HTTP payloads, regardless of how they're compressed or transmitted over the wire.

Additionally, we can at least write out to .html.gz or similar if it's gzip and decompression is disabled/unavailable.

from tcpflow.

Sounds great. I'm most appreciative!
On Nov 27, 2012, at 11:34 PM, Will Glynn [email protected] wrote:

I'll take a stab at handling gzip/deflate. I think I'll do it a little differently -- write an HTTPBODY file in all cases, but if it's a compressed Content-Encoding that we have the ability to decompress, then decompress while writing to the output file instead of writing two output files. That seems most appropriate to me; the output should correspond to HTTP payloads, regardless of how they're compressed or transmitted over the wire.

Additionally, we can at least write out to .html.gz or similar if it's gzip and decompression is disabled/unavailable.

—
Reply to this email directly or view it on GitHub.

from tcpflow.