downloads keep getting aborted at random points about xdccjs HOT 13 CLOSED

@JiPaix I think this issue should probably be re-opened. The problem(s) still persists for me.

Thanks, I didn't have time to do so 😆, I'll investigate some more this weekend.

You said in a previous comment the download failed right at the end, is that behavior also random?

from xdccjs.

Here's an update since i don't think i'll be able to implement changes until next week:

The failure at the very end (few milliseconds left + md5 not matching) is happening consistently on every download

I took a look at how HexChat (an open-source desktop IRC client written in C) handles dcc transfers and found out the issue.
I've got a fix running on a test build, i know just need to implement this for production.

The failing somewhere randomly during the transfer is happening very randomly and sporadically

After finding what was the cause of the first problem i'm pretty sure both are linked.
I ran the following test without any issues:

• 5 packages over 5 different bots
• 4 packages being >5GB one ~500mb.
• Multiple locations (vpn): US (california, florida, virginia) EU (ireland, germany, france) and Japan

I have the feeling that chat messages (e.g. new packet announcements) from the bot might disturb the transfer randomly, but was not able to validate that hypothesis.

File transfers and IRC are handled by two separated connections and aren't linked in any way.
The only possible thing to happen is for the machine hosting the bot to have network issues, but that would affect both connections.

from xdccjs.

how this thing works

as i said before:

xdccJS then connects to the bot and there's a back-and-forth conversation between the two (via TCP).
Every time the bot sends a chunk of data, xdccJS has to reply with the number of bytes received so far. 

This is a mechanism called ACK (acknowledgement) to tell the bot which chunk of bytes it should send next.

in addition to that:

the ACK has to be stored in a Buffer, either 32-bit or 64-bit.

what was the problem?

mIRC has been dictating how IRC should work for more than a decade.
In 2008 they released an update:

If mIRC sees that it is sending or receiving a file with a size larger than a 32bit value,
it sends and expects to receive 64bit acks.

Meaning that any XDCC script/bots that was written around this era abide by this rule.
And 64-bit wasn't much of a thing back in the day...

32-bit buffers have a limitation where the ACK can't exceed a value corresponding to 4GB (in bytes).
Since most bot can now read either 64-bit or 32-bit I used 64-bit buffers by default which max value is 18446744.1 TB.
But then older bots would "confuse" this 64-bit for 32-bit and "misread" the value.

what's the fix?

• Before starting a download bots send the expected size of the file
• if the file size is larger than 4GB we use a 64-bit buffer
• if it's less we use a 32-bit buffer.
• This method works with both old and newer bots.

from xdccjs.

investigation

• while download xdccJS times out if no data is received after 10s (read ECONNRESET)
• These bots do not follow conventions
  • What's supposed to happen:

    After sending the request the bot gives you its IP:PORT,
    sometimes there's also somekind of password (a series of numbers) to authentificate the connection.
    
    xdccJS then connects to the bot and there's a back-and-forth conversation between the two.
    Every time the bot sends a chunk of data, xdccJS has to reply with the number of bytes received so far. 
    
    This is a mechanism that tells the bot which chunk of bytes it should send next.
    
    Once the last byte of data is received, xdccJS sends a final message,
    The bot then knows you received every bytes from the files, and closes the connection.
    
    medetashimedetashi.
    

  • What really happens

    The Passwords aren't numbers.
    
    Once the last byte of data is received, xdccJS sends its last message,
    But these bot already disconnected.
    
     And it miserably fails
    
    i believe the reason behind this is:
    There's nothing to send anymore so i don't need to know where your download is at.
    

Solutions

• Let the user set the timeout using the already existing option timeout
  • implement
  • add the option for the CLI
• Accept tokens (passwords) that aren't numbers
• Do not throw errors when a download is complete and the bot disconnects abruptly

from xdccjs.

Thanks again for updating xdccJS so fast! However, unfortunately after updating to v4.5.28 the problem (or one that looks like it) still persists 😢

[i] downloading : <PACKET NAME>
      / [====================] ETA: 411ms @ 6.03 MB/s - 99%[X] Connection error: read ECONNRESET
     [X] Connection error: read ECONNRESET
      [i] retrying: 1/1
[X] couldn't connect to <BOT IP>:<BOT PORT>
       [X] couldn't connect to <BOT IP>:<BOT PORT>
      [X] skipped pack: <PACKET NUMBER>
      [X] couldn't download pack: <PACKET NUMBER> from <BOT NAME>
      ...

I think you're on the right track though. The download curiously aborts at just 411 milliseconds left (I got one where there were only 91ms left) - so some odd behavior regarding the last chunk seems likely. The downloaded has about the size I expect, but the md5 doesn't match, so something must have gotten amiss.

One additional thing. I did not pay much attention to this before, but I was getting download aborts both very shortly before the DL was supposed to finish (error case above, a few milliseconds before finish) as well as aborts randomly throughout the transfer (see log in previous comment, notice how the remaining time and error messages are very different). I initially thought that both were a symptom of the same problem, but it looks like they may not be related after all.

from xdccjs.

@JiPaix I think this issue should probably be re-opened. The problem(s) still persists for me.

from xdccjs.

The failure at the very end (few milliseconds left + md5 not matching) is happening consistently on every download (problem 1). The failing somewhere randomly during the transfer is happening very randomly and sporadically (problem 2). I was able to exclude a shaky internet connection on my side though. I have the feeling that chat messages (e.g. new packet announcements) from the bot might disturb the transfer randomly, but was not able to validate that hypothesis.

from xdccjs.

Works like a charm. No more random aborts, no more errors at 99.9999% and the MD5 sums match. Thanks a lot!! 🥳

from xdccjs.

I unfortunately introduced a new bug with files between 2.1GB and 4.3GB...
Consider updating xdccJS once again as i just published a fix to the fix. 🥲

from xdccjs.

I was just about to write you about that 😁

node:internal/buffer:72
    throw new ERR_OUT_OF_RANGE('value', range, value);
    ^
RangeError [ERR_OUT_OF_RANGE]: The value of "value" is out of range. It must be
>= -2147483648 and <= 2147483647. Received 2147485020
    at new NodeError (node:internal/errors:371:5)
    at checkInt (node:internal/buffer:72:11)
    at writeU_Int32BE (node:internal/buffer:802:3)
    at Buffer.writeInt32BE (node:internal/buffer:890:10)
    at Downloader.onData (/usr/local/lib/node_modules/xdccjs/dist/downloader.js:190:24)
    at Socket.<anonymous> (/usr/local/lib/node_modules/xdccjs/dist/downloader.js:103:42)
    at Socket.emit (node:events:390:28)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:289:9)
    at Socket.Readable.push (node:internal/streams/readable:228:10) {
  code: 'ERR_OUT_OF_RANGE'
}

This one seems to be fixed in v4.5.33.

from xdccjs.

And the error at 99% bug seems to be back with v4.5.33:

      \ [====================] ETA: 264ms @ 16.72 MB/s - 99%[X] Connection error: read ECONNRESET
     [X] Connection error: read ECONNRESET
      [i] retrying: 1/1

xdccJS errors out a few milliseconds before finishing and the MD5 doesn't match.

from xdccjs.

I didn't have a hard time replicating the issue but it took me quite a while to understand what was going on after all i've done.
It was again an issue with how the bot and xdccjs communicates with each other during transfers.

I pushed [email protected] hopefully this is it

from xdccjs.

Tested a file between 2.1GB and 4.3GB and it's working nicely! No error at 99% and the MD5 matches. Great work, thanks a lot!!! 👍

from xdccjs.