Comments (14)
4.5.11 has been released.
from swcompression.
Hello,
This is an intriguing issue. What do you mean when you say "sporadic crashes"? Some files cause them and some files don't, or that one single file sometimes causes a crash and other times doesn't?
Before discussing potential solutions, it would be interesting to understand the root cause. I can foresee one of three options:
- The TAR file(s), that cause(s) this crash, is (are) malformed. In this case, your suggested solution would be appropriate.
- The TAR file is well-formed. This would mean that there is a problem with the implementation in SWCompression, and in that case it should be fixed instead.
- The TAR file doesn't technically conform to any of the several TAR formats implemented in SWCompression, however such files are consistently produced by another implementation (for example, due to a bug in this other implementation). In this case, a workaround should be added to account for such "non-standard" files.
I don't know how this is possible, as the tarfiles are under my control and all have been tested to work.
This sentence makes me think that this is either the 2nd or 3rd option, but to be absolutely sure, I would need to investigate this problem further. For this I would need additional information, like an example of a file that causes the crash and/or the name of the program that produced it.
Though it seems unlikely, it may also be the case that there is an issue within Swift itself, which surfaces only on specific platforms and/or specific versions of Swift. There was a precedent before, see this code.
P.S. I guess, your suggested fix would be useful in any case, but I still think that I should better understand what is happening.
from swcompression.
Hi @tsolomko, thanks for the quick reply!
Unfortunately, the crash report doesn't indicate which file(s) were involved. However, this is for an app that has only seven tarfiles that it uses.
These files were created using the stock bsdtar
that comes with macOS:
bsdtar 3.3.2 - libarchive 3.3.2 zlib/1.2.11 liblzma/5.0.5 bz2lib/1.0.6
I'd be happy to send you one of these files privately. Please contact me at phil at larkwire dot com.
from swcompression.
I've sent you an email.
Also, am I correct to assume that you use Swift 5.4 on one of the darwin platforms (like ios)?
from swcompression.
Yes, these crashes are occurring on iOS, however they are on Swift 5.3.2.
from swcompression.
Hmm, I've tested with your files both on macOS and iOS, both with Swift 5.4 and Swift 5.3.2, and I haven't encountered any crashes.
I am going to add a check for indices anyway (like this), but for now I cannot reproduce the issue.
from swcompression.
Well, I'm not surprised ... it happens only sporadically!
Your fix looks good, except maybe >=
test for the start index (and possibly <=
for the end index)?
from swcompression.
The start index can't be equal to data.startIndex
since by that point in code at least one tar header (which size is equal to 512 bytes) has been processed.
The end index can't be equal to data.endIndex
since a TAR file must end with 1024 bytes of zeros, which aren't included into any entry. These zero bytes are actually crucial to determine the end of a TAR file, and their presence is checked separately every time before trying to parse an entry.
(After writing these lines I've got a feeling that this new check is redundant...)
from swcompression.
That makes sense. Could entryInfo.size
be wrong ... sporadically?!
from swcompression.
The size property, as every other numeric field of TAR, is non-trivially determined depending on the format extensions used, but if it was wrong, it would be a deterministic problem. I also just in case compared the sizes determined by SWCompression with the output of tar -tvf
and they match exactly.
Looking at the crash stack trace again, assuming that it represents a real-life scenario and guessing from the functions' names, could it be possible that TAR files just aren't fully downloaded, before being processed by SWCompression?
from swcompression.
Good question! The download occurs via system calls that call back when the download is finished.
If the network is bad and the file doesn't download completely, wouldn't the initial parse fail (no closing bytes)?
from swcompression.
But let's say that the parse succeeds, despite the fact that not all of the file contents are there. It seems that testing against data.endIndex
should catch this?
from swcompression.
What I was trying to imply in my previous message is that no matter what SWCompression does (crashes or throws an error), it still won't provide you the results, so it may or may not be a good idea to check that the data is finished downloading before invoking SWCompression.
If the network is bad and the file doesn't download completely, wouldn't the initial parse fail (no closing bytes)?
This is a good question. Apparently, no: the currently implemented check actually conflates two situations: when the EOF marker is reached and when a file is truncated.
But let's say that the parse succeeds, despite the fact that not all of the file contents are there. It seems that testing against
data.endIndex
should catch this?
Yes, it will catch it.
Anyhow, I am going to release an update with this fix some time soon (this week?), but the logic in the "initial parse" is definitely incorrect and I should reconsider it in the future.
from swcompression.
That sounds good. Thanks so much for addressing this so promptly. Really much appreciated!
from swcompression.
Related Issues (20)
- [CRASH] LittleEndianByteReader.swift:33:20 HOT 2
- Progress when decompression is needed HOT 1
- Memory issues when working with larger files HOT 8
- [CRASH] MsbBitReader.bit() HOT 4
- AR Format Support HOT 7
- Can Gzip unarchive return Member? HOT 1
- GzipArchive.unarchive wrongMagic for short data HOT 1
- Brotli Support? HOT 2
- Symlinks in tarballs are created with absolute paths HOT 3
- BZip2Error.wrongCRC when decompressing HOT 2
- Support File I/O for large archives and resource-constrained environments
- Error when decompressing a certain 7z file HOT 2
- Some zip files can't be extracted HOT 2
- LZ4 format not compatible? HOT 8
- Compiling for watchOS fails HOT 1
- LZ4 decompress speed HOT 10
- Support splitting zips into multiple files
- Is there a way to extract a single file from a 7z compressed archive? HOT 4
- LZFSE Support
- Please update BitByteData dependency HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from swcompression.