Code Monkey home page Code Monkey logo

datacompression's Introduction

DataCompression

A libcompression wrapper extension for the Data type

Supported compression algorithms are:

Requirements

  • iOS deployment target >= 9.0
  • macOS deployment target >= 10.11
  • tvOS deployment target >= 9.0
  • watchOS deployment target >= 2.0

Swift version support

Library Version Swift Version
3.8.0 5.7 (Xcode 14)
3.6.0 5.1 (Xcode 11)
3.5.0 5.0
3.1.0 4.2
3.0.0 3.0 -> 4.1
2.0.1 < 3.0

Usage example

Try all algorithms and compare the compression ratio

let raw: Data! = String(repeating: "There is no place like 127.0.0.1", count: 25).data(using: .utf8)

print("raw   =>   \(raw.count) bytes")

for algo: Data.CompressionAlgorithm in [.zlib, .lzfse, .lz4, .lzma] {
    let compressedData: Data! = raw.compress(withAlgorithm: algo)

    let ratio = Double(raw.count) / Double(compressedData.count)
    print("\(algo)   =>   \(compressedData.count) bytes, ratio: \(ratio)")
    
    assert(compressedData.decompress(withAlgorithm: algo)! == raw)
}

Will print something like:

raw    =>   800 bytes
zlib   =>    40 bytes, ratio:  20.00
lzfse  =>    69 bytes, ratio:  11.59
lz4    =>   181 bytes, ratio:   4.42
lzma   =>   100 bytes, ratio:   8.00

Container formats

The famous zlib deflate algorithm (RFC-1951) can also be used with the shortcuts .deflate() and .inflate()
let data: Data! = "https://www.ietf.org/rfc/rfc1951.txt".data(using: .utf8)
let deflated: Data! = data.deflate()
let inflated: Data? = deflated?.inflate()
assert(data == inflated)
Data in gzip format (RFC-1952) can be handled with .gzip() and .gunzip()
let data: Data! = "https://www.ietf.org/rfc/rfc1952.txt".data(using: .utf8)
let gzipped: Data! = data.gzip()
let gunzipped: Data? = gzipped.gunzip()
assert(data == gunzipped)

Note: Compressed data in gzip format will always be 18 bytes larger than raw deflated data and will append/perform a crc32 checksum based data integrity test .

Data in zip format (RFC-1950) can be handled with .zip() and .unzip()
let data: Data! = "https://www.ietf.org/rfc/rfc1950.txt".data(using: .utf8)
let zipped: Data! = data.zip()
let unzipped: Data? = zipped.unzip()
assert(data == unzipped)

Note: Compressed data in zip format will always be 6 bytes larger than raw deflated data and will append/perform a adler32 checksum based data integrity test .

Compress a file on the command line and decompress it in Swift

The easiest way is using the already installed gzip command line tool. Assuming you have a file called file.txt, after calling

gzip -9 file.txt

the file should have been compressed to file.txt.gz. You can now load and uncompress the contents of your file with:

let compressedData = try? Data(contentsOf: URL(fileURLWithPath: "/path/to/your/file.txt.gz"))

if let uncompressedData = compressedData?.gunzip() {
    print(String(data: uncompressedData, encoding: .utf8) ?? "Can't decode UTF-8")
}

Checksum extensions

Unrelated to compression but for convenience Crc32 and Adler32 methods are also exposed on the Data type which may come in handy.

let classicExample = "The quick brown fox jumps over the lazy dog".data(using: .utf8)!
let crc32    = classicExample.crc32()
let adler32  = classicExample.adler32()
print("crc32: \(crc32), adler32: \(adler32)")

Will print:

crc32: 414fa339, adler32: 5bdc0fda

Install

Cocoa Pods

To integrate DataCompression into your Xcode project using CocoaPods, add it to your Podfile:

target '<your_target_name>' do
    pod 'DataCompression'
end

Then, run the following command:

$ pod install

You then will need to add import DataCompression at the top of your swift source files to use the extension.

Carthage

Note: DataCompression versions < 3.3.0 are not compatible with carthage. That means Swift 5 only.

To integrate DataCompression into your Xcode project using Carthage, add it as a dependency to your Cartfile. Just add:

github "mw99/DataCompression"

You will then have to add the framework paths in the carthage copy-frameworks run script phase of your Xcode project. The paths may differ depending on you have setup your project in relation to Carthage.

Input:
$(SRCROOT)/Carthage/Build/iOS/DataCompression.framework
Output:
$(BUILT_PRODUCTS_DIR)/$(FRAMEWORKS_FOLDER_PATH)/DataCompression.framework

You then will need to add import DataCompression at the top of your swift source files to use the extension.

Swift Package Manager

To integrate DataCompression into your Xcode project using the swift package manager, add it as a dependency to your Package.swift file:

import PackageDescription

let package = Package(
    name: "<your_package_name>",
    dependencies: [
        .Package(url: "https://github.com/mw99/DataCompression.git", majorVersion: 3)
    ]
)

You then will need to add import DataCompression at the top of your swift source files to use the extension.

The next time you run swift build, the new dependencies will be resolved.

$ swift build

Or just copy the file into your project

You only need one file located at Sources/DataCompression.swift. Drag and drop it into the Xcode project navigator.

Change log / Upgrading guide

Version 3.7.0 to 3.8.0
  • Solved a bug causing crashes when using .unzip(), because of unaligned pointer loading caused by internal changes in Swift 5.7.
Version 3.6.0 to 3.7.0
  • Support for Xcode 14 with Swift 5.7
Version 3.5.0 to 3.6.0
  • Target platforms finally added to the SPM Package file
  • Carthage support improved
  • Support for Xcode 11 SPM integration
Version 3.4.0 to 3.5.0
  • Fix that prevents a bug in Apples lzfse compressor when working with large chunks of data.
Version 3.3.0 to 3.4.0
  • Swift 5 release had further deprecation warnings than in the Swift 5 beta. Fixed.
Version 3.2.0 to 3.3.0
  • Added support for Carthage
Version 3.1.0 to 3.2.0
  • Podspec swift version set to 5.0
  • Library file structure updated to fit the new swift package manager layout
Version 3.0.0 to 3.1.0
  • Podspec swift version set to 4.2
Version 2.0.X to 3.0.0
  • The encoded data in zip format is not copied anymore, which should improve performance.
  • Checksum validation is now always performed with libz and way faster.
  • The skipCheckSumValidation: parameter of .unzip() was removed.
  • Items of the algorithm enum type are now Swift like lowercase, e.g. .LZMA.lzma

License

Apache License, Version 2.0
Copyright 2016, Markus Wanke

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

datacompression's People

Contributors

mw99 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

datacompression's Issues

fatalErrors on XCode14/iOS16

Been using this happily for years. Updated phone to iOS 16 and XCode to 14 today and now I'm getting fatalErrors in

func unzip(skipCheckSumValidation: Bool = true) -> Data?

Somewhere inside the

return last.withMemoryRebound(to: UInt32.self, capacity: 1)

statement. I don't get a lot of useful info from it, just

Swift/arm64-apple-ios.swiftinterface:23320: Fatal error
2022-09-14 16:03:58.690178-0700 TWiG MC[668:25066] Swift/arm64-apple-ios.swiftinterface:23320: Fatal error

in my console. My Data argument headed in was:

78.DA.0D.CD.3B.4E.43.31.10.40.51.B3.81.B4.94.D0.23.9E.C6.E3.4F.6C.29.EB.A0.45.E3.F9.C0.53.88.8D.EC.14.B0.FB.BC.F6.16.E7.3E.7D.5E.DC.29.55.2D.6C.31.A8.17.0B.A1.24.3A.37.23.00.89.39.86.AC.18.6B.33.03.C5.1C.19.53.40.CB.D2.72.80.7A.14.9F.C8.17.41.36.16.43.4E.44.29.85.5C.9B.A8.86.B3.C5.C4.B1.98.04.40.AA.15.98.35.A2.57.F6.26.02.10.92.02.D6.82.8D.51.01.DC.2F.AD.75.D5.7F.EF.5E.9A.B7.63.17.A9.94.06.87.AB.DF.63.DD.77.71.CF.DB.EB.C7.BB.07.78.EB.74.53.77.9A.8B.3A.FD.E9.DA.BA.FE.AC.D1.F7.39.F7.2F.BA.EF.A3.6F.3C.6E.AE.CD.71.D5.E9.DC.03.5C.71.40.90

I'll do some digging/experimenting. This has been rocksolid for so long, surprised to see it start misbehaving all of a sudden

Unable to unzip large files zip on MacOS command line.

Your DataCompression.swift is exactly the kind of utility that I need to unzip downloaded files in a mobile App. The files are sqlite database files that are about 6MB compressed and 22MB uncompressed. I have tested that each of your compression algorithms will compress and decompress this data correctly. However, when I compress a file with MacOS command-line zip.

$ zip myFile.zip myFile

None of your decompression algorithms is able to decompress the 'myFile.zip'.

.inflate() and .ZLIB return the error 'Invalid BTYPE 3 block'. All of the others return no error. They all output a nil output Data.

Are there any settings that can be tweaked to solve that problem? Thanks so much for any help you can provide. I would really like to be able to use your code.

Below is the sample code that I have used to test this:

private func zip_unzip(sourceFile: String, targetFile: String, doZip: Bool) -> Void {
do {
let sourceUrl = URL(fileURLWithPath: NSHomeDirectory() + sourceFile)
print("source URL (sourceUrl)")
let source = try Data(contentsOf: sourceUrl)
print("source Data (source)")
let target: Data? = (doZip) ? source.compress(withAlgorithm: .ZLIB) : source.decompress(withAlgorithm: .ZLIB)
print("target Data (String(describing: target))")
let targetUrl = URL(fileURLWithPath: NSHomeDirectory() + targetFile)
print("target url (targetUrl)")
try target?.write(to: targetUrl, options: Data.WritingOptions.atomic)
} catch {
print("caught exception did not complete zip/unzip")
}
}

Xcode deprecation warnings for withUnsafeBytes

Is there any plan to fix the deprecations for

'withUnsafeBytes is deprecated: use withUnsafeBytes<R>(_: (UnsafeRawBufferPointer) throws -> R) rethrows -> R instead

or is there a reasoning behind?

Thanks for the great work

CFBundleVersion is missing

Build for Release failed because of Mantis:

ERROR ITMS-90056: "This bundle Payload/.../Frameworks/Mantis.framework is invalid. The Info.plist file is missing the required key: CFBundleVersion. Please find more information about CFBundleVersion at https://developer.apple.com/documentation/bundleresources/information_property_list/cfbundleversion"
Return status of iTunes Transporter was 1: ERROR ITMS-90056: "This bundle Payload/.../Frameworks/Mantis.framework is invalid. The Info.plist file is missing the required key: CFBundleVersion. Please find more information about CFBundleVersion at https://developer.apple.com/documentation/bundleresources/information_property_list/cfbundleversion"
The call to the iTMSTransporter completed with a non-zero exit status: 1. This indicates a failure.

I can't find the reason why CFBundleVersion is missing but by adding it manually inside /iOS/Carthage/Build/iOS/Mantis.framework fixed the problem temporary.

Can't unarchive .zip

I try to unzip my .zip file (downloaded from the Internet) using the .unzip () command and get a nil. Also I cannot open on my computer the file zipped by .zip(). With gzip all ok. Please help!

Test Failure

Hey, first of all thanks for this great library! I use this in an app of mine- it seems to work very well.

However, I suddenly seem to get test failures (macOS 10.4.4, Xcode 10.2).

The falling test is testRandomDataBlob, specifically the .lzfse decompression test. Apples data compression lib seems to make it hard to find out what the actual error is though. Do you have any idea whats up with that?

Btw the way this test is set up makes it look like more tests are failing after a first test failure. However really only lzfse decompression fails.

Single file zip decompresion

Apple convert single files to .zip when running "Compress this file", and would be good to be able to unzip the file. .unzip() isn't for this case. For example, Perfetto only has gzip support. I have C++ code to handle this, but was just missing this omission in this otherwise amazing library.

I ended up renaming .zip()/.unzip() -> .zlib()/unzlib() in my local copy.

Missing CFBundleVersion when uploading to ITMS

When using carthage as a dependency manager, i receive following Error if I try to upload my build

ERROR ITMS-90056: "This bundle Payload/..../Frameworks/DataCompression.framework is invalid. The Info.plist file is missing the required key: CFBundleVersion. Please find more information about CFBundleVersion at https://developer.apple.com/documentation/bundleresources/information_property_list/cfbundleversion"
Return status of iTunes Transporter was 1: ERROR ITMS-90056: "This bundle Payload/..../Frameworks/DataCompression.framework is invalid. The Info.plist file is missing the required key: CFBundleVersion. Please find more information about CFBundleVersion at https://developer.apple.com/documentation/bundleresources/information_property_list/cfbundleversion"
The call to the iTMSTransporter completed with a non-zero exit status: 1. This indicates a failure.

Unzip a .gz folder

Can you show a code example of unzipping a .gz folder and storing the folder on an ios device in order to access the files stored in the archive?

That would be highly appreciated.

NOT work for ZLIB in iOS 9.x

I got zlib compressed data from network and invoke data.inflate() to decompress.
It works fine in iOS 11 simulator, but fails in iOS 9 simulator.
The function compression_stream_process in DataCompression.swift always return -1.
Neither zlib level-5 nor level-10 can be decompressed
Same data can be decompressed by SWCompression in iOS9, so I think it's a bug in this library.

XCode Version 9.1 (9B55)
iOS 9.x simulator

compress array of json issue

I am using gzip with Compression of array of json [{ }, {}, {}...]
it compressed at iOS side and also decompress successfully at iOS side but when I upload compressed data to sever it fail to decompress.
even it fails here.
https://www.multiutil.com/text-to-gzip-compress/

I compress data and save it to document directory and then read compressed text then try to decompress text(array of json) by using https://www.multiutil.com/text-to-gzip-compress/ but it fails.

what could be issue?

inflate fails with guard stream.dst_size == 0 returns nil

I'm bring back some binary data from a web server using URLSession and POST.

let (data, response) = try await m_oURLSession.data(for: request)

after checking the content-encoding header is "deflate"

I run this:

inflatedData = data.inflate()

Which fails down deep and returns nil because stream.dst_size is not zero:

switch compression_stream_process(&stream, flags)
{
case COMPRESSION_STATUS_OK:
guard stream.dst_size == 0 else

I know the data is actually deflate because I've got an Android app which processes the same deflated data successfully.
Any ideas on what might be happening here?
Am I missing some conversion of data before calling inflate ?
TY

Swift 5

Awful lot of warnings about deprecated uses of withUnsafeBytes..

Some parts of the LICENSE file are left as templates

Thank you for providing us with a great library.

I am trying to create an application that uses this library.
I wanted to display the license clause of this library in my application.
However, I found the following in the LICENSE file

Copyright {yyyy} {name of copyright owner}

Perhaps this is an item that should be modified from the original template.

I would appreciate it if you could fix it if you could.

ZLIB compression and decompression

I have been able to get compression and decompression to work just fine with every algorithm except the ones pertaining to ZLIB. So pretty much just deflate/inflate and compress/decompress with .ZLIB. I am taking a URL, getting it's data representation and passing it into the compression method and then posting it to the server. For decompression I am pulling the data down from the server, passing it into the decompression method and the creating a URL with the decompressed data. Like I said this process has worked every time as long as I am not using ZLIB. I am pretty much trying to see if anyone has had this problem in the past. Thanks.

Gunzip problems

I have problems using gunzip with Swift 5, Xcode 11.3.1

It doesn’t return the inflated content, instead I ‘ve got back the same deflated input data. Executed under test environment or common debug conditions in Xcode. Therefore in the playground environment the same code works perfect as expected!

The file gzipped XML file is about 36kb, inflated expected to be about 4MB

Are there some compiler settings I have to consider? Memory alignment? Any ideas?

DataCompression module generates a different adler32 checksum

Marcus,

I hate getting issues where someone says "there is a problem in your code..."
without giving details... so I hope you'll forgive my verbosity.

In C++, the following will generate a checksum of 0xA490238:

std::string test_string = "TEST CRC";
unsigned int a32 =  adler32(0L, (Bytef *)test_string.data(), test_string.length());

With the DataCompression module the same string generates a checksum of 0xA490239:

let test_crc = "TEST CRC"
var a32 = Adler32()
a32.advance(withChunk: test_crc.data(using:.utf8)!)
print ( "a32 = \(a32)" )

The issue appears to be related to the initialization of checksum for adler32:

--- DataCompression.swift:384 ---

/// Raw checksum. Updated after a every call to `advance(withChunk:)`
public var checksum:UInt32 = 1

--- DataCompression.swift:384 ---

In zlib/adler32, a 1L is returned only if the chunk being evaluated is null.
Otherwise, checksum starts with 0.

--- zlib-1.2.11/adler32.c:87 ---

 if (buf == Z_NULL)
        return 1L;

--- zlib-1.2.11/adler32.c:87 ---

Suggested fix:
initialize checksum to 0 (DataCompression.swift:385) and then,when returning checksum, return 1 if checksum == 0

--- DataCompression.swift:402 ---

    /// Formatted checksum.
    public var description: String
    {
        if checksum == 0 { // THB
            return "1"     // THB
        }                  // THB
        return String(format: "%08x", checksum)
    }

--- DataCompression.swift:402 ---

You would still have to account for checksum being read in other ways...

Thanks!

Tom Bowden
email: [email protected]

compression_stream calls only available on OS X 10.11 or newer

I just added DataCompression to a Swift 5 based SwiftPM tool this one but I am now getting errors stating:

'compression_stream_init' is only available on OS X 10.11 or newer
'compression_stream_destroy' is only available on OS X 10.11 or newer

Here's my Package.swift file content:

// swift-tools-version:5.0

import PackageDescription

let package = Package(
    name: "Accio",
    platforms: [.macOS(.v10_12)],
    products: [
        .executable(name: "accio", targets: ["Accio"]),
        .library(name: "AccioKit", type: .dynamic, targets: ["AccioKit"])
    ],
    dependencies: [
        .package(url: "https://github.com/mw99/DataCompression.git", .upToNextMajor(from: "3.4.0")),
        .package(url: "https://github.com/Flinesoft/HandySwift.git", .upToNextMajor(from: "2.8.0")),
        .package(url: "https://github.com/onevcat/Rainbow.git", .upToNextMajor(from: "3.1.4")),
        .package(url: "https://github.com/jakeheis/SwiftCLI.git", .upToNextMajor(from: "5.2.2")),
        .package(url: "https://github.com/kareman/SwiftShell.git", .upToNextMajor(from: "4.1.2")),
        .package(url: "https://github.com/mengxiangyue/xcodeproj.git", .branch("master")), // TODO: switch back to original once https://github.com/tuist/xcodeproj/issues/385 is fixed
    ],
    targets: [
        .target(
            name: "Accio",
            dependencies: ["AccioKit"]
        ),
        .target(
            name: "AccioKit",
            dependencies: [
                "DataCompression",
                "HandySwift",
                "Rainbow",
                "SwiftCLI",
                "SwiftShell",
                "xcodeproj",
            ]
        ),
        .testTarget(
            name: "AccioKitTests",
            dependencies: ["AccioKit", "HandySwift", "xcodeproj"]
        )
    ]
)

Work with SQLite database?

I stored the compressData as into a SQLite database.
When I read it, it's a SQLite.Blob.
I tried but could not find a way to convert the back to .
Any idea?

Property if data is compressed

Is there a way to check if a given set of data is compressed in any way?
like data.isGzipped -> true or data.compression -> Compression.gzip || Compression.zip ,....

If not this would be a feature request

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.