saagarjha / unxip Goto Github PK

View Code? Open in Web Editor NEW

822.0 822.0 38.0 62 KB

A fast Xcode unarchiver

License: GNU Lesser General Public License v3.0

Swift 97.55% Shell 1.45% Makefile 1.00%

unxip's People

Contributors

Stargazers

Watchers

unxip's Issues

Progress reporting

Progress reporting is very useful when unxipping large files (such as Xcode), especially on devices where it takes a long time. Would it be possible to add some kind of progress reporting to this tool?

Output argument

It would be a nice addition if it was possible to specify an output path for the extracted file. It is easily doable with a mv, but skipping the move to have it directly where it belongs would be better

Fails to extract Xcode 12.4

As titled, it manages to extract Xcode 12.4 to 2.04GB worth of broken app bundle and nothing more. Perhaps I failed to download it correctly?

Unxip-ing Xcode 14.1 RC2 on macOS 12.5 results in failure due to illegal hardware instruction

Title pretty much says it all. The resulting version of Xcode 14.1 was 9.9 gigs on disk and would not verify correctly. Got this output from unxip when it failed.

$ ./unxip Xcode_14.1_Release_Candidate_2.xip 
[1]    31188 illegal hardware instruction  ./unxip Xcode_14.1_Release_Candidate_2.xip

I deleted Xcode, re-ran the same command, and it succeeded without issue, so unsure how relevant this is but thought it worth raising.

(Maybe?) Relevant details:
Late 2019 Macbook Pro with Intel i9
MacOS 12.5
Unxip 1.1.1 release

add -v / --version flags

Amazing utility! @saagarjha

Would you consider adding a commandline switch to print the version? (-v/--version)

I would like to include this tool in some build scripts etc so it would be useful to be able to check if the correct version was installed already.

MacOS 11.6.6: dyld: Library not loaded: /usr/lib/swift/libswift_Concurrency.dylib

+ unxip Xcode_13.2.1.xip
dyld: Library not loaded: /usr/lib/swift/libswift_Concurrency.dylib
  Referenced from: /usr/local/bin/unxip
  Reason: image not found

Brew Formula

It would be great to have this available for install via brew.

Compilation error.

Hi @saagarjha Can you please update the installation instruction please.

With current installation instructions. I am gettting compilation errors.

unxip-main dks0580224$ swiftc -parse-as-library -O unxip.swift
unxip.swift:16:11: error: cannot find type 'AsyncStream' in scope
extension AsyncStream.Continuation {
          ^~~~~~~~~~~
unxip.swift:25:37: error: cannot find type 'Sendable' in scope
struct ConcurrentStream<TaskResult: Sendable> {
                                    ^~~~~~~~
unxip.swift:29:15: error: cannot find type 'AsyncStream' in scope
        var results: AsyncStream<TaskResult> {
                     ^~~~~~~~~~~
unxip.swift:72:89: error: cannot find type 'Task' in scope
        mutating func addRunningTask(operation: @escaping @Sendable () async -> TaskResult) -> Task<TaskResult, Never> {
                                                                                               ^~~~
unxip.swift:83:20: error: cannot find type 'Sendable' in scope
final class Chunk: Sendable {
                   ^~~~~~~~
unxip.swift:341:80: error: cannot find type 'AsyncStream' in scope
        static func files<ChunkStream: AsyncSequence>(in chunkStream: ChunkStream) -> AsyncStream<File> where ChunkStream.Element == Chunk {
                                                                                      ^~~~~~~~~~~
unxip.swift:341:33: error: cannot find type 'AsyncSequence' in scope
        static func files<ChunkStream: AsyncSequence>(in chunkStream: ChunkStream) -> AsyncStream<File> where ChunkStream.Element == Chunk {
                                       ^~~~~~~~~~~~~
unxip.swift:341:116: error: 'Element' is not a member type of type 'ChunkStream'
        static func files<ChunkStream: AsyncSequence>(in chunkStream: ChunkStream) -> AsyncStream<File> where ChunkStream.Element == Chunk {
                                                                                                              ~~~~~~~~~~~ ^
unxip.swift:73:14: error: cannot find 'Task' in scope
                let task = Task {
                           ^~~~
unxip.swift:128:9: error: cannot find 'Task' in scope
                                try Task.checkCancellation()
                                    ^~~~
unxip.swift:145:3: error: missing protocol 'AsyncSequence'
                for await chunk in compressionStream.results {
                ^
unxip.swift:407:46: error: cannot find 'Task' in scope
                var hardlinks = [File.Identifier: (String, Task<Void, Never>)]()
                                                           ^~~~
unxip.swift:408:33: error: cannot find 'Task' in scope
                var directories = [Substring: Task<Void, Never>]()
                                              ^~~~
unxip.swift:409:3: error: missing protocol 'AsyncSequence'
                for await file in files(in: chunks(from: content).results) {
                ^
unxip.swift:515:3: error: missing protocol 'AsyncSequence'
                for await _ in taskStream.results {
                ^
unxip.swift:423:43: error: cannot find type 'Task' in scope
                        func parentDirectoryTask(for: File) -> Task<Void, Never>? {
                                                               ^~~~
unxip.swift:547:21: error: '_Concurrency' module not imported, required for async main
        static func main() async throws {
                           ^

Normal to see "File exists" in output?

When I run unxip, I see my terminal spammed with

creating directory at ./Xcode.app/Contents/Developer/Applications/Xcode Server Builder.app/Contents/MacOS failed: File exists
creating directory at ./Xcode.app/Contents/Developer/Applications/Xcode Server Builder.app/Contents/XPCServices/XCSPortalService.xpc/Contents/_CodeSignature failed: File exists

This normal?

Kernel panic during unxip on 12.4 Beta

I'm trying to unxip Xcode_13.3.1.xip. It's my first time using unxip, but three times in a row I've had a panic 10–30 seconds into the process.

I'm on 12.4 Beta (21F5048e), maybe it's a bug with that given it's attributed to AirPlayXPCHelper. I'm not really familiar with panic logs as much as crash logs, I don't know where to start looking into this.

I have plenty of disk space, I'm unxip-ing from my ~/Downloads, I have a core dump I can send you somehow if that's useful.

Kernel-2022-04-14-114904.panic

panic(cpu 4 caller 0xffffff801c5d2943): Kernel trap at 0xffffff801c8b53ce, type 14=page fault, registers:
CR0: 0x000000008001003b, CR2: 0x0000000000000028, CR3: 0x000000046507802b, CR4: 0x00000000003626e0
RAX: 0x0000000000000000, RBX: 0xffffff870b7c9aa0, RCX: 0x0000000000000000, RDX: 0x00000000fbfee027
RSP: 0xffffffe5bc4b3d70, RBP: 0xffffffe5bc4b3dc0, RSI: 0xffffff8bd60cad10, RDI: 0x0000000000000000
R8:  0x0000000000000000, R9:  0x0000000000003d46, R10: 0x0000000000000002, R11: 0xffffff90a426e000
R12: 0xffffff801cda2395, R13: 0xffffff9a3a21cf01, R14: 0xffffff870b7c9e68, R15: 0xffffff7f7fffffff
RFL: 0x0000000000010282, RIP: 0xffffff801c8b53ce, CS:  0x0000000000000008, SS:  0x0000000000000000
Fault CR2: 0x0000000000000028, Error code: 0x0000000000000002, Fault CPU: 0x4, PL: 0, VF: 0

Panicked task 0xffffff956ea65090: 5 threads: pid 164: AirPlayXPCHelper
Backtrace (CPU 4), panicked thread: 0xffffff90a426e000, Frame : Return Address
0xffffffe5bc4b3720 : 0xffffff801c483efd 
0xffffffe5bc4b3770 : 0xffffff801c5e3186 
0xffffffe5bc4b37b0 : 0xffffff801c5d255d 
0xffffffe5bc4b3800 : 0xffffff801c423a60 
0xffffffe5bc4b3820 : 0xffffff801c4842cd 
0xffffffe5bc4b3940 : 0xffffff801c483a86 
0xffffffe5bc4b39a0 : 0xffffff801cd1580d 
0xffffffe5bc4b3a90 : 0xffffff801c5d2943 
0xffffffe5bc4b3c10 : 0xffffff801c5d2632 
0xffffffe5bc4b3c60 : 0xffffff801c423a60 
0xffffffe5bc4b3c80 : 0xffffff801c8b53ce 
0xffffffe5bc4b3dc0 : 0xffffff801cab870a 
0xffffffe5bc4b3e10 : 0xffffff801cab92eb 
0xffffffe5bc4b3e30 : 0xffffff801ca21f13 
0xffffffe5bc4b3ea0 : 0xffffff801ca229b5 
0xffffffe5bc4b3f40 : 0xffffff801cb8a72b 
0xffffffe5bc4b3fa0 : 0xffffff801c424226 

Process name corresponding to current thread (0xffffff90a426e000): AirPlayXPCHelper

Mac OS version:
21F5048e

Kernel version:
Darwin Kernel Version 21.5.0: Mon Mar 28 19:53:00 PDT 2022; root:xnu-8020.120.43.111.1~1/RELEASE_X86_64
Kernel UUID: ACC5F23C-A2E0-3E73-86C9-38119B418C2F
KernelCache slide: 0x000000001c200000
KernelCache base:  0xffffff801c400000
Kernel slide:      0x000000001c210000
Kernel text base:  0xffffff801c410000
__HIB  text base: 0xffffff801c300000
System model name: MacBookPro14,3 (Mac-551B86E5744E2388)
System shutdown begun: NO
Panic diags file available: YES (0x0)
Hibernation exit count: 0

System uptime in nanoseconds: 806517166554
Last Sleep:           absolute           base_tsc          base_nano
  Uptime  : 0x000000bbc82b5505
  Sleep   : 0x0000000000000000 0x0000000000000000 0x0000000000000000
  Wake    : 0x0000000000000000 0x00000009d63b8f14 0x0000000000000000
Compressor Info: 0% of compressed pages limit (OK) and 0% of segments limit (OK) with 0 swapfiles and OK swap space
Zone info:
  Zone map: 0xffffff80a10d8000 - 0xffffffa0a10d8000
  . PGZ   : 0xffffff80a10d8000 - 0xffffff80a30d9000
  . VM    : 0xffffff80a30d9000 - 0xffffff856f8d8000
  . RO    : 0xffffff856f8d8000 - 0xffffff87090d8000
  . GEN0  : 0xffffff87090d8000 - 0xffffff8bd58d8000
  . GEN1  : 0xffffff8bd58d8000 - 0xffffff90a20d8000
  . GEN2  : 0xffffff90a20d8000 - 0xffffff956e8d8000
  . GEN3  : 0xffffff956e8d8000 - 0xffffff9a3b0d8000
  . DATA  : 0xffffff9a3b0d8000 - 0xffffffa0a10d8000
  Metadata: 0xffffffe586172000 - 0xffffffe5a6172000
  Bitmaps : 0xffffffe5a6172000 - 0xffffffe5ac172000

last started kext at 51006612420: >driverkit.serial	6.0.0 (addr 0xffffff7fb56b5000, size 28672)
loaded kexts:
com.intel.driver.EnergyDriver	3.5.5
>AudioAUUC	1.70
>X86PlatformShim	1.0.0
>AGPM	128
>!APlatformEnabler	2.7.0d0
@filesystems.autofs	3.0
>!AHIDALSService	1
@kext.AMDFramebuffer	4.0.8
>!AUpstreamUserClient	3.6.9
@kext.AMDRadeonX4000	4.0.8
@kext.AMDRadeonServiceManager	4.0.8
@UVCService	1
>!A!IPCHPMC	2.0.1
@kext.AMD9500!C	4.0.8
>!AHDAHardwareConfigDriver	340.2
>!AGraphicsDevicePolicy	6.5.7
>!AHDA	340.2
>!A!IKBLGraphics	18.0.7
@AGDCPluginDisplayMetrics	6.5.7
>SMCMotionSensor	3.0.4d1
>pmtelemetry	1
|IOUserEthernet	1.0.1
>usb.!UUserHCI	1
>!AHV	1
>!ADiskImages2	126.100.13
>eficheck	1
>!AGFXHDA	140.3
>AGDCBacklightControl	6.5.7
>!AMuxControl	6.5.7
>!AEmbeddedOSSupportHost	1
>!AFIVRDriver	4.1.0
>!ABacklight	180.7
>ACPI_SMC_PlatformPlugin	1.0.0
>!A!IKBLGraphicsFramebuffer	18.0.7
>!AThunderboltIP	4.0.3
>!AMCCSControl	1.16
>!A!ISlowAdaptiveClocking	4.0.0
>!ATopCaseHIDEventDriver	5450.2
>!AFileSystemDriver	3.0.1
@filesystems.tmpfs	1
@filesystems.lifs	1
@filesystems.hfs.kext	583.100.10
@BootCache	40
@!AFSCompression.!AFSCompressionTypeZlib	1.0.0
@!AFSCompression.!AFSCompressionTypeDataless	1.0.0d1
@filesystems.apfs	1934.120.3
>AirPort.BrcmNIC	1400.1.1
@private.KextAudit	1.0
>!ASmartBatteryManager	161.0.0
>!AACPIButtons	6.1
>!ARTC	2.0.1
>!ASMBIOS	2.1
>!AACPIEC	6.1
>!AAPIC	1.7
@!ASystemPolicy	2.0.0
@nke.applicationfirewall	402
|IOKitRegistryCompatibility	1
|EndpointSecurity	1
@Dont_Steal_Mac_OS_X	7.0.0
@kec.!AEncryptedArchive	1
>driverkit.serial	6.0.0
@kext.triggers	1.0
@kext.AMDRadeonX4100HWLibs	1.0
@kext.AMDRadeonX4000HWServices	4.0.8
>usb.IOUSBHostHIDDevice	1.2
>!ASMBusPCI	1.0.14d1
@kext.AMDSupport	4.0.8
>DspFuncLib	340.2
@kext.OSvKernDSPLib	529
|IO!BSerialManager	9.0.0
|IO!BPacketLogger	9.0.0
|IO!BHost!CUSBTransport	9.0.0
|IO!BHost!CUARTTransport	9.0.0
|IO!BHost!CTransport	9.0.0
>IO!BHost!CPCIeTransport	9.0.0
|IOAVB!F	1040.6
@plugin.IOgPTPPlugin	1040.3
|IOEthernetAVB!C	1.1.0
|CSR!BHost!CUSBTransport	9.0.0
|Broadcom!BHost!CUSBTransport	9.0.0
|Broadcom!B20703USBTransport	9.0.0
>!AIPAppender	1.0
>!A!ILpssUARTv1	3.0.60
>!A!ILpssUARTCommon	3.0.60
>!AOnboardSerial	1.0
|IOSerial!F	11
>!AHDA!C	340.2
|IOHDA!F	340.2
|IOAudio!F	340.2
@vecLib.kext	1.2.0
>!AGraphicsControl	6.5.7
>!ABacklightExpert	1.1.0
|IONDRVSupport	597
>IOPlatformPluginLegacy	1.0.0
>X86PlatformPlugin	1.0.0
>IOPlatformPlugin!F	6.0.0d8
|IOAccelerator!F2	462.8
>!AThunderboltEDMSink	5.0.3
>!AThunderboltDPOutAdapter	8.5.1
>!ASMBus!C	1.0.18d1
@!AGPUWrangler	6.5.7
@!AGraphicsDeviceControl	6.5.7
|IOGraphics!F	597
|IOSlowAdaptiveClocking!F	1.0.0
>usb.cdc.ecm	5.0.0
>usb.cdc.ncm	5.0.0
>usb.!UiBridge	1.0
>usb.cdc	5.0.0
>usb.networking	5.0.0
>usb.!UHostCompositeDevice	1.2
>!AActuatorDriver	5450.3
>!AHIDKeyboard	228.2
>!AMultitouchDriver	5450.3
>!AInputDeviceSupport	5450.2
>!AHS!BDriver	5450.2
>IO!BHIDDriver	9.0.0
>!AHSSPIHIDDriver	63
>!AThunderboltDPInAdapter	8.5.1
>!AThunderboltDPAdapter!F	8.5.1
>!AThunderboltPCIDownAdapter	4.1.1
>!ABSDKextStarter	3
|IOSurface	302.14
@filesystems.hfs.encodings.kext	1
>!AXsanScheme	3
>!AThunderboltNHI	7.2.81
|IONVMe!F	2.1.0
>!AHSSPISupport	63
|IO80211!FLegacy	1200.12.2b1
|IOSkywalk!F	1.0
>mDNSOffloadUserClient	1.0.1b8
>corecapture	1.0.4
>!A!ILpssSpi!C	3.0.60
>!AHPM	3.4.4
|IOThunderbolt!F	9.3.3
>!A!ILpssI2C!C	3.0.60
>!A!ILpssDmac	3.0.60
>!A!ILpssI2C	3.0.60
>!A!ILpssGspi	3.0.60
>usb.!UXHCIPCI	1.2
>usb.!UXHCI	1.2
>usb.!UHostPacketFilter	1.0
|IOUSB!F	900.4.2
>!AEFINVRAM	2.1
>!AEFIRuntime	2.1
|IOSMBus!F	1.1
|IOHID!F	2.0.0
|IOTimeSync!F	1040.3
|IONetworking!F	3.4
>DiskImages	493.0.0
|IO!B!F	9.0.0
|IOReport!F	47
$quarantine	4
$sandbox	300.0
@kext.!AMatch	1.0.0d1
|CoreAnalytics!F	1
>!ASSE	1.0
>!AKeyStore	2
>!UTDM	533.120.2
|IOUSBMass!SDriver	210.120.3
|IOSCSIBlockCommandsDevice	456.100.7
|IO!S!F	2.1
|IOSCSIArchitectureModel!F	456.100.7
>!AMobileFileIntegrity	1.0.5
$!AImage4	4.2.0
@kext.CoreTrust	1
>!AFDEKeyStore	28.30
>!AEffaceable!S	1.0
>!ACredentialManager	1.0
>KernelRelayHost	1
|IOUSBHost!F	1.2
>!UHostMergeProperties	1.2
>usb.!UCommon	1.0
>!ABusPower!C	1.0
>!ASEPManager	1.0.1
>IOSlaveProcessor	1
>!AACPIPlatform	6.1
>!ASMC	3.1.9
|IOPCI!F	2.9
|IOACPI!F	1.4
>watchdog	1
@kec.pthread	1
@kec.Libm	1
@kec.corecrypto	12.0

header error how to fix

swiftc -parse-as-library -O unxip.swift

/usr/local/include/module.modulemap:15:9: error: header 'compression.h' not found
header "compression.h"
^
:0: error: could not build Objective-C module 'Compression'

Corrupted unpacking of Xcode with unxip 1.1.2

When I unpack Xcode 14.1 RC 2, the resulting application bundle is corrupt and won't launch. When I use /usr/bin/xip it works fine.

I'm not sure what the issue is, but right off I know Xcode.app/Contents/Info.plist is corrupted.

Unpacked with 'xip':

$ sha256sum ./Xcode.app/Contents/Info.plist
847d9996dbdbae2916775368c26e3fe3c2a34eb0e738a700368f6f637fb4608c  ./Xcode.app/Contents/Info.plist

Unpacked with unxip 1.1.2:

$ sha256sum /Applications/Xcode.app/Contents/Info.plist
b798bf5b5850661c83a72022a13af0e62d71679b09485b4f95b8e66e2f4e5e25  /Applications/Xcode.app/Contents/Info.plist

$ plutil -convert xml1 -o - Xcode.app/Contents/Info.plist
Xcode.app/Contents/Info.plist: Property List error: Unexpected character b at line 1 / JSON error: JSON text did not start with array or object and option to allow fragments not set. around line 1, column 0.

unxip hash:

$ sha256sum "$(which unxip)"
253b5d2e4602cd52eef1c01d5280802d72c8a55cdc7711ad9a16dd543a84d19c  /opt/homebrew/bin/unxip

Xcode hash:

$ sha256sum Xcode_14.1_Release_Candidate_2.xip
12f8a3aef78bf354470ad8b351addd925c8edad888137d138ca50a8130eb9f2f  Xcode_14.1_Release_Candidate_2.xip

macOS version:

$ sw_vers
ProductName:		macOS
ProductVersion:		13.0
BuildVersion:		22A380

I've attached the corrupted Info.plist (zipped because GitHub doesn't accept .plist)

Let me know if I can provide anything else

Info.plist.zip

Separate core extraction logic from CLI tool

It would be great if the code was modularized so that one could only use the extraction logic without the CLI stuff (main, error printing, ...) around. This would enable tools like https://github.com/xcodereleases/xcinfo to use it.

Remark while installing

Hi. This is what it's showing me when I'm trying to install it.

<unknown>:0: remark: did not find a prebuilt standard library for target 'x86_64-apple-macos' compatible with this Swift compiler; building it may take a few minutes, but it should only happen once for this combination of compiler and target

error: type 'Void' does not conform to protocol 'Sendable'

unxip cannot be compiled

nick@nick-pc ~ % swiftc -parse-as-library -O unxip.swift 
2022-03-20 15:32:40.618 xcodebuild[16798:122541] XType: com.apple.fonts is not accessible.
2022-03-20 15:32:40.619 xcodebuild[16798:122541] XType: XTFontStaticRegistry is enabled.
unxip.swift:405:20: error: type 'Void' does not conform to protocol 'Sendable'
                var taskStream = ConcurrentStream<Void>(batchSize: 64)  // Worst case, should allow for files up to 64 * 16MB = 1GB

on macOS Monterey x86_64, 5GB RAM

Excessive memory usage

Hi,
I just wanted to let you know that this script, while it seems to be faster than Apple's implementation, used 80GB of RAM out of 32 i have installed, throwing more than half into swap, obviously. Is there a way to limit RAM usage to 75% of availbale memory, so it only rarely goes into swap (i believe macOS starts to use swap when 75% memory usage is reached)? Swapping slows down the process a lot too, so this would probably improve performance

saagarjha / unxip Goto Github PK

unxip's People

Contributors

Stargazers

Watchers

Forkers

unxip's Issues

Recommend Projects

Recommend Topics

Recommend Org