myfreeer / chrome-pak-customizer Goto Github PK

https://github.com/myfreeer/chrome-pak-customizer/tree/3.x

• Readme
• Changelog
• Updated guide
• CI-driven release
• Switch off appveyor build
• Statistics output like 2.x
• More unit tests

subj, where to find pack_mingw64.exe

Target: branch 3.x

Description

Currently chrome-pak-customizer reads the whole pak file for unpacking, while allocating all the memory needed of pak file for packing. On early stage of this project hard disk drive (HDD) is widely used, which good at sequential reading or writing while has a slow random access speed, so the tradeoff is to read and write the whole pak file and then operate it in memory.
Nowadays solid-state drive (SSD) is taking over, which could have much better random access speed than HDDs, so maybe it is time for making the tradeoff optional, using mmap or MapViewOfFile can create a view of file managed by operating system, so the file is automatically read and written on reading and writing memory, which could save memory or make better performance in some cases.

Proposal

• Add an command-line parameter to use mmap for reading and writing pak files whenever available.
• Print availablity of mmaping for help.
• Print a warning if the parameter is on and mmap is unavailable.
• Use mmap for reading and writing pak files if the parameter is on and mmap is available by OS.
• Unsafe could be used if needed.

Non-Goals

• It is not a goal to implement this on branch develop or master
• It is not a goal to make the parameter on by default.
• It is not a goal to implement block-based reading and writing for pak files.
• It is not a goal to use mmap for non-pak files, such as contents inside pak files and pak-index.ini

Hi,

I got an index out of range error when I tried to unpack a chrome string resource (see attachment):

C:\Users\Günter\Documents\Raccoon\com.android.chrome_65.0.3325.109-332510901_minAPI21(armeabi-v7a)(nodpi)\full-package-win32-and-win64>unpack
node-chrome-pak
Written by Hwang, C. W. ([email protected])

unpacking at C:\Users\Günter\Documents\Raccoon\com.android.chrome_65.0.3325.109-332510901_minAPI21(armeabi-v7a)(nodpi)\full-package-win32-and-win64\unpacked
buffer.js:830
throw new RangeError('Index out of range');
^
RangeError: Index out of range
at checkOffset (buffer.js:830:11)
at Buffer.readUInt32BE (buffer.js:904:5)
at unpack_proc (C:\Users\Günter\Documents\Raccoon\com.android.chrome_65.0.3325.109-332510901_minAPI21(armeabi-v7a)(nodpi)\full-package-win32-and-win64\node-chrome-pak.js:159:21)
at Object. (C:\Users\Günter\Documents\Raccoon\com.android.chrome_65.0.3325.109-332510901_minAPI21(armeabi-v7a)(nodpi)\full-package-win32-and-win64\node-chrome-pak.js:36:5)
at Module._compile (module.js:541:32)
at Object.Module._extensions..js (module.js:550:10)
at Module.load (module.js:458:32)
at tryModuleLoad (module.js:417:12)
at Function.Module._load (module.js:409:3)
at Function.Module.runMain (module.js:575:10)

Gewartet wird 1 Sekunden. Weiter mit beliebiger Taste...

C:\Users\Günter\Documents\Raccoon\com.android.chrome_65.0.3325.109-332510901_minAPI21(armeabi-v7a)(nodpi)\full-package-win32-and-win64>

pak file:
de.zip

Cancel

subj

Hello, I make my own chromium fork for linux, win, and macos called Thorium > https://github.com/Alex313031/Thorium/
I recently compiled pak for linux, and downloaded the windows releases, and am including them in the installers of Thorium, with a credit to you in the readme. I include it because Thorium is designed to be dev friendly, with a new UI for the reload button that includes "hard reload" and "empty cache and hard reload", and the inclusion of chromedriver, content_shell, and now pak in the installer to ease debugging, testing, and web development.

This link > https://github.com/shuax/ChromePAK in your readme needs to be updated to > https://github.com/shuax/ChromePAK-V5

On the note of that repo, Is yours up to date with his code? His latest commit is a year newer. Maybe his has a fix for the brotli thing in it already?

About the recent brotli thing, is this likely to affect someone who just wants to unpack a .pak in Thorium just to see the contents?

subj

this should have macOS binaries available, since not everyone has the tools to build it.
i built one myself but it may be out of date, but it does work on catalina
pak.zip

A while back, Chromium started using Brotli compression on textual resources. It doesn't look like the unpacker handles that?

If I strip 8 bytes off the front of the file extracted from unpack.bat, I can then use brotli.exe to uncompress the resource.

Is there an index or a map that contains the file locations or file names within a built version of chromium? If so where would it be located and how would it be encoded?

I'm not nearly smart enough to sift through chromium's source, but, from what I found in i18n-grit, no such index is contained within the pak file itself. Right now I'm assuming the index is kept within an executable file or some other binary.

Attempted compile on Linux 64 bit (Mageia release 6), with cmake v3.7.2 and ninja v1.7.2
cmake completed without errors displayed
The following is the output during compile with ninja.
(The constant MAX_PATH used was not defined when first used.
There are a number of other errors.)

build $ ninja
[1/3] Building C object CMakeFiles/pak.dir/main.c.o
FAILED: CMakeFiles/pak.dir/main.c.o
/usr/bin/cc -O3 -Os -pipe -Wall -Wextra -fmerge-all-constants -Wl,--gc-sections,--build-id=none -s -MD -MT CMakeFiles/pak.dir/main.c.o -MF CMakeFiles/pak.dir/main.c.o.d -o CMakeFiles/pak.dir/main.c.o -c /pdev/dev/chrome-pak-customizer/main.c
/pdev/dev/chrome-pak-customizer/main.c: Dans la fonction ‘printHelp’:
/pdev/dev/chrome-pak-customizer/main.c:21:19: erreur : ‘MAX_PATH’ undeclared (first use in this function)
char selfName[MAX_PATH];
^
/pdev/dev/chrome-pak-customizer/main.c:21:19: note : each undeclared identifier is reported only once for each function it appears in
/pdev/dev/chrome-pak-customizer/main.c:22:5: attention : implicit declaration of function ‘GetModuleFileName’ [-Wimplicit-function-declaration]
GetModuleFileName(NULL, selfName, MAX_PATH);
^
/pdev/dev/chrome-pak-customizer/main.c:21:10: attention : unused variable ‘selfName’ [-Wunused-variable]
char selfName[MAX_PATH];
^
/pdev/dev/chrome-pak-customizer/main.c: Dans la fonction ‘main’:
/pdev/dev/chrome-pak-customizer/main.c:124:16: erreur : ‘PATH_MAX’ undeclared (first use in this function)
char path1[PATH_MAX];
^
/pdev/dev/chrome-pak-customizer/main.c:126:10: attention : unused variable ‘path2’ [-Wunused-variable]
char path2[PATH_MAX];
^
/pdev/dev/chrome-pak-customizer/main.c:124:10: attention : unused variable ‘path1’ [-Wunused-variable]
char path1[PATH_MAX];
^
[2/3] Building C object CMakeFiles/pak.dir/pak_pack.c.o
FAILED: CMakeFiles/pak.dir/pak_pack.c.o
/usr/bin/cc -O3 -Os -pipe -Wall -Wextra -fmerge-all-constants -Wl,--gc-sections,--build-id=none -s -MD -MT CMakeFiles/pak.dir/pak_pack.c.o -MF CMakeFiles/pak.dir/pak_pack.c.o.d -o CMakeFiles/pak.dir/pak_pack.c.o -c /pdev/dev/chrome-pak-customizer/pak_pack.c
/pdev/dev/chrome-pak-customizer/pak_pack.c: Dans la fonction ‘pakUnpack’:
/pdev/dev/chrome-pak-customizer/pak_pack.c:14:18: erreur : ‘PATH_MAX’ undeclared (first use in this function)
char pathBuf[PATH_MAX];
^
/pdev/dev/chrome-pak-customizer/pak_pack.c:14:18: note : each undeclared identifier is reported only once for each function it appears in
In file included from /pdev/dev/chrome-pak-customizer/pak_pack.h:16:0,
from /pdev/dev/chrome-pak-customizer/pak_pack.c:1:
/pdev/dev/chrome-pak-customizer/pak_defs.h:86:30: attention : format ‘%u’ expects argument of type ‘unsigned int’, but argument 3 has type ‘uint_fast32_t {alias long unsigned int}’ [-Wformat=]
#define PAK_INDEX_GLOBAL_TAG "[Global]"
^
/pdev/dev/chrome-pak-customizer/pak_pack.c:30:39: note : in expansion of macro ‘PAK_INDEX_GLOBAL_TAG’
sprintf(pakIndexStr + offset, PAK_INDEX_GLOBAL_TAG "\r\nversion=%u\r\n",
^
/pdev/dev/chrome-pak-customizer/pak_pack.c:36:30: attention : format ‘%u’ expects argument of type ‘unsigned int’, but argument 3 has type ‘uint_fast16_t {alias long unsigned int}’ [-Wformat=]
sprintf(fileNameBuf, "%u%s", files[i].id, pakGetFileType(files[i]));
^
/pdev/dev/chrome-pak-customizer/pak_pack.c:37:49: attention : format ‘%u’ expects argument of type ‘unsigned int’, but argument 3 has type ‘uint_fast16_t {alias long unsigned int}’ [-Wformat=]
offset += sprintf(pakIndexStr + offset, "%u=%s\r\n", files[i].id,
^
/pdev/dev/chrome-pak-customizer/pak_pack.c:14:10: attention : unused variable ‘pathBuf’ [-Wunused-variable]
char pathBuf[PATH_MAX];
^
/pdev/dev/chrome-pak-customizer/pak_pack.c: Dans la fonction ‘pakPack’:
/pdev/dev/chrome-pak-customizer/pak_pack.c:93:34: attention : format ‘%u’ expects argument of type ‘unsigned int *’, but argument 3 has type ‘uint_fast32_t * {alias long unsigned int *}’ [-Wformat=]
sscanf(pakIndexBuf + offset, " version=%u%n", &myHeader.version, &count);
^
/pdev/dev/chrome-pak-customizer/pak_pack.c:134:18: erreur : ‘PATH_MAX’ undeclared (first use in this function)
char pathBuf[PATH_MAX];
^
/pdev/dev/chrome-pak-customizer/pak_pack.c:180:12: attention : format ‘%u’ expects argument of type ‘unsigned int’, but argument 2 has type ‘uint_fast32_t {alias long unsigned int}’ [-Wformat=]
printf("\nresource_count=%u\nalias_count=%u\n", myHeader.resource_count,
^
/pdev/dev/chrome-pak-customizer/pak_pack.c:180:12: attention : format ‘%u’ expects argument of type ‘unsigned int’, but argument 3 has type ‘uint_fast16_t {alias long unsigned int}’ [-Wformat=]
/pdev/dev/chrome-pak-customizer/pak_pack.c:182:12: attention : format ‘%u’ expects argument of type ‘unsigned int’, but argument 2 has type ‘uint_fast32_t {alias long unsigned int}’ [-Wformat=]
printf("version=%u\nencoding=%u\n", myHeader.version, myHeader.encoding);
^
/pdev/dev/chrome-pak-customizer/pak_pack.c:183:12: attention : format ‘%u’ expects argument of type ‘unsigned int’, but argument 2 has type ‘uint_fast32_t {alias long unsigned int}’ [-Wformat=]
printf("\npak size: %u\n", pakFile.size);
^
/pdev/dev/chrome-pak-customizer/pak_pack.c:134:10: attention : unused variable ‘pathBuf’ [-Wunused-variable]
char pathBuf[PATH_MAX];
^
ninja: build stopped: subcommand failed.

Chromium started using a custom variant or brotli compression to compress resources inside pak file. Unpacking this would be simple, detect the binary prefix and strip 8 bytes makes it compatible with brotli utils, but packing would be difficult even if with brotli-compressed file.

The prefix contains uncompressed size of brotli stream, which, according to official issue, can not be obtained without uncompressing it, but incroducing a brotli uncompressor would make binary size of the program many times larger than current.

Binary size matters from the very beginning of the C version of this tool, but since it's years from last release, things might change.

Click on the upper right corner for perfering to keep its current status and small binary size.
Click on the upper right corner for perfering to introduce the feature even if binary size could be more than 1MB.

Hello, I have made some changes to pak, like using -O3, and adding color to --help, etc.
I compiled for linux, but now I want to make an updated windows binary. The readme and wiki only has instructions for linux, except the LGPL 2.1 caveat.

How would I compile pak for windows?