Code Monkey home page Code Monkey logo

archive's Introduction

Archive

A port of libarchive to WebAssembly, with a simple JS wrapper.

It is highly recommended that you offload reading and writing to a Worker. For the browser or deno, that'd be through Worker and for node that would be through the worker_threads module. There isn't an official worker wrapper yet, but using a library like Comlink can get you up and running with one in minutes.

Install

# if you use yarn:
yarn add @magnetardev/archive

# if you use npm:
npm install @magnetardev/archive

Why?

Aren't there a few of these ports? Well, yes. In fact, this uses the exact same Dockerfile from libarchivejs. However, libarchivejs does not support Node and forces you to use a Worker. I also haven't found one that supports creating archives. Finally, the APIs most of the ports offer are not as simple as they can be, which is a shame. So, I made this.

Examples

Reading an archive

import * as archive from '@magnetardev/archive';

const buffer = ...; // Obtain a Uint8Array version of your archive.
const reader = await archive.createReader(buffer);
for (const entry of reader) {
  console.log(entry.path, entry.extract());
}
reader.close();

Creating an archive

import * as archive from "@magnetardev/archive";

// Create a writer.
const writer = await archive.createWriter();

// Add a file
const contents = new TextEncoder().encode("hello, world!");
writer.add("hello.txt", contents);

// Finalize the archive and get a Uint8Array version.
const zip = writer.close();
console.log(zip);

Building

Building is kind of a mess, but is quite easy with Docker. If you choose to do so without Docker, you need to compile OpenSSL, LZMA, and libarchive with emcc. This short guide covers the Docker build process:

  1. Run yarn build:wasm. This can take quite a while the first time, but consecutive builds will be much faster. This builds just the WebAssembly port and the Emscripten wrapper.
  2. Run yarn build. This builds the JS wrapper.
  3. Done.

archive's People

Contributors

magnetardev avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

archive's Issues

Use Jest for Testing

Right now it's just "seeing if it works." That's not very optimal. We should use jest instead to make debugging easier and faster. Testing for browsers is also an option (via playwright or puppeteer), but is probably redundant.

[Docs] Needs proper documentation

There's a few hurdles when it comes to using the lib right now, namely the locateFile stuff. Should be documented, along with all the functions and constants.

Support More Compression Formats in Writer

Currently, the writer only supports zip files. Which should be good enough for most people, but different use cases exist. We should at least support writing 7z, rar, and tar archives. Ideally, we should support as many formats as libarchive supports.

Adding a format is as easy as:

  • Adding another string to ArchiveType in src/writer.ts
  • Checking the format in archive_write_memory in lib/main.c and applying the right functions to the archive.

It's just a matter of knowing the right functions more than anything.

This issue will be closed once the following formats are supported:

  • 7z
  • rar (libarchive can only read, not write, RARs)
  • tar

Update Dockerfile

Right now the Dockerfile uses trzeci/emscripten, which is fine, but that uses Emscripten v1.39.4. Emscripten is currently on v2.0.16, so needless to say it should be updated. We should use emscripten/emsdk instead. From my attempts at this, it's not the easiest thing to do. The closest I got was compiling up to libarchive, but it was missing bz2 symbols, disabling bz2 support should be avoided.

Asyncify Breaks Reading

In an attempt at implementing #10, reading the next entry fails.

I'm not knowledgable with Asyncify, but will look into more.

[Tests] Write tests!

  • Both for node & browsers (Playwright)
  • All compression formats
  • Decompression
  • Test password protection

Create Worker Wrapper

Workers are good, and the JS community needs to normalize using them. Offloading reading and writing archives can be intensive tasks and should not be ran on the main thread. The worker versions should be prioritized, but you should be able to import the wrapper directly if you need it. This would be incredibly simple if we use Comlink, but we should try to minimize dependencies. The API should be essentially identical to what we have now.

usage functions for OpenSSL, LZMA, and libarchive for nodejs and the browser

I am looking for usage functions for OpenSSL, LZMA, and libarchive for nodejs and the browser

  • First, Is this archive going to be maintained with appropriate releases? I am contemplating to use it for nodejs and for the browser.
  • Second, I am intending to use it for OpenSSL, LZMA, and libarchive function as well. Any demos?

Streamable Reading

To some extent, reading a compressed file should be streamed. Right now the lib loads the entire archive into memory, which is... bad. Obviously some compression formats cannot be streamed, though that isn't as big an issue as buffered read.

In theory, libarchive supports this already, by specifying a custom read function: List contents of Archive with custom read functions

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.