lovell / 64 Goto Github PK

High performance Base64 encoding and decoding for Node.js using SIMD (AVX2, AVX, SSE4.2, SSE4.1, SSSE3) acceleration

License: Apache License 2.0

Python 28.40% JavaScript 34.90% C++ 20.32% HTML 16.38%

base64 nodejs encode decode simd

64's Introduction

64

High performance Base64 encoding and decoding for Node.js using SIMD (AVX2, AVX, SSE4.2, SSE4.1, SSSE3) acceleration. Uses Alfred Klomp's base64 library.

When compared with Base64 encoding/decoding via Node's Buffer object, expected performance gains are up to ~6x depending on length.

Due to the cost of making a shared library call, it is less suitable for very short Buffers, typically those less than ~300 bytes when encoding and less than ~150 bytes when decoding.

Pre-compiled binaries are provided for Node 6, 8 and 10 on the most common platforms.

Requirements

Node.js v6+
Linux, OS X or Windows
x86 or x64 CPU

Install

npm install 64

Usage

import { encode, decode } from '64';

const input = Buffer.from('The quick brown fox jumped over the lazy sleeping dog');

const encoded = encode(input); // VGhlIHF1aWNrIGJyb3duIGZveCBqdW1wZWQgb3ZlciB0aGUgbGF6eSBzbGVlcGluZyBkb2c=
const decoded = decode(encoded);

API

encode(input)

input is a Buffer containing the data to be Base64 encoded.

Returns a Buffer containing the Base64 encoded version of input.

A TypeError will be thrown if input is not a Buffer.

decode(encoded)

encoded is a Buffer containing Base64 encoded data.

Returns a Buffer containing the Base64 decoded version of encoded.

A TypeError will be thrown if encoded is not a Buffer.

Performance

Intel i3-4170
Ubuntu 16.04.3 LTS
Node.js v8.1.3

Licence

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

 http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

64's People

Contributors

Stargazers

Watchers

Forkers

tuananh evaluation-alex icyberon nicknaso geek jdrew1303 cakoose jradonich rk-forfuns olvresc y21 jqk6 gqqnbig suryatmodulus der-ofenmeister

64's Issues

returns buffer on AWS Lambda

I'm doing npm install with docker run -v "$PWD":/var/task lambci/lambda:build-nodejs8.10 npm install

const base64 = require('64');

const responseFromBuffer = (buffer, { cacheTag, eTag }) => ({
	headers: {
		'Content-Type': 'image/jpeg',
	},
	isBase64Encoded: true,
	body: base64.encode(buffer),
});
...
...

The lambda logs show the output of body to be

"body":{"type":"Buffer","data":[105,86,66,79,82,119,48...

N-API support

Hi @lovell ,
I worked on this native addon and I ported it to N-API
You can find my work here: https://github.com/NickNaso/64/tree/napi
If you want experiment with N-API you can create a branch and call it "napi" so I can execute the PR on it.
After that we can iterate to over it and at the end publish a tagged version of 64 like reported here: https://nodejs.org/en/docs/guides/publishing-napi-modules/

Enhancement: Add support for Windows

Enhancement: Add support for ARM CPUs

OS X slower than Linux, slower than native for short strings

I was benchmarking this library on my laptop and it is more than 2 times slower than native implementation. Is the testing code wrong or the benchmark in readme is wrong.

HW
MacBook Pro (15-inch, 2016)
2,7 GHz Intel Core i7
16 GB 2133 MHz LPDDR3

Code

var now = require('performance-now')
var b64 = require('64')

function run (fn, desc) {
  var start = now()
  for (var i = 0; i < 1000000; i++) {
    fn()
  }
  var end = now()
  console.log(desc, (end - start).toFixed(2), 'ms')
}

console.log('NODE', process.version)
run(() => b64.decode(Buffer.from(b64.encode(Buffer.from('test string')), 'base64')), '64')
run(() => Buffer.from(Buffer.from('test string').toString('base64'), 'base64').toString('ascii'), 'native')

Results

NODE v8.1.4
64     2304.52 ms
native 1058.34 ms

NODE v7.6.0
64     2644.51 ms
native 1180.36 ms

NODE v6.10.2
64     2964.18 ms
native 1373.59 ms

64 module compiled against a different Node.js

Hi,

I'm having an issue to ship the 64 module with my NodeJS standalone app on macOS High Sierra v10.13.6.

$ node --version
v15.1.0

$ npm --version
7.0.8

$ npm install
[...]
all good

## package the app as a binary
$ make pkg

## test it
$ ./build/flash
pkg/prelude/bootstrap.js:1244
      throw error;
      ^

Error: The module '/Users/micka/flash/node_src/node_modules/64/build/Release/base64.node'
was compiled against a different Node.js version using
NODE_MODULE_VERSION 88. This version of Node.js requires
NODE_MODULE_VERSION 64. Please try re-compiling or re-installing
the module (for instance, using `npm rebuild` or `npm install`).
    at Object.Module._extensions..node (internal/modules/cjs/loader.js:805:18)
    at Module.load (internal/modules/cjs/loader.js:651:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:591:12)
    at Function.Module._load (internal/modules/cjs/loader.js:583:3)
    at Module.require (internal/modules/cjs/loader.js:690:17)
    at Module.require (pkg/prelude/bootstrap.js:1225:31)
    at require (internal/modules/cjs/helpers.js:25:18)
    at Object.<anonymous> (/snapshot/node_src/node_modules/64/index.js:3:16)
    at Module._compile (pkg/prelude/bootstrap.js:1320:22)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)

That being said, if i run my app like this, there's no issue at all:

$ node ./flash.js

I even tried to recompile from source (npm install --build-from-source), but I got the same error.

Help appreciated.