fuse-friends / fuse-native Goto Github PK

View Code? Open in Web Editor NEW

200.0 200.0 29.0 257 KB

Multithreaded FUSE bindings for Node JS.

C 42.63% Python 0.99% JavaScript 56.38%

fuse-native's People

Contributors

Stargazers

Watchers

fuse-native's Issues

Not working with electron

Hey, I'm trying to use this package with electron v25.2.0. The filesystem mounts correctly however, when I try to read any file, I get the following error:

Transport endpoint is not connected

I have tried to recompile the package to target electron using this command:

prebuildify --napi --strip --target [email protected]

it compiles fine, but the same error still happens.

When I use it through nodejs directly, this error goes away and I'm able to read without any problem.

For this library, is there a required version of osxfuse? I tried updating my osxfuse to the latest 4.0.4 and I get a mount error. I know the npm package that this package uses to install osxfuse, installs the 3.10.4.

read(..): when writing binary data in a Buffer, it throws ERR_INVALID_ARG_TYPE "TypeError: argument must be a string"

https://github.com/fuse-friends/fuse-native/blob/master/example.js#L63

buf.write(slice) with a buffer containing binary data doesn't work. It throws ERR_INVALID_ARG_TYPE "TypeError: argument must be a string"

How would I return binary data? If I just do buf.write(slice.toString()) then the encoding seems incorrect and the hex data received by the terminal is wrong.

fuse: device not found on WSL2

I'm testing fuse-native on Ubuntu running on WSL2. I've Fuse installed and fuse-native is-configured says true. Despite that, fuse.mount() crashes the script with fuse: device not found, try 'modprobe fuse' first error. WSL2 doesn't seem to be able to do modprobe so that doesn't help.

Any idea what to try to fix this?

Clarify supported node versions

Which node versions are currently supported?

I had errors building fuse-native on node 13 and node 12, node 10 worked well.

Adding Mobile Support

Hello! Would I realistically be able to add iOS/Android support to this library? I want to try using it in a React Native app running nodejs-mobile

I see that this is meant for Linux and MacOS, but I've never worked on this kind of code before so I'm not sure where to begin when adding mobile compatibility, if it's possible to do so

support >4gb file sizes

we seem to truncate to uin32 in too many places

Windows Support Possible?

Hi, I'm wondering if Windows support is possible, with something like Dokany?

Cannot overwrite files

$ echo "Hello World" > file
warning: An error occurred while redirecting file 'file'
open: Function not implemented

function open(path, flags, cb) {
    console.log(path, flags);
    return cb(0, 42);
}

It returns this flag:

/file 32769

Appending data or overwriting specific parts without overwriting whole file works fine.
When appending data the flag is 33793.

But when I try to overwrite file with bash, I don't know how to implement it.

Any suggestions how to handle file overwrites?
What shall I return instead of (0, 42)?

Debug (overwrite):

unique: 740, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 1810488
getattr /
   unique: 740, success, outsize: 120
unique: 742, opcode: OPENDIR (27), nodeid: 1, insize: 48, pid: 1810488
   unique: 742, success, outsize: 32
unique: 744, opcode: RELEASEDIR (29), nodeid: 1, insize: 64, pid: 0
   unique: 744, success, outsize: 16
unique: 746, opcode: LOOKUP (1), nodeid: 1, insize: 45, pid: 1810488
LOOKUP /file
getattr /file
   NODEID: 40
   unique: 746, success, outsize: 144
unique: 748, opcode: OPEN (14), nodeid: 40, insize: 48, pid: 1810488
open flags: 0x8001 /file
/file 32769
   open[42] flags: 0x8001 /file
   unique: 748, success, outsize: 32
unique: 750, opcode: SETATTR (4), nodeid: 40, insize: 128, pid: 1810488
   unique: 750, error: -38 (Function not implemented), outsize: 16
unique: 752, opcode: RELEASE (18), nodeid: 40, insize: 64, pid: 0
release[42] flags: 0x8001
   unique: 752, success, outsize: 16
unique: 754, opcode: LOOKUP (1), nodeid: 1, insize: 45, pid: 2435855

Debug (appending/overwriting specific parts, not whole file which works):

unique: 890, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 1810488
getattr /
   unique: 890, success, outsize: 120
unique: 892, opcode: OPENDIR (27), nodeid: 1, insize: 48, pid: 1810488
   unique: 892, success, outsize: 32
unique: 894, opcode: RELEASEDIR (29), nodeid: 1, insize: 64, pid: 0
   unique: 894, success, outsize: 16
unique: 896, opcode: LOOKUP (1), nodeid: 1, insize: 45, pid: 1810488
LOOKUP /file
getattr /file
   NODEID: 40
   unique: 896, success, outsize: 144
unique: 898, opcode: OPEN (14), nodeid: 40, insize: 48, pid: 1810488
open flags: 0x8401 /file
/file 33793
   open[42] flags: 0x8401 /file
   unique: 898, success, outsize: 32
unique: 900, opcode: WRITE (16), nodeid: 40, insize: 81, pid: 1810488
write[42] 1 bytes to 0 flags: 0x8401
...

Having problems with this too:

$ touch hi
touch: setting times of 'hi': Function not implemented

But it does create the file, so it's not a huge deal.

"Unsupported macOS Version"

Am I doing something obviously wrong?

> node index.js
{ Error [fuse failed]: fuse failed
    at fs.stat (/Users/wmhilton/code/wmhilton/foofs/node_modules/fuse-native/index.js:241:19)
    at FSReqWrap.oncomplete (fs.js:154:5) code: 'fuse failed', name: 'Error [fuse failed]' }

> node -v
v10.15.3
> npm ls
[email protected] /Users/wmhilton/code/wmhilton/foofs
└─┬ [email protected]
  ├─┬ [email protected]
  │ ├── [email protected]
  │ └── [email protected]
  ├─┬ [email protected]
  │ └── [email protected]
  ├── [email protected]
  └── [email protected]

macOS Catalina, Version 10.15.2 (19C57)

Modified and Access times

Using the example as a basis I am passing an object containing mtime, atime and ctime

            mtime: attr.modified,
            atime: attr.modified,
            ctime: attr.modified,
            nlink: 1,
            size: attr.size,
            mode: mode,
            uid: process.getuid(),
            gid: process.getgid(),

back from to readdir and gettattr, where attr is my object with correct data. No matter what JS date object I put as mtime / atime / ctime, when looking at properties in nautilus, it always reads back with the current date time.

I have tried manually setting the times to new Date(5000000) in the callbacks, but they still always give me the current date and time.

Is this a limitation of the current implementation or am I doing something wrong?

Invalidate cache

I've been using this library successfully with Linux and node 13, so thanks for creating it.

I was wondering if there was a way to force the OS to request a readdir? I found fuse_invalidate_path which sounded promising. But I couldn't find anything similar in this library.

What I'm trying to achieve is the file browser automatically updating when my program updates the filesystem. Without the user having to refresh.

Fuse-native does not work on worker-thread (FATAL ERROR)

When we put fuse code into worker thread (instead of main thread) we get an error :

FATAL ERROR: HandleScope::HandleScope Entering the V8 API without proper locking in place

FUSE library version: 2.9.7
nullpath_ok: 0
nopath: 0
utime_omit_ok: 0
unique: 1, opcode: INIT (26), nodeid: 0, insize: 56, pid: 0
INIT: 7.26
flags=0x001ffffb
max_readahead=0x00020000
FATAL ERROR: HandleScope::HandleScope Entering the V8 API without proper locking in place
1: 0xa5d530 node::Abort() [/usr/local/bin/node]
2: 0x9927fd node::FatalError(char const*, char const*) [/usr/local/bin/node]
3: 0xc34eca v8::Utils::ReportApiFailure(char const*, char const*) [/usr/local/bin/node]
4: 0xc3665c v8::HandleScope::HandleScope(v8::Isolate*) [/usr/local/bin/node]
5: 0xa19866 napi_open_handle_scope [/usr/local/bin/node]
6: 0x7fe038da5882
7: 0x143dd96 [/usr/local/bin/node]
8: 0x1450875 [/usr/local/bin/node]
9: 0x143e6c8 uv_run [/usr/local/bin/node]
10: 0x9b1ed5 node::SpinEventLoop(node::Environment*) [/usr/local/bin/node]
11: 0xa9e790 node::NodeMainInstance::Run(node::EnvSerializeInfo const*) [/usr/local/bin/node]
12: 0xa2a9ba node::Start(int, char**) [/usr/local/bin/node]
13: 0x7fe03bbdabf7 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
14: 0x9afc1c [/usr/local/bin/node]
Aborted (core dumped)

Code example

const USE_THREADS = true; //to switch between 


(async function () {
    const {
        Worker, isMainThread, parentPort, workerData
    } = require('worker_threads');


    if (USE_THREADS && isMainThread) {
        function runFuseOnWorkerThread() {
            return new Promise((resolve, reject) => {
                const worker = new Worker(__filename, {
                    workerData: {}
                });
                worker.on('message', resolve);
                worker.on('error', reject);
                worker.on('exit', (code) => {
                    if (code !== 0)
                        reject(new Error(`Worker stopped with exit code ${code}`));
                });
            });
        };

        try {

            await runFuseOnWorkerThread()
        }
        catch(e) {
            console.log(e);
        }


    } else {

        if (isMainThread) {
           console.log('this is main and only thread');
        } else {
            console.log('this is worker thread');
        }

        const Fuse = require("fuse-native");
        const fs = require("fs");
        const pfs = require("fs").promises;
        const path = require("path");
        const os = require("os");



        const ops = {
            readdir: function (path, cb) {
                if (path === '/') return cb(null, ['test'])
                return cb(Fuse.ENOENT)
            },
            getattr: function (path, cb) {
                if (path === '/') return cb(null,
                    {
                        mtime: new Date(),
                        atime: new Date(),
                        ctime: new Date(),
                        nlink: 1,
                        size: 0,
                        mode: 16877,
                        uid: 0,
                        gid: 0,
                    });
                if (path === '/test') return cb(null, {
                    mtime: new Date(),
                    atime: new Date(),
                    ctime: new Date(),
                    nlink: 1,
                    size: 'hello world'.length,
                    mode: 33188,
                    uid: 0,
                    gid: 0,
                })
                return cb(Fuse.ENOENT)
            },
            open: function (path, flags, cb) {
                return cb(0, 42)
            },
            release: function (path, fd, cb) {
                return cb(0)
            },
            read: function (path, fd, buf, len, pos, cb) {
                var str = 'hello world'.slice(pos, pos + len)
                if (!str) return cb(0)
                buf.write(str)
                return cb(str.length)
            }
        }

        const mnt = '/tmp/fuse';
        const fuse = new Fuse(mnt, ops, {debug: true, force: true, mkdir: true})
        fuse.mount(function (err) {
            console.log('err', err);
            //parentPort.postMessage(true);
            fs.readFile(path.join(mnt, 'test'), function (err, buf) {
                // buf should be 'hello world'
                if (err) {
                    console.log('fuse error', err);
                }
                if (buf) {
                    console.log('buf should be hello world', buf.toString());
                }
            })
        })
    }
}());

Calling fs.readDirSync fails

I have a file system implemented using Fuse-Native that I can navigate via OS X Finder. I am making a tester for the file system to ensure it works after making changes. I am trying to get the contents of a directory using a script in Node.js, but it is failing.

Example dir: /Users/me/Documents/fusemount/virtualDir

The error: 'EPERM: operation not permitted, scandir '/Users/me/Documents/fusemount/virtualDir'

The code I am using:

  fs.readdirSync('/Users/me/Documents/fusemount/virtualDir').forEach(file => {
    console.log(file);
  });

When I run this I have successful requests for:

access('\')
getAttr('\virtualDir')
getAttr('\')

The debug console shows:

unique: 9, opcode: ACCESS (34), nodeid: 1, insize: 48, pid: 37685
access / 01
   unique: 9, success, outsize: 16
unique: 3, opcode: LOOKUP (1), nodeid: 1, insize: 48, pid: 37685
LOOKUP /virtualDir
getattr /[Mounted Part of Path]
   NODEID: 28
   unique: 3, success, outsize: 160
unique: 4, opcode: GETXATTR (22), nodeid: 28, insize: 71, pid: 0
getxattr /virtualDir com.apple.macl 0 0
   unique: 4, error: -45 (Operation not supported), outsize: 16
unique: 11, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 37685
getattr /
   unique: 11, success, outsize: 136
unique: 2, opcode: GETXATTR (22), nodeid: 1, insize: 71, pid: 0
getxattr / com.apple.macl 0 0
   unique: 2, error: -45 (Operation not supported), outsize: 16

I have not implemented the following operations:

releaseDir
ftruncate
fgetAttr
utimens
fsync
fsyncDir

My statfs operation is returning the same information found in the readme for this project.

getAttr returns the following for folders:

{
  mtime: new Date(2020, 8, 16, 3, 20, 0),
  atime: new Date(2020, 8, 16, 3, 20, 0),
  ctime: new Date(2020, 8, 16, 3, 20, 0),
  nlink: 1,
  size: 100,
  mode: 0o40777,
  uid: process.getuid ? process.getuid() : 0,
  gid: process.getgid ? process.getgid() : 0
};

Any help would be great. Thanks.

Design (multithreaded ?) question regarding fuse-native/vs fuse-bindings

Previously, using fuse-bindings, and reading an mkv file, VLC requested for continious ranges of bytes (e.g. 4=>6, 6=>8, 8=>10, ..)

Now using fuse-native, it sometimes read ranges in a non serial way (e.g. requesting 4=>6, 8=>10, 6=>8)

Is it the "normal" behavior due to the multitread gain of fuse-native vs fuse-bindings ? If so, I've got a lot of changes in random-read-http to make !

fuse.context()

In fuse-bindings there was a function fuse.context() that gets the information about caller process uid and gid.

what is the equivalent of this function in fuse-native?

"Illegal hardware instruction"

Hello,
I am using the example in the README, but I get an error after I enter the mounted path and I ls twice (in the first one everything is ok):

FUSE library version: 2.9.9
nullpath_ok: 0
nopath: 0
utime_omit_ok: 0
unique: 1, opcode: INIT (26), nodeid: 0, insize: 56, pid: 0
INIT: 7.27
flags=0x003ffffb
max_readahead=0x00020000
null mount
   INIT: 7.19
   flags=0x00000011
   max_readahead=0x00020000
   max_write=0x00020000
   max_background=0
   congestion_threshold=0
   unique: 1, success, outsize: 40
unique: 2, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 28915
getattr /
   unique: 2, success, outsize: 120
unique: 3, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 28922
getattr /


#
# Fatal error in , line 0
# Check failed: result.second.
#
#
#
#FailureMessage Object: 0x7fffb9a45270
 1: 0xa96131  [node]
 2: 0x19614f4 V8_Fatal(char const*, ...) [node]
 3: 0xe58879 v8::internal::GlobalBackingStoreRegistry::Register(std::shared_ptr<v8::internal::BackingStore>) [node]
 4: 0xbc5798 v8::ArrayBuffer::GetBackingStore() [node]
 5: 0xa08559 node::Buffer::New(node::Environment*, char*, unsigned long, void (*)(char*, void*), void*) [node]
 6: 0xa089c3 node::Buffer::New(v8::Isolate*, char*, unsigned long, void (*)(char*, void*), void*) [node]
 7: 0x9fdbad napi_create_external_buffer [node]
 8: 0x7fcd588e29e5  [/home/project/fuse-native/build/Release/fuse.node]
 9: 0x1320b69  [node]
10: 0x1332fd0  [node]
11: 0x1321478 uv_run [node]
12: 0xa6aa64 node::NodeMainInstance::Run() [node]
13: 0x9f9571 node::Start(int, char**) [node]
14: 0x7fcd5b12809b __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
15: 0x99279c  [node]
[1]    28899 illegal hardware instruction  node index.js

npm install on Windows causes errors

We are using fuse-naive in an Electron application that will run on Windows and Mac. We have fuse-native in our package.json. When we run npm install we receive the error message below. Is there a suggested work around for this so that it installs on Mac and not on Windows? Would the os attribute in the package.json help in this case? I have seen the ability to use optionalDependencies, but that could hide issues if for some reason fuse-native failed to install on Mac.

D:\code\work\workspace\app\app-engine\node_modules\fuse-native>if not defined npm_config_node_gyp (node "D:\Program Files\nodejs\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild )  else (node "D:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" rebuild )
Traceback (most recent call last):
  File "D:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\gyp_main.py", line 50, in <module>
    sys.exit(gyp.script_main())
  File "D:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\__init__.py", line 554, in script_main
    return main(sys.argv[1:])
  File "D:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\__init__.py", line 547, in main
    return gyp_main(args)
  File "D:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\__init__.py", line 520, in gyp_main
    [generator, flat_list, targets, data] = Load(
  File "D:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\__init__.py", line 136, in Load
    result = gyp.input.Load(build_files, default_variables, includes[:],
  File "D:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\input.py", line 2778, in Load
    LoadTargetBuildFile(build_file, data, aux_data,
  File "D:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\input.py", line 416, in LoadTargetBuildFile
    ProcessVariablesAndConditionsInDict(
  File "D:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\input.py", line 1295, in ProcessVariablesAndConditionsInDict
    ProcessVariablesAndConditionsInList(value, phase, variables,
  File "D:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\input.py", line 1311, in ProcessVariablesAndConditionsInList
    ProcessVariablesAndConditionsInDict(item, phase, variables, build_file)
  File "D:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\input.py", line 1295, in ProcessVariablesAndConditionsInDict
    ProcessVariablesAndConditionsInList(value, phase, variables,
  File "D:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\input.py", line 1315, in ProcessVariablesAndConditionsInList
    expanded = ExpandVariables(item, phase, variables, build_file)
  File "D:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\input.py", line 914, in ExpandVariables
    sys.stderr.write(p_stderr)
TypeError: write() argument must be str, not bytes while trying to load binding.gyp
gyp ERR! configure error
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (D:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\lib\configure.js:351:16)
gyp ERR! stack     at ChildProcess.emit (events.js:210:5)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:272:12)
gyp ERR! System Windows_NT 10.0.19041
gyp ERR! command "D:\\Program Files\\nodejs\\node.exe" "D:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
gyp ERR! cwd D:\code\work\workspace\app\app-engine\node_modules\fuse-native
gyp ERR! node -v v12.13.0
gyp ERR! node-gyp -v v5.0.5
gyp ERR! not ok
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules\fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"win32","arch":"x64"})

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `node-gyp-build`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\user\AppData\Roaming\npm-cache\_logs\2020-09-10T18_54_10_653Z-debug.log

Is this project dead

We have a project that uses it and it will not work in Big Sur because the version required of osxfuse is not compatible with Big Sur and more recent OS X releases.

If this project is dead, do you know of any alternatives?

Add missing fuse option: nonempty

The nonempty fuse option is missing:

nonempty
    Allows mounts over a non-empty file or directory. By default these
    mounts are rejected to prevent accidental covering up of data, which
    could for example prevent automatic backup.

I added it myself at line number 195 of index.js in _fuseOptions():

    if (this.opts.nonempty) options.push('nonempty')

Support for macFUSE

Hi, will the newer versions of osxfuse (now macFUSE) be supported in the future?

[BUG] useIno option does not appear to use custom ino values in stat struct

After inode 1 for the root fs node I set starting inode number at 100 and increment it with every created node.
I use the ino field in the stat struct.
When doing a stat on any of my fuse nodes I can see the count starts from 1 and does not jump to 100+.
const fuse = new Fuse('./mnt', ops, {
debug: false,
displayFolder: false,
autoUnmount: true,
force: true,
kernelCache: false,
useIno: true // <- this does not appear to be used
});

Looking at _fuseOptions in index.js I can see that use_ino is not something that is pushed into the -o option returned from the function for the fuse mount options.

Added option in PR33

[BUG] Issue creating symlink caused by incomplete path for src passed to callback

When i do the following:
cd somedir
ln -s theFile theLink

the src param is just the file name theFile
the dest param is the full path /somedir/theLink

I cant resolve the src param without knowing the current working dir.
The src param should be the full path /somedir/theFile

> hyperdrive fuse-setup
Configuring FUSE...
Password:
FUSE successfully configured:
  * Your root drive will be mounted at ~/Hyperdrive when the daemon is next started.
  * If your mountpoint ever becomes unresponsive, try running `hyperdrive force-unmount`.
> hyperdrive start
dyld: lazy symbol binding failed: Symbol not found: _fuse_mount
  Referenced from: ~/.nvm/versions/node/v15.4.0/lib/node_modules/@hyperspace/hyperdrive/node_modules/fuse-native/build/Release/fuse.node
  Expected in: flat namespace

dyld: Symbol not found: _fuse_mount
  Referenced from: ~/.nvm/versions/node/v15.4.0/lib/node_modules/@hyperspace/hyperdrive/node_modules/fuse-native/build/Release/fuse.node
  Expected in: flat namespace

fuse-friends / fuse-native Goto Github PK

fuse-native's People

Contributors

Stargazers

Watchers

Forkers

fuse-native's Issues

Recommend Projects

Recommend Topics

Recommend Org