tus / tus-node-server Goto Github PK

• lib/stores/FileStore.js (currently 62.71)
• lib/handlers/PostHandler.js (currently 65.0)
• lib/handlers/HeadHandler.js (currently 77.78)
• lib/models/File.js (currently 85.71)

We will need https support. Though, maybe we can just use an ELB in front of two tus-node-servers to load balance and terminate SSL there.

hi there,

i am new with node.js as with tus, i tried to start an example und run in one problem after the other. first your examples are with const, but the code i got wit npm install uses strict and so const does not work. there are even module in the installbase with const.

then:
npm run example

[email protected] example /usr/local/node_modules/tus-node-server
nodemon example/server.js

[nodemon] 1.10.0
[nodemon] to restart at any time, enter rs
[nodemon] watching: .
[nodemon] starting node example/server.js

/usr/local/node_modules/tus-node-server/index.js:9
Server,
^
SyntaxError: Unexpected token ,
at Module._compile (module.js:439:25)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.require (module.js:364:17)
at require (module.js:380:17)
at Object. (/usr/local/node_modules/tus-node-server/example/server.js:1:73)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
[nodemon] app crashed - waiting for file changes before starting...

where inds.js contains:
module.exports = {
Server,
DataStore,
FileStore,
GCSDataStore,
};

....

Implementations should be able to provide filename for POST requests, and only if it is not provided, it can default back to Uid.

Thanks for the suggestion @js3692!

File is never written in a vagrant box with shared folder.

When the uploaded file size is less than the chunksize, the event handler 'EVENT_UPLOAD_COMPLETE' happens.
But if the file size is bigger, nothing happens!
The other 2 handlers are working fine in both scenarios!

How can I use this together with express?

Once a file is deleted from GCS, there's currently knowing it once existed. Therefore, GCSDataStore.getOffset() can't return ERRORS.FILE_NO_LONGER_EXISTS when appropriate.

The GoogleCloudPlatform node library does not currently support resuming interrupted uploads. Once #11 is completed, GCSDataStore.write() can be amended to actually resume an interrupted upload, finishing the implementation.

Following #10, the remaining tasks for FileStore are:

• Create file (via creation extension)
• Store metadata (key-value pairs from Upload-Metadata header)
• Files must expire (via expiration extension)
• Delete file (via termination extension)
• Accept chunks (via concatenation extension)

The Client MAY supply the Upload-Metadata header to add additional metadata to the upload creation request. The Server MAY decide to ignore or use this information to further process the request or to reject it. If an upload contains additional metadata, responses to HEAD requests MUST include the Upload-Metadata header and its value as specified by the Client during the creation.

JSON validate the Upload-Metadata header value, and save to configstore in FileStore.create

hi there,

i got the tus-server running by starting ist by hand, but i would like to start it up automatically.

as i am on centos 7 i still could use initv-scripts, but also systemd-manifests. both did not work as i thought.

i have a server.js in /usr/local/tus-node-server/hest
so starting by hand is: npm run hest

in the startscript i tried without and with cd to /usr/local/tus-node-server, but get:
sh: nodemon: command not found

the npm-debug.log shows this:

0 info it worked if it ends with ok
1 verbose cli [ 'node', '/bin/npm', 'run', 'hest' ]
2 info using [email protected]
3 info using [email protected]
4 verbose node symlink /bin/node
5 verbose run-script [ 'prehest', 'hest', 'posthest' ]
6 info lifecycle [email protected]prehest: [email protected]
7 silly lifecycle [email protected]prehest: no script for prehest, continuing
8 info lifecycle [email protected]hest: [email protected]
9 verbose lifecycle [email protected]hest: unsafe-perm in lifecycle true
10 verbose lifecycle [email protected]hest: PATH: /usr/local/lib/node_modules/npm/bin/node-gyp-bin:/usr/local/tus-node-server/node_modules/.bin:/bin:/sbin:/usr/sbin:/bin:/usr/bin,/usr/local/bin/
11 verbose lifecycle [email protected]hest: CWD: /usr/local/tus-node-server
12 silly lifecycle [email protected]hest: Args: [ '-c', 'nodemon hest/server.js' ]
13 info lifecycle [email protected]hest: Failed to exec hest script
14 verbose stack Error: [email protected] hest: nodemon hest/server.js
14 verbose stack spawn ENOENT
14 verbose stack at ChildProcess. (/usr/local/lib/node_modules/npm/lib/utils/spawn.js:33:16)
14 verbose stack at ChildProcess.emit (events.js:98:17)
14 verbose stack at maybeClose (child_process.js:766:16)
14 verbose stack at Process.ChildProcess._handle.onexit (child_process.js:833:5)
15 verbose pkgid [email protected]
16 verbose cwd /usr/local/tus-node-server
17 error Linux 3.10.0-327.18.2.el7.x86_64
18 error argv "node" "/bin/npm" "run" "hest"
19 error node v0.10.42
20 error npm v3.10.3
21 error file sh
22 error code ELIFECYCLE
23 error errno ENOENT
24 error syscall spawn
25 error [email protected] hest: nodemon hest/server.js
25 error spawn ENOENT
26 error Failed at the [email protected] hest script 'nodemon hest/server.js'.
26 error Make sure you have the latest version of node.js and npm installed.
26 error If you do, this is most likely a problem with the tus-node-server package,
26 error not with npm itself.
26 error Tell the author that this fails on your system:
26 error nodemon hest/server.js
26 error You can get information on how to open an issue for this project with:
26 error npm bugs tus-node-server
26 error Or if that isn't available, you can get their info via:
26 error npm owner ls tus-node-server
26 error There is likely additional logging output above.
27 verbose exit [ 1, true ]

where do i a mistake?

regards
andreas

Right now the creation extension omit's thee protocol from a created url, which can cause some issues using tus-js-client server side (tus/tus-java-client#10 (comment)). We should consider a new approach that works in the browser and server

I start uploading a large file of 1GB and let it upload around halfway before closing the tab (Chrome). I open the page again and start to upload the same file. I always receive this message:

upload.js:286 HEAD http://localhost:8080/demo/null 404 (Not Found)_resumeUpload @ upload.js:286start @ upload.js:109(anonymous function) @ demo.js:82
upload.js:206 Refused to get unsafe header "Location"

I am using tus-js-client to test this (the demo) and it works perfectly fine with tusd (in Go). This problem only occurs with tus-node-server.

My server is only a copy paste of the Quick Start code provided in the README under "Build a standalone server".

Uploading files works fine too, but never resumes correctly, it always starts from the beginning.

Hello,
Is there any plans to support FTP as data source so files can be uploaded directly from FTP?

Is there any particular reason why the POST handler includes 'http://' as part of it's response?

const url = `http://${req.headers.host}${this.store.path}/${File.id}`;

Can this be changed to:

const url = `//${req.headers.host}${this.store.path}/${File.id}`;

I'm running into a scenario where a POST is issued over HTTPS, and then the subsequent PATCH is over HTTP because of this.

It appears you can't save metadata to an GCS object until the upload is complete. We need to figure out how to set/get upload_length, upload_defer_length from a GCS upload_id for POST/HEAD requests.

Hello,

I am not sure if this has been answered/addressed already, but I couldn't find anything related. I am using the FileStore datastore, and I was wondering what is the recommended way (or if there is already a mechanism in place) to auto-purge expired files in order to not consume disk space.

Thank you for your time.

Abstract OPTIONS handler into its own component which:

• Returns server's configuration

As the topic says, the files uploaded with tus-node-server are saved on the server side using an ID as the fileName. But I need to list the files that are on the server and let the user download them (not only list the new uploads, but list all uploaded files). Is there a way to get the list of files on the server, including their name and url?

Thank you.

Storing Metadata in ~/.config/configstore is rather impractical.
You should at least provide an option to set the path.

Looking at the example resumable upload curl commands and the tus.io protocol spec: should the response have a upload-offset header giving the amount uploaded so far and an upload-length of the full file size from the initial POST?

Abstract POST handler into its own component which] creates the file, following the DataStore interface (but only using local file store for now)

I can't run the server, nor demo files. I keep getting Error: Cannot find module 'google-cloud' from node.
Is there any workaround? Am I doing something wrong?

You should bug the author to publish it

Currently, events like the EVENT_FILE_CREATED are emitted from the DataStore level (like so), and Error's that include response information are thrown from the DataStore level as well (like so).

Since neither of these are the concern of the DataStore, and rather the HTTP request handler, they should be abstracted one level above to the handler. The DataStore should be purely focused on reading & writing.

This will also make #61 possible.

Hi (again).

If I upload a file and then try to re-upload it, the EVENT_UPLOAD_COMPLETE is not firing. I assume this is by design, but it would be nice to know if the file is already there with some other event, like EVENT_UPLOAD_ALREADY_EXISTS in order to handle it or re-firing EVENT_UPLOAD_COMPLETE might also be just fine as well.

We are looking at using tus-node-server for our file uploads. Excited to try it out. Happy to help out with a s3 storage backend plugin, can anybody give guidance on where to start?

Allow configuration in the server construction.

Server#listen should use https.createServer, with the config passed in the tus construction

const tus = require('tus-node-server');
const config = {
    tls: {
        key: fs.readFileSync('/agent2-key.pem'),
        cert: fs.readFileSync('/agent2-cert.pem'),
    }
};
const tus_server = new tus.Server(config);

Also need to update PostHandler to choose http|https based on the configuration

Hi all.
How can i get original filename?
In file object i get only
{ file: File { id: '-', upload_length: '3957145', upload_defer_length: undefined, upload_metadata: undefined } }

PatchHandler needs to:

• writes chunks to file store

anyway to do a callback after a file is completed uploaded ? let say there is a need to write a record to database once file is successfully uploaded

There should be a way to set contentType on the uploaded object as such...

var stream = file.createWriteStream({
metadata: {
contentType:
}
});

I already upload it in AWS but i got an error 502 Bad Gateway

In order to store the metadata from the POST request, a GCS object needs to be created, since the metadata isn't stored in a resumable URI. To fix that, GCSDataStore.create() pipes an empty stream to create an empty GCS object, containing the data.

If we maintain the GCS upload id, (passed back in the 'x-guploader-uploadid' header) to use the same resumable URI, the content type won't match in GCSDataStore.write(), and GCS will delete the object:

Error: The uploaded data did not match the data from the server. As a precaution, the file has been deleted. To be sure the content is the same, you should try uploading the file again

So, right now, the Upload-Offset header does not translate through to the gcloud-node package as a resume session, but rather a replacement upload to the same object.

Resuming an upload will currently replace the GCS object with the new data piped in, and override the starting bytes of the file at the resumed offset 😢 .

Hello, I dont really know how to get more infos about tus FileStore object options so I ask here, is there a way to set a custom location and name for the configstore file ?
Thank yo uby advance and sorry for using this issue tracker to do that.

Have a great day ;)

Errors should all be constants. Create an error constants object, then use it for

promise rejections

if (upload_length === undefined && upload_defer_length === undefined) {
    return reject(412);
}

// should be
if (upload_length === undefined && upload_defer_length === undefined) {
    return reject(ERRORS.LENGTH_HEADER_REQUIRED);
}

and response writing

super.send(res, error, {}, 'Upload-Length or Upload-Defer-Length Required\n');

// should be
super.send(ERRORS.LENGTH_HEADER_REQUIRED);

I start uploading a large file of 1GB and let it upload around halfway before closing the tab (Chrome). I open the page again and start to upload the same file. I always receive this message:

Failed because: Error: tus: invalid or missing offset value
Do you want to retry?

The error in the console is:
upload.js:257 Refused to get unsafe header "Upload-Offset"xhr.onload @ upload.js:257

I am using tus-js-client to test this (the demo) and it works perfectly fine with tusd (in Go). This problem only occurs with tus-node-server.

My server is only a copy paste of the Quick Start code provided in the README under "Build a standalone server".

Uploading files works fine too, but never resumes correctly.

What I'm trying to find out is if tus server could tell the client to re-upload or just patch that certain difference. My test situation was that I started to upload a text file until around 50% or so, stopped the uploading, changed the text on the beginning then resumed the process. Of course changing the text at the end was OK but it didn't synced the differences on the beginning of the text file. Although the server didn't fired the upload completed event my client was informed as it's fully uploaded. I'm using the tus-node-server demo and https://github.com/gerdus/tus-dotnet-client

Hi, my implementation was failing due to missing headers in response. File uploads were not resuming. I inspected the http://tus.io/demo.html network requests and noticed this header, which was missing in my node server's responses:

'Access-Control-Expose-Headers' : 'Upload-Offset, Location, Upload-Length, Tus-Version, Tus-Resumable, Tus-Max-Size, Tus-Extension, Upload-Metadata'

I examined the node.js library code a little bit, and discovered that on Server.js file you are controlling CORS with the following lines :

if (req.headers.origin) {
     res.setHeader('Access-Control-Allow-Origin', req.headers.origin);
}

I believe it should also set the "Access-Control-Expose-Headers" header too. So i changed to following piece of code and it worked :

if (req.headers.origin) {
     res.setHeader('Access-Control-Allow-Origin', req.headers.origin);
     res.setHeader('Access-Control-Expose-Headers' , 'Upload-Offset, Location, Upload-Length, Tus-Version, Tus-Resumable, Tus-Max-Size, Tus-Extension, Upload-Metadata');
}

Is there a better way to achieve CORS, or is this the way to do it? Thank you.

For using Google Cloud Storage as the file store.

HeadHandler needs to:

• Return Upload-Offset or 404, 410, 403

Hello,
I'm currently working on MongoDB and I need to upload some large files into DB. Is there a plan to support GridFS?

I was not able to install tus-node-server using node 7.4.0. I received the following errors:

node-pre-gyp ERR! Tried to download: https://storage.googleapis.com/grpc-precompiled-binaries/node/grpc/v0.15.0/node-v51-darwin-x64.tar.gz
node-pre-gyp ERR! Pre-built binaries not found for [email protected] and [email protected] (node-v51 ABI) (falling back to source compile with node-gyp)

and building it failed too:
CXX(target) Release/obj.target/grpc_node/src/node/ext/call.o
../src/node/ext/call.cc:616:20: error: no member named 'SetHiddenValue' in 'v8::Object'
info.This()->SetHiddenValue(Nan::New("channel_").ToLocalChecked(),

I downgraded to node 6.9.1 and install worked just fine. This solution works for me for now but supporting the latest major node release would be nice.

Cheers!

Users should be able to choose their appropriate DataStore, and not deal with all stores. Right now all stores are included in this module, which causes issues such as #56 and #54.

This will serve as a progress issue, leading to the closing of #61.

• tus-datastore-abstract
• tus-datastore-filesystem
• tus-datastore-gcs

Hi,
I cant use tus-node-server because google-gax raises compile errors.
But the case is, I don't need any store (or rather implement my own) so it would be really good to split it up into single packages:

• tus-server
• tus-server-store-fs
• tus-server-store-google
• tus-server-store-gridfs
  etc...

server.on(EVENTS.EVENT_UPLOAD_COMPLETE, (event) => {
console.log(event.file.upload_length);
console.log(event.file.upload_metadata );
});

How to get event.file.upload_length and event.file.upload_metadata ? I can't get those value from event hook complete. Possible to get my req.get(..) from event ?

• Individual method unit tests
• End to end system tests?

path and directory don't have to be the same thing and in many cases shouldn't.

FileStore construction should be:

server.datastore = new tus.FileStore({
  directory: 'files/videos/user-uploads',
  path: '/api/videos/tus-upload'
});

Thanks for the suggestion @js3692!

The DataStore defines the interface for storing files.

The local FileStore will implement it for storing the files on the server:

const server = new Tus();
server.datastore = new FileStore({
    path: '/files'
});

Then we can further build out AWS S3, GCS, etc:

const server = new Tus();
server.datastore = new S3Store({
    KEY: 'ABC',
    SECRET: 'XYZ'
    bucket: 'tus-uploads'
});