Comments (17)
Thanks for the reproduction case! I have received the error from the test, and will be looking into it.
from nodejs-storage.
A quick question about:
In the real-world, we experienced this problem when an HTTP server behind a reverse proxy was piping from GCS to the HTTP response, and the upstream proxy closed the socket.
When the upstream proxy closed the socket, did you get an error event? I believe the right thing to do at that point is to close the GCS stream by calling .end()
.
Alternatively, this can be handled for you by using pumpify:
pumpify(sourceStream, brake(10), targetStream)
.on('error', err => {
// err === expected error about the socket breaking.
// sourceStream (the GCS stream) was auto-closed
})
from nodejs-storage.
I am using pump already to handle the piping. When the situation occurs, pump
calls the callback with a premature close
error, and after a little while the content mismatch error is emitted.
Since we're piping directly to the end-user HTTP response, there really isn't any good way to retry the fetch, so I've simply disabled the validation as there's really nothing I can do when this situation occurs - even if it was an actual content mismatch.
I still thought you might want to check if there is something you can do on your end to stop the error from being emitted, but I couldn't manage to reproduce the case 100% - using pump
solves the test case I created, but somehow doesn't solve the problem we have in a real-world case :/
It seems determining premature closed streams is non-trivial.
from nodejs-storage.
It still seems like there should only be one error emitted since you're using pump
. Is the GCS stream being piped anywhere else?
from nodejs-storage.
We have the same issues.
We create a readstream and pipe()
it directly to a Socket, these errors pop up.
Is there any way to fix this properly?
we only started getting these errors since updated to latested version of @google-cloud/storage
from nodejs-storage.
Could you provide reproduction code? When you get an error from any other stream, the GCS stream needs to be ended as well.
from nodejs-storage.
Seems like we are experiencing same that we can only reproduce it when it is in a docker container in the google kubernetes container, it always seems to happen with the same file. (96.03 KB size). somehow it seems to happen differently on different clients so i'm wondering if it might have to do with chunking.
also we have the same issue if we pipe createReadStream() through a transform:
const readStream = file.createReadStream()
const transform = new MetaDatTransform();
const dest = readStream.pipe(transform);
readStream.on('error', (err) => {
dest.destroy(err);
});
return dest;
i was able to get rid of this error by first reading stream till end and then reading it manually into a new stream. (bit of a work around but it worked).
from nodejs-storage.
Can't seem to reproduce it now :/ it seems to happen only when an OPH scanner requests (directly over TCP socket). but i wonder how this can influence the google library to throw this error.
this is what we do with the readstream:
const rs = file.createReadStream();
rs.pause();
rs.on('error', () => {...})//do logging here, close tcp socket
rs.on('data', function (buffer) {
readLength += buffer.length;
});
rs.on('end', function (info) {}); //close tcp socket.
self.respond('150 Opening ' + self.mode.toUpperCase() + ' mode data connection', () => {
rs.pipe(passiveSocket);
rs.resume();
});
(this code is from lib: https://github.com/nodeftpd/nodeftpd)
from nodejs-storage.
@joelharkes @rexxars any updates on how this is going / reproduction code?
from nodejs-storage.
Sorry, we were using as port of a relatively large and complex application and I found it hard to reproduce in an isolated fashion. Having said that, I find Node streams notoriously hard to get right, so I can't rule out that I was doing something wrong.
We disabled the verification, since there's no way to retry the request when piping directly to an HTTP response. Sorry I can't be of more help here.
from nodejs-storage.
Idem, we had:
- Error in combination with http sockets to OPH scanner.
- Using stream transformer
The first we fixed by downgrading to storage version 1.2
The second by first converting to a promise and then transforming string.
Couldn't find the source problem debugging through the node modules. Maybe it's one of the sub dependencies.
from nodejs-storage.
Having said that, I find Node streams notoriously hard to get right
That is definitely true. Feel free to ping me if you ever need an extra set of eyes on a problem.
I'm going to close this issue for now, because as said above, streams are hard to get right, and the blame could be one of many different points in an application, making it very hard to debug. If it's possible to create a reproduction case, I'll be happy to keep digging.
from nodejs-storage.
Hey @stephenplusplus I get the same with a cloud function where I react to upload events and try to append files to a zip in the bucket.
If I put a timeout between creating the write stream of the zip file and start writing to it is fine otherwise I get that error.
This happens only when I open the write stream in 'a' mode not 'w'
from nodejs-storage.
@DavideArgellati would you mind opening a new issue with reproduction code I can try?
from nodejs-storage.
Cool good to know there is a 5000ms second timout that was removed.
We also found that downloading and uploading the same file at the same time. could create this message.
Downloads where done with streams and by clients with slower connections and small buffers.
it could take a while to download the file. (Im not sure if this could be related to the 5000ms timeout as well.)
changing the upload schedule seemed to have reduced/removed these problems.
from nodejs-storage.
I am getting CONTENT_DOWNLOAD_MISMATCH every time I upload and download an image.
Using the completely vanilla upload and download samples here:
https://github.com/googleapis/nodejs-storage/blob/master/samples/files.js#L101
https://github.com/googleapis/nodejs-storage/blob/master/samples/files.js#L155
I try with any non-image file => always works.
I try with any image file => always CONTENT_DOWNLOAD_MISMATCH when downloading.
I thought maybe I need to set contentType explicitly, so I followed
https://stackoverflow.com/questions/41229918/google-storage-nodejs-upload-mime-type, but that didn't help.
This is super vanilla setup, no concurrency. Running on Electron on OSX. Does anyone know what could be the problem here?
from nodejs-storage.
By the way the download is actually succeeding, in that the correct file is getting saved to disk. So this is just a false positive.
from nodejs-storage.
Related Issues (20)
- storage requester pays existing bucket methods that accept userProject: file#getMetadata failed HOT 2
- Add object retention samples
- storage requester pays existing bucket methods that accept userProject: file#exists failed HOT 1
- Missing / undocumented feature in File.getSignedUrl() HOT 10
- Clarify the limitation message associated with bucket signed URLs
- storage HMAC keys: should get metadata for an HMAC key failed HOT 5
- storage HMAC keys: should make the key INACTIVE failed HOT 2
- File.publicUrl() unexpectedly escapes forward slashes in > 7.6.0 HOT 11
- storage HMAC keys: should delete the key failed HOT 3
- How to list the subdirectories/subfolders in a folder HOT 3
- storage public access prevention enforced public access prevention behavior: object cannot be made public via ACL failed HOT 3
- how to access storage with impersonated token? HOT 3
- what is the page limit of getFiles with autoPaginate=false HOT 2
- combining gzipped files
- Support Storage Control quickstart HOT 8
- Update mime dependency to ^4.0.0 HOT 10
- deps!: Remove form-data dependency in favor of Node 18 implementation
- FileMetadata type missing null on metadata for delete HOT 1
- The build failed HOT 2
- GetServiceAccountOptions missing projectIdentifier field
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nodejs-storage.