Code Monkey home page Code Monkey logo

Comments (3)

vbshnsk avatar vbshnsk commented on June 28, 2024

I have since tried to DELETE the resumable upload by crc32 when I destroy my write/validate streams but it also doesn't affect the underlying streams, so the handle stays open.

from nodejs-storage.

xXACCEXx avatar xXACCEXx commented on June 28, 2024

Not sure if this is the same behaviour I'm seeing or not.

I'm seeing streams aren't closing when using WebStream apis in Node.
To me, it feels like the same problem, the handler on the web stream side completes correctly, but the event is not emitted on the GCS lib side.

This has the effect of any means of closing the stream does not work... Either destroy or end... Neither actually closes the handle.

from nodejs-storage.

xXACCEXx avatar xXACCEXx commented on June 28, 2024

Here's a little example of what I mean.

I tested this on node v20.10.0 using ts-node.
You can get the CSV from here
https://github.com/datablist/sample-csv-files/raw/main/files/people/people-2000000.zip

import { join } from 'path';
import { createReadStream } from 'fs';
import { Writable, Readable } from 'stream'
import { TransformStream } from 'stream/web'

import { Storage } from '@google-cloud/storage'

const _sourcePathname = join(process.cwd(), 'test/people-2000000.csv');
const _storage = new Storage({ keyFilename: './secrets/gcp.json' });
const _bucket = _storage.bucket('journeywise_test_bucket');

const sourceFile = createReadStream(_sourcePathname, { encoding: 'utf-8' })
const _readable = Readable.toWeb(sourceFile);

const destinationFile = _bucket.file('output.csv');
const destinationStream = destinationFile.createWriteStream();
const _writable = Writable.toWeb(destinationStream);


const assertSize = (assertSize: number) => {
	let size = 0;
	return new TransformStream({
		transform(chunk, ctrl) {
			size += chunk.length;
			ctrl.enqueue(chunk);
		},
		flush() {
			console.log('Closed stream. Size match?', size == assertSize);
		}
	})
}

_readable
	.pipeThrough(assertSize(235121126))
	.pipeTo(_writable)
	.then(() => console.log('stream completed successfully'))

from nodejs-storage.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.