Comments (10)
looks like the same thing, probably just a case of calling the same thing two different names
from concat-stream.
If it's the same thing, would it make sense to try and merge the projects?
from concat-stream.
They are very similar. However one of accum's original use cases was to make sure that if multibyte chars were split across packets in a stream, that they were properly concatenated back together before converting to string otherwise they can be corrupted.
Since I didn't have that use case in a test, the code regressed over time, but after realizing that today I added a test and fixed the code back so it handles this situation properly.
See multibyte char split test -jeffbski/accum@858c22c
and the fix - jeffbski/accum@33d2961
To fix concat-stream similarly it involves doing Buffer.concat before the toString('utf8') or similar rather than just converting the individual packets.
from concat-stream.
@dmitrig01 there are plenty of modules with similar functionality on NPM, IMO it isnt worth the effort to try and merge projects in this scenario. if you want to send me a PR to the concat-stream readme that mentions similar modules in an effort to avoid confusion then I would merge it.
@jeffbski at the moment concat-stream doesn't call .toString() for streams of buffers, it just does Buffer.concat so multibyte chars shouldnt get messed up
from concat-stream.
@maxogden I may have misread the concat-stream code but I thought that if one specifies
concat({encoding: 'string'}, cb)
That this would force a conversion to string from the buffer packets by iterating over the packets and calling toString on each one individually which causes the problem if a multi byte char happen to get split over two packets in the stream.
Please ignore me if I am crazy, that's just what it looked like from my quick look at the code.
Ps. I plan to add concat-stream to my readme under related projects so others can find it. It is nice that you have made it browser compatible.
All the best,
Jeff
from concat-stream.
@jeffbski oh yea good point, so:
- if encoding is buffer, and data are buffers, it uses Buffer.concat() and returns a buffer
- if encoding is string, and data are buffers, it will convert buffers to strings before concat, returns a string
however, the latter case, while possible, isn't recommended since you can just do the first case instead and then to buffer.toString(). string encoding is really intended for when your incoming chunks are strings. but I think i can probably fix multibyte char preservation for this edge case... cheers!
from concat-stream.
Yes, it is one of those edge cases that would be tricky to understand after it happened, so if it can be made foolproof, all the better. I had hit this on a project and it had me confused for a bit.
from concat-stream.
@jeffbski fixed in 54444db. thanks for bringing it up
from concat-stream.
Excellent!
from concat-stream.
Hahahahah now the two are even more similar! I'll submit a PR :-)
from concat-stream.
Related Issues (20)
- Publish 1.4.5 to npm
- git tags HOT 2
- Stream empty for concatenated passthrough streams HOT 2
- concat-stream - buffer object in callback is empty. HOT 4
- handling errors HOT 1
- CI HOT 2
- Use standard, error-first callbacks HOT 4
- introduce Readable and Duplex concat streams HOT 1
- Use end-of-stream to determine end
- Convert to real stream HOT 4
- Bump readable-stream version to 2.1.x HOT 1
- Callback never fired if read stream emits an error HOT 2
- Security Vulnerability HOT 2
- Buffer Overhead Vulnerability HOT 2
- Buffer overread vulnerability HOT 2
- Buffer constructor runtime deprecation - this package emits a warning on Node 10
- How big is the difference from 1.6.* to 2.* ? HOT 3
- Is this package still maintained? HOT 1
- Stream consumers: Alternative builtin nodejs solution
- Drop polyfills in a next semver major?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from concat-stream.