golang / snappy Goto Github PK
View Code? Open in Web Editor NEWThe Snappy compression format in the Go programming language.
License: BSD 3-Clause "New" or "Revised" License
The Snappy compression format in the Go programming language.
License: BSD 3-Clause "New" or "Revised" License
The Snappy compression format in the Go programming language. To use as a library: $ go get github.com/golang/snappy To use as a binary: $ go install github.com/golang/snappy/cmd/snappytool@latest $ cat decoded | ~/go/bin/snappytool -e > encoded $ cat encoded | ~/go/bin/snappytool -d > decoded Unless otherwise noted, the Snappy-Go source files are distributed under the BSD-style license found in the LICENSE file. Benchmarks. The golang/snappy benchmarks include compressing (Z) and decompressing (U) ten or so files, the same set used by the C++ Snappy code (github.com/google/snappy and note the "google", not "golang"). On an "Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz", Go's GOARCH=amd64 numbers as of 2016-05-29: "go test -test.bench=." _UFlat0-8 2.19GB/s ± 0% html _UFlat1-8 1.41GB/s ± 0% urls _UFlat2-8 23.5GB/s ± 2% jpg _UFlat3-8 1.91GB/s ± 0% jpg_200 _UFlat4-8 14.0GB/s ± 1% pdf _UFlat5-8 1.97GB/s ± 0% html4 _UFlat6-8 814MB/s ± 0% txt1 _UFlat7-8 785MB/s ± 0% txt2 _UFlat8-8 857MB/s ± 0% txt3 _UFlat9-8 719MB/s ± 1% txt4 _UFlat10-8 2.84GB/s ± 0% pb _UFlat11-8 1.05GB/s ± 0% gaviota _ZFlat0-8 1.04GB/s ± 0% html _ZFlat1-8 534MB/s ± 0% urls _ZFlat2-8 15.7GB/s ± 1% jpg _ZFlat3-8 740MB/s ± 3% jpg_200 _ZFlat4-8 9.20GB/s ± 1% pdf _ZFlat5-8 991MB/s ± 0% html4 _ZFlat6-8 379MB/s ± 0% txt1 _ZFlat7-8 352MB/s ± 0% txt2 _ZFlat8-8 396MB/s ± 1% txt3 _ZFlat9-8 327MB/s ± 1% txt4 _ZFlat10-8 1.33GB/s ± 1% pb _ZFlat11-8 605MB/s ± 1% gaviota "go test -test.bench=. -tags=noasm" _UFlat0-8 621MB/s ± 2% html _UFlat1-8 494MB/s ± 1% urls _UFlat2-8 23.2GB/s ± 1% jpg _UFlat3-8 1.12GB/s ± 1% jpg_200 _UFlat4-8 4.35GB/s ± 1% pdf _UFlat5-8 609MB/s ± 0% html4 _UFlat6-8 296MB/s ± 0% txt1 _UFlat7-8 288MB/s ± 0% txt2 _UFlat8-8 309MB/s ± 1% txt3 _UFlat9-8 280MB/s ± 1% txt4 _UFlat10-8 753MB/s ± 0% pb _UFlat11-8 400MB/s ± 0% gaviota _ZFlat0-8 409MB/s ± 1% html _ZFlat1-8 250MB/s ± 1% urls _ZFlat2-8 12.3GB/s ± 1% jpg _ZFlat3-8 132MB/s ± 0% jpg_200 _ZFlat4-8 2.92GB/s ± 0% pdf _ZFlat5-8 405MB/s ± 1% html4 _ZFlat6-8 179MB/s ± 1% txt1 _ZFlat7-8 170MB/s ± 1% txt2 _ZFlat8-8 189MB/s ± 1% txt3 _ZFlat9-8 164MB/s ± 1% txt4 _ZFlat10-8 479MB/s ± 1% pb _ZFlat11-8 270MB/s ± 1% gaviota For comparison (Go's encoded output is byte-for-byte identical to C++'s), here are the numbers from C++ Snappy's make CXXFLAGS="-O2 -DNDEBUG -g" clean snappy_unittest.log && cat snappy_unittest.log BM_UFlat/0 2.4GB/s html BM_UFlat/1 1.4GB/s urls BM_UFlat/2 21.8GB/s jpg BM_UFlat/3 1.5GB/s jpg_200 BM_UFlat/4 13.3GB/s pdf BM_UFlat/5 2.1GB/s html4 BM_UFlat/6 1.0GB/s txt1 BM_UFlat/7 959.4MB/s txt2 BM_UFlat/8 1.0GB/s txt3 BM_UFlat/9 864.5MB/s txt4 BM_UFlat/10 2.9GB/s pb BM_UFlat/11 1.2GB/s gaviota BM_ZFlat/0 944.3MB/s html (22.31 %) BM_ZFlat/1 501.6MB/s urls (47.78 %) BM_ZFlat/2 14.3GB/s jpg (99.95 %) BM_ZFlat/3 538.3MB/s jpg_200 (73.00 %) BM_ZFlat/4 8.3GB/s pdf (83.30 %) BM_ZFlat/5 903.5MB/s html4 (22.52 %) BM_ZFlat/6 336.0MB/s txt1 (57.88 %) BM_ZFlat/7 312.3MB/s txt2 (61.91 %) BM_ZFlat/8 353.1MB/s txt3 (54.99 %) BM_ZFlat/9 289.9MB/s txt4 (66.26 %) BM_ZFlat/10 1.2GB/s pb (19.68 %) BM_ZFlat/11 527.4MB/s gaviota (37.72 %)
Interested to find out why there's no streaming interface for snappy? Wouldn't
the protocol support it? Would be nice to know that data streams could use
snappy too like the json stuff: `json.NewEncoder(w).Encode(obj)`
Original issue reported on code.google.com by [email protected]
on 25 Jul 2014 at 5:20
Hello,
From my simple benchmark for a util that is using snappy/gzip for compression,
I noticed that snappy is 10x faster than gzip in compression, but it's slower by 2x in decompression.
I'm wondering if I'm using incorrectly or whether this is an expected performance.
Thanks,
The comment (and docs) to NewBufferedWriter
state that "Users must call Close to guarantee all data has been forwarded to the underlying io.Writer":
Line 132 in 674baa8
However, in looking at the Flush/Close implementations, and their usages in tests, it seems that users only need to call Flush
, not Close
, which would allow a caller to continue using the buffered writer.
I would find the Encode
and Decode
functions easier to use if they would extend their dst
argument to its capacity. This would let the user optimally reuse a dst
buffer when processing multiple blocks, without worrying about reallocating or preallocating.
var dst []byte
src := readSomething()
for len(src) > 0 {
dst = snappy.Encode(dst, src)
writeSomething(dst)
src = readSomething()
}
The changes would be pretty simple, swapping a couple len
for cap
calls in the functions. However strictly speaking this is a breaking change; if someone is not expecting the codec to touch anything past len(dst)
they would be unpleasantly surprised. So I wanted to check if there's interest before I prepare a PR (of course also with documentation updates and a benchmark demoing/justifying the change).
code:
stringReader := strings.NewReader(pbMsg.GetBody().GetReqData().GetContent())
snappyReader := snappy.NewReader(stringReader)
rNum, _ := snappyReader.Read(conBytes)
line:pbMsg.GetBody().GetReqData().GetContent() return a string . when using the stringReader as the paramater , the rNum is 0, why? 3q very much
Hey,
I try to use snappy with golang, I see JS and Java has a function named isValidCompressed
which can validate the encoded buffer, but I can't see that function with golang.
I am thinking to use decode function directly. And if it returns error, it means the encoded buffer is invalid. But I am not sure that's safe/correct or not.
Such as below:
byte[] bytes = Snappy.isValidCompressedBuffer(recordBytes) ? Snappy.uncompress(recordBytes) : recordBytes;
Anyone have any ideas on how to safe and quickly validate the encoded buffer?
I'm trying to get a full working repro, but my current best guess is this was while trying to read a file which had experienced a partial write of the compressed data:
panic: runtime error: slice bounds out of range
goroutine 69 [running]:
panic(0xf4cf00, 0xc82000e0b0)
/go/src/runtime/panic.go:464 +0x3e6
github.com/golang/snappy.(*Reader).Read(0xc89eb43730, 0xcb1ae08000, 0x10000, 0x10000, 0xa, 0x0, 0x0)
/gopath/src/github.com/golang/snappy/decode.go:198 +0xba4
bufio.(*Reader).fill(0xc9eb4603c0)
/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Read(0xc9eb4603c0, 0xc93f20326d, 0xd93, 0xd93, 0xc82b48fd00, 0x0, 0x0)
/go/src/bufio/bufio.go:207 +0x260
io/ioutil.(*nopCloser).Read(0xc91fe04ec0, 0xc93f20326d, 0xd93, 0xd93, 0x20e1843c1a693, 0x0, 0x0)
<autogenerated>:4 +0x82
bufio.(*Scanner).Scan(0xc837df2500, 0x11b6370)
/go/src/bufio/scan.go:208 +0x97b
...
I've encountered an issue with the golang/snappy library where I'm unable to decompress a Snappy compressed JSON file. The error I receive is Failed to decompress content: snappy: corrupt input. However, I've verified that the file is not corrupt by successfully decompressing it using the snzip tool.
Steps to Reproduce:
.option("compression", "snappy")
and write it to s3.package main
import (
"bytes"
"fmt"
"io/ioutil"
"log"
"github.com/golang/snappy"
)
func main() {
// Read the compressed file
content, err := ioutil.ReadFile("path_to_your_snappy_file.snappy")
if err != nil {
log.Fatalf("Failed to read file: %v", err)
}
// Decompress using golang/snappy
decompressed, err := snappy.Decode(nil, content)
if err != nil {
log.Fatalf("Failed to decompress content: %v", err)
}
// Print the decompressed content
fmt.Println(string(decompressed))
}
Observe the error: Failed to decompress content: snappy: corrupt input.
Expected Behavior:
The Snappy compressed JSON file should be decompressed without errors.
Actual Behavior:
Received an error indicating the input is corrupt, even though other tools like snzip can decompress the file without issues.
Additional Information:
The Snappy compressed file is a JSON file where each line is a separate JSON object.
I've verified the integrity of the file by decompressing it using snzip.
The issue might be related to the specific Snappy format or framing used, but I'm not certain.
The existing Go decoded was no doubt tested with go-fuzz. Does the new asm decoder also need heavy testing with bad inputs?
s/goinstall/go get/
Original issue reported on code.google.com by [email protected]
on 17 Jan 2013 at 2:48
Hello,
When I try to test the cmd/influxd/run in the project github.com/influxdata/influxdb in version 1.4.2 it fails with
/usr/share/gocode/src/github.com/golang/snappy/decode_amd64.go:14: missing function body for "decode"
/usr/share/gocode/src/github.com/golang/snappy/encode_amd64.go:14: missing function body for "emitLiteral"
/usr/share/gocode/src/github.com/golang/snappy/encode_amd64.go:19: missing function body for "emitCopy"
/usr/share/gocode/src/github.com/golang/snappy/encode_amd64.go:24: missing function body for "extendMatch"
/usr/share/gocode/src/github.com/golang/snappy/encode_amd64.go:29: missing function body for "encodeBlock"
Thank you for your help
The signature is:
func Encode(dst, src []byte) ([]byte, error)
The current implementation only ever returns nil for the errors.
Furthermore, how could this function ever return an error? Aren't any possible
values of dst and src valid?
For comparison, encoding/hex has
func Encode(dst, src []byte) int
func Decode(dst, src []byte) (int, error)
Original issue reported on code.google.com by [email protected]
on 1 Oct 2014 at 11:22
I tried compressing a dd
generated file of bigger size using snappy.
And i found the resulting compressed size to be larger than the actual size.
How to avoid it ?
Is fixed size compression possible?
Hi team, we met an error snappy: corrupt input
while using snappy to compress through a TCP connection.
On the source side:
conn.reader = snappy.NewReader(io.Reader(net.Conn))
conn.writeCloser = snappy.NewBufferedWriter(io.WriteCloser(net.Conn))
conn.Write(buf)
On the destination side:
conn.reader = snappy.NewReader(io.Reader(net.Conn))
conn.writeCloser = snappy.NewBufferedWriter(io.WriteCloser(net.Conn))
conn.Read(buf)
We get the error snappy: corrupt input
when conn.Read(buf)
. And this error happens intermittently.
We read some snappy code, and we know that this error is reported because the checksum or the decode result length is wrong. But we use snappy based on a TCP connection and TCP can guarantee data integrity. So if snappy needs several network packages to decode the complete data? Or this is caused by some problem with the network card, the hardware cannot verify the data correctly.
I also find some discussions on the network that suspect this is caused by memory overflow or runtime memory limit not enough. But I cannot make sure. Since we don't get any other error messages.
I'd appreciate any help or suggestions on how to debug. Thanks!
Hi,
why is the encoded (compressed) string larger than the original?
https://play.golang.org/p/J7iL2AwQ37
I believe the contributing instructions in the README are no longer accurate.
They suggest that users follow Go's practice for contributing to snappy-go,
citing the following link:
http://golang.org/doc/contribute.html
But the contributing instructions for Go have changed since the switch to git.
And, it's no longer clear to me how one would submit a patch to this repo for
code review. Further, while Go now uses the Gerrit system for code review it
looks like the older Rietveld system may still be in use here? But I just don't
know for sure.
Original issue reported on code.google.com by bryan.matsuo
on 23 Feb 2015 at 7:29
I would like to see some functions to facilitate building a framed snappy
stream. Some functions that would be helpful:
EncodeFrame(dst, src []byte)
StartFrames(dst)
FinishFrames(dst)
Please provide any additional information below.
The official snappy repository has a document describing the framing format for
snappy (http://code.google.com/p/snappy/source/browse/trunk/framing_format.txt)
Original issue reported on code.google.com by jehiah
on 19 Jul 2012 at 3:58
version 0.0.2, 0.0.3: crash
version 0.0.1: OK
crashes in encode_arm64.s
error cause is
related to snappy, arm64
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:741 +0x230 fp=0x1400159e140 sp=0x1400159e100 pc=0x10303b9f0
github.com/golang/snappy.encodeBlock(0x14000dc7502, 0x1304, 0x1304, 0x14000dc6000, 0x102f, 0x13bc, 0x14000195b01)
$HOME/go/pkg/mod/github.com/golang/[email protected]/encode_arm64.s:666 +0x360 fp=0x140015a61e0 sp=0x1400159e150 pc=0x1037118c0
github.com/golang/snappy.Encode(0x14000dc7500, 0x1306, 0x1306, 0x0, 0x0, 0x0, 0x2, 0x4, 0x140015a62f8)
$HOME/go/pkg/mod/github.com/golang/[email protected]/encode.go:39 +0x17c fp=0x140015a6230 sp=0x140015a61e0 pc=0x103710dfc
I'm sorry if this is flat-out wrong or just too unimportant to fix; I'm new
here.
Encode allocates MaxEncodedLen(len(src)) bytes for dst. However, the (very
well-explained) scenario motivating the formula in MaxEncodedLen(srcLen)
involves an encoder emitting 5-byte-long copy elements each copying four bytes.
But snappy-go can't encounter that scenario because it emits at most
3-byte-long copy elements, each copying at least four bytes, regardless of
block length.
Given the restrictions on what snappy-go emits, the worst case I can see is a
3-byte-long copy element that copies four bytes, followed by a 2-byte-long
literal element with length 61, followed by the literal bytes, repeated for the
whole block: then it takes 66 bytes to encode every 65 bytes. Add three bytes
for the length, and 64K encodes to 65536*(66/65)+3=66547 bytes, 10393 fewer
than the 76940 bytes actually allocated. Maybe someone can find a worse case.
If snappy-go did, in the future, emit 5-byte copy elements, presumably it still
wouldn't do so in a 64K block, because offsets within a 64K block always fit in
two bytes.
(If MaxEncodedLen is meant to represent the worst possible blowup with the
Snappy encoding, rather than the worst possible blowup with snappy-go or other
currently-existing encoders, note that the encoding appears to allow
5-byte-long copy elements to only copy one byte, so the absolute most
pessimistic upper bound may be len(src)*5 plus space for the preamble.)
Original issue reported on code.google.com by [email protected]
on 6 Aug 2014 at 6:18
If Decode() is called with a nil destination buffer and the uncompressed length
is zero, then a nil slice will be returned. This contradicts the docs, which
say that "a newly allocated slice will be returned" in this case.
Repro: http://play.golang.org/p/NiNH49J4M2 . This doesn't actually run because
the playground can't import snappy, but you can copy/paste this and run it on
your machine.
There are two alternatives for fixing this problem:
1. If the current behavior is "as designed", update the docs to say that a nil
slice will be returned of the decompressed output has length 0 and the dst
input is nil.
2. Add a nil check and allocate a zero-length byte slice to handle this case.
I've attached patches for each of these alternatives (but don't apply them both
since the two solutions are mutually exclusive).
Original issue reported on code.google.com by [email protected]
on 28 Aug 2014 at 5:25
Attachments:
go get github.com/golang/snappy
src/github.com/golang/snappy/decode_amd64.s: Assembler messages:
src/github.com/golang/snappy/decode_amd64.s:34: Error: no such instruction: text ·decode(SB),NOSPLIT,$48-56' src/github.com/golang/snappy/decode_amd64.s:36: Error: junk
(FP)' after expression
src/github.com/golang/snappy/decode_amd64.s:36: Error: too many memory references for movq' src/github.com/golang/snappy/decode_amd64.s:37: Error: junk
(FP)' after expression
src/github.com/golang/snappy/decode_amd64.s:37: Error: too many memory references for movq' src/github.com/golang/snappy/decode_amd64.s:38: Error: too many memory references for
movq'
src/github.com/golang/snappy/decode_amd64.s:39: Error: too many memory references for movq' src/github.com/golang/snappy/decode_amd64.s:40: Error: too many memory references for
add'
src/github.com/golang/snappy/decode_amd64.s:41: Error: junk `(FP)' after expression
go version
go version xgcc (Ubuntu 4.9.1-0ubuntu1) 4.9.1 linux/amd64
What steps will reproduce the problem?
Call snappy.Decode(nil, x) where x has the form 0xff 0xff 0xff 0xff ... 0xff
0x00. The function call issues a runtime panic.
What is the expected output? What do you see instead?
No panic and a non-nil error returned, "snappy: corrupt input" or something
reasonable. This is the behavior if the input has fewer than five 0xff bytes.
What version of the product are you using? On what operating system?
Revision 12e4b4183793 (go1.3.1-linux-amd64)
Please provide any additional information below.
There seem to be panics in three different locations, depending on the length
of the input. I put together a gist showing the lengths at which each different
panic occurs.
https://gist.github.com/bmatsuo/2cad6e15c1e3a04d9879
I'm not actually positive these inputs are invalid by the spec. I just stumbled
into them testing behavior of "github.com/mreiferson/go-snappystream" when
decoding junk input.
Original issue reported on code.google.com by bryan.matsuo
on 8 Sep 2014 at 2:11
Until now we've been importing snappy using code.google.com/p/snappy-go/snappy
but recently this has caused this issue:
package code.google.com/p/snappy-go/snappy: unable to detect version control system for code.google.com/ path
By changing the import to github.com/golang/snappy/snappy
, we seem to have resolved the issue.
Is this the recommended way to reference this library?
Can you please include the file needed for testing (snapp_test.go):
words, err := ioutil.ReadFile("/usr/share/dict/words")
Original issue reported on code.google.com by [email protected]
on 14 Nov 2011 at 8:05
Any chance you'd be willing to tag releases following semver?
Why we can not customize encode/decode buffer size when initializing a new reader or writer through a function argument.
NSQ use snappy for its message encode/decode with one connection one snappy encoder/decoder. This will consume more and more memory according to the connections number as each connection will use 64KB memory.
Also, this is more friendly for uses cases where the uncompressed data are almost all in small size.
With version 0.0.2 on Golang 1.15, the tests are failing on ARMv7hl and i686:
Testing in: /builddir/build/BUILD/snappy-0.0.2/_build/src
PATH: /builddir/build/BUILD/snappy-0.0.2/_build/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin
GOPATH: /builddir/build/BUILD/snappy-0.0.2/_build:/usr/share/gocode
GO111MODULE: off
command: go test -buildmode pie -compiler gc -ldflags " -X github.com/golang/snappy/version=0.0.2 -extldflags '-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld '"
testing: github.com/golang/snappy
github.com/golang/snappy
--- FAIL: TestDecode (0.00s)
snappy_test.go:365: #29 (decodedLen=0; tagCopy4, 4 extra length|offset bytes; with msb set (0x93); discovered by go-fuzz):
got "", snappy: unsupported literal length
want "", snappy: corrupt input
FAIL
exit status 1
FAIL github.com/golang/snappy 0.628s
As of commit 6880122, I see the following errors. checking out the previous commit resolves.
$ git checkout HEAD^
Previous HEAD position was 45da9d9... Zero out only that part of the hash table in use.
HEAD is now at 6880122... Write the encoder's encodeBlock in asm.
[lloyd@scylla snappy]$ go test
# github.com/golang/snappy
asm: invalid instruction: 00231 (/home/lloyd/dev/go/src/github.com/golang/snappy/encode_amd64.s:335) MOVWQZX table+120(SP)(R11*2), R15
asm: invalid instruction: 00237 (/home/lloyd/dev/go/src/github.com/golang/snappy/encode_amd64.s:340) MOVW AX, table+120(SP)(R11*2)
asm: invalid instruction: 00586 (/home/lloyd/dev/go/src/github.com/golang/snappy/encode_amd64.s:476) MOVW AX, table+120(SP)(R11*2)
asm: invalid instruction: 00603 (/home/lloyd/dev/go/src/github.com/golang/snappy/encode_amd64.s:485) MOVWQZX table+120(SP)(R11*2), R15
asm: invalid instruction: 00607 (/home/lloyd/dev/go/src/github.com/golang/snappy/encode_amd64.s:489) MOVW AX, table+120(SP)(R11*2)
asm: asm: assembly of ./encode_amd64.s failed
FAIL github.com/golang/snappy [build failed]
[lloyd@scylla snappy]$ git checkout HEAD^
Previous HEAD position was 6880122... Write the encoder's encodeBlock in asm.
HEAD is now at 17e4358... Restrict the scope of the tableSize variable.
[lloyd@scylla snappy]$ go test
PASS
ok github.com/golang/snappy 0.532s
It is great that this snappy golang library is getting some love, especially assembly language love. Nice!
A while back I wrote a Go implementation of the framing/streaming spec for snappy. It is here:
https://github.com/glycerine/go-unsnap-stream
If the maintainers of golang/snappy would like to adopt my implementation and integrate the framing implementation with golang/snappy library or a future Go standard lib for snappy, you have my blessing. (Especially if you want to re-write parts in assembly to make it even faster.)
Thanks again for improving golang/snappy.
I have an application which writes large volume of data to Victoria Metrics using prometheus remote write API. The API specs requires compressing the data being written using snappy encoding. When I do memory profile of my application, the snappy encoding alone is using ~900MB of memory.
This is problematic because I process data in real time using multiple replicas of my application and then write the processed data to Victoria Metrics. So the total memory usage by all replicas becomes too high.
A similar issue, where snappy decode is consuming high memory in prometheus: prometheus/prometheus#13174
In decodedLen()
,
if uint64(int(v)) != v {
return 0, 0, errors.New("snappy: decoded block is too large")
}
However, the snappy format description (and the C and C++ implementations) limit this varint to 32-bits:
https://github.com/google/snappy/blob/master/format_description.txt
The stream starts with the uncompressed length (up to a maximum of 2^32 - 1),
stored as a little-endian varint.
This check should instead use
if uint64(uint32(v)) != v {
return 0, 0, errors.New("snappy: decoded block is too large")
}
I can file a PR here or a CL if this project is using gerrit if you'd like.
This bug dates from the Go 1.1 integer size change.
Hi,
this is a follow-up to #30. The same build but for golang 1.6 fails for another reason:
# github.com/golang/snappy
asm: invalid instruction: 00228 (/home/travis/gopath/src/github.com/golang/snappy/encode_amd64.s:338) MOVWQZX table+120(SP)(R11*2), R15
asm: invalid instruction: 00234 (/home/travis/gopath/src/github.com/golang/snappy/encode_amd64.s:343) MOVW AX, table+120(SP)(R11*2)
asm: invalid instruction: 00589 (/home/travis/gopath/src/github.com/golang/snappy/encode_amd64.s:506) MOVW AX, table+120(SP)(R11*2)
asm: invalid instruction: 00606 (/home/travis/gopath/src/github.com/golang/snappy/encode_amd64.s:515) MOVWQZX table+120(SP)(R11*2), R15
asm: invalid instruction: 00610 (/home/travis/gopath/src/github.com/golang/snappy/encode_amd64.s:519) MOVW AX, table+120(SP)(R11*2)
asm: asm: assembly of ../../golang/snappy/encode_amd64.s failed
Full log is located here, with the error message starting on line 348 - https://gist.github.com/serejja/b0c40abc844ab4ce1c8630afa988e3bb
Just to provide a bit more info (which is in that log anyway):
Please let me know if this is not related to golang/snappy itself and I should open a ticket on Travis CI.
Thanks!
I saw all example used this code
rawContent, err := ioutil.ReadFile(localSnappyFilePath)
if err != nil {
return err
}
content, err := snappy.Decode(nil, rawContent)
if err != nil {
return err
}
but my VM just 1 GB RAM, and the files size is 2GB, i can't read it all into memory
thanks your hard work and your patience 🙏
The following program crashes with a panic:
package main
import "github.com/golang/snappy/snappy"
func main() {
data := []byte("\xff\xff\xff\xff\xff\xff\xff\xff\xec0")
n, err := snappy.DecodedLen(data)
if err != nil || n > 1e6 {
return
}
dec, err := snappy.Decode(nil, data)
if err != nil {
if dec != nil {
panic("dec is not nil")
}
return
}
if len(dec) != n {
println(len(dec), n)
panic("bad decoded len")
}
n = snappy.MaxEncodedLen(len(dec))
enc, err := snappy.Encode(nil, dec)
if err != nil {
panic(err)
}
if len(enc) > n {
panic("bad encoded len")
}
}
panic: runtime error: index out of range
goroutine 1 [running]:
github.com/golang/snappy/snappy.Decode(0x0, 0x0, 0x0, 0xc208041f00, 0xa, 0x20, 0x0, 0x0, 0x0, 0x0, ...)
src/github.com/golang/snappy/snappy/decode.go:54 +0xaca
main.main()
snappy.go:11 +0xf6
on commit 156a073
I guess adding 5.8 Mb snappy.test
in 6ffc20e was a mistake?
Getting this error upon attempting to run:
`go get github.com/golang/snappy
asm: 00001 (/root/go/src/github.com/golang/snappy/encode_arm64.s:30) TEXT "".emitLiteral(SB), NOSPLIT, $32-56: unaligned frame size 32 - must be 8 mod 16 (or 0)
asm: 00119 (/root/go/src/github.com/golang/snappy/encode_arm64.s:264) TEXT "".encodeBlock(SB), $32896-56: unaligned frame size 32896 - must be 8 mod 16 (or 0)
asm: assembly failed`
I looked at the history of snappytool and it seems like the tool was purposely rewritten to use block format instead of stream format. What's the reason behind this? Why can't the snappytool support both stream and block format?
I would love to have stream format support because, then, I can use the tool to decode a snappy encoded stream (that is stored in some DB) for troubleshooting and debugging purposes.
Hi,
one of our projects is experiencing Travis CI build failures because of golang/snappy.
The build job on golang 1.4 is erroring with a following message:
# github.com/golang/snappy
../../golang/snappy/encode_amd64.s:208 redeclaration of end
../../golang/snappy/encode_amd64.s:564 redeclaration of end
Full log is located here, with the error message starting on line 350 - https://gist.github.com/serejja/e4593342cb002370c3c95326a6cce1c0
Just to provide a bit more info (which is in that log anyway):
Thanks!
Will this project be migrating to github as code.google.com will be shutting
down in January 2016?
Original issue reported on code.google.com by jehiah
on 16 Apr 2015 at 1:53
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.