unique process of compression about fastpfor HOT 13 CLOSED

Can you provide a test case? (source code)

I do not understand the issue you are reporting.

from fastpfor.

I want to compress two ipv4 addresses, so i define a vector contained two
32 bits integers. After that, i need to know if the compressed data is
writen in 32 bits max or no, for thus i need to display the compressed data
in form of decimal. The later will be considered as another address (writen
in maximum 32 bits). It is possible ?? Transform two address on one can be
performed with this process of compression?? I can deal with the original
addresses differently (vector contained eight 8 bits integers), my goal is
to compress the information in max 32 bits word.

2015-05-09 1:53 GMT+02:00 Daniel Lemire [email protected]:

Can you provide a test case? (source code)

I do not understand the issue you are reporting.

—
Reply to this email directly or view it on GitHub
#20 (comment).

from fastpfor.

@mounamouna

This library is ill-suited for the purpose you describe. Though you can certainly can encode an array containing two 32-bit integers, it is unlikely that the result will be a single 32-bit integer in general.

This library is meant for computing arrays containing many integers. Please see the example:

https://github.com/lemire/FastPFor/blob/master/example.cpp

I am closing this issue as invalid.

If you do find a bug, please provide a reproducible test case.

from fastpfor.

Please Sir ,just a final question, "computing arrays containing many
integers",
that means we are able to compute each integer in the initial vector
through the compressed data???? so the compressed data is a vector contains
integers smaller then integers in the initial vector???That is right??

Thanks in advance Sir.

2015-05-09 2:42 GMT+02:00 Daniel Lemire [email protected]:

@mounamouna https://github.com/mounamouna

This library is ill-suited for the purpose you describe. Though you can
certainly can encode an array containing two 32-bit integers, it is
unlikely that the result will be a single 32-bit integer in general.

This library is meant for computing arrays containing many integers.
Please see the example:

https://github.com/lemire/FastPFor/blob/master/example.cpp

I am closing this issue as invalid.

If you do find a bug, please provide a reproducible test case.

—
Reply to this email directly or view it on GitHub
#20 (comment).

from fastpfor.

that means we are able to compute each integer in the initial vector
through the compressed data?

Of course.

so the compressed data is a vector contains integers smaller then integers in the initial vector?

The goal of the library is to have fewer integers in the compressed vector. Yes.

from fastpfor.

It isn't logical to have the same compressed vector for two different
initial vectors. that's right? I tested with two different initial vectors
(a,b) and (a1,b) but the compressed vector compressed_output.data()[] is
the same. Is it a bug ??

2015-05-09 3:33 GMT+02:00 Daniel Lemire [email protected]:

that means we are able to compute each integer in the initial vector
through the compressed data?

Of course.

so the compressed data is a vector contains integers smaller then integers
in the initial vector?

The goal of the library is to have fewer integers in the compressed
vector. Yes.

—
Reply to this email directly or view it on GitHub
#20 (comment).

from fastpfor.

Yes it is a bug. It is most likely a bug in your code.

from fastpfor.

mouna@ubuntu:~/newtmp/FastPFor$ ./example

Compressed data 19984

Compressed data 23

You are using 0.109 bits per integer.

Decompressed data 1 4294967295

Decompressed data 2 4294967295

mouna@ubuntu:~/newtmp/FastPFor$ make example

[ 85%] Built target FastPFor

Scanning dependencies of target example

[100%] Building CXX object CMakeFiles/example.dir/example.cpp.o

Linking CXX executable example

[100%] Built target example

mouna@ubuntu:~/newtmp/FastPFor$ ./example

Compressed data 19984

Compressed data 23

You are using 0.109 bits per integer.

Decompressed data 1 4967295

Decompressed data 2 4294967295

What is the problem? I used 32 bits integers, i changed the first integer
but the compressed vector still the same.

2015-05-08 18:51 GMT-07:00 Daniel Lemire [email protected]:

Yes it is a bug. It is most likely a bug in your code.

—
Reply to this email directly or view it on GitHub
#20 (comment).

from fastpfor.

mouna@ubuntu:~/newtmp/FastPFor$ ./example

Compressed data 19984

Compressed data 23

You are using 0.109 bits per integer.

Decompressed data 1 4294967295

Decompressed data 2 4294967295

mouna@ubuntu:~/newtmp/FastPFor$ make example

[ 85%] Built target FastPFor

Scanning dependencies of target example

[100%] Building CXX object CMakeFiles/example.dir/example.cpp.o

Linking CXX executable example

[100%] Built target example

mouna@ubuntu:~/newtmp/FastPFor$ ./example

Compressed data 19984

Compressed data 23

You are using 0.109 bits per integer.

Decompressed data 1 4967295

Decompressed data 2 4294967295

What is the problem? I used 32 bits integers, i changed the first one but
the compressed vector still (19984 , 23).

2015-05-09 3:51 GMT+02:00 Daniel Lemire [email protected]:

Yes it is a bug. It is most likely a bug in your code.

—
Reply to this email directly or view it on GitHub
#20 (comment).

from fastpfor.

If you think you have found a bug, please submit a test case.

from fastpfor.

test case
first:
mydata[0] = 4294967295;
mydata[1] = 4294967295;
second:
mydata[0] = 4967295;
mydata[1] = 4294967295;

2015-05-08 19:30 GMT-07:00 Daniel Lemire [email protected]:

If you think you have found a bug, please submit a test case.

—
Reply to this email directly or view it on GitHub
#20 (comment).

from fastpfor.

The size of the compressed vector is most certainly more than two words. Probably four words. That is, the "compressed" vector is probably larger than the input vector.

These arrays you provide are not compressible using this library. They are too short.

Please read the papers, study carefully the code and the examples.

• Daniel Lemire and Leonid Boytsov, Decoding billions of integers per second through vectorization, Software Practice & Experience 45 (1), 2015. http://arxiv.org/abs/1209.2137 http://onlinelibrary.wiley.com/doi/10.1002/spe.2203/abstract
• Daniel Lemire, Leonid Boytsov, Nathan Kurz, SIMD Compression and the Intersection of Sorted Integers, Software Practice & Experience (to appear) http://arxiv.org/abs/1401.6399
• Jeff Plaisance, Nathan Kurz, Daniel Lemire, Vectorized VByte Decoding, International Symposium on Web Algorithms 2015, 2015. http://arxiv.org/abs/1503.07387
• Wayne Xin Zhao, Xudong Zhang, Daniel Lemire, Dongdong Shan, Jian-Yun Nie, Hongfei Yan, Ji-Rong Wen, A General SIMD-based Approach to Accelerating Compression Algorithms, ACM Transactions on Information Systems 33 (3), 2015. http://arxiv.org/abs/1502.01916

I am not going to be able to help you further.

from fastpfor.

Thank you Sir.

2015-05-09 5:09 GMT+02:00 Daniel Lemire [email protected]:

The size of the compressed vector is most certainly more than two words.
Probably four words. That is, the "compressed" vector is probably larger
than the input vector.

These arrays you provide are not compressible using this library. They are
too short.

Please read the papers, study carefully the code and the examples.

• Daniel Lemire and Leonid Boytsov, Decoding billions of integers per
  second through vectorization, Software Practice & Experience 45 (1), 2015.
  http://arxiv.org/abs/1209.2137
  http://onlinelibrary.wiley.com/doi/10.1002/spe.2203/abstract
• Daniel Lemire, Leonid Boytsov, Nathan Kurz, SIMD Compression and the
  Intersection of Sorted Integers, Software Practice & Experience (to appear)
  http://arxiv.org/abs/1401.6399
• Jeff Plaisance, Nathan Kurz, Daniel Lemire, Vectorized VByte
  Decoding, International Symposium on Web Algorithms 2015, 2015.
  http://arxiv.org/abs/1503.07387
• Wayne Xin Zhao, Xudong Zhang, Daniel Lemire, Dongdong Shan, Jian-Yun
  Nie, Hongfei Yan, Ji-Rong Wen, A General SIMD-based Approach to
  Accelerating Compression Algorithms, ACM Transactions on Information
  Systems 33 (3), 2015. http://arxiv.org/abs/1502.01916

I am not going to be able to help you further.

—
Reply to this email directly or view it on GitHub
#20 (comment).

from fastpfor.