I have added an option to group multiple bit planes into a chunk (default is 4). To change it, provide, say --bit_planes_per_chunk 8, when encoding. With this change, the chunks are about 150KB on average.

The chunk size can also be changed with two options when encoding:

• --bricks_per_chunk (default is 512). For example, --bricks_per_chunk 4096 should create chunks of size 1 MB ish.
• --brick_size (default is 32 32 32). For example, --brick_size 64 64 64 should also create chunks of size 1 MB ish.

The first option (increasing the bricks-per-chunk) will result in more metadata to be stored, but is better in that the bricks are kept small so fewer samples need to be decoded when we ask for a small region of interest (e.g., a slice).

TODO

• Can we plot the impact of the chunk size on data retrieval time?
• Can we allow fetching parts of an object? (so that each chunk does not have to be stored as a file)

e.g. FileSystem.h, StackTrace.h

Current status:

idx2 dataset can be uploaded and retrieved from object storage.

Limitations

@scrgiorgio: can you upload your powerpoint slides with the TODO list?

• Small chunk size (see #8)

TODO

@scrgiorgio: can you add instructions for the following?

• How to generate an idx2 dataset that can be uploaded to the cloud
• How to upload such a dataset to the cloud?
• How to retrieve such a dataset from visusviewer/OpenViSUS python?

Hi,

I'm getting some error when installing the idx2. Please check it.

zhangbo@zhangbo-PC:~/Codes/idx2/build$ make install
Consolidate compiler generated dependencies of target faster
[ 10%] Built target faster
Consolidate compiler generated dependencies of target idx2Core
[ 79%] Built target idx2Core
Consolidate compiler generated dependencies of target idx2
[ 83%] Built target idx2
Consolidate compiler generated dependencies of target idx2App
[ 87%] Built target idx2App
Consolidate compiler generated dependencies of target idx2App_v2
[ 91%] Built target idx2App_v2
Consolidate compiler generated dependencies of target idx2Samples
[ 95%] Built target idx2Samples
Consolidate compiler generated dependencies of target idx2CompareVolumes
[100%] Built target idx2CompareVolumes
Install the project...
-- Install configuration: ""
-- Installing: /home/zhangbo/Codes/idx2/install/lib/libidx2.a
CMake Error at Source/cmake_install.cmake:50 (file):
  file INSTALL cannot find "/home/zhangbo/Codes/idx2/Source/idx2.h
  idx2_v2.h": No such file or directory.
Call Stack (most recent call first):
  cmake_install.cmake:47 (include)

It looks like either idx2Config.cmake is missing from the repo, or the cmake configuration to generate it is missing. I get:

CMake Error at Source/cmake_install.cmake:73 (file):
  file INSTALL cannot find "/home/pdavis/idx2/Source/idx2Config.cmake": No
  such file or directory.
Call Stack (most recent call first):
  cmake_install.cmake:47 (include)

and indeed that file does not exist.

1. Receiving an error:

 oid idx2::Print(idx2::printer*, const idx2::array<t>&)’:
/home/pdavis/idx2/Source/Core/Format.h:33:3: error: there are no arguments to ‘idx2_Print’ that depend on a template parameter, so a declaration of ‘idx2_Print’ must be available

Which can be reduced to a warning with '-fpermissive`.

 #if defined(__clang__)

Has a trailing tick (`) at a couple different places, which gcc doesn't like.

3. The following define in FileSystem.cpp (line 27) causes gcc some parsing confusion:

#define stat struct stat

in combination with the define on line 27, it causes line 220 to expand to a syntax error. I'm not sure what the portable solution is, but for linux I could just remove the line 27 define and change line 219 to struct stat S.

4. Pthread isn't being linked for linux machines.

5. GCC scope struct stat from the idx2 namespace on line 10786 of idx2.hpp, and I can't figure out how to make it not do so. I finally resorted to renaming idx2::stat to idx2::idx_stat, which is somewhat ugly and requires changing about a dozen variable types just after line 40609.

6. Source/Core/String.cpp has to explicitly include cerrno in gcc