decibelcooper / proio Goto Github PK

This is so that parallel threads working on separate events can take on the burden of serialization.

In addition, add code to C++ and Python Readers to construct descriptors in memory from the file descriptor protos.

Hi, David

Now I've noticed that we are missing one variable in mc.proto

optional uint32 id = 12; // original position in the MC generator

It is important to have it. After slimming event records, I need to know the original position of particles in the generator. So far I was using "barcode" for this, but "barcode" is reserved for something else (it is used by pileup mixing tool. It keeps event numbers from which the particle originates.

Can you please add it? After this change, mc.proto record should be 100% compatible with any generator or tool I know.

best, Sergei

This issue is described by @sbinet here.

In the Go implementation, entry Types are cached:

The C++ implementation should do the same with Descriptors so that a string-based lookup is not done for every entry.

Thanks to Alexander Kiselev for pointing this out.

Identified by protobuf errors (wrong wiretype and field id). Saw this when simulating and digitizing high-energy events and keeping all calorimeter eic::SimHits. Events were on O(10 MB) each, uncompressed. It seems to be an issue with the github.com/pierrec/lz4 reader when decompressing these large frames, but I can't be sure yet.

We need to include "model" mc.proto and other template files inside created PROIO files.
Therefore, I would suggested to copy the directory "model" to the installation location. Then one can link such files using the path $PROIO/model/mc.proto (as an example) during file creation of files.

There should be also a mechanism to store such files (and log files) inside PROIO

best, Sergei

This is already in the Go implementation.

• Reader::NextHeader() is for scanning the document without reading bucket data.
• Event::RemoveEntry() is for, well...

Needs to be recreated for the new scheme.

We need a support for URL for proio/java-proio/src/main/java/proio/Reader.java
(I've implemented it in my version of this class). It should look as this:

import java.net.*;

.....



 public Reader(String filename) throws IOException {


     // first URL
     if (filename.startsWith("http") || filename.startsWith("ftp")) {
                                URL url = null;
                                try {
                                        url = new URL(filename);
                                } catch (MalformedURLException e) {
                                        System.out.println("Error in accessing the URL="
                                                        + url.toString());
                                }
                               try {
                                        // System.out.println("Reading from URL: " + file);
                                        URLConnection urlConn = url.openConnection();
                                        urlConn.setDoInput(true);
                                        urlConn.setUseCaches(false);
                                        InputStream inStream = urlConn.getInputStream();
                                        stream = CodedInputStream.newInstance(inStream);
                                } catch (MalformedURLException e) {
                                        System.out.println(e.toString());
                                } catch (IOException e) {
                                        System.out.println(e.toString());
                                }

    } else {

    // normal file
    fileStream = new FileInputStream(filename);
    stream = CodedInputStream.newInstance(fileStream);

    }

  }

The C++ LZ4 library is compact enough and enough of a hassle that packaging it with cpp-proio might be a good idea.

Need...

• good introduction
• comprehensive examples
• clear build/install instructions

Ideally these will be generated by the respective builds on the user's machine to match the user's protobuf library. The go generated code is a special case, and will stay for the foreseeable future.

It might be nice to implement either a fwk Input/OutputStreamer or a standalone Component in the go-proio source code itself. @sbinet, any thoughts?

This comes about from discussion with Alexander Kiselev. The idea is to alleviate the entry object allocation overhead (that is significant in some scenarios) by Clearing previously allocated entries of the desired type.

Entries with IDs are owned by the Event at all times, and are accessible and mutable in memory up until the point that the Event is either Pushed or destroyed. Currently, Events at this point destroy allocated entries. Instead, it would be advantageous to Clear the entries and return them to a global store to be used again.

Would like to provide root dictionaries and a TTree implementation

Needs to be recreated for the new scheme

Hi, David

I've typed "make", and after 5 min of dong something with the singularity container, it failed to compile

Importing: /tmp/singularity-cache/docker/sha256:1024a55a68004f50fe1cfdb2c6353faeadc033c343d632c8e198091901bcfc45.tar.gz
/bin/tar: opt/Go/.wh..wh..opq: implausibly old time stamp 1969-12-31 18:00:00
Importing: /tmp/singularity-cache/metadata/sha256:90a76215fbf315cbb77cb867de4eaf6e7bc8b44ea70ddf6bab70378ab8ea112f.tar.gz
Done. Container is at: proio-gen.img
singularity exec -C -H /proio proio-gen.img protoc --gofast_out=go-proio/model proio.proto
ERROR  : Base home directory does not exist within the container: /users
ABORT  : Retval = 255
make: *** [go-proio/model/proio.pb.go] Error 255

Is it possible to have an option for cpp version using make, assumed gcc is installed?

Looks like I've gimped the Go Skip() functionality with a bug here:

Proio Readers should be able to handle errors and recover from them. This should be added to each implementation's tests.

Hi, David

It looks like the class proio/Reader.java cannot use the method seekToStart for streams (it works only for local files).
This creates a problem for random access of events using URL (the method skip() is not sufficient for this). Can you add a support for streams in the seekToStart() method?

best, Sergei

This may ideally be in Markdown in a new model/doc/eic directory.

@whit2333 in case you're interested

Also authors/contributors lists. Model after https://github.com/go-hep/hep.