Comments (2)
Moelf
commented on June 5, 2024
To find out if this is a bug in the Reco_tf.py with RNTuple step, or in our Julia library, I have opened up the same resultant RNTuple.data22_13p6TeV.00440447.physics_TLA.merge.DAOD_TLA._lb0635._SFO-ALL._0001.root in (Py)ROOT and now compare the information in the first 100 events:
import ROOT
df = ROOT.RDataFrame("CollectionTree", "./RNTuple.data22_13p6TeV.00440447.physics_TLA.merge.DAOD_TLA._lb0635._SFO-ALL._0001.root")
df2 = df.Range(100)
pts = [list(i) for i in list(df2.Take['ROOT::VecOps::RVec<std::float_t>']("HLT_AntiKt4EMPFlowJets_subresjesgscIS_ftf_TLAAux:.pt"))]
pbs = [list(i) for i in list(df2.Take['ROOT::VecOps::RVec<std::float_t>']("HLT_AntiKt4EMPFlowJets_subresjesgscIS_ftf_TLAAux::fastDIPS20211215_pb"))]
for pt, pb in zip(pts, pbs):
print(pt," => ", pb)[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[225832.0625, 199013.65625] => [0.059916406869888306, 0.0674041360616684]
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[270950.78125, 135270.375, 80851.171875, 36513.28515625, 23011.748046875, 32175.6640625] => [0.04527727887034416, 0.03434162959456444, 0.0377415232360363, 0.04799623787403107, 0.05184456706047058, 0.0377415232360363]
[] => []
[] => []
[] => []
[] => []
[114320.1484375, 81667.375, 29592.568359375, 23742.60546875, 25626.587890625] => [0.04169735312461853, 0.5882907509803772, 0.02801903337240219, 0.13336534798145294, 0.016203496605157852]
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[118424.234375, 50642.171875, 40561.140625, 35867.19140625] => [0.10440044105052948, 0.024244185537099838, 0.4974711239337921, 0.7921126484870911]
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[204712.078125, 198915.0, 25886.50390625, 24531.337890625, 31552.33984375] => [0.03016129694879055, 0.3731847107410431, 0.09278646856546402, 0.026132898405194283, 0.16553984582424164]
[171653.78125, 149278.453125, 36532.859375, 20924.8828125] => [0.023812301456928253, 0.04269582778215408, 0.05409770458936691, 0.08165643364191055]
[] => []
[] => []
[] => []
[] => []
[] => []
[] => []
[55841.7265625, 50764.0078125, 32384.1015625, 28708.12890625] => [0.5095839500427246, 0.11230865865945816, 0.08891720324754715, 0.020344357937574387]
[109319.671875, 100309.6953125, 28350.01953125] => [0.03540632501244545, 0.111589215695858, 0.09328745305538177]
[] => []
[] => []
[47301.1796875, 46569.56640625, 27481.79296875] => [0.0411888025701046, 0.04025974124670029, 0.0377415232360363]
[190133.96875, 153411.8125, 144305.515625, 94753.984375, 38096.0625, 21210.4375, 22168.083984375, 30078.56640625] => [0.008741471916437149, 0.34812411665916443, 0.0377415232360363, 0.0377415232360363, 0.07163631916046143, 0.11629883199930191, 0.0954534038901329, 0.23611751198768616]
[] => []
[192810.296875, 93130.6796875, 78054.328125, 59670.0546875, 29768.62109375, 30117.498046875] => [0.05335315316915512, 0.895403265953064, 0.02925291657447815, 0.09248718619346619, 0.0377415232360363, 0.0377415232360363]
[93971.734375, 49757.83203125, 40307.98046875, 40243.8359375, 34162.44921875, 29140.591796875, 25661.353515625, 20558.78125, 21356.53125] => [0.23926283419132233, 0.0412287563085556, 0.06981765478849411, 0.23113372921943665, 0.02090582810342312, 0.1142696663737297, 0.07911603152751923, 0.111677385866642, 0.044584933668375015]
[] => []
[] => []
[] => []
[] => []
[] => []Conclusion
looks like our Julia library is bugged
from unroot.jl.
Moelf
commented on June 5, 2024
So I believe this is a case where the schema extension in the footer bites us:
julia> findfirst(!isempty, rnt_pts)
37
julia> footer = data22_aod_rnt.var"HLT_AntiKt4EMPFlowJets_subresjesgscIS_ftf_TLAAux::fastDIPS20211215_pb".rn.footer;
julia> println.(footer.extension_header_links.field_records);
...
UnROOT.FieldRecord(field_version=0x00000000, type_version=0x00000000, parent_field_id=0x00000431, struct_role=0x0001, flags=0x0000, repetition=0, field_name="HLT_AntiKt4EMPFlowJets_subresjesgscIS_ftf_TLAAux::fastDIPS20211215_pb", type_name="std::vector<float>", type_alias="", field_desc="", )
UnROOT.FieldRecord(field_version=0x00000000, type_version=0x00000000, parent_field_id=0x00000431, struct_role=0x0000, flags=0x0000, repetition=0, field_name="_0", type_name="float", type_alias="", field_desc="", )
UnROOT.FieldRecord(field_version=0x00000000, type_version=0x00000000, parent_field_id=0x00000433, struct_role=0x0001, flags=0x0000, repetition=0, field_name="HLT_AntiKt4EMPFlowJets_subresjesgscIS_ftf_TLAAux::fastDIPS20211215_pc", type_name="std::vector<float>", type_alias="", field_desc="", )
UnROOT.FieldRecord(field_version=0x00000000, type_version=0x00000000, parent_field_id=0x00000433, struct_role=0x0000, flags=0x0000, repetition=0, field_name="_0", type_name="float", type_alias="", field_desc="", )
UnROOT.FieldRecord(field_version=0x00000000, type_version=0x00000000, parent_field_id=0x00000435, struct_role=0x0001, flags=0x0000, repetition=0, field_name="HLT_AntiKt4EMPFlowJets_subresjesgscIS_ftf_TLAAux::fastDIPS20211215_pu", type_name="std::vector<float>", type_alias="", field_desc="", )
...
julia> footer.extension_header_links.column_records
103-element Vector{UnROOT.ColumnRecord}:
UnROOT.ColumnRecord(type=0x000e, nbits=0x0040, field_id=0x000003ef, flags=0x00000008, first_ele_idx=36, )
UnROOT.ColumnRecord(type=0x0011, nbits=0x0020, field_id=0x000003f0, flags=0x00000000, first_ele_idx=0, )
UnROOT.ColumnRecord(type=0x000e, nbits=0x0040, field_id=0x000003f1, flags=0x00000008, first_ele_idx=36, )
UnROOT.ColumnRecord(type=0x0011, nbits=0x0020, field_id=0x000003f2, flags=0x00000000, first_ele_idx=0, )
UnROOT.ColumnRecord(type=0x000e, nbits=0x0040, field_id=0x000003f3, flags=0x00000008, first_ele_idx=36, )
UnROOT.ColumnRecord(type=0x0011, nbits=0x0020, field_id=0x000003f4, flags=0x00000000, first_ele_idx=0, )
UnROOT.ColumnRecord(type=0x000e, nbits=0x0040, field_id=0x000003f5, flags=0x00000008, first_ele_idx=36, )
UnROOT.ColumnRecord(type=0x0011, nbits=0x0020, field_id=0x000003f6, flags=0x00000000, first_ele_idx=0, )
UnROOT.ColumnRecord(type=0x000e, nbits=0x0040, field_id=0x000003f7, flags=0x00000008, first_ele_idx=36, )
UnROOT.ColumnRecord(type=0x0011, nbits=0x0020, field_id=0x000003f8, flags=0x00000000, first_ele_idx=0, )
UnROOT.ColumnRecord(type=0x000e, nbits=0x0040, field_id=0x000003f9, flags=0x00000008, first_ele_idx=36, )
...As you can see, there are a bunch of things aligned to start with index 36 (37 in Julia), which is when the HLT_AntiKt4EMPFlowJets_subresjesgscIS_ftf_TLAAux becomes non-empty, thus these fieds and columns are added in.
Including our POI:
"HLT_AntiKt4EMPFlowJets_subresjesgscIS_ftf_TLAAux::fastDIPS20211215_pb"
from unroot.jl.
Related Issues (20)
- Opening `km3net_online.root` causes huge memory usage spike HOT 2
- `LazyTree()` hang regression in 0.10.16
- Pre-compilation failure after upgrading to v1.9.3 HOT 6
- Performance for trees with a large number of branches HOT 13
- Fix Documentation due to their 1.0 release
- `RNTuple` reading extremely slow
- `nanoAOD_ttbar` latency HOT 26
- CI broken on nighly due to MD5.jl using SHA.jl internals
- RNTuple RC2 compatibility
- Do not manage to read a TTree with a structure of arrays of basic types HOT 17
- Cannot read empty collections from a RNTuple file HOT 1
- ConcurrencyViolationError when reading with XRootD HOT 2
- [RNTuple] Wrong offset `Index32/Index64` array when read from multiple pages HOT 7
- [RNTuple] accessing nested structs is not lazy enough HOT 1
- [WIP] 0.11.0 breaking changes items
- Re-write resources with `Base.Lockable`
- [RNTuple] OutOfMemoryError in show() HOT 2
- Reading a file with Branches and Leafs HOT 11
- xrootd doesn't handle XCache
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from unroot.jl.