Comments (7)
aminnj
commented on May 22, 2024
1
Just to make sure... we see that 3 baskets are read when only one should be read if the printout were lazy
julia> @time using UnROOT ; @time const tf = LazyTree(ROOTFile("Run2012BC_DoubleMuParked_Muons.root"),"Events", ["Muon_pt"])
julia> ENV["JULIA_DEBUG"] = UnROOT
julia> keep = vcat([1:15 ,10^7 .+ (1:15), 5*10^7 .+ (1:15)]...) ; println(keep)
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 10000001, 10000002, 10000003, 10000004, 10000005, 10000006, 10000007, 10000008, 10000009, 10000010, 10000011, 10000012, 10000013, 10000014, 10000015, 50000001, 50000002, 50000003, 50000004, 50000005, 50000006, 50000007, 50000008, 50000009, 50000010, 50000011, 50000012, 50000013, 50000014, 50000015]
julia> @time LazyTree(UnROOT.innertable(tf)[keep])
┌ Debug: Read branch Muon_pt, basket 1, 0.0177459716796875 MB compressed, 0.02355194091796875 MB uncompressed
└ @ UnROOT ~/.julia/dev/UnROOT/src/root.jl:460
┌ Debug: Read branch Muon_pt, basket 461, 7.881964683532715 MB compressed, 10.52783203125 MB uncompressed
└ @ UnROOT ~/.julia/dev/UnROOT/src/root.jl:460
┌ Debug: Read branch Muon_pt, basket 509, 8.197038650512695 MB compressed, 10.862678527832031 MB uncompressed
└ @ UnROOT ~/.julia/dev/UnROOT/src/root.jl:460
0.251167 seconds (621 allocations: 81.347 MiB, 55.20% gc time)
Row │ Muon_pt
│ SubArray{Float3
─────┼──────────────────────────────────────
1 │ [10.8, 15.7]
2 │ [10.5, 16.3]
3 │ [3.28]
4 │ [11.4, 17.6, 9.62, 3.5]
5 │ [3.28, 3.64, 32.9, 23.7]
6 │ [3.57, 4.57, 4.37]
7 │ [57.6, 53.0]
8 │ [11.3, 23.9]
9 │ [10.2, 14.2]
10 │ [11.5, 3.47]
11 │ [8.82, 17.6]
12 │ [14.6, 12.3]
⋮ │ ⋮
33 rows omittedfrom unroot.jl.
Moelf
commented on May 22, 2024
is this just printing? I forgot what this does exactly:
https://github.com/tamasgal/UnROOT.jl/blob/6dc0838a939ff21413c0cf61e7208829dfa8f8a4/src/iteration.jl#L216-L218
I thought innertable(lt)[rang] materialize those rows
from unroot.jl.
aminnj
commented on May 22, 2024
Yeah it's materializing it, so technically that UnitRange snippet is not giving a truly LazyTree either
from unroot.jl.
Moelf
commented on May 22, 2024
so technically that UnitRange snippet is not giving a truly LazyTree either
it can't, because that means we need a view-like structure where we note the source LazyTree and what index we're looking at.
I wonder what @view does?
from unroot.jl.
aminnj
commented on May 22, 2024
Interesting. The view gives the same printout but skips reading basket 509. Though I don't know why it's 1/461/1.
julia> LazyTree(UnROOT.innertable(tf)[keep])
┌ Debug: Read branch Muon_pt, basket 1, 0.0177459716796875 MB compressed, 0.02355194091796875 MB uncompressed
└ @ UnROOT ~/.julia/dev/UnROOT/src/root.jl:460
┌ Debug: Read branch Muon_pt, basket 461, 7.881964683532715 MB compressed, 10.52783203125 MB uncompressed
└ @ UnROOT ~/.julia/dev/UnROOT/src/root.jl:460
┌ Debug: Read branch Muon_pt, basket 509, 8.197038650512695 MB compressed, 10.862678527832031 MB uncompressed
└ @ UnROOT ~/.julia/dev/UnROOT/src/root.jl:460
julia> @views LazyTree(UnROOT.innertable(tf)[keep])
┌ Debug: Read branch Muon_pt, basket 1, 0.0177459716796875 MB compressed, 0.02355194091796875 MB uncompressed
└ @ UnROOT ~/.julia/dev/UnROOT/src/root.jl:460
┌ Debug: Read branch Muon_pt, basket 461, 7.881964683532715 MB compressed, 10.52783203125 MB uncompressed
└ @ UnROOT ~/.julia/dev/UnROOT/src/root.jl:460
┌ Debug: Read branch Muon_pt, basket 1, 0.0177459716796875 MB compressed, 0.02355194091796875 MB uncompressed
└ @ UnROOT ~/.julia/dev/UnROOT/src/root.jl:460from unroot.jl.
Moelf
commented on May 22, 2024
I think the printout you're seeing are purely from displaying at REPL:
julia> @time LazyTree(UnROOT.innertable(tf)[10^7:2*10^7:end]);
┌ Debug: Read branch Muon_pt, basket 461, 7.881964683532715 MB compressed, 10.52783203125 MB uncompressed
└ @ UnROOT ~/.julia/dev/UnROOT/src/root.jl:460
┌ Debug: Read branch Muon_pt, basket 485, 8.226861000061035 MB compressed, 10.893505096435547 MB uncompressed
└ @ UnROOT ~/.julia/dev/UnROOT/src/root.jl:460
┌ Debug: Read branch Muon_pt, basket 509, 8.197038650512695 MB compressed, 10.862678527832031 MB uncompressed
└ @ UnROOT ~/.julia/dev/UnROOT/src/root.jl:460
0.148089 seconds (649 allocations: 120.398 MiB, 16.84% gc time)
julia> @time @views LazyTree(UnROOT.innertable(tf)[10^7:2*10^7:end]);
0.000001 secondsbtw, we already return a collection of LazyEvent now when the slicing is not UnitRange:
julia> @time tf[10^7:2*10^7:end];
0.000002 seconds (1 allocation: 112 bytes)
julia> @time tf[[1, end-100, end]];
0.000003 seconds (2 allocations: 192 bytes)
julia> typeof(tf[10^7:2*10^7:end])
Vector{LazyEvent{from unroot.jl.
Moelf
commented on May 22, 2024
julia> LazyTree("./test/samples/tree_with_int_array_zstd.root", "t1")[[1,3]]
Row │ a
│ Int32
─────┼───────
1 │ 0
2 │ 2this is now solved
from unroot.jl.
Related Issues (20)
- Opening `km3net_online.root` causes huge memory usage spike HOT 2
- `LazyTree()` hang regression in 0.10.16
- Pre-compilation failure after upgrading to v1.9.3 HOT 6
- Performance for trees with a large number of branches HOT 13
- Fix Documentation due to their 1.0 release
- `RNTuple` reading extremely slow
- `nanoAOD_ttbar` latency HOT 26
- CI broken on nighly due to MD5.jl using SHA.jl internals
- RNTuple RC2 compatibility
- Do not manage to read a TTree with a structure of arrays of basic types HOT 17
- Cannot read empty collections from a RNTuple file HOT 1
- ConcurrencyViolationError when reading with XRootD HOT 2
- [RNTuple] Wrong offset `Index32/Index64` array when read from multiple pages HOT 7
- [RNTuple] accessing nested structs is not lazy enough HOT 1
- [WIP] 0.11.0 breaking changes items
- Re-write resources with `Base.Lockable`
- [RNTuple] OutOfMemoryError in show() HOT 2
- Reading a file with Branches and Leafs HOT 11
- [RNTuple] miss-aligned column in DAOD_TLA with RNTuple RC2 HOT 2
- xrootd doesn't handle XCache
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from unroot.jl.