Comments (7)
I'd like to take this, but wanted to clarify:
- When you say constructing work units is a multi-hour operation, do you mean it takes hours to construct work units from
fits
files ? If so, I think we can provide feedback on the number of images loaded from thefits
file during the loading phase. - In terms of feedback for reprojection, an easy implementation can just display
Image i / total
done. We can use\r
character to overwrite the previous line so we avoid havingtotal
number of rows printed. I can't think of a good way to estimate the time, but if my proposed solution is good enough we can start with that.
from kbmod.
- I think this refers to the creation of a
WorkUnit
from anImageCollection
, although it might be good to updates to all of the phases of reprojection including loading in aWorkUnit
from disk/FITs files (which takes a couple of minutes). - In general for other lincc-frameworks projects we tend to use tqdm for tracking progress because it works pretty well out of the box. There are ways to configure it with
multiprocessing
futures as well for the parallel version of reproject.
from kbmod.
Thank you for those clarifications - I'm looking at discrete_piles_e2e.ipynb
to see an example of creating a WorkUnit
from an ImageCollection
. I installed the lsst
package from here (pip install git+https://github.com/lsst/daf_butler.git
). However, I am missing the test data specified in REPO
path in the notebook. Is there some test data you can share ? Otherwise, I can try to use the MockButler
from kbmod.
I think you shouldn't have to worry about installing the LSST stack to do development on this portion of KBMOD! If you just want to test against a basic WorkUnit
we have one we use for testing included in the test directory (this one, it's also the one we run against for the reprojection unit tests). If we wanted to run tests against actual data we could try doing some runs with the data we have our HPC clusters, but I think the unit test one would be a good place to start !
from kbmod.
^ this is just in relation to reprojection, thinking about it you might need the stack to work with ImageCollection
generation, but I might need to defer to another dev about working with that
from kbmod.
@maxwest-uw Made a PR with a start on this. Some points to confirm but let's address them there
from kbmod.
Need to add the TQDM loop to a few of the ImageCollection and WorkUnit functions so that we can track the progress of various large tasks in the reproject/KBMOD search pipeline.
Important requirement: needs to interact nicely with output files on HPCs like klone.
Required Changes
tqdm should be added to the following places in the code:
- https://github.com/dirac-institute/kbmod/blob/main/src/kbmod/image_collection.py#L531
- https://github.com/dirac-institute/kbmod/blob/main/src/kbmod/work_unit.py#L288
- https://github.com/dirac-institute/kbmod/blob/main/src/kbmod/reprojection.py#L129
- [Stretch Goal] https://github.com/dirac-institute/kbmod/blob/main/src/kbmod/reprojection.py#L271
- Working with parallel jobs and tqdm is kind of a PITA, so Iām marking this one as a stretch goal if you can find an easy way to do this.
Testing
- Test piping the output to a file and make sure that we can read the TQDM output nicely when not running directly from the command line.
from kbmod.
Related Issues (20)
- Single Method to Reproject an ImageCollection
- Parallelize reprojection.reproject_work_unit HOT 1
- correct_parallax fails for small r HOT 3
- Specify parallax minimizer algorithm and bounds HOT 1
- WorkUnit .to_fits overwrite option not working
- We lose geocentric distance from original workunit during reprojection HOT 1
- Need to preserve visit as part of the provenance.
- Support numpy v2.0
- Rename ImageCollection `mjd` to `mjd_mid`
- Change how WCS and BBOX work in ImageCollection HOT 1
- Assess zero points / normalization (mosaic and temporal) HOT 1
- Add more exhaustive tests for the standardizer values.
- Exctract the equivalent of Earth Location in standardizers
- Image collection add rename column methods
- add ability for non-lazy reprojection to write to a sharded fits
- add method to load a `WorkUnit` after being lazily initialized
- Need a matching vectorized inverse parallax function
- Need instrument column in ImageCollection
- progress bars should be optional
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
š Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ššš
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ā¤ļø Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kbmod.