This will be a bit tricky, but we should allow a given galaxy injection to decide between different injection types (e.g. postage stamp vs model fits). This will be useful for cases where the seeing of the deep-field coadd is worse than the SE chip we're injecting into.

The bal_config file should keep track of whether the tile injection for each realization was successful or not - this will help immensely when keeping track of long tile re-runs.

There is a directory config where we can put example config files (see config/balrog-test01.yaml which is an example full balrog config)

We can move e.g. single_epoch.yaml there

The latest HexGrid run seems to duplicate positions for every other column, adding multiple galaxies together. This is a recent change, and is likely due to the added center position in the HexGrid.

We're going to start using issues again! To start off with, we need to implement star injection. Sahar's star lists are found at: /data/des20.b/data/sallam/Yanny-balrog/5sigma/, where we can select what star density we want (for our fiducial 10% realization density, we would grab the list from Extra_10_percent_16.5-26.5). There is a list with RA/DEC/MAG w/ zeropoint of 30 for all tiles.

Star injections will be done in SE same as galaxies. We will need new IO functions for Sahar's catalogs, write a new method for finding stars contained in a chip, and to update add_gs_injection() to take care of both (but allow for only one!).

The tricky part seems to be with the GalSim config file. We can set multiple galaxy types (w/ 'galaxy' type 2 being a star) with the following:

gal :
    type : List
    items : 
        - type : ngmixGalaxy
          # Existing galaxy injection
          #...
          index : [pos_1, pos_2, ...]
        
        - type: delta_function
           index : [pos_1, pos_2, ...]
           # Rest of star parameters, like mag/flux
    
    index : # Decide how to sample from type 1 vs type 2

But this also requires setting gal[index] to something sensible to determine when to injection from type 1 or type 2. So far, I haven't gotten the config to work using the above format while also specifying exact image coordinates for stars / galaxies.

To get galsim single_epoch.yaml to work, one needs to replace the call to os.environ['BALROG_DIR'] with the full path to BALROG_DIR instead. Might need an import os somewhere?

To speed things up, we should allow CSV files to pass tile geometry information as well as check that each file has the necessary columns first.

To minimize code redundancy, we should have a main plotting library that everyone can use for data exploration and test making. We should also have a standardized plotting suite to produce all of the common figures.

The main injection code balrog_injection.py has blown up way out of scale; I think it's well past time to break up this monolithic code into smaller, well contained files.

Some of the stars in Sahar's catalogs lie outside of the unique tile region. They won't be rendered in the final coadd image but are still contained in the truth catalog, so they give misleading detection efficiencies. We should add a check in generate_stars that removes these objects.

By the end of the Balrog workshop at Fermilab next week, we should converge on all flag cuts (e.g. flags, obj_flags, mof_cm_flags, etc.) and value cuts (e.g. s2n, T, T_err, etc.) that will be used in the main DES Y3 run.

A quirk with megamixer requires us to replace the output path directory structure from .../$realization/... to .../$realization/$medsversion/... to run successfully.

Large realization runs (~10) are approaching the maximum time limit on Fermi machines so we should split the job into single-realization chunks. We should allow bal_config to pass not just the total number of realizations n_realizations but also which ones are to be computed with that job in the following ways:
(1) The total number of realizations to produce (what we're currently doing),
(2) A range of realizations (e.g. 2-4),
(3) A list of realizations (e.g. [0, 3, 4, 7]).
This should mostly be an input parsing fix as the realization number is only important for names and directory structures.

As injection requests get more complicated, it may be a good idea to separate Balrog configuration from GalSim configuration. This could go a long way in modularizing Balrog, allowing for custom versions of all simulation steps, and making Balrog more general #7.

While completing #5, I ran into a few test cases where there were no stars to be injected into a few chips. It now handles this correctly but there are a few misleading warnings that are printed (like the star catalog defaulting to using g-band even though no stars will be injected).

We need to (1) complete the same checks for galaxies (even though this should never occur) and (2) find a way around the misleading warnings while keeping the current injection structure.

Most of this is already handled by GalSim - but since we deal with 'modified' GalSim yaml files, we'll have to update a bit of the checks done in config._load_input_catalogs() to make sure we don't mess up the input indexing for add_gs_injection(). This isn't an issue for adding a constant shear but there are a few edge cases for PowerSpectrumShear, for example.

We should also add the relevant ellipticity-ellipticity (difference) correlation function to the standard plots (#19). There is a prototype plotting routine there but it needs some updating.

This will be open for a while - we will identify slow parts of the code that we can use to speed up injection. The latest run has injection taking nearly as long as runing SOF which is far too slow!

First up is handling large objects. I profiled the injection code and roughly half of the runtime is in _drawKImage(). Messing with max_T and min_T shows that a few very large objects (cm_T>10) can slow down a chip injection by well over 100% the mean time. This appears to be a bigger issue with the new combined catalog y3_balrog_test_input_mof as it has a larger cm_T tail.

We need to find a way to either:

1. Make reasonable cuts combining cm_T and cm_T_err to reduce the size of the large T tail, or
2. Make more aggressive cuts on max_T (possible selection issues), or
3. Make a change to GalSim to allow error handling for the case of insufficient max_fft_size and explore its effect on selection.
4. Some combination of all 3!

The other approach is to solve #6 which will certainly help speed up runtime, but it will have diminishing returns as 1% of objects can take ~90% of runtime in the worst cases.

Not enough people are raising issues.

More people should do this.

I needed a feature for this work, which I added to desmeds

please git pull and update when you get a chance

For the TAMU run, it would be better to use a deeper input catalog like the COSMOS catalog that ships with GalSim (or the DES mof catalog).

Currently we do not add noise. For faint objects, the existing sky noise is probably sufficient, but for bright objects, the current implementation will underestimate the actual noise.

We need to get ready for potential deep-field coadd injections. GalSim already supports real images, but it will take some linking to have it compatible with the current version of balrog_injection.py.

Currently, an index array is generated for all Balrog injections given the number of injected objects. However, this causes issues for simulation runs for input types that are not catalogs. We should add a check that generates index only for classes that inherit from galsim.catalog (or something similar).

As CCD defects tend to be masked in rows or columns, it could be useful to rotate an injection grid off of the RA/DEC axis. We would simply add an additional (optional) parameter to grid initialization.

Right now we are using a single (randomly chosen) Y3 MOF catalog as inputs for testing. Now that we are larger tests with multiple tiles at (integrated) 100% DES density, we should have a larger input catalog to inject from. This could be as simple as a union of multiple Y3 MOF catalogs.

Currently, balrog_injection.py expects a certain directory structure for the input nullweight images and can only do injections on a tile-wide basis. This was a design choice for the main survey-wide Balrog run, but there have already been requests for using Balrog for clusters and transient studies that don't conform to this structure.

We should think about how to implement the current structure to allow for more general injection strategies in the future. I think we have to live with full-tile injections to replicate the DESDM stack properly, but we should allow for tile-subset injections, different property (including RA,DEC) sampling methods, and possibly transient injections.

Any other thoughts?

balrog_injection.py has strange behavior when running using multiple processes on Fermi machines. Locally, I can run injection w/ nproc : 8 without issues; but on DES machines, running with nproc>2 quickly (~6 injections) leads to the following error:

Traceback (most recent call last):
  File "/cvmfs/des.opensciencegrid.org/eeups/fnaleups/Linux64/galsim/1.5.0alpha/bin/galsim", line 275, in <module>
    main()
  File "/cvmfs/des.opensciencegrid.org/eeups/fnaleups/Linux64/galsim/1.5.0alpha/bin/galsim", line 256, in main
    except_abort=args.except_abort)
  File "/cvmfs/des.opensciencegrid.org/eeups/fnaleups/Linux64/galsim/1.5.0alpha/lib/python2.7/site-packages/galsim/config/process.py", line 780, in Process
    except_abort=except_abort)
  File "/cvmfs/des.opensciencegrid.org/eeups/fnaleups/Linux64/galsim/1.5.0alpha/lib/python2.7/site-packages/galsim/config/output.py", line 144, in BuildFiles
    except_abort = except_abort)
  File "/cvmfs/des.opensciencegrid.org/eeups/fnaleups/Linux64/galsim/1.5.0alpha/lib/python2.7/site-packages/galsim/config/process.py", line 971, in MultiProcess
    result = job_func(**kwargs)
  File "/cvmfs/des.opensciencegrid.org/eeups/fnaleups/Linux64/galsim/1.5.0alpha/lib/python2.7/site-packages/galsim/config/output.py", line 222, in BuildFile
    data = builder.buildImages(output, config, file_num, image_num, obj_num, ignore, logger)
  File "/cvmfs/des.opensciencegrid.org/eeups/fnaleups/Linux64/galsim/1.5.0alpha/lib/python2.7/site-packages/galsim/config/output.py", line 473, in buildImages
    image = galsim.config.BuildImage(base, image_num, obj_num, logger=logger)
  File "/cvmfs/des.opensciencegrid.org/eeups/fnaleups/Linux64/galsim/1.5.0alpha/lib/python2.7/site-packages/galsim/config/image.py", line 237, in BuildImage
    image, current_var = builder.buildImage(cfg_image, config, image_num, obj_num, logger)
  File "./injector.py", line 44, in buildImage
    return super(BalrogImageBuilder, self).buildImage(config, base, image_num, obj_num, logger)
  File "./injector.py", line 14, in buildImage
    im, cv = super(AddOnImageBuilder, self).buildImage(config, base, image_num, obj_num, logger)
  File "/cvmfs/des.opensciencegrid.org/eeups/fnaleups/Linux64/galsim/1.5.0alpha/lib/python2.7/site-packages/galsim/config/image_scattered.py", line 118, in buildImage
    self.nobjects, base, logger=logger, obj_num=obj_num, do_noise=False)
  File "/cvmfs/des.opensciencegrid.org/eeups/fnaleups/Linux64/galsim/1.5.0alpha/lib/python2.7/site-packages/galsim/config/stamp.py", line 106, in BuildStamps
    except_func = except_func)
  File "/cvmfs/des.opensciencegrid.org/eeups/fnaleups/Linux64/galsim/1.5.0alpha/lib/python2.7/site-packages/galsim/config/process.py", line 939, in MultiProcess
    raise res
EOFError

However: It always runs successfully for nproc=2 ! This is baffling to me, but w/ 2 processors galaxy injection is fast enough that it hasn't been a pressing issue. Should still look into soon.

When making the grid image, a few (~3 or so out of 4,000) of the objects have a smaller postage stamp added on top:


I have no idea what could be causing this! We should look for similar cases in non-grid images.

@mcclearyj found the following error when using image files with the image not in HDU extension 0:

ValueError: WCS does not have longitude type of 'RA', therefore (ra, dec) data can not be returned

While GalSim natively can handle this case with ext : 1, for example, this doesn't currently propagate to injector.py. I believe the fix is simply to add the following to injector.AddOnImageBuilder.buildImage():

try:
    ext = config['wcs']['ext']
except KeyError:
    ext = 0
initial_image = galsim.fits.read(initial_image_name, hdu=ext)

@mcclearyj: Can you confirm that this solved your issue? Thanks!

The most convenient way to implement #5 is to have all star-related inputs and variables be set in the bal_config fields input and image : Balrog. We should make this uniform for all other inputs; e.g. tilelist, geom_file, tile_dir, etc.

This will also help with #7.

When the code was first written, we were extremely conservative about what sorts of input and config types were allowed to be passed in the bal_config to ensure that we were doing things correctly.

However, this is too conservative: It leads to some unexpected and unnecessary compatibility issues when doing simple tests using typical galsim types.

We should change all of the allowed_types into safe_types or supported_types and only warn the user if using one of the unsupported types. This shifts some of the responsibility to the user to do reasonable things, but this is an ok compromise for flexibility.

For grid runs, it will be a common desire to remove any injection-injection blends due to objects whose size are of order the grid. This should be an additional config parameter for grid objects.

The title says it all - we want to do MOF vs SOF tests, so sof needs to (1) be added as a valid input for ngmix_catalog, and (2) any flag or column specifics need to be accounted for.

Currently, Balrog galaxies are injected without noise. Most noise contributions will be inherited from the existing image but we still need to include shot noise with the injections.

Importantly, we should pick a single seed for this and save to the truth catalog metadata for future reference.

Now that we have many injection runs it is easy to get confused which run came from which version (as Nikolay's pipeline needs the same directory structure). The easiest solution is to add simulation metadata (including which injection output dir run it came from) to the truth table header.

Random position sampling currently accounts for issues with the RA=0/360 degree overlap region. We should make sure that grid objects handle this case correctly as well.

Currently injections are all (incorrectly) assuming zeropoints of 30, which is consistent with the input catalog. See Brian's slack comment:

For example for tile DES0347-5540, there's a list file here: /data/des71.a/data/kuropat/balrog_test/y3v02/DES0347-5540/lists/DES0347-5540_r_nullwt-flist-y3v02.dat
The first line of this file is: /data/des71.a/data/kuropat/balrog_test/y3v02/DES0347-5540/nullwt-r/D00253910_r_c54_r2366p01_immasked_nullwt.fits 31.70017433166504
So for any r-band flux laid down on expnum 253910 ccnum=54, that flux should be multiplied by pow(10,(31.70017-30)/2.5) = 4.787 and so forth.

This means we need the following changes:

• Pass data version in bal_config : image
• New chip method to grab the zeropoint from the .dat file
• Either scale flux explicitly, or tell galsim to do so (which I prefer)
  All of this is making me think about how to make Balrog IO more general (see #7), but for now we can assume that the current Balrog image class will really become a Balrog_DES class, or something similar.

The current definitions of these input parameters is the total/final number of galaxies or galaxy density after all injection realizations. This has shown to be counterintuitive, and we should change the definitions to be the injection number/density per realization.

We should update the docs to reflect this as well.

This is currently done by Nikolay's pipeline, but we should move it into balrog_injection.py as (1) it would be the user's expected behaviour and (2) it will allow us to modify them when needed like with grid tests.

There is currently a small issue with the grid and blank tests in which nullwt chips that do not contain any injections are correctly skipped - but then their values are not reset for the test! We should think of the right way to handle this without having to trick GalSim into running over a chip with no injections.

The first pass implementation of pos_sampling : grid was not very flexible. We should allow additional inputs to configure how the grid is constructed.

Looking to the future, we may also want to add some kind of rejection sampling on the grid given the shapes / sizes of certain grid objects.

One possible issue for the grid test is that the SCI images no longer match the BKG images. Instead of adding onto a blank background and adding noise consistent with the WGT map, we could instead inject on top of the BKG images and pass these along to MEDs, along with the original BKG images.

Currently, setting the simulation header with

gal :
    rotate : Random

Will cause different rotation angles of the same object between different bands. This will have to be set in generate_galaxies so it can be called consistently in add_gs_injection().

It looks like one of the previous updates for handling directory path inputs has broken the parsing of relative paths set in bal_config. The constructor of Config now converts all paths to be absolute, so it is likely just a change of which os function is being used to concatenate paths.

Let's try and get a simple script that runs:
i) Downloading necessary input images (desmeds prep tile command)
ii) Galsim (can start with a dummy Galsim run i.e. just copy the SE images to a different directory iii) Swarp
iv) SExtractor light - some comparison of resulting SExtractor catalogs to the DESDM ones would be good here.

(Once step iv) looks ok, move on to making meds files and running MOF)

Stolen from slack:
The ngmix_catalog should know which band to inject in given the nullwt name. But it is being overwritten by the global config that is passed in bal_config.yaml. It was copied over a long time ago when ngmix_catalog simulations were done in a slightly different way. So the global

input : ngmix_catalog
    bands : 'griz'
    ....

(which is being used to determine how many bands to iterate over) is also overwriting each chip injection to inject all bands at every injection

Star injection is currently handled by dividing up the total star catalog for a given tile into n_realizations=len(reals) groups. However, this doesn't allow for consistent star injection across multiple single realization runs as the star groups are randomly shuffled.

We should allow n_realizations to be a simultaneous config input along with realizations, and n_realizations>len(realizations) would indicate that the star catalog should not be shuffled so future runs can complete the total catalog injection.

Blank and Grid runs ignore the initial images but don't currently change the background noise in the header! This leads to some significant diagnostics issues:
Blank Run:

Normal Run:

We should read the background noise for each chip and then add the noise level for blank and grid runs.

Moved this from #36 to distinguish issues. We now have a base Grid class and an implemented RectGrid that take inputted grid parameters as well as tile metadata. We now should implement HexGrid which does the same on a hexagonal grid. This will likely be used for the positions of the main Balrog run if we do not use uniform scatter to minimize Balrog blends.

Now that we are adding stars, we need to be more careful to screen out stars from the input galaxy catalog. This can be done by adding some additional checks (modest_cuts?) in the ngmix_catalog initialization.