yaml_generator.py currently fails when trying to simulate one of the newer "TIGHT" NIRCam dither patterns. The addition of "TIGHT" to the dither pattern name (e.g. FULL 3TIGHT), means that the yaml_generator currently cannot pull out the 3, in order to determine the number of primary dithers.

From Leonardo Ubeda:
/Users/lubeda/anaconda2/envs/jwst_dev/lib/python3.5/site-packages/nircam_simulator-1.0-py3.5.egg/nircam_simulator/scripts/yaml_generator.py

primarytot = np.int(file_dict['PrimaryDithers'])
try:
subpixtot = np.int(file_dict['SubpixelPositions'])
except:
subpixtot = np.int(file_dict['SubpixelPositions'][0])
primary_dither = np.ceil(1. * tot_dith / subpixtot)
subpix_dither = tot_dith - (primary_dither * primarytot * subpixtot - subpixtot)

The primarytot variable is read as a number so the case of dither = 3 works fine, and the case of dither = 3TIGHT does not.

Tyler Desjardins mentions that we should consider moving emails from help[at]stsci.edu to point to the web portal where possible and appropriate. For HST (or any non-JWST), it is https://hsthelp.stsci.edu . For JWST, it is https://jwsthelp.stsci.edu . Please update info in setup.py, setup.cfg, documentation, etc as appropriate.

Please close this issue if it is irrelevant to your repository. This is an automated issue. If this is opened in error, please let pllim know!

xref spacetelescope/hstcal#317

Testing of the "stmag" option for a point source list showed that this option did not work for me on the witserv3 machine. I found that the condition used to test whether to switch from abmag to stmag was never returning True. In fixing this I also added an option to use the "vegamag" string to enable use of the vegamag column in the NIRCam_zeropoints.list or niriss_zeropoints.list file. Hence the addition of the normal magnitudes should work for both instruments. I have only tested the change with NIRISS.

Add a method that takes a yaml input parameter file and outputs a traps filled file. This way a user can specify a random scene, create a persistence map from it, and then use that as input for their own observations.

Then you just need to modify Kevin’s function to calculate persistence in an observation given the initial traps filled file.

Currently the suffix is 'IPC_NIRCam_XX' where XX is the detector name. Simplify this to 'IPC_NIRCam' and have the code add the detector name. This will make the creation of yaml files easier.

Target types other than point source can be specified at the moment, but if you specify a 2d sersic, there is currently no way to specify size, position angle, sersic index, etc.

I was trying to use the simulator on an input dark exposure, and it errored out on the invert_ipc_kernel function (inside obs_generator.py) with a NameError, I think because "nyc" and "nyx" are not defined.

For simulations in preparation of astrometric commissioning, it is very useful to implement a no-distortion mode in which the optical system is assumed to be perfect.

I get an error that the FITS import is missing in unlinearize.py.

/ifs/jwst/wit/witserv/data7/nrc/acanipe/simulated_images/newest_version/nircam_simulator/nircam_simulator/scripts/unlinearize.py in unlinearize(image, coeffs, sat, lin_satmap, maxiter, accuracy, robberto, save_accuracy_map, accuracy_file)
74 devcheck = np.copy(dev)
75 devcheck[i2] = -1.
---> 76 h0 = fits.PrimaryHDU()
77 h1 = fits.ImageHDU(devcheck)
78 hl = fits.HDUList([h0,h1])

NameError: name 'fits' is not defined

Currently PSFs are constant across a detector and in fact the same for all detectors. New webbpsf includes field dependence.

Also need to update to more realistic wavefront error values. Current PSF library is too conservative in terms of WFE.

I very well might be mistaken here, but in my use of the simulator, I have found that the generation of the observationlist.yaml file seems to be a non-essential step. I have been using scripts/write_observationlist.py to automatically generate these .yaml files from APT XML and pointing files.

If there are no cases where including the observationlist.yaml provides something that cannot otherwise be parsed from APT output (and please let me know if so!), it seems like it might be more efficient to skip this step, and go straight from the APT output to .yaml generation.

Mimic what is done by the ETC, and allow a user to choose 'low', 'medium', or 'high' for the backgound level, in addition to the option of allowing the user to enter a countrate for the background

In order to make more clear that this is not a full-up instrument simulator, and due to the plans to add support for NIRISS, we need to rename the repo. This will necessitate a few changes in the code, mostly related to import statements. I believe I will have to ask the spacetelescope admins to actually do the renaming. I believe only they have the proper permissions.

Suggested new name:
Multi-Instrument RAmp GEnerator (MIRAGE).

Update the way that the pixel area map is used. Currently the final seed image is divided by the pixel area map, which is not correct for point sources.

Volk says that the PAM should be applied only to extended sources as well as zodi/backgrounds.

At the moment there are lots of overlapping Level 3 inputs in the yaml files. For example, the visit_id is an input. But the visit_id can be created by combining the program ID, visit number, and observation number. Simplify the yaml file by requiring only the basic information, and leave it to the code to construct the other metadata.

Assuming we are carrying the linearized as well as raw darks in the reference file collection, it makes more sense for the yaml generator to populate yaml files with linearized darks rather than raw darks, in order to save computation time.

This simulator should be expanded to support FGS imaging as well (in addition to the recently added NIRISS capability).

The main body of work is probably to assemble all the necessary input files and the PSF library, although Kevin Volk has done most of that work already.

Kevin Stevenson mentioned the Batman package (https://www.cfa.harvard.edu/~lkreidberg/batman/) that can be used to create a simulated lightcurve for a transiting exoplanet.

Also may need to look into file formats, assuming that time series data are composed of many integrations that may need to span multiple files.

Currently the simulator does not do quite the correct thing when adding IPC effects into the data. IPC is added in at the end, after the seed image has been added to the dark ramp. But the dark ramp, regardless of whether it is raw or linearized, actually already has IPC effects in it.

My quick test showed that the ideal case of adding IPC effects to a seed+dark ramp that has no IPC effects is not quite the same as adding IPC effects to the seed image and then adding that to a dark ramp that has IPC effects. (Differences in my tests were less than 1 DN between the two cases.)

So to do things properly, what we really should do is include IPC correction in the dark current linearization process. Then when IPC effects are added at the end, they will be added to a combined seed+dark ramp that has no IPC effects at all. This would necessitate changing the linearized darks in the reference file library, as well as adding IpcStep to the linearization process. This also means the simulator would need both forward and backwards IPC kernels. The inversion function can translate from one to the other, but then we'd need a way for the user to specify which kernel they provided in the input yaml file.

The other option would be to get to the case that is almost correct, by moving the addition of IPC effects to somewhere before the seed image is added to the dark ramp (but after the addition of cosmic rays).

Make updates to the basic switches/modes that are needed to allow NIRISS imaging to run.

Kevin Volk will collect and supply the necessary reference/configuration files.

Currently, the simulator is looking for NIRCam, NIRISS, and FGS reference files in both of the following directories (and maybe others?):

• /ifs/jwst/wit/nircam/nircam_simulator_data/
• /ifs/jwst/wit/niriss/nircam_ramp_simulation_files/

Standardizing how reference files are organized, located, and named would serve to more easily keep the simulator references up-to-date, and would also make it easier to expand to instruments besides NIRCam, as is being discussed in issue #62.

I don't know much about reference files or CRDS, but another question: should we ultimately aim to be getting the most recent versions of reference files from crds, in /grp/crds/cache/references/jwst?

Enhance the yaml_generator script (and dependents) to convert the content of an NIRISS APT file into simulator input files.

To simplify the repository structure and improve maintainability, mirage should depend of pysiaf for accessing SIAF information, e.g. the focal plane geometry and the location of subarrays.

https://github.com/spacetelescope/pysiaf

Investigate whether the source locations are correct, or if a tangent plane projection adjustment is needed.

@Johannes-Sahlmann already has a function that makes this adjustment.

For instance, these two files have inconsistent nomenclature:
config/NIRCam_subarray_definitions.list
config/niriss_subarrays.list

This may partially be superseded when the dependence on pysiaf will be introduced.

catalogs.py, with separate functions that take in arrays of appropriate values and write out each kind of catalog.

catalogs.write_pointsource()
catalogs.write_galaxy()
catalogs.write_extended()
catalogs.write_moving_target()

Initial cropping and blotting functions have been developed and tested. Package up in order to produce a proper seed image that matches the format of the seed image created via catalog.

I found a silly bug where PSF library file names are not correct if you happen to have a source that has a subpixel location that is ~exactly at a tenth of a pixel.

For example, the NIRISS PSF library has PSFs at 0.1 pixel intervals across the pixel. I have a source that happens to fall at x,y = 100.2, 100.3. The PSF filename is derived by finding the subpixel location of the object and rounding to two decimal places before making it a string. So 0.2 should become "0.20". But, it appears that python's round function ignores your requested number of digits after the decimal point if there are trailing zeros. See the example below:

Request 2 digits after the decimal:
In [12]: round(0.199999999, 2)
Out[12]: 0.2

Request 5 digits after the decimal:
In [13]: round(0.199999999, 5)
Out[13]: 0.2

So, catalog_source_image.py needs to be updated to force the number of digits after the decimal to be 2 in all cases.

including use of hdf5 files. Also examples of galaxy with star forming region through use of two sources in ~the same place.

Currently the disperser expects a collection of 2D imaging seed images, which are translated into a single 2d dispersed seed image.

For moving targets, the imaging seed images are 4D. Need to find a way to either call the disperser multiple times while passing one frame of the imaging seed images, or modifying the disperser to accept 4D seed images and return a 4D dispersed seed image. The former is probably the way to go.

Create one APT file for each instrument that can be simulated (for now, NIRCam and NIRISS), with all necessary templates in each file.

Ultimately, compare .yaml output with some kind of reference .yaml.

Use the STScI package template to model test structure for continuous integration:
https://github.com/spacetelescope/stsci-package-template

Basically, run get_catalogs.py and make sure it works. Will require an internet connection to access Vizier.

Add the option to specify the input source magnitude in the Vega systems (in addition to AB and ST mag).

@KevinVolkSTScI has some of the necessary conversion relationships, at least for NIRISS.

Testing of the simulation code showed that the frame time calculated in the code for NIRISS is not correct. The full frame time was given as 10.742 seconds. This is off by 0.05% but the discrepancy was worse for subarrays such as SUB80.
To fix this the calcFrameTime subroutine needs to be revised. This routine is in both obs_generator.py and catalog_seed_image.py. It is not clear why there are two versions of the same code differing only in a print statement in the routine in obs_generator.py. It would be better to have only one calcFrameTime routine.
The fix leaves the NIRCam frame time calculation as it was. I note however that the JDox page ttps://jwst-docs.stsci.edu/display/JTI/NIRCam+Detector+Readout gives the same frame time for NIRCam as for NIRISS in full frame read-out, 10.73677 seconds.
I have coded a fix and tested it for a few NIRISS sub-arrays and the values are as expected. I have not been able to upload the changes (I get an error about my username or password). Since I am using the password that works in the browser to get this issue up, I do not know what the problem is with pulling the changes back.

Need to modify observation list yaml file to be able to supply more than one filter per observation. Code development was done using an example APT file that had only a single filter per observation, and was never expanded to accept multiple filters.

Need to write a technical report in order to fully describe the simulator (also be sure to include ancillary useful code, like the yaml_generator).

TR is in progress. Needs lots of updates from when I started writing it months ago to describe a much less mature code base.

Right now different input catalogs assume different input velocities. Make everything consistent. Maybe arcsec per hour?

This way a user could create a simulator instance, read in the parameter file, but then adjust some parameters before running.

After discussing the current way the simulator incorporates PSFs with Marshall, we feel that multiple collaborators should come together to think about the PSF library in the following contexts:

• The limits of having a discrete, 5x5 sub-pixel grid onto which stars are placed (especially in the context of precision astrometry)
• Redundancy and symmetry in the current organization of the library
• Benefits (and challenges) of referencing one single oversampled PSF and interpolating as appropriate, instead of querying pre-computed PSFs from files
• More complicated versions of interpolation (convolution?)
• Generally, balancing computation time with disk space

The code does not generally locate stars correctly from a point source list when a subarray is being simulated, at least for NIRISS (but probably for NIRCam as well). This was found to be due to having point source positions relative to the subarray in the code, whereas the RADectoXY routine produces return values of (x,y) for the pixel in the full frame coordinates. One needs to subtract the self.subarray_bounds[0] and self.subarray_bounds[1] values to get pixel positions relative to the sub-array. Related to this, the XYtoRADec routine needs input pixel values in the full frame but the code is passing values relative to the subarray so the RA/Dec output values are incorrect. One in that case needs to add self.subarray_bounds[0] and self.subarray_bounds[1] to the pixel values that are being passed into XYtoRADec. I have corrections for these problems ready to post when I get the account issues straightened out.

The fix needs to be put into catalog_seed_image.py in three places: for point sources, for moving targets, and for galaxies/extended sources.

An update to the niriss_subarrays.list file is needed. There was an error in the bounds for SUBSTRIP256 and the SUB80 subarray needed to be replaced by the AMI1 to AMI4 cases as in the SIAF file. Branch niriss-subarray-fix has the corrected subarray file and no other changes.

At the moment the segmentation map is complete if using a point source and a galaxy catalog, but other combos don't work, I think. Maybe create a self.indexbase that can be incremented after each catalog is read in.

When using an installation of mirage, yaml_generator.py writes files into site_packages/*.egg/config/ directory. I don't think that is the desired default location.

I suspect the reason is in line 879 ff of yaml_generator.py:
scripts_path = os.path.dirname(os.path.realpath(__file__))

I stripped photon yield out quite a while ago because NIRCam uses 1.0 for all filters. But NIRISS and FGS require non-zero values, so these calculations will have to be added back in.

Note that there appears to be some old functions left-over from this still in catalog_seed_image.py, although for some reason I was calling it "quantum yield" at the time. All that's left is a function that reads in the values from the old-format zeropoint file, which used to have photon yield coefficients in it.

The updated calculations will have to be more sophisticated than the old ones. From Kevin:
"For the Guider the yield will be strongly correlated with the spectral type of the source because of the wide range of wavelength response. Hence while for NIRISS I just have a mean value of the photon yield per filter, for the Guider that is not going to work very well."

I believe at the moment the Poisson noise is being determined from the signal in ADU. This means the noise levels will be too high by a factor of ~2.2.

The simulator fails in dark_prep.py when requesting an exposure with multiple integrations and starting with a raw dark. It completes successfully when requesting multiple integrations but starting with a linearized dark.

Traceback for this error is below. It looks like when reading in the raw dark to create duplicate integrations, the code is looking for a zero attribute, which isn't there (but is there in the linearized dark).
Requested readout pattern RAPID is valid. Using the nframe = 1 and nskip = 0 Dark shape as read in: (1, 5, 2048, 2048) Requested output has 2 integrations, while input dark has only 1. Adding integrations to input dark by making copies of the input. Traceback (most recent call last): File "test_simulator.py", line 8, in <module> m.create() File "/ifs/jwst/wit/nircam/hilbert/python_repos/nircam_simulator/nircam_simulator/scripts/imaging_simulator.py", line 70, in create d.prepare() File "/ifs/jwst/wit/nircam/hilbert/python_repos/nircam_simulator/nircam_simulator/scripts/dark_prep.py", line 98, in prepare self.darkints() File "/ifs/jwst/wit/nircam/hilbert/python_repos/nircam_simulator/nircam_simulator/scripts/dark_prep.py", line 744, in darkints self.integration_copy(reqints, ndarkints) File "/ifs/jwst/wit/nircam/hilbert/python_repos/nircam_simulator/nircam_simulator/scripts/dark_prep.py", line 759, in integration_copy copyzero = self.dark.zero AttributeError: 'Read_fits' object has no attribute 'zero'

Nor has requested that WCS information is added to seed images, so that his updated tool can potentially extract spectra from the seed images.

This should be relatively straightforward...

With SIAF as an input, aperture reference location V2, V3 values should be pulled from there. They should not be necessary/used from the subarray definition file.

Long-term solution: use pysiaf when it becomes public. At that point there should be no need for SIAF as input either.

Currently the simulator uses some aperture names that differ from the official SIAF naming schemes. These are listed in the "Name" column of the subarray definitions list. For example, the SIAF name NRCA3_DHSPIL_SUB96 is converted to SUB96DHSPILA. This has been causing some problems with parsing APT files, and trying to match the given name to the SIAF name.

I think it would make more sense to just use the SIAF name to refer to the aperture everywhere in the code?

(These efforts could certainly be combined with renaming instruments in issue #67 and renaming config files in PR #69)

Create iPython notebook detailing how to simulate NIRCam (and other instruments?) images during OTE Commissioning