aabdurrouf / pixedfit Goto Github PK

View Code? Open in Web Editor NEW

40.0 40.0 7.0 350.99 MB

piXedfit is a Python package designed for analyzing spatially resolved SEDs of galaxies

Home Page: https://pixedfit.readthedocs.io/en/latest/index.html

License: MIT License

Python 100.00%

astronomical-images astronomical-spectroscopy astrophysics galaxies pixel-binning sed-fitting

pixedfit's People

Contributors

Stargazers

Watchers

Forkers

chihuanbin pastroe ra1nl nikhil0504 tylerastro b1geasy ywis

pixedfit's Issues

singleSEDfit error

I have got the latest version of piXedfit, when I run singleSEDfit for the example fluxmap file, I got the KeyError:"Unable to open object (object 'gas_logu' doesn't exist)". I have tried to download the lastest model file from "https://drive.google.com/drive/folders/1YjZGg97dPT8S95NJmO5tiFH9jWhbxuVy?usp=sharing", but it does not work now. The page can not find now.

`images_processing` function not working as expected

Hey @aabdurrouf, thank you for this amazing code!

I am trying using piXedfit for some JWST analysis but I can't seem to get the images_processing function working as I want to.

I have PSF matched images and don't need reprojection ether cause I already did them myself for some other analysis.

Here is my function:

%tb
img_process = images_processing(filters, sci_img, var_img, gal_ra, gal_dec, dir_images=dir_imgs, img_unit=img_unit, 
                                img_scale=img_scale, img_pixsizes=img_pixsizes, kernels=kernels, gal_z=gal_z, 
                                stamp_size=stamp_size, remove_files=False, run_image_processing=True, flag_psfmatch = 1, 
                                flag_reproject = 1, flag_crop = 1)

So the first error that popped up was:

---------------------------------------------------------------------------
UnboundLocalError                         Traceback (most recent call last)
[/Users/nikhilgaruda/Documents/Astronomy_Research/G191/pixedfit/image_processing.ipynb](https://file+.vscode-resource.vscode-cdn.net/Users/nikhilgaruda/Documents/Astronomy_Research/G191/pixedfit/image_processing.ipynb) Cell 11 line 2
      [1](vscode-notebook-cell:/Users/nikhilgaruda/Documents/Astronomy_Research/G191/pixedfit/image_processing.ipynb#X13sZmlsZQ%3D%3D?line=0) get_ipython().run_line_magic('tb', '')
----> [2](vscode-notebook-cell:/Users/nikhilgaruda/Documents/Astronomy_Research/G191/pixedfit/image_processing.ipynb#X13sZmlsZQ%3D%3D?line=1) img_process = images_processing(filters, sci_img, var_img, gal_ra, gal_dec, dir_images=dir_imgs, img_unit=img_unit, 
      [3](vscode-notebook-cell:/Users/nikhilgaruda/Documents/Astronomy_Research/G191/pixedfit/image_processing.ipynb#X13sZmlsZQ%3D%3D?line=2)                                 img_scale=img_scale, img_pixsizes=img_pixsizes, kernels=kernels, gal_z=gal_z, 
      [4](vscode-notebook-cell:/Users/nikhilgaruda/Documents/Astronomy_Research/G191/pixedfit/image_processing.ipynb#X13sZmlsZQ%3D%3D?line=3)                                 stamp_size=stamp_size, remove_files=False, run_image_processing=True, flag_psfmatch = 1, flag_reproject = 1, flag_crop = 1)

File [/opt/homebrew/Caskroom/miniforge/base/envs/pixedfit/lib/python3.10/site-packages/piXedfit/piXedfit_images/images_process.py:130](https://file+.vscode-resource.vscode-cdn.net/opt/homebrew/Caskroom/miniforge/base/envs/pixedfit/lib/python3.10/site-packages/piXedfit/piXedfit_images/images_process.py:130), in images_processing.__init__(self, filters, sci_img, var_img, gal_ra, gal_dec, dir_images, img_unit, img_scale, img_pixsizes, run_image_processing, flag_psfmatch, flag_reproject, flag_crop, kernels, gal_z, stamp_size, remove_files, idfil_align)
    127 self.run_image_processing = run_image_processing
    129 if run_image_processing == True:
--> 130 	self.reduced_stamps()

File [/opt/homebrew/Caskroom/miniforge/base/envs/pixedfit/lib/python3.10/site-packages/piXedfit/piXedfit_images/images_process.py:219](https://file+.vscode-resource.vscode-cdn.net/opt/homebrew/Caskroom/miniforge/base/envs/pixedfit/lib/python3.10/site-packages/piXedfit/piXedfit_images/images_process.py:219), in images_processing.reduced_stamps(self)
    216 		temp_file_names.append(name_out)
    217 else:
    218 	#++> science image
--> 219 	hdu = fits.open(sci_img_name[filters[bb]])
    220 	header = hdu[0].header
    221 	#new_data = remove_naninf_image_2dinterpolation(hdu[0].data)

UnboundLocalError: local variable 'bb' referenced before assignment

This happened because you haven't added a for loop for to access each filter with bb

But, adding a for loop after

piXedfit/piXedfit/piXedfit_images/images_process.py

Lines 217 to 218 in 726cca4

    
           else: 
        
           	#++> science image

like this: for bb in range(0,nbands):

gave me a new error

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
[/Users/nikhilgaruda/Documents/Astronomy_Research/G191/pixedfit/image_processing.ipynb](https://file+.vscode-resource.vscode-cdn.net/Users/nikhilgaruda/Documents/Astronomy_Research/G191/pixedfit/image_processing.ipynb) Cell 11 line 2
      [1](vscode-notebook-cell:/Users/nikhilgaruda/Documents/Astronomy_Research/G191/pixedfit/image_processing.ipynb#X13sZmlsZQ%3D%3D?line=0) get_ipython().run_line_magic('tb', '')
----> [2](vscode-notebook-cell:/Users/nikhilgaruda/Documents/Astronomy_Research/G191/pixedfit/image_processing.ipynb#X13sZmlsZQ%3D%3D?line=1) img_process = images_processing(filters, sci_img, var_img, gal_ra, gal_dec, dir_images=dir_imgs, img_unit=img_unit, 
      [3](vscode-notebook-cell:/Users/nikhilgaruda/Documents/Astronomy_Research/G191/pixedfit/image_processing.ipynb#X13sZmlsZQ%3D%3D?line=2)                                 img_scale=img_scale, img_pixsizes=img_pixsizes, kernels=kernels, gal_z=gal_z, 
      [4](vscode-notebook-cell:/Users/nikhilgaruda/Documents/Astronomy_Research/G191/pixedfit/image_processing.ipynb#X13sZmlsZQ%3D%3D?line=3)                                 stamp_size=stamp_size, remove_files=False, run_image_processing=True, flag_psfmatch = 1, 
      [5](vscode-notebook-cell:/Users/nikhilgaruda/Documents/Astronomy_Research/G191/pixedfit/image_processing.ipynb#X13sZmlsZQ%3D%3D?line=4)                                 flag_reproject = 1, flag_crop = 1)

File [/opt/homebrew/Caskroom/miniforge/base/envs/pixedfit/lib/python3.10/site-packages/piXedfit/piXedfit_images/images_process.py:130](https://file+.vscode-resource.vscode-cdn.net/opt/homebrew/Caskroom/miniforge/base/envs/pixedfit/lib/python3.10/site-packages/piXedfit/piXedfit_images/images_process.py:130), in images_processing.__init__(self, filters, sci_img, var_img, gal_ra, gal_dec, dir_images, img_unit, img_scale, img_pixsizes, run_image_processing, flag_psfmatch, flag_reproject, flag_crop, kernels, gal_z, stamp_size, remove_files, idfil_align)
    127 self.run_image_processing = run_image_processing
    129 if run_image_processing == True:
--> 130 	self.reduced_stamps()

File [/opt/homebrew/Caskroom/miniforge/base/envs/pixedfit/lib/python3.10/site-packages/piXedfit/piXedfit_images/images_process.py:226](https://file+.vscode-resource.vscode-cdn.net/opt/homebrew/Caskroom/miniforge/base/envs/pixedfit/lib/python3.10/site-packages/piXedfit/piXedfit_images/images_process.py:226), in images_processing.reduced_stamps(self)
    224 hdu.close()
    225 name_out = "crop_%s" % check_name_remove_dir(sci_img_name[filters[bb]],dir_images)
--> 226 fits.writeto(name_out,new_data,header, overwrite=True)
    227 sci_img_name[filters[bb]] = name_out
    229 #++> variance image

File [/opt/homebrew/Caskroom/miniforge/base/envs/pixedfit/lib/python3.10/site-packages/astropy/io/fits/convenience.py:457](https://file+.vscode-resource.vscode-cdn.net/opt/homebrew/Caskroom/miniforge/base/envs/pixedfit/lib/python3.10/site-packages/astropy/io/fits/convenience.py:457), in writeto(filename, data, header, output_verify, overwrite, checksum)
    455 if hdu.is_image and not isinstance(hdu, PrimaryHDU):
    456     hdu = PrimaryHDU(data, header=header)
--> 457 hdu.writeto(
    458     filename, overwrite=overwrite, output_verify=output_verify, checksum=checksum
    459 )

File [/opt/homebrew/Caskroom/miniforge/base/envs/pixedfit/lib/python3.10/site-packages/astropy/io/fits/hdu/base.py:400](https://file+.vscode-resource.vscode-cdn.net/opt/homebrew/Caskroom/miniforge/base/envs/pixedfit/lib/python3.10/site-packages/astropy/io/fits/hdu/base.py:400), in _BaseHDU.writeto(self, name, output_verify, overwrite, checksum)
    397 from .hdulist import HDUList
    399 hdulist = HDUList([self])
--> 400 hdulist.writeto(name, output_verify, overwrite=overwrite, checksum=checksum)

File [/opt/homebrew/Caskroom/miniforge/base/envs/pixedfit/lib/python3.10/site-packages/astropy/io/fits/hdu/hdulist.py:1010](https://file+.vscode-resource.vscode-cdn.net/opt/homebrew/Caskroom/miniforge/base/envs/pixedfit/lib/python3.10/site-packages/astropy/io/fits/hdu/hdulist.py:1010), in HDUList.writeto(self, fileobj, output_verify, overwrite, checksum)
   1006 mode = FILE_MODES[fileobj_mode(fileobj)] if isfile(fileobj) else "ostream"
   1008 # This can accept an open file object that's open to write only, or in
   1009 # append/update modes but only if the file doesn't exist.
-> 1010 fileobj = _File(fileobj, mode=mode, overwrite=overwrite)
   1011 hdulist = self.fromfile(fileobj)
   1012 try:

File [/opt/homebrew/Caskroom/miniforge/base/envs/pixedfit/lib/python3.10/site-packages/astropy/io/fits/file.py:217](https://file+.vscode-resource.vscode-cdn.net/opt/homebrew/Caskroom/miniforge/base/envs/pixedfit/lib/python3.10/site-packages/astropy/io/fits/file.py:217), in _File.__init__(self, fileobj, mode, memmap, overwrite, cache, use_fsspec, fsspec_kwargs)
    215     self._open_fileobj(fileobj, mode, overwrite)
    216 elif isinstance(fileobj, (str, bytes)):
--> 217     self._open_filename(fileobj, mode, overwrite)
    218 else:
    219     self._open_filelike(fileobj, mode, overwrite)

File [/opt/homebrew/Caskroom/miniforge/base/envs/pixedfit/lib/python3.10/site-packages/astropy/io/fits/file.py:626](https://file+.vscode-resource.vscode-cdn.net/opt/homebrew/Caskroom/miniforge/base/envs/pixedfit/lib/python3.10/site-packages/astropy/io/fits/file.py:626), in _File._open_filename(self, filename, mode, overwrite)
    623 ext = os.path.splitext(self.name)[1]
    625 if not self._try_read_compressed(self.name, magic, mode, ext=ext):
--> 626     self._file = open(self.name, IO_FITS_MODES[mode])
    627     self.close_on_error = True
    629 # Make certain we're back at the beginning of the file
    630 # BZ2File does not support seek when the file is open for writing, but
    631 # when opening a file for write, bz2.BZ2File always truncates anyway.

FileNotFoundError: [Errno 2] No such file or directory: 'crop_drz/mosaic_plckg191_nircam_f090w_30mas_20230718_drz.fits'

So, do I need to create crop_*, stamp_*, name_img_* and name_var_* myself in the folder where I run my piXedfit code?
Here are the lines:

piXedfit/piXedfit/piXedfit_images/images_process.py

Line 224 in 726cca4

    
           name_out = "crop_%s" % check_name_remove_dir(sci_img_name[filters[bb]],dir_images)

piXedfit/piXedfit/piXedfit_images/images_process.py

Line 402 in 726cca4

    
           name_out = 'stamp_%s' % check_name_remove_dir(psfmatch_sci_img_name[filters[bb]], dir_images)

piXedfit/piXedfit/piXedfit_images/images_process.py

Line 498 in 726cca4

str_temp = "name_img_%s" % filters[bb]

piXedfit/piXedfit/piXedfit_images/images_process.py

Line 501 in 726cca4

str_temp = "name_var_%s" % filters[bb]

log10 Dust mass issue when generating models

Hi, Abdurro'uf!
It's a very cool pipeline for modeling the SED of galaxies, But when I try to generate the model of the as the blow code, the dust mass is always 0, so the log10(dust_mass) is always nan. The above questions will lead to a series of follow-up issues, when select the models for fitting.
`

from piXedfit.piXedfit_model import save_models_rest_spec
nmodels =10
nproc = 2
params_range = {'log_age':[-1.0,1.14], 'dust1':[0.0,3.0], 'dust2':[0.0,3.0]
}
imf_type = 1 # Chabrier (2003)
sfh_form = 4 # double power law SFH form
dust_law = 0 # charlot & Fall (2000) two components dust attenuation law
duste_switch = 1 # turn off dust emission
add_neb_emission = 1 # turn on nebular emission
add_agn = 0 # turn off AGN dusty torus emission
name_out = 's_cb_dpl_cf_de_na_100k.hdf5'
save_models_rest_spec(imf_type=imf_type, sfh_form=sfh_form, dust_law=dust_law,
params_range=params_range,
duste_switch=duste_switch, add_neb_emission=add_neb_emission, add_agn=add_agn,
nmodels=nmodels, nproc=nproc, name_out=name_out)
`

Could you please take a look and see if there are any issues with the code generated above? If everything is fine, I'd like to offer a suggestion for your pipeline to avoid the mentioned problems. After checking your pipeline, I propose adding the following lines at line 104 in /piXedfit_model/save_models_rest_spec.py:

if dust_mass > 0:
mod_log_dustmass_temp[int(count)] = log10(dust_mass)
else:
mod_log_dustmass_temp[int(count)] = -99

I am not familiar with the characteristics of galaxy spectral synthesis and fsps code as I am an outsider in galaxy physics. Therefore, I'm unsure if this correction is appropriate, and I'm concerned it might lead to subsequent errors. If you have better suggestions to enhance the robustness of the code, please let me know. Thank you!

About SED fitting (based on bin map and pixel-by-pixel)

Hi, Abdurro'uf!
First of all, thank you for making these wonderful code available!
We are now working on performing pixel-by-pixel spatially resolved analysis on JWST data. Fortunately, we find that your have already applied spatially resolved analysis on JWST observation and the result is interesting and inspiring! After reading most of your recent works which involve spatially resolved SED fitting, I have been trying to use your code and find the 'image_process' and 'pixel_binning' modules are quite satisfying and convenient!
I follow the instructions of the example tutorial of fitting NGC-5194 carefully and I am confident that the SFH model and flux_map are generated properly.
But the SED fitting have run for 26 hours till now and it does not seem to come to an end.
It is just repeating the same information in the terminal:

It seems like the programme cannot find a hdf5 file in the '/data/temp/' directory. (But, with this error message, the code can still be running)
Could you please tell me how to fix this or I just need to ignore it and let the machine running?
By the way, I was wondering whether it is caused by my machine.
(I am using a DELL workstation with a intel i5-12500 CPU and 32G RAM)
In the same time, I was trying to do the SED fitting with Bagpipes but the SPS method is different from your code and my laptop run slowly and still cannot give me a result.
Thanks in advance!
Best.

Mingfeng Liu
Nanjing Normal University

Questions about fitting process

Hi！Abdurro'uf

Thank you very much for your timely and accurate reply last time. Through this time of learning and using, the excellent code you have written has given encouraging results. However, during this period, I encountered several questions, which I could not find the answer to by myself.

First of all, regarding the range of parameters fitted, such as metallicity, the default value is [-2,2], I want to change the parameter limit range to [-5,5]. When I tried to modify the scope using the save_models_rest_spec function to create the model, it didn't seem to work. Maybe I'm changing the scope the wrong way? Below is the configuration file and the result, which does not seem to really fit in the range [-5,5], but only increases the display range.

from piXedfit.piXedfit_model import save_models_rest_spec
imf_type = 1                    #  IMF
sfh_form = 3                   # SFH
dust_law = 1                    # dust attenuation law
duste_switch = 0                # dust emission
add_neb_emission = 1            # nebular emission
add_agn = 0                     # AGN dusty torus emission

nmodels = 100000                # number of model spectra to be generated
nproc = 30                      # number of processors to be used in the calculation


params_range = {'log_age':[-1.0,1.14], 'dust1':[0.0,3.0], 'dust2':[0.0,3.0], 'logzsol':[-5.0,5.0]}

name_out = 'model_test_MCMC_c'     # name of the output HDF5 file
save_models_rest_spec(imf_type=imf_type, sfh_form=sfh_form, dust_law=dust_law, params_range=params_range,
                        duste_switch=duste_switch, add_neb_emission=add_neb_emission, add_agn=add_agn,
                        nmodels=nmodels,nproc=nproc, name_out=name_out)

We know that some values in the results, such as QPAHs, can be fixed as a typical value according to other people's research. Can pixedfit fix a parameter value to participate in the fitting? I didn't get the answer in the manual, so how do you do this, which is very important for fitting.
How to derive the proportion information of different components from the figure below, such as the ratio of AGN or nebula emission to the total. Maybe it is my omission, I did not find the data interface where.

In addition: I use MCMC to fit, and the fitting time of 1000 points is expected to be up to 300 days, is this normal?

There are so many questions here that maybe I should open another issue for ask the 3 ? Thank you very much for your wonderful work and support, and I am looking forward to your reply !

My computer information：
Ubuntu 18.04.6
Intel(R) Xeon(R) Gold 6230R CPU @ 2.10GHz *104
9.3 GiB RAM

【Question】Does piXedfit_bin average or sum the flux inside a bin?

I think pixel binning should be "average of the flux". Because if we sum the flux in several pixels , then a bin will correspond to a larger solid angle(arcsec^2) ,or physical scale.（e.g. several kpc in a galaxy） Meanwhile, in different bins there are different pixel numbers, so they will represent different physical scales of a galaxy. I'm confused about whether piXedfit_bin uses average or sum.

Error for fitting with AGN model

Hi, Abdurro'uf!
I found a strange error that comes up every time I try to add AGN components for fitting, here are two different forms. I wonder if it has to do with the fact that the fitting region has to be in the center of the galaxy?I don't understand the prompt of this error. It would be great if you had any comments！

%%time
# Fitting spatial bin at the galactic center

# call the function
from piXedfit.piXedfit_fitting import SEDfit_from_binmap

fits_binmap = "pixbin_fluxmap_ngc2992_ref2_2284.fits"
binid_range = [2281,2284]                                                # range of the bins to be fit: central bin

# Name of HDF5 file containing model spectra at rest-frame for initial fitting
models_spec = "s_cb_dpl_cf_nde_na_AGN_30k.hdf5"

# call function for defining priors in SED fitting                      
from piXedfit.piXedfit_fitting import priors

# define ranges of some parameters
ranges = { 'dust1':[0.0,3.0], 'dust2':[0.0,3.0], 'log_age':[-1.0,1.14]}
pr = priors(ranges)
params_ranges = pr.params_ranges()

# define prior forms
prior1 = pr.uniform('logzsol')
prior2 = pr.uniform('dust1')
prior3 = pr.uniform('dust2')
prior4 = pr.uniform('log_age')
params_priors = [prior1, prior2, prior3, prior4]

# choice of SED fitting method
fit_method = 'mcmc'

nproc = 10            # number of cores to be used in calculation
SEDfit_from_binmap(fits_binmap, binid_range=binid_range, models_spec=models_spec,
                             params_ranges=params_ranges, params_priors=params_priors,
                             fit_method=fit_method, nwalkers=100, nsteps=1000, nproc=nproc,
                                initfit_nmodels_mcmc=30000, cosmo=0, H0=70.0, Om0=0.3,
                             store_full_samplers=1)
#------------------------------------
Number of free parameters: 11
Free parameters: 
['logzsol', 'log_tau', 'log_alpha', 'log_beta', 'log_age', 'dust_index', 'dust1', 'dust2', 'log_fagn', 'log_tauagn', 'log_mass']
Number of fix parameters: 0
Number of parent models in models_spec: 30000
Number of models for initial fitting: 30000
rank 3 --> progress: 3000 of 3000 (100%)ank 6 --> progress: 2838 of 3000 (94%)

100%|██████████| 1000/1000 [10:23<00:00,  1.60it/s]

rank: 4  Calculation process: 5508 from 9500  --->  57%rank: 5  Calculation process: 4748 from 9500  --->  49%rank: 4  Calculation process: 5108 from 9500  --->  53%

IOPub data rate exceeded.
The Jupyter server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--ServerApp.iopub_data_rate_limit`.

Current values:
ServerApp.iopub_data_rate_limit=1000000.0 (bytes/sec)
ServerApp.rate_limit_window=3.0 (secs)

rank: 5  Calculation process: 4543 from 9500  --->  47%rank: 7  Calculation process: 4634 from 9500  --->  48%rank: 5  Calculation process: 1244 from 9500  --->  13%

IOPub data rate exceeded.
The Jupyter server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--ServerApp.iopub_data_rate_limit`.

Current values:
ServerApp.iopub_data_rate_limit=1000000.0 (bytes/sec)
ServerApp.rate_limit_window=3.0 (secs)

rank: 1  Calculation process: 9500 from 9500  --->  100%ank: 3  Calculation process: 8973 from 9500  --->  94%

Traceback (most recent call last):
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p2.py", line 838, in <module>
    store_to_fits(nsamples=nsamples,sampler_params=sampler_params,sampler_log_sfr=sampler_log_sfr,
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p2.py", line 185, in store_to_fits
    col = fits.Column(name='log_fagn_bol', format='D', array=np.array(sampler_log_fagn_bol))
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/astropy/io/fits/column.py", line 715, in __init__
    array = self._convert_to_valid_data_type(array)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/astropy/io/fits/column.py", line 1434, in _convert_to_valid_data_type
    return _convert_array(array, dtype)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/astropy/io/fits/util.py", line 700, in _convert_array
    return array.view(dtype)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/numpy/core/_internal.py", line 548, in _view_is_safe
    raise TypeError("Cannot change data-type for object array.")
TypeError: Cannot change data-type for object array.
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[21917,1],0]
  Exit code:    1
--------------------------------------------------------------------------

Number of free parameters: 11
Free parameters: 
['logzsol', 'log_tau', 'log_alpha', 'log_beta', 'log_age', 'dust_index', 'dust1', 'dust2', 'log_fagn', 'log_tauagn', 'log_mass']
Number of fix parameters: 0
Number of parent models in models_spec: 30000
Number of models for initial fitting: 30000
rank 0 --> progress: 3000 of 3000 (100%)ank 9 --> progress: 2873 of 3000 (95%)

100%|██████████| 1000/1000 [09:49<00:00,  1.70it/s]

rank: 1  Calculation process: 4972 from 9500  --->  52%rank: 8  Calculation process: 5155 from 9500  --->  54%rank: 8  Calculation process: 2130 from 9500  --->  22%

IOPub data rate exceeded.
The Jupyter server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--ServerApp.iopub_data_rate_limit`.

Current values:
ServerApp.iopub_data_rate_limit=1000000.0 (bytes/sec)
ServerApp.rate_limit_window=3.0 (secs)

rank: 4  Calculation process: 4138 from 9500  --->  43%rank: 5  Calculation process: 3779 from 9500  --->  39%

IOPub data rate exceeded.
The Jupyter server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--ServerApp.iopub_data_rate_limit`.

Current values:
ServerApp.iopub_data_rate_limit=1000000.0 (bytes/sec)
ServerApp.rate_limit_window=3.0 (secs)

rank: 1  Calculation process: 9500 from 9500  --->  100%ank: 0  Calculation process: 9275 from 9500  --->  97%

Traceback (most recent call last):
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p2.py", line 838, in <module>
    store_to_fits(nsamples=nsamples,sampler_params=sampler_params,sampler_log_sfr=sampler_log_sfr,
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p2.py", line 185, in store_to_fits
    col = fits.Column(name='log_fagn_bol', format='D', array=np.array(sampler_log_fagn_bol))
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/astropy/io/fits/column.py", line 715, in __init__
    array = self._convert_to_valid_data_type(array)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/astropy/io/fits/column.py", line 1434, in _convert_to_valid_data_type
    return _convert_array(array, dtype)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/astropy/io/fits/util.py", line 700, in _convert_array
    return array.view(dtype)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/numpy/core/_internal.py", line 548, in _view_is_safe
    raise TypeError("Cannot change data-type for object array.")
TypeError: Cannot change data-type for object array.
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[43520,1],0]
  Exit code:    1
--------------------------------------------------------------------------

Number of free parameters: 11
Free parameters: 
['logzsol', 'log_tau', 'log_alpha', 'log_beta', 'log_age', 'dust_index', 'dust1', 'dust2', 'log_fagn', 'log_tauagn', 'log_mass']
Number of fix parameters: 0
Number of parent models in models_spec: 30000
Number of models for initial fitting: 30000
rank 1 --> progress: 3000 of 3000 (100%)ank 6 --> progress: 2016 of 3000 (67%)

100%|██████████| 1000/1000 [09:02<00:00,  1.84it/s]

rank: 3  Calculation process: 4521 from 9500  --->  47%rank: 7  Calculation process: 4867 from 9500  --->  51%rank: 9  Calculation process: 4700 from 9500  --->  49%

IOPub data rate exceeded.
The Jupyter server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--ServerApp.iopub_data_rate_limit`.

Current values:
ServerApp.iopub_data_rate_limit=1000000.0 (bytes/sec)
ServerApp.rate_limit_window=3.0 (secs)

rank: 2  Calculation process: 4378 from 9500  --->  46%rank: 7  Calculation process: 4902 from 9500  --->  51%rank: 7  Calculation process: 3143 from 9500  --->  33%

IOPub data rate exceeded.
The Jupyter server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--ServerApp.iopub_data_rate_limit`.

Current values:
ServerApp.iopub_data_rate_limit=1000000.0 (bytes/sec)
ServerApp.rate_limit_window=3.0 (secs)

rank: 1  Calculation process: 9500 from 9500  --->  100%ank: 1  Calculation process: 8408 from 9500  --->  88%

Traceback (most recent call last):
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p2.py", line 838, in <module>
    store_to_fits(nsamples=nsamples,sampler_params=sampler_params,sampler_log_sfr=sampler_log_sfr,
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p2.py", line 185, in store_to_fits
    col = fits.Column(name='log_fagn_bol', format='D', array=np.array(sampler_log_fagn_bol))
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/astropy/io/fits/column.py", line 715, in __init__
    array = self._convert_to_valid_data_type(array)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/astropy/io/fits/column.py", line 1434, in _convert_to_valid_data_type
    return _convert_array(array, dtype)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/astropy/io/fits/util.py", line 700, in _convert_array
    return array.view(dtype)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/numpy/core/_internal.py", line 548, in _view_is_safe
    raise TypeError("Cannot change data-type for object array.")
TypeError: Cannot change data-type for object array.
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------

CPU times: user 10.2 s, sys: 5.69 s, total: 15.9 s
Wall time: 34min 3s

--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[65436,1],0]
  Exit code:    1
--------------------------------------------------------------------------

%%time
# Fitting spatial bin at the galactic center

# call the function
from piXedfit.piXedfit_fitting import SEDfit_from_binmap

fits_binmap = "pixbin_fluxmap_ngc2992_ref1_1942.fits"
binid_range = [1940,1942]                                                # range of the bins to be fit: central bin

# Name of HDF5 file containing model spectra at rest-frame for initial fitting
models_spec = "model_MCMC_131011_30k.hdf5"

# call function for defining priors in SED fitting                      
from piXedfit.piXedfit_fitting import priors

# define ranges of some parameters
ranges = { 'dust1':[0.0,3.0], 'dust2':[0.0,3.0], 'log_age':[-1.0,1.14]}
pr = priors(ranges)
params_ranges = pr.params_ranges()

# define prior forms
prior1 = pr.uniform('logzsol')
prior2 = pr.uniform('dust1')
prior3 = pr.uniform('dust2')
prior4 = pr.uniform('log_age')

params_priors = [prior1, prior2, prior3, prior4]

# choice of SED fitting method
fit_method = 'mcmc'

nproc = 10            # number of cores to be used in calculation
SEDfit_from_binmap(fits_binmap, binid_range=binid_range, models_spec=models_spec,
                             params_ranges=params_ranges, params_priors=params_priors,
                             fit_method=fit_method, nwalkers=100, nsteps=1000, nproc=nproc,
                                initfit_nmodels_mcmc=30000, cosmo=0, H0=70.0, Om0=0.3,
                             store_full_samplers=1)
#-----------------------------------------
Number of free parameters: 8
Free parameters: 
['logzsol', 'log_tau', 'log_t0', 'log_age', 'dust2', 'log_fagn', 'log_tauagn', 'log_mass']
Number of fix parameters: 0
Number of parent models in models_spec: 30000
Number of models for initial fitting: 30000
rank 3 --> progress: 3000 of 3000 (100%)ank 9 --> progress: 2744 of 3000 (91%)rank 1 --> progress: 2132 of 3000 (71%)

Traceback (most recent call last):
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p1.py", line 629, in <module>
    sampler.run_mcmc(pos, nsteps, progress=True)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/emcee/ensemble.py", line 443, in run_mcmc
    for results in self.sample(initial_state, iterations=nsteps, **kwargs):
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/emcee/ensemble.py", line 312, in sample
    raise ValueError(
ValueError: Initial state has a large condition number. Make sure that your walkers are linearly independent for the best performance
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[2605,1],0]
  Exit code:    1
--------------------------------------------------------------------------
mv: cannot stat 'samplers_2800.hdf5': No such file or directory
Traceback (most recent call last):
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
    f = h5py.File(name_file, 'r')
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 567, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 231, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/kongxiangming/Downloads/piXedfit2/data/temp/samplers_2800.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
Traceback (most recent call last):
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
    f = h5py.File(name_file, 'r')
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 567, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 231, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/kongxiangming/Downloads/piXedfit2/data/temp/samplers_2800.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
Traceback (most recent call last):
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
    f = h5py.File(name_file, 'r')
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 567, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 231, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/kongxiangming/Downloads/piXedfit2/data/temp/samplers_2800.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
Traceback (most recent call last):
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
    f = h5py.File(name_file, 'r')
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 567, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 231, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/kongxiangming/Downloads/piXedfit2/data/temp/samplers_2800.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
Traceback (most recent call last):
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
    f = h5py.File(name_file, 'r')
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 567, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 231, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/kongxiangming/Downloads/piXedfit2/data/temp/samplers_2800.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
Traceback (most recent call last):
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
    f = h5py.File(name_file, 'r')
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 567, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 231, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/kongxiangming/Downloads/piXedfit2/data/temp/samplers_2800.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
Traceback (most recent call last):
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
Traceback (most recent call last):
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
    f = h5py.File(name_file, 'r')
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 567, in __init__
    f = h5py.File(name_file, 'r')
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 567, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 231, in make_fid
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 231, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/kongxiangming/Downloads/piXedfit2/data/temp/samplers_2800.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/kongxiangming/Downloads/piXedfit2/data/temp/samplers_2800.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
Traceback (most recent call last):
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
    f = h5py.File(name_file, 'r')
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 567, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 231, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/kongxiangming/Downloads/piXedfit2/data/temp/samplers_2800.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
Traceback (most recent call last):
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
    f = h5py.File(name_file, 'r')
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 567, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 231, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/kongxiangming/Downloads/piXedfit2/data/temp/samplers_2800.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[1092,1],7]
  Exit code:    1
--------------------------------------------------------------------------
rm: cannot remove '/home/kongxiangming/Downloads/piXedfit2/data/temp/samplers_2800.hdf5': No such file or directory

Number of free parameters: 8
Free parameters: 
['logzsol', 'log_tau', 'log_t0', 'log_age', 'dust2', 'log_fagn', 'log_tauagn', 'log_mass']
Number of fix parameters: 0
Number of parent models in models_spec: 30000
Number of models for initial fitting: 30000
rank 4 --> progress: 3000 of 3000 (100%)ank 3 --> progress: 2037 of 3000 (67%)

Traceback (most recent call last):
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p1.py", line 629, in <module>
    sampler.run_mcmc(pos, nsteps, progress=True)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/emcee/ensemble.py", line 443, in run_mcmc
    for results in self.sample(initial_state, iterations=nsteps, **kwargs):
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/emcee/ensemble.py", line 312, in sample
    raise ValueError(
ValueError: Initial state has a large condition number. Make sure that your walkers are linearly independent for the best performance
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[1150,1],0]
  Exit code:    1
--------------------------------------------------------------------------
mv: cannot stat 'samplers_26345.hdf5': No such file or directory
Traceback (most recent call last):
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
    f = h5py.File(name_file, 'r')
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 567, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 231, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/kongxiangming/Downloads/piXedfit2/data/temp/samplers_26345.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
Traceback (most recent call last):
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
    f = h5py.File(name_file, 'r')
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 567, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 231, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/kongxiangming/Downloads/piXedfit2/data/temp/samplers_26345.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
Traceback (most recent call last):
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
Traceback (most recent call last):
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
    f = h5py.File(name_file, 'r')
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 567, in __init__
    f = h5py.File(name_file, 'r')
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 567, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 231, in make_fid
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 231, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/kongxiangming/Downloads/piXedfit2/data/temp/samplers_26345.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/kongxiangming/Downloads/piXedfit2/data/temp/samplers_26345.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
Traceback (most recent call last):
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
    f = h5py.File(name_file, 'r')
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 567, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 231, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/kongxiangming/Downloads/piXedfit2/data/temp/samplers_26345.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
Traceback (most recent call last):
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
    f = h5py.File(name_file, 'r')
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 567, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 231, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/kongxiangming/Downloads/piXedfit2/data/temp/samplers_26345.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
Traceback (most recent call last):
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
    f = h5py.File(name_file, 'r')
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 567, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 231, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/kongxiangming/Downloads/piXedfit2/data/temp/samplers_26345.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
Traceback (most recent call last):
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
    f = h5py.File(name_file, 'r')
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 567, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 231, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/kongxiangming/Downloads/piXedfit2/data/temp/samplers_26345.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
Traceback (most recent call last):
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
    f = h5py.File(name_file, 'r')
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 567, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 231, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/kongxiangming/Downloads/piXedfit2/data/temp/samplers_26345.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
Traceback (most recent call last):
  File "/home/kongxiangming/Downloads/piXedfit2/piXedfit/piXedfit_fitting/./mc_p2.py", line 665, in <module>
    f = h5py.File(name_file, 'r')
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 567, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/home/kongxiangming/.conda/envs/pixedfit2/lib/python3.10/site-packages/h5py/_hl/files.py", line 231, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/home/kongxiangming/Downloads/piXedfit2/data/temp/samplers_26345.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)

CPU times: user 839 ms, sys: 927 ms, total: 1.77 s
Wall time: 1min 20s

--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[1663,1],0]
  Exit code:    1
--------------------------------------------------------------------------
rm: cannot remove '/home/kongxiangming/Downloads/piXedfit2/data/temp/samplers_26345.hdf5': No such file or directory

WISE W3 uncertainty

https://github.com/aabdurrouf/piXedfit/blob/main/piXedfit/piXedfit_images/images_utils.py line 627, the last parenthesis should be moved before + np.square(sigma_src) term.

several possible bugs

Hi, I've encountered several possible bug in the image processing code. Here's a list of them:

in image_process.py, line 197, 211, 377, 390, 436, 449
The "hdu.header.update()" seems not to update the WCS correctly. I fixed it by replacing the "hdu.writeto(...)" with "fits.writeto(name_out, cutout.data, cutout.wcs.to_header(), overwrite=True)"

in image_process.py, line 107:
get_img_pixsizes() will fail if the "dir_images" parameter is provided.

in image_utils.py, line 906:
Add .astype(int) to np.zeros() or use np.full(..., 0). The "mask_region1" can't be a float array.

in image_utils.py, line 907:
"or" should be replaced with "|" or "np.logical_or()".

piXedfit not working in HPC

Hello,

I am trying to run the code for a JWST project on HPC. But, when I try to make the model rest frame spectra my code breaks due to some MPI error that is being cause by piXedfit.

I did test the MPI installation in my cluster and everything seems to be fine there with no conflicts and only breaks when I run piXedfit.

The error seems to be some networking error but upon checking with my HPC support team they didn't find any problems with the way I set up my cluster and seems to be leaning in the way that it's the problem of the way MPI implementation might be written here.

Please let me know how I can fix it because this code takes a very long time to run on my laptop (Macbook Air M1). I would like to also know if there are any other new features implemented for JWST images and spectroscopy.

Thanks,
Nikhil

FILE DUMPS:

Here is HPC job shell file:

#!/bin/bash
# a name for this job; letters and digits only please. (optional)
#SBATCH --job-name=run_pixed_fit

# Researchers can buy in for priority queueing. However, the amount of time 
# they can use for this is limited.  Everyone has access to unlimited
# windfall, but any priority jobs will go first.  The partitions (queues) on
# Puma are windfall, standard, and high_pri
# non-windfall partitions require a "#SBATCH --account=your_PI_group line"
#SBATCH --account=bfrye
#SBATCH --partition=standard

# Standard Puma Nodes have 94 available cores and 480GB of available RAM each.
# Since no memory allocation is set, the ram available will be 5GB per core
# (Task in Sluem).
# Note: the fewer resources you request, the less time your job will spend in
# waiting and the more resources will be available for others to use.
#SBATCH --nodes=4
#SBATCH --ntasks=32
#SBATCH --mem=32gb

#SBATCH --mail-type=ALL
#SBATCH [email protected]

# This is the amount of time you think your job will take to run.
# 240 houra (10 days) is the maximum.
# This request shows 5 minutes
#SBATCH --time=10:00:00

# Reset modules, so we have a known starting point...
module purge
module load autotools prun/1.3 gnu8/8.3.0 openmpi3/3.1.4 ohpc
module load anaconda
module list

source /home/u4/nikhilgaruda/.bashrc

export SPS_HOME=/home/u4/nikhilgaruda/Software/fsps
export PIXEDFIT_HOME=/home/u4/nikhilgaruda/Software/piXedfit
export OMPI_MCA_opal_cuda_support=true

conda activate /home/u4/nikhilgaruda/.conda/envs/pixedfit
echo $CONDA_DEFAULT_ENV

conda list

conda install astropy -y


cd /groups/bfrye/G191/pixedfit
strace mpirun model_rest_frame_spectra.py

# *** If you need assistance using Puma, please email
# *** [email protected]

exit 0
#
#-end of puma_cpu sample script.

Here is the output from the run:
slurm-1653886.out.txt

keyError of img_process.reduced_stamps()

When i run the image processing,it happens.
And i'm sure this filter is in the list_fliters(),why cant recognize？
No problem with the input cell.

$output_stamps = img_process.reduced_stamps()

KeyError Traceback (most recent call last)
Input In [26], in <cell line: 2>()
1 # run
----> 2 output_stamps = img_process.reduced_stamps()

File ~/opt/anaconda3/envs/pixedfit/lib/python3.8/site-packages/piXedfit/piXedfit_images/images_process.py:178, in images_processing.reduced_stamps(self)
173 idfil_psfmatch = 0
175 elif flag_psfmatch == 0:
176 ####============== (2) PSF matching ============#####
177 ##==> (a) Get filter index with largest PSF size:
--> 178 idfil_psfmatch = get_largest_FWHM_PSF(filters=filters)
179 print ("[PSF matching to %s]" % filters[idfil_psfmatch])
181 ##==> (b) Get Kernels:

File ~/opt/anaconda3/envs/pixedfit/lib/python3.8/site-packages/piXedfit/piXedfit_images/images_utils.py:1137, in get_largest_FWHM_PSF(filters)
1135 fil_fwhm = np.zeros(nbands)
1136 for bb in range(0,nbands):
-> 1137 fil_fwhm[bb] = col_fwhm_psf[filters[bb]]
1139 idx_fil_max, max_val = max(enumerate(fil_fwhm), key=itemgetter(1))
1141 return idx_fil_max

KeyError: 'hst_wfc3_uvis_f275w'

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.