Wrapper for Py6S with automatic download of atmospheric composition from MODIS.
pip install -e .[test]
pytest -v
docker build -t dhigras/atmcorr-env -f Dockerfile.environment .
docker push dhigras/atmcorr-env
Atmospheric correction with Py6S and MODIS data
License: MIT License
We need to make the retrieval and selection of calibration factors for WorldView 2 and 3 more robust.
It also appears that there are new values out: https://apollomapping.com/image_downloads/DigitalGlobe_Absolute_Radiometric_Calibration_All_Satellites_2106.pdf
Should we update to those?
For some S2 images, atmcorr can return negative reflectance values for some bands (so far seen in the red band) when deglint is set to false.
Example can be found here \\DKCPH1-NCR3612\d$\Velux_DK_HabitatMapping\neg_refl
Reported by @milyra.
The problem seems to be worse in winter than in summer. This could hint at a problem with the Sun angles.
Some places it is called dg_calibration (like in the requirements in setup.py and /atmcorr/metadata/worldview.py) but the package is called https://github.com/DHI-GRAS/dg-calibration
But in that package, there is a folder called /tree/master/dg_calibration
I'm confused. Does Python not care about the difference between - and _ ?
e.g.
atmcorr\wrap_6S.py:331: RuntimeWarning: invalid value encountered in less
refl[refl < 0.0] = 0.0
A new package for py6s is now avalable in anaconda
https://anaconda.org/pypi/py6s
This makes it possiable to use the command below
Insted pip install -i https://pypi.anaconda.org/pypi/simple py6s
insted of having a version of py6s laying the the atmospheric correction folder
In images where the initial 'no data' value is 0, if dark waters reach 0 as a result of the atmospheric correction, the value gets set to 'no data', which then causes problems when the results from the atmcorr is sent further through the chain.
Can we possibly keep 0 and 'no data' separate? Meaning, if there were a value before, it shouldn't by accident end up as 'no data' when just using atmcorr.
(bathytest) C:\Users\mlra>pip install C:\Users\mlra\Desktop\atmcorr-master.zip
WARNING: Value for scheme.headers does not match. Please report this to <https://github.com/pypa/pip/issues/9617>
distutils: C:\Users\mlra\AppData\Local\Continuum\miniconda3\envs\bathytest\Include\UNKNOWN
sysconfig: C:\Users\mlra\AppData\Local\Continuum\miniconda3\envs\bathytest\Include
WARNING: Additional context:
user = False
home = None
root = None
prefix = None
Processing c:\users\mlra\desktop\atmcorr-master.zip
Requirement already satisfied: dg_calibration in c:\users\mlra\appdata\local\continuum\miniconda3\envs\bathytest\lib\site-packages (from atmcorr===unknown-62f606e077c4539c4bf868d13bf3a00fabb4c428) (0.4)
Requirement already satisfied: sensor_response_curves in c:\users\mlra\appdata\local\continuum\miniconda3\envs\bathytest\lib\site-packages (from atmcorr===unknown-62f606e077c4539c4bf868d13bf3a00fabb4c428) (unknown-657f78e2129a2eb294ca538b7d515d72edfadc31)
Requirement already satisfied: satmeta in c:\users\mlra\appdata\local\continuum\miniconda3\envs\bathytest\lib\site-packages (from atmcorr===unknown-62f606e077c4539c4bf868d13bf3a00fabb4c428) (1.3.0)
Requirement already satisfied: shapely in c:\users\mlra\appdata\local\continuum\miniconda3\envs\bathytest\lib\site-packages (from satmeta->atmcorr===unknown-62f606e077c4539c4bf868d13bf3a00fabb4c428) (1.7.1)
Requirement already satisfied: python-dateutil in c:\users\mlra\appdata\local\continuum\miniconda3\envs\bathytest\lib\site-packages (from satmeta->atmcorr===unknown-62f606e077c4539c4bf868d13bf3a00fabb4c428) (2.8.1)
Requirement already satisfied: lxml in c:\users\mlra\appdata\local\continuum\miniconda3\envs\bathytest\lib\site-packages (from satmeta->atmcorr===unknown-62f606e077c4539c4bf868d13bf3a00fabb4c428) (4.6.3)
Requirement already satisfied: affine in c:\users\mlra\appdata\local\continuum\miniconda3\envs\bathytest\lib\site-packages (from satmeta->atmcorr===unknown-62f606e077c4539c4bf868d13bf3a00fabb4c428) (2.3.0)
Requirement already satisfied: six>=1.5 in c:\users\mlra\appdata\local\continuum\miniconda3\envs\bathytest\lib\site-packages (from python-dateutil->satmeta->atmcorr===unknown-62f606e077c4539c4bf868d13bf3a00fabb4c428) (1.15.0)
Requirement already satisfied: scipy in c:\users\mlra\appdata\local\continuum\miniconda3\envs\bathytest\lib\site-packages (from sensor_response_curves->atmcorr===unknown-62f606e077c4539c4bf868d13bf3a00fabb4c428) (1.6.2)
Requirement already satisfied: numpy in c:\users\mlra\appdata\local\continuum\miniconda3\envs\bathytest\lib\site-packages (from sensor_response_curves->atmcorr===unknown-62f606e077c4539c4bf868d13bf3a00fabb4c428) (1.20.2)
Building wheels for collected packages: atmcorr
Building wheel for atmcorr (setup.py) ... done
Created wheel for atmcorr: filename=atmcorr-unknown_62f606e077c4539c4bf868d13bf3a00fabb4c428-py3-none-any.whl size=29575 sha256=38d4c8d5b82181d2c1976bc7efc420533854b029ab603ef2fa98072d5a1e659b
Stored in directory: c:\users\mlra\appdata\local\pip\cache\wheels\6c\8a\ef\0fd0c53fd63871b78026fe3c66ef8aac4257dd43363a4cc346
WARNING: Built wheel for atmcorr is invalid: Metadata 1.2 mandates PEP 440 version, but 'unknown-62f606e077c4539c4bf868d13bf3a00fabb4c428' is not
Failed to build atmcorr
Installing collected packages: atmcorr
Running setup.py install for atmcorr ... done
DEPRECATION: atmcorr was installed using the legacy 'setup.py install' method, because a wheel could not be built for it. A possible replacement is to fix the wheel build issue reported above. You can find discussion regarding this at https://github.com/pypa/pip/issues/8368.
WARNING: Value for scheme.headers does not match. Please report this to <https://github.com/pypa/pip/issues/9617>
distutils: C:\Users\mlra\AppData\Local\Continuum\miniconda3\envs\bathytest\Include\UNKNOWN
sysconfig: C:\Users\mlra\AppData\Local\Continuum\miniconda3\envs\bathytest\Include
WARNING: Additional context:
user = False
home = None
root = None
prefix = None
Successfully installed atmcorr-unknown-62f606e077c4539c4bf868d13bf3a00fabb4c428
When looking at overlapping S2 tiles acquired at the same time, it becomes clear that the atmospheric correction is not robust.
As a fix, we should look at atmospheric parameters for all of the swath at once, and then explode the parameters down to the correct scale. This will allow all the tiles to fail together, or pass together, instead of cases where partial results are retrieved and some of the MODIS downloads fail. In turn, we will get a consistent product across tile-borders.
We should also consider implementing a sanity check on the MODIS files. Especially we should look at the values retrieved in cloudy areas, to see how MODIS deals with these, and if we should filter them away.
This feature is lower priority than establishing a operational product chain for SEGES / NAER.
Setting the tileSize parameter in the preprocessing resulted in the following error message (input was a Sen2 image)
bathymetry production options_chain_preprocessing_v07.yaml
unknown+93442f60efde999ab6f515bf2490b2e60762630f
INFO | Reading input data
INFO | Start pre-processing
INFO | Starting atmospheric correction ...
Getting 6S params: 0%| | 0/605 [00:00<?, ?job/s]b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'
Getting 6S params: 0%| | 1/605 [00:07<1:14:32, 7.41s/job]bGetting 6S params: 8%|█████▏ | 46/605 [00:08<01:38, 5.67job/s]b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
b'The system cannot execute the specified program.\r\n'
ERROR | Uncaught exception
Traceback (most recent call last):
File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\bathymetry\scripts\cli.py", line 66, in main
cli(obj=obj)
File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\click\core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\click\core.py", line 717, in main
rv = self.invoke(ctx)
File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\click\core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\click\core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\click\core.py", line 555, in invoke
return callback(*args, **kwargs)
File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\click\decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\bathymetry\scripts\production.py", line 24, in production
main.run_chain(options, processing_backend=backend)
File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\bathymetry\chain\main.py", line 109, in run_chain
deepwater_vectorfile=o_chain.get('deepWaterShapeFile', None)
File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\bathymetry\chain\preprocess.py", line 56, in main
refl_data_bandfirst, profile = atmcorr_main(**o_atmcorr)
File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\atmcorr\processing.py", line 266, in main
for tilecp, adjcoef, idx in pbar(executor.map(wrap_6S.run_sixs_job, jobgen)):
File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\tqdm\_tqdm.py", line 979, in __iter__
for obj in iterable:
File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\concurrent\futures\_base.py", line 586, in result_iterator
yield fs.pop().result()
File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\concurrent\futures\_base.py", line 432, in result
return self.__get_result()
File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\concurrent\futures\_base.py", line 384, in __get_result
raise self._exception
File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\concurrent\futures\thread.py", line 56, in run
result = self.fn(*self.args, **self.kwargs)
File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\atmcorr\wrap_6S.py", line 141, in run_sixs_job
mysixs.run()
File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\Py6S\sixs.py", line 315, in run
self.outputs = Outputs(outputs[0], outputs[1])
File "c:\users\litn\appdata\local\continuum\miniconda3\envs\bathymetry\lib\site-packages\Py6S\outputs.py", line 64, in __init__
raise OutputParsingError("6S returned an error (shown above) - check for invalid parameter inputs")
Py6S.sixs_exceptions.OutputParsingError: 6S returned an error (shown above) - check for invalid parameter inputs
Having the option of using 6S on Landsat 4 & 5 data would be good for historical studies.
Some sort of choice has to be made for the old data, where MODIS atmospheric parameters are unavailable.
When using tiling to get finer-grained viewing angles, the results do not mach for two neighboring (and overlapping) Sentinel 2 tiles from the same overpass.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.