ubarsc / python-fmask Goto Github PK

I am trying to run f-mask algorith with this simple demo code:

and I got this result:

while .SAFE folder which I use looks:

Can you help me solve this problem?

Thank You. Looking forward to hearing from you

Hi,

Considering this annoucement :

https://sentinels.copernicus.eu/web/sentinel/-/copernicus-sentinel-2-major-products-upgrade-upcoming-l1c-and-l2a-sample-products

I tried fmask on a sample of the new SAFE format (can be found here) and it seems broken:
fmask_sentinel2Stacked.py -o mask.tif --granuledir ./S2B_MSIL1C_20210517T103619_N7990_R008_T30QVE_20210929T075738.SAFE/GRANULE Traceback (most recent call last): File "/b/home/ipgs/a2s_prod/A2S/installs/venv_a2s/bin/fmask_sentinel2Stacked.py", line 22, in <module> sentinel2Stacked.mainRoutine() File "/b/home/ipgs/a2s_prod/A2S/installs/venv_a2s/lib/python3.6/site-packages/fmask/cmdline/sentinel2Stacked.py", line 285, in mainRoutine resampledBands = makeStackAndAngles(cmdargs) File "/b/home/ipgs/a2s_prod/A2S/installs/venv_a2s/lib/python3.6/site-packages/fmask/cmdline/sentinel2Stacked.py", line 177, in makeStackAndAngles sentinel2makeAnglesImage.makeAngles(xmlfile, anglesfile) File "/b/home/ipgs/a2s_prod/A2S/installs/venv_a2s/lib/python3.6/site-packages/fmask/cmdline/sentinel2makeAnglesImage.py", line 86, in makeAngles info = sen2meta.Sen2TileMeta(filename=infile) File "/b/home/ipgs/a2s_prod/A2S/installs/venv_a2s/lib/python3.6/site-packages/fmask/sen2meta.py", line 43, in __init__ f = open(filename) FileNotFoundError: [Errno 2] No such file or directory: './S2B_MSIL1C_20210517T103619_N7990_R008_T30QVE_20210929T075738.SAFE/GRANULE/MTD_TL.xml'

Is there a workaround ?

best regards
David

Original report by Malyuk Maksim (Bitbucket: Malyuk_M, ).

Hi all!

When I run fmask_sentinel2makeAnglesImage I have "RuntimeWarning: invalid value encountered in rint" error:

But I've worked with Fmask for Python 1 mounth ago, and everything was ok.

Original report by Anonymous.

Hi

I am experimenting with the test-parallax branch and was trying to process the data bundle S2A_MSIL1C_20171031T101141_N0206_R022_T34WDS_20171031T122112.SAFE using the commands

gdalbuildvrt -resolution user -tr 20 20 -separate allbands.vrt *_B0[1-8].jp2 *_B8A.jp2 *_B09.jp2 *_B1[0-2].jp2

fmask_sentinel2makeAnglesImage.py -i ../*.xml -o angles.img

fmask_sentinel2Stacked.py -a allbands.vrt -z angles.img -o cloud.img

I have managed to run the code on other tiles but on this one I received this error

S2A_MSIL1C_20171031T101141_N0206_R022_T34WDS_20171031T122112.SAFE/GRANULE/L1C_T34WDS_A012317_20171031T101142/IMG_DATA$ fmask_sentinel2Stacked.py -a allbands.vrt -z angles.img -o cloud.img
Traceback (most recent call last):
File "/usr/local/bin/fmask_sentinel2Stacked.py", line 152, in
mainRoutine()
File "/usr/local/bin/fmask_sentinel2Stacked.py", line 144, in mainRoutine
fmask.doFmask(fmaskFilenames, fmaskConfig)
File "/usr/local/lib/python2.7/dist-packages/fmask/fmask.py", line 143, in doFmask
fmaskFilenames, fmaskConfig, missingThermal)
File "/usr/local/lib/python2.7/dist-packages/fmask/fmask.py", line 269, in doPotentialCloudFirstPass
applier.apply(potentialCloudFirstPass, infiles, outfiles, otherargs, controls=controls)
File "/usr/local/lib/python2.7/dist-packages/rios/applier.py", line 675, in apply
userFunction(*params)
File "/usr/local/lib/python2.7/dist-packages/fmask/fmask.py", line 360, in potentialCloudFirstPass
(ratio8a8, ratio8a7, v8a8, v8a7, cdi) = calcCDI(ref, fmaskConfig, otherargs.refBands)
File "/usr/local/lib/python2.7/dist-packages/fmask/fmask.py", line 1314, in calcCDI
cdi = (v8a8 - v8a7) / (v8a8 + v8a7)
FloatingPointError: invalid value encountered in true_divide

Would you consider making python-fmask and official python package?

This would be great since it

• avoids future naming conflicts if it is registered on pypi
• makes it automatically installable via pip if another package depends on it

Happy to help if you have any questions.

Hi,

I wonder if any function or command line can apply the mask to the original RGB file for sentinel 2? Thanks.

Original report by David Ketchum (Bitbucket: dgketchum, GitHub: dgketchum).

Hi,
I love your project. I was reading the source and came across the values for ESUN, which my need updating. I'm probably missing something, but the values for Landsat 7 on fmask.LandsatTOA aren't as they appear at https://landsat.usgs.gov/esun. I'm probably missing something that takes care of this. Is this the case?

Thanks for all your work on fmask, it makes my life easier!

Original report by Thomas Maschler (Bitbucket: thomas-maschler, GitHub: thomas-maschler).

Hi,

I am trying using Fmask using Sentinel 2 images downloaded from AWS.

I can build the stacked layer and create the angles file, when trying to build the cloud mask I get an error:

_fillminima.error: parameter 6 and 7 must be int64 arrays

Any hints what went wrong?

Original report by Anonymous.

Hi! Sorry for being a bother...using 0.5.2 version of python-fmask on Ubuntu 14.04 i keep getting the same error:

Traceback (most recent call last):
File "/usr/local/bin/fmask_usgsLandsatStacked.py", line 146, in
mainRoutine()
File "/usr/local/bin/fmask_usgsLandsatStacked.py", line 143, in mainRoutine
fmask.doFmask(fmaskFilenames, fmaskConfig)
File "/usr/local/lib/python2.7/dist-packages/fmask/fmask.py", line 143, in doFmask
fmaskFilenames, fmaskConfig, missingThermal)
File "/usr/local/lib/python2.7/dist-packages/fmask/fmask.py", line 295, in doPotentialCloudFirstPass
raise fmaskerrors.FmaskFileError(msg)
fmask.fmaskerrors.FmaskFileError: Expected 2 bands in Thermal file. Found 1 bands

I tried using python-fmask on scenes LE71920282015005SG100, LE71920282000252FUI00 and LC81920282015285LGN00

Original report by Chu Yaw Tai (Bitbucket: [Chuyaw Tai](https://bitbucket.org/Chuyaw Tai), ).

Hi, I have some issues for the FMask installation and usage in Window.
I did:
python setup.py build
python setup.py install

how to run and generate cloud.img?
fmask_sentinel2Stacked.py -o cloud.img --safedir <product>.SAFE

Original report by Xavier Corredor Llano (Bitbucket: XavierCLL, GitHub: XavierCLL).

HI,

I want to process fmask in parallel for improve the time of run. I tried implement the parallel process with jobmanager in Rios library described in here

I search the "controls = applier.ApplierControls()" in python-fmask code and change with:

#!python
controls = applier.ApplierControls()
controls.setNumThreads(multiprocessing.cpu_count())
controls.setJobManagerType("multiprocessing")

For some process work fine, but in others not:

In landsatTOA.py changed in line 167, produce the following error:

#!python
  File "/usr/bin/fmask_usgsLandsatTOA.py", line 48, in mainRoutine
    landsatTOA.makeTOAReflectance(cmdargs.infile, cmdargs.mtl, cmdargs.anglesfile, cmdargs.output)
  File "/usr/lib/python2.7/site-packages/fmask/landsatTOA.py", line 175, in makeTOAReflectance
    applier.apply(riosTOA, inputs, outputs, otherinputs, controls=controls)
  File "/usr/lib/python2.7/site-packages/rios/applier.py", line 674, in apply
    outBlocksList = jobmgr.runSubJobs(userFunction, jobInputs)
  File "/usr/lib/python2.7/site-packages/rios/parallel/jobmanager.py", line 209, in runSubJobs
    outputBlocksList = self.gatherAllOutputs(jobIDlist)
  File "/usr/lib/python2.7/site-packages/rios/parallel/jobmanager.py", line 808, in gatherAllOutputs
    output = job.get(timeout=None)            
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 567, in get
    raise self._value
KeyError: 139830344649824

Happens the same if tried enable parallel rios in fmask.py line 217 and line 1189.

Any idea of this issue?

Thanks

Original report by Anonymous.

Hi,

I am trying to create a docker image for python-fmask. How can one do a test to check if the build and instaltion of python-fmask ran successfuly?

Many thanks,
Lufuno Vhengani

Can you provide a download link if necessary, except Google driver, thank you!!!

Original report by oaznay (Bitbucket: oaznay, GitHub: oaznay).

Hi,

I have an issue using python FMASK (0.4-3) with S2-L1C products. I use GDAL-2.1.3 and rios-1.4.3; I tried with several images and I still get the same error.

Can you help me please?

Thank you by advance.

Regards,

O.

Hello, i would like know how i can use the fmask funcionalities associated whit a qgis code, for create a cloud mask

Original report by Anonymous.

Dear All,

I have this error when I try to run fmask_sentinel2makeAnglesImage.py:

Traceback (most recent call last):
File "/home/userdata/python-fmask-0.4.5/bin/fmask_sentinel2makeAnglesImage.py", line 122, in
mainRoutine()
File "/home/userdata/python-fmask-0.4.5/bin/fmask_sentinel2makeAnglesImage.py", line 64, in mainRoutine
info = sen2meta.Sen2TileMeta(filename=cmdargs.infile)
File "/usr/local/lib/python2.7/dist-packages/fmask/sen2meta.py", line 37, in init
sensingTimeStr = sensingTimeNode.text.strip()
AttributeError: 'NoneType' object has no attribute 'text'

And I think the module is working,

fmask_sentinel2makeAnglesImage.py -h
usage: fmask_sentinel2makeAnglesImage.py [-h] [-i INFILE] [-o OUTFILE]

optional arguments:
-h, --help show this help message and exit
-i INFILE, --infile INFILE
Input sentinel-2 tile metafile
-o OUTFILE, --outfile OUTFILE
Output angles image file

Best regards,

filler issue created by bitbucket_issue_migration

Hi! Thanks for making python fmask! I usually don’t work in a python environment, but I found the installation straightforward on my ubuntu laptop, and it works great.

This is either a question or a feature request. I’ve read on the python fmask website that most people use the --safedir option to access and apply the fmask to Sentinel images, and that the older way which necessitated, among other things, the separate creation of Anglefiles is now obsolete.

I would like to use my own stack of Sentinel TOA reflectance files in python fmask, instead of using the -- safedir option. I want to run my own crop mask on the Sentinel files first to see if that will reduce the errors of commission. One question I have is will python fmask work with other file types as input files, i.e. geotiffs? (I assume it does, but just wanted to check)

Is there any documentation on how one would go about using python fmask on a stack of TOA reflectances produced by the user. In particular, I wonder if there is some documentation on what one would need to do to <sentinel2makeAnglesImage.py> to produce your own Anglefiles. I usually work in an R environment and my python abilities are not so good.

The feature request part of it is if there is a way to produce the ImageAngle files from the command line. That might be too much work for you, particularly if there is not so much interest in such a feature, but I thought I would throw that out there.

Thanks again for making python fmask!

Original report by Xavier Corredor Llano (Bitbucket: XavierCLL, GitHub: XavierCLL).

Hi,

Python-fmask cannot process old MTL format such as LT50120541987003XXX10

Key error when try parse file:

#!bash
... ".../fmask/landsatangles.py", line 236, in sunAnglesForExtent

dateStr = mtlInfo['DATE_ACQUIRED']
KeyError: 'DATE_ACQUIRED'

Regards

Xavier

Original report by Anonymous.

Hi,

I am trying to run python-fmask, I can do all the steps but when I try to run the last one:

*fmask_usgsLandsatStacked.py -t thermal.img -a toa.img -m _MTL.txt -z angles.img -s saturationmask.img -o cloud.img

I get this error:

*Traceback (most recent call last):
File "/home/diego/anaconda3/envs/fmask/bin/fmask_usgsLandsatStacked.py", line 146, in
mainRoutine()
File "/home/diego/anaconda3/envs/fmask/bin/fmask_usgsLandsatStacked.py", line 143, in mainRoutine
fmask.doFmask(fmaskFilenames, fmaskConfig)
File "/home/diego/anaconda3/envs/fmask/lib/python3.6/site-packages/fmask/fmask.py", line 143, in doFmask
fmaskFilenames, fmaskConfig, missingThermal)
File "/home/diego/anaconda3/envs/fmask/lib/python3.6/site-packages/fmask/fmask.py", line 273, in doPotentialCloudFirstPass
applier.apply(potentialCloudFirstPass, infiles, outfiles, otherargs, controls=controls)
File "/home/diego/anaconda3/envs/fmask/lib/python3.6/site-packages/rios/applier.py", line 675, in apply
userFunction(params)
File "/home/diego/anaconda3/envs/fmask/lib/python3.6/site-packages/fmask/fmask.py", line 312, in potentialCloudFirstPass
refNullmask = (inputs.toaref[otherargs.bandsForRefNull] == otherargs.refNull).any(axis=0)
IndexError: index 6 is out of bounds for axis 0 with size 6

All the intermediate products has been well generated, so I can't figure out where is the error.

I'd appreciate your help. Anyway, thanks for this tool.

Im tryng execute this comand : fmask_usgsLandsatStacked.py -o cloud.img scenedir LC08_L1TP_217076_20180901_20180912_01_T1 to get a LandSat-8 image whith de 5 classifications, but im having this error:
fmask_usgsLandsatStacked.py: error: unrecognized arguments: scenedir LC08_L1TP_217076_20180901_20180912_01_T1
im starting now to use fmask, and dont know how resolve this error

Hi everyone:) I'm really sorry for this little naive and less technical question. I'm kind of new to this. But I'd like to use your great package and include it in a little GRASS-GIS add-on. I already spent way too much time in trying to get this done and didn't achieve anything due to the lack of my knowledge.

So actually my question just is if there is any way in downloading and installing python-fmask via conda-forge and use it without being a conda environment. Probably that's a little strange explication, but what I mean is more or less the following:

I installed python-fmask with
conda install -c conda-forge python-fmask
in my base environment.

Then I can perfectly run:
which fmask_sentinel2Stacked.py

and it gives me:
~/miniconda3/bin/fmask_sentinel2Stacked.py

I then did a conda deactivate. Afterwards I'm unable to find the fmask-commandline-tools.

So is there any chance to get this run without being in a conda environment?

Thank you very much in advance for any help and sorry again about the ingenuous question...

Hey, I was wondering if the fmask can be implemented using google collab. So far I wasn't able to download it there using !pip install python-fmask.

I've received this error when run package with Sentinel 2 file:

It was run in python 3.8.1, with gdal GDAL 2.2.3, Ubuntu 18.04, with pythonfmask-0.5.4.

fmask_sentinel2Stacked.py -o cloud.img --safedir ./img_files/S2A_MSIL1C_20200311T125301_N0209_R052_T24LXN_20200311T160933.SAFE
Traceback (most recent call last):
  File "/home/diogo/.ve/fmask/bin/fmask_sentinel2Stacked.py", line 22, in <module>
    sentinel2Stacked.mainRoutine()
  File "/home/diogo/.ve/fmask/lib/python3.8/site-packages/fmask/cmdline/sentinel2Stacked.py", line 311, in mainRoutine
    fmask.doFmask(fmaskFilenames, fmaskConfig)
  File "/home/diogo/.ve/fmask/lib/python3.8/site-packages/fmask/fmask.py", line 157, in doFmask
    potentialShadowsFile = doPotentialShadows(fmaskFilenames, fmaskConfig, NIR_17)
  File "/home/diogo/.ve/fmask/lib/python3.8/site-packages/fmask/fmask.py", line 725, in doPotentialShadows
    scaledNIR_filled = fillminima.fillMinima(scaledNIR, nullval, NIR_17_dn)
  File "/home/diogo/.ve/fmask/lib/python3.8/site-packages/fmask/fillminima.py", line 72, in fillMinima
    fillminima.fillMinima(img, img2, hMin, hMax, nullmask, boundaryval,
TypeError: fillMinima() takes 3 positional arguments but 8 were given

Original report by Ian Cooke (Bitbucket: ircwaves, GitHub: ircwaves).

First let me say that we are very grateful that you have contributed this fmask implementation to the open source world. It is an excellent package that provides a key component for the python ecosystem for Sentinel-2 data users.

Version 0.5.0 indicates that passing fmask_sentinel2Stacked.py pre-stacked file of reflectance bands (--toa), and the image of angles (--anglesfile) is obsoleted. This use-case worked out very well for our application where we download the individual bands and the metadata XML, instead of the entire SAFE archive.

@vincentschut is mentioned here indicating that VRT-based resampling may result in pulling shifted data from the overviews. We haven't observed any issue thus far, but perhaps a link to an example (or discussion) can be posted? I've grappled with many resampling/registration issues, so I am quite familiar with how subtle they can be.

I've labelled this minor, as there are various options to work around, but I think it might be helpful for general understanding of the reason for this change.

Thanks,

Ian

Hi,

When I try to run with python3.6 (by default) I have the next error:

$ fmask_sentinel2Stacked.py --safedir /tmp/tmpc6i7ri38/S2A_MSIL1C_20200101T142731_N0208_R053_T19JFH_20200101T174749.SAFE -o cloud.img
Traceback (most recent call last):
  File "/usr/local/bin/fmask_sentinel2Stacked.py", line 22, in <module>
    sentinel2Stacked.mainRoutine()
  File "/usr/local/lib/python3.6/dist-packages/fmask/cmdline/sentinel2Stacked.py", line 285, in mainRoutine
    resampledBands = makeStackAndAngles(cmdargs)
  File "/usr/local/lib/python3.6/dist-packages/fmask/cmdline/sentinel2Stacked.py", line 214, in makeStackAndAngles
    '-o', cmdargs.toa] + resampledBands)
  File "/usr/lib/python3.6/subprocess.py", line 306, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/lib/python3.6/subprocess.py", line 287, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.6/subprocess.py", line 1295, in _execute_child
    restore_signals, start_new_session, preexec_fn)
TypeError: expected str, bytes or os.PathLike object, not NoneType

But when I python3.7 (by default) any problems:

$ fmask_sentinel2Stacked.py --safedir /tmp/tmpc6i7ri38/S2A_MSIL1C_20200101T142731_N0208_R053_T19JFH_20200101T174749.SAFE -o cloud.img
$ 

I don't investigate the error, but could be possible this?

I'm trying to work out what dilations are used by default by python-fmask when it's applied to Landsat data. In the documentation I can see several different options where dilations are specified or mentioned:

https://www.pythonfmask.org/en/latest/fmask_fmask.html#fmask.fmask.maskAndBuffer
https://www.pythonfmask.org/en/latest/fmask_fmask.html#fmask.fmask.matchShadows
https://www.pythonfmask.org/en/latest/fmask_config.html#fmask.config.FmaskConfig.setCloudBufferSize
https://www.pythonfmask.org/en/latest/fmask_config.html#fmask.config.FmaskConfig.setShadowBufferSize

My reading of this is that there seems to be a default 3 pixel buffer that is applied to shadows and perhaps clouds, as well as an additional optional (but default) 5 pixel cloud buffer and a 10 pixel shadow buffer, giving a total of 8 pixel buffer for clouds, and 13 pixel buffer for cloud shadow.

Does this sound correct?

Original report by Anonymous.

Hello,

I've just installed fmask to my python 2.7.
I'm going to use it to make cloud mask files and not familiar with cmd commands.
Could you please show me how to use fmask in scripts properly?

Regards,
Grigory

I would like to run fmask with Sentinel 2A. Theoretically speaking, is there any problem?

Original report by Christof Kaufmann (Bitbucket: ChristofKaufmann, GitHub: ChristofKaufmann).

Dear all,

this is rather a discussion than a report. When working with python-fmask I noticed that importing parts of it change the GDAL exception setting. So the following - maybe unusual - example fails for me:

from fmask import landsatTOA

# Setting the exceptions off (default) make gdal_merge work
#from osgeo import gdal
#gdal.DontUseExceptions()

import sys
sys.path.append('/usr/bin')
import gdal_merge
gdal_merge.main(['gdal_merge.py', '-separate', '-o', 'test.tif', 'band1.tif', 'band2.tif'])

This should actually stack band1.tif and band2.tif into a single file test.tif. However, gdal_merge assumes that exceptions are turned off (which is by itself an issue maybe). I can work around this, by using

import contextlib
from osgeo import gdal

@contextlib.contextmanager
def gdal_exceptions(state=None):
    orig_state = gdal.GetUseExceptions()
    if state is not None:
        if state:
            gdal.UseExceptions()
        else:
            gdal.DontUseExceptions()
    yield
    if orig_state:
        gdal.UseExceptions()
    else:
        gdal.DontUseExceptions()

and then I can surround the imports of fmask like:

with gdal_exceptions(): # preserves the original exception state
    from fmask import landsatTOA

and calls to fmask functions like:

with gdal_exceptions(True):  # set exception state temporarily True
    landsatTOA.makeTOAReflectance(...)

But it is just a work around.

So a possible solution could be to wrap gdal_merge in a subprocess, but I don't really like spawning multiple processes. Another possible solution would be for python-fmask to not change the exception setting. However, I tried to start a pull request with a similar solution as gdal_exceptions, but if the exceptions are really important for the python-fmask code, there would be way too many with statements required in python-fmask. What is your opinion on this?

BTW: I really import the fmask command line scripts and use them similarly as gdal_merge in the example above. I just simplified the problem here as much as possible.

Hello, Team,

How are you.

I added a PR: #54, which aims to get the Sentinel-2 run with TOA feature back.

Please let me know how do you think this solution.

Cheers

Sai

Original report by Anonymous.

I encountered IndexError when I compute cloud mask with my Landsat 8 scene LC08_L1TP_150033_20150413_20170410_01_T1 downloaded from USGS Collection L1T products.

its all okey in

#!python

gdal_merge.py -separate -of HFA -co COMPRESSED=YES -o ref.img LC8*_B[1-7,9].TIF
gdal_merge.py -separate -of HFA -co COMPRESSED=YES -o thermal.img LC8*_B1[0,1].TIF
fmask_usgsLandsatMakeAnglesImage.py -m *_MTL.txt -t ref.img -o angles.img
mask_usgsLandsatSaturationMask.py -i ref.img -m *_MTL.txt -o saturationmask.img
fmask_usgsLandsatTOA.py -i ref.img -m *_MTL.txt -z angles.img -o toa.img

but it given me IndexError in :

#!python

fmask_usgsLandsatStacked.py -t thermal.img -a toa.img -m *_MTL.txt -z angles.img -s saturationmask.img -o cloud.img

below are the detailed error information

#!python

Traceback (most recent call last):
  File "/home/deyu1/anaconda2/envs/fmaskenv/bin/fmask_usgsLandsatStacked.py", line 146, in <module>
    mainRoutine()
  File "/home/deyu1/anaconda2/envs/fmaskenv/bin/fmask_usgsLandsatStacked.py", line 143, in mainRoutine
    fmask.doFmask(fmaskFilenames, fmaskConfig)
  File "/home/deyu1/anaconda2/envs/fmaskenv/lib/python2.7/site-packages/fmask/fmask.py", line 143, in doFmask
    fmaskFilenames, fmaskConfig, missingThermal)
  File "/home/deyu1/anaconda2/envs/fmaskenv/lib/python2.7/site-packages/fmask/fmask.py", line 269, in doPotentialCloudFirstPass
    applier.apply(potentialCloudFirstPass, infiles, outfiles, otherargs, controls=controls)
  File "/home/deyu1/anaconda2/envs/fmaskenv/lib/python2.7/site-packages/rios/applier.py", line 667, in apply
    userFunction(*params)
  File "/home/deyu1/anaconda2/envs/fmaskenv/lib/python2.7/site-packages/fmask/fmask.py", line 308, in potentialCloudFirstPass
    refNullmask = (inputs.toaref[otherargs.bandsForRefNull] == otherargs.refNull).any(axis=0)
IndexError: index 1 is out of bounds for axis 0 with size 1

its be appreciate anyone who give me why. thank u...

Hi there,
It is so cool to publish the python version of Fmask. Thank you so much for your work. But I am wondering if the version 0.5.5 is also for Fmask4.0? And when I tried to run version 0.5.5, I got some errors:

1. in the file fillminia.py, Line 71，72, the fillMinima function have eight variables. But in the definition of this function, there are three variables.
   2)in the file valueindex.py, Line 143，144, there is no function called valndxFunc.
   Did I miss something or the code is not fully finished until now.
   Thank you so much,
   Yiming

How should Fmask handle cloud shadows at the two edges of the scene nearest to the sun?

Example (LC81520742013143LGN02): Landsat 8, path 152, row 74, 2013-05-23T06:11:31.604832Z

It appears that a cloud object touching the scene edge has been assigned a height, and a shadow has been projected from it. Pixels very close to the edge seem to be masked clear, because there is no data to indicate how far the cloud extends beyond the scene edge. Would it be more conservative to default to marking these pixels as shadow, where there is shadow being projected through the air column above them?

Original report by Felipe de Lucia Lobo (Bitbucket: FLLobo, ).

Hello,

I am trying to run fmask on python for the first time. I have installed phython_fmask in my computer just fine. However, when try the first command line (gdalbuildvrt -resolution user -tr 20 20 -separate allbands.vrt *_B0[1-8].jp2 *_B8A.jp2 *_B09.jp2 *_B1[0-2].jp2) it does not recognize the input file. I tried several forms for the input image file, no luck.

Two questions: 1) What folder in the image data should I call the function? the GRANULE one?
2) What is the standard input data format for gsalbuildvrt? For example: S2A_OPER_MSI_L1C_TL_SGS__20160305T231645_A003668_T19KHA_B01.jp2 ?

I am stuck here, any help, please? I am using Windows 10 OS.

thank you

Felipe

Hi,

Is there a plan to support Landsat Collection 2?
does it require a lot of changes in the equations or in its settings?

with minimum changes, I could run python-fmask in an image of Landsat 8 C2, but the result was bad:

Thanks, regards,
Xavier

Hi,

I try to install python-fmask in a clean venv and when I run the fmask_sentinel2Stacked.py give me error that numpy, rios ... are not installed.

I can see the documentation that say: Installation requires Python, numpy, scipy, GDAL and RIOS but maybe will be interest has that written on the README or just a requirements.txt file

Original report by Anonymous.

I've managed to successfully run python-fmask, producing an output .img file (please see #2 for my issues with creating output in another format). However, the final application I will be using it in does not support .img files, so I have tried to convert it to a GeoTIFF file using gdal_translate.

However, looking at the output image I find that it is a RGB image showing the FMask classes as different colours. I would like to get this output as a more 'standard' FMask output, without the colour palette or RAT...just raw digital numbers as in the original FMask.

I've tried playing around with the various options to gdal_translate, and I've had a brief look at the python-fmask code, but I can't quite see how to get this working.

Any ideas?

Fmask for Sentinel-2 does work . I was using the code exactly the same way in May and it was fine.
Thank you!

Here is the log:
(base) C:\landEcology\analysis\RS\1.matupi\1.raster_data\S2_2019>fmask_sentinel2Stacked.py -o cloud.img --safedir S2A_MSIL1C_20190605T142751_N0207_R053_T20MPS_20190605T191007.SAFE
Traceback (most recent call last):
File "C:\Users\almetha\Anaconda3\Scripts\fmask_sentinel2Stacked.py-script.py", line 10, in
sys.exit(mainRoutine())
File "C:\Users\almetha\Anaconda3\lib\site-packages\fmask\cmdline\sentinel2Stacked.py", line 285, in mainRoutine
resampledBands = makeStackAndAngles(cmdargs)
File "C:\Users\almetha\Anaconda3\lib\site-packages\fmask\cmdline\sentinel2Stacked.py", line 197, in makeStackAndAngles
resampleMethod = chooseResampleMethod(cmdargs.pixsize, inBandImg)
File "C:\Users\almetha\Anaconda3\lib\site-packages\fmask\cmdline\sentinel2Stacked.py", line 226, in chooseResampleMethod
imginfo = fileinfo.ImageInfo(inBandImg)
File "C:\Users\almetha\Anaconda3\lib\site-packages\rios\fileinfo.py", line 79, in init
ds = gdal.Open(str(filename), gdal.GA_ReadOnly)
File "C:\Users\almetha\Anaconda3\lib\site-packages\osgeo\gdal.py", line 3087, in Open
return _gdal.Open(*args)
RuntimeError: `S2A_MSIL1C_20190605T142751_N0207_R053_T20MPS_20190605T191007.SAFE/GRANULE\L1C_T20MPS_A020642_20190605T143120/IMG_DATA\T20MPS_20190605T142751_B01.jp2' not recognized as a supported file format.

Original report by Anonymous.

Dear all.

I am trying to work with Fmask in a Xubuntu Machine. I already installed Numpy, scipy and RIOS. GDAL was installed before with Miniconda.

Running the script lines from the documentation I got an error in the third one, any help?:

Python-fmask$ ./bin/fmask_usgsLandsatTOA.py -i ref.img -m LE7*_MTL.txt -o toa.img
Traceback (most recent call last):
File "./bin/fmask_usgsLandsatTOA.py", line 23, in
from fmask import landsatTOA
File "/Python-fmask/fmask/landsatTOA.py", line 27, in
from . import fmask
File "/Python-fmask/fmask/fmask.py", line 62, in
from . import fillminima
File "/Python-fmask/fmask/fillminima.py", line 33, in
from . import _fillminima
ImportError: cannot import name '_fillminima'

Original report by Anonymous.

When running the following from the Windows command-line:

#!python

python C:\Anaconda3\envs\HOTBAR\Scripts\fmask_usgsLandsatStacked.py -t thermal.img -a toa.img -m LE72020242003106EDC00_MTL.txt -s saturationmask.img -o cloud_NEW.bsq

I get the following output:

#!python

Traceback (most recent call last):
  File "C:\Anaconda3\envs\HOTBAR\Scripts\fmask_usgsLandsatStacked.py", line 107, in <module>
    mainRoutine()
  File "C:\Anaconda3\envs\HOTBAR\Scripts\fmask_usgsLandsatStacked.py", line 104, in mainRoutine
    fmask.doFmask(fmaskFilenames, fmaskConfig)
  File "C:\Anaconda3\envs\hotbar\lib\site-packages\fmask\fmask.py", line 154, in doFmask
    pass1file)
  File "C:\Anaconda3\envs\hotbar\lib\site-packages\fmask\fmask.py", line 1155, in finalizeAll
    b"Cloud Shadow", b"Snow"])
  File "C:\Anaconda3\envs\hotbar\lib\site-packages\rios\rat.py", line 332, in writeColumn
    writeColumnToBand(gdalBand, colName, sequence, colType, colUsage)
  File "C:\Anaconda3\envs\hotbar\lib\site-packages\rios\rat.py", line 231, in writeColumnToBand
    if not attrTbl.ChangesAreWrittenToFile():
AttributeError: 'NoneType' object has no attribute 'ChangesAreWrittenToFile'

In this case I have the default RIOS output set to 'ENVI' - but I get a similar error if I set the default output to GTIFF.

Any ideas?

Original report by Xavier Corredor Llano (Bitbucket: XavierCLL, GitHub: XavierCLL).

Hi,

In some cases of run, the python-fmask stop with TypeError:

#!bash
fmask/fillminima.py, line 67, in fillMinima
    innerBoundary = nullmaskDilated - nullmask
TypeError: numpy boolean subtract, the `-` operator, is deprecated, use the bitwise_xor, the `^` operator, or the logical_xor function instead.

I checked the variables nullmaskDilated and nullmask and both are boolean matrix with True values. I don't know exactly what append and how fix that

Regards

Original report by Anonymous.

Hello guys,

when a try execute this command:

#!Shell Script

fmask_usgsLandsatTOA.py -i LC08_L1TP_223071_20161226_20170315_01_T1_.vrt -m LC08_L1TP_223071_20161226_20170315_01_T1_MTL.txt -z LC08_L1TP_223071_20161226_20170315_01_T1_angle.img -o LC08_L1TP_223071_20161226_20170315_01_T1_toa.img

I got this error:

#!Shell Script
  File "/usr/local/bin/fmask_usgsLandsatTOA.py", line 23, in <module>
    from fmask import landsatTOA
  File "/usr/local/lib/python2.7/dist-packages/fmask/landsatTOA.py", line 29, in <module>
    from . import fmask
  File "/usr/local/lib/python2.7/dist-packages/fmask/fmask.py", line 68, in <module>
    from . import fillminima
  File "/usr/local/lib/python2.7/dist-packages/fmask/fillminima.py", line 37, in <module>
    from fmask import fillminima
ImportError: cannot import name fillminima

So I commented thoss part, and work:

fillminima.py file:

#!python

#if os.getenv('READTHEDOCS', default='False') != 'True':
#    from . import _fillminima

valueindexes.py

#!python

#if os.getenv('READTHEDOCS', default='False') != 'True':
#    from . import _valueindexes

The problem is occuring in python2.7 and python3.5.

PS: Your API is awesome. I love RIOS too. Great Job !

Original report by Stefan Milutinovic (Bitbucket: Milutinke, GitHub: Milutinke).

When i want to run "/home/stefan/Downloads/python-fmask-0.4.2/bin/fmask_sentinel2Stacked.py -a allbands.vrt -z angles.img -o cloud.img
", I get this error.

I am using this for one Granule of sentinel 2 image.

Is it my fault or no?

Thanks.

Original report by Jens2121 (Bitbucket: Jens2121, GitHub: Jens2121).

I am unable to pass my arguments into the cmd line to "Make a 4-layer image of satellite and sun angles, from the given tile metadata file."

I am following the guide at "http://pythonfmask.org/en/latest/":

Make a separate image of the per-pixel sun and satellite angles.

#!python
fmask_sentinel2makeAnglesImage.py -i ../*.xml -o angles.img

The exact command I put into the cmd in windows is, as well as the output is as follows:

#!python
F:\1_Cloud_Masking_Test\s2a_35jpm_2016_12_23>fmask_sentinel2makeAnglesImage.py -i metdata.xml -o allangles.img
usage: fmask_sentinel2makeAnglesImage.py [-h] [-i INFILE] [-o OUTFILE]

optional arguments:
  -h, --help            show this help message and exit
  -i INFILE, --infile INFILE
                        Input sentinel-2 tile metafile
  -o OUTFILE, --outfile OUTFILE
                        Output angles image file

As you can see it is just returning the 'print_help()' command.

Is there anything I am doing incorrectly?

Original report by Anonymous.

Dear developers,

I greatly appreciate your work and hope that you won't disapprove of this way of communicating with you.

I was very happy finding python-fmask and would like to use it operationally in a Landsat & Sentinel-2 processing chain. For that purpose I processed some S2A scenes in Northern & Southern Europe as well as India to get a feeling for the results. I also compared the python-fmask (0.4.3) results to Sen2Cor on S2A imagery. I was very satisfied with the python-fmask results over various summer images. In most cases it seemed more robust than the Sen2Cor cloud mask.

But I discovered that python-fmask seems to classify all winter scenes in Northern Europe as 100% cloudy even if mostly clear. Do you have any idea where this comes from or how to solve it?

It seems that other people had the same problem (http://forum.step.esa.int/t/sentinel-2-cloud-mask-with-fmask/4152/19).

Another question: Could you please tell me where to find the "cloud probability threshold" mentioned on the "original" Fmask website (https://github.com/prs021/fmask) if it exists? I don't find it in fmask-python. Maybe this could help me...?

At the moment I use python-fmask in anaconda on a Mac. I tried configuring several parameters (cloudBufferSize, shadowBufferSize, strictFmask, minCloudSize_pixels) which improved things, but not substantially.

Thanks a lot for your time!

I hope you don't feel offended by my way of contacting you.

Kind regards

Florian Schlenz
([email protected])

Original report by Johan Mollevik (Bitbucket: [Johan Mollevik](https://bitbucket.org/Johan Mollevik), ).

Hi

Now that the SAFE format is suported an added feature would be support for reading directly from zip compressed SAFE archives. This is something which several graphical GIS programs does quite happily (SNAP and QGIS for example) and from my knowledge of the zip format should be quite doable.

The gain for doing this would be space savings at the processing machine and simplified processing as the zip files are whats downloaded from ESA.

Thanks a lot for maintaining this for the comunity

Hi all,

I would like to use Fmask with data in Google Earth Engine (GEE). Currently, only two algorithms have been integrated to GEE: ee.Algorithms.FMask.matchClouds, and ee.Algorithms.FMask.fillMinima. The Cloud Displacement Index algorithm for Sentinel-2 data is also available in GEE (see ee.Algorithms.Sentinel2.CDI).

It would be ideal if we could adapt the functions in python-fmask to be able to get the necessary inputs into the ee.Algorithms.FMask.matchClouds algorithm, i.e. the potential cloud and shadow mask images.

Does anyone have any experience using the GEE python API to do this?

I have been experimenting with this and so far have developed two functions in my python-fmask fork.

The first one creates the angles images for Sentinel-2 data (see notebook here):

The second function performs the potential cloud first pass (see notebook here) - including the use of the ee.Algorithms.Sentinel2.CDI function:

If there is anyone interested in helping developing this, please let me know and I will create a pull request with what I have so far.

Also, if there is anyone who has successfully used the ee.Algorithms.FMask.matchClouds with Sentinel-2 L1C data in GEE without having to go through all this trouble, please share your solution!

Hi all, we've noticed some strange "terraced" artefacts in the cloud shadow data produced by Fmask in Digital Earth Australia's Landsat Collection 3 datasets:

The example above is from 2020-02-26 offshore of Kurnell, Sydney, however the artefacts are common and affect most timesteps for that location and others (I haven't exhaustively tested this, but I've seen it come up regularly in other locations too). I can provide a reproducible DEA datacube query if it would help track down the issue.

The Fmask data was produced using v0.5.4 of python-fmask:

Does anyone know what is causing these artefacts?

Original report by Anonymous.

The function sunAnglesForExtent in landsatangls.py

Line 227
Original code : (ul_long, ul_lat, ur_long, ur_lat, lr_long, lr_lat, ll_long, ll_lat) = cornerLatLong

Changed code : (ul_lat, ul_long, ur_lat, ur_long, lr_lat, lr_long, ll_lat, ll_long) = cornerLatLong

Then the output : angle map is correct.