semiautomaticgit / remotior_sensus Goto Github PK

When i select the MTL file for Landsat data in image conversion, SCP 8.1.3 shows error message: "Warning: no band found. Check metadata inside the directory". If i process without the metadata, k1 and k2 constants, for example, are not loaded.

Hello,

I recently attempted to build remotior_sensus and run the tests and found that a majority of them failed. Specifically, only 7 out of 32 tests passed. This is concerning as it suggests potential issues with the library's functionality and reliability.

Could you please confirm if these tests are expected to be operational at this stage? If so, I would appreciate any guidance on possible causes for these failures.

Here's a brief overview of my environment:

• Operating System: Archlinux
• Python Version: 3.11.6
• Library Version: 0.2.01

Any insights or assistance you can provide in resolving these test failures would be greatly appreciated.

Thank you.

Description
When using version 0.4.2 of the remotior_sensus library, a ResourceWarning: unclosed file occurs due to unclosed file handlers in the Log class. This issue was not present in version 0.3.5.

Steps to Reproduce
Initialize a session with remotior_sensus.Session.
Observe the warning message about unclosed files in the logs.

Proposed Solution
In the log.py file, add a method to properly close all handlers to ensure that file handlers are correctly closed. Below is the modified code for the Log class:

Modified Code Snippet
Add the following method to ensure that all file handlers are properly closed:

def close_handlers(self, logger):
    if logger.hasHandlers():
        for handler in logger.handlers:
            try:
                handler.close()
            except Exception as err:
                print(f"Error closing handler: {err}")
        logger.handlers.clear()

And update the init method to call close_handlers:

if not multiprocess:
    # create file handler
    fh = logging.FileHandler(file_path)
    fh.setLevel(level)
    fhf = logging.Formatter(
        '%(levelname)s|%(asctime)s.%(msecs)03d|%(module)s|%(funcName)s'
        '|%(lineno)s|%(message)s', '%Y-%m-%dT%H:%M:%S'
    )
    fh.setFormatter(fhf)
    self.close_handlers(logger)  # Ensure handlers are closed before adding new ones
    logger.addHandler(fh)
    # create console handler
    if stream_handler:
        ch = logging.StreamHandler()
        ch.setLevel(level)
        if time:
            chf = logging.Formatter(
                '%(levelname)s[%(asctime)s.%(msecs)03d] '
                '%(module)s.%(funcName)s[%(lineno)s] %(message)s',
                '%Y-%m-%dT%H:%M:%S'
            )
        else:
            chf = logging.Formatter(
                '%(levelname)s %(module)s.%(funcName)s[%(lineno)s] '
                '%(message)s'
            )
        ch.setFormatter(chf)
        logger.addHandler(ch)
    logger.propagate = False
    self.log = logger
    self.file_path = file_path
    self.stream = None
    self.level = level
else:
    # create stream handler
    stream = io.StringIO()
    ch = logging.StreamHandler(stream)
    ch.setLevel(level)
    if time:
        chf = logging.Formatter(
            '%(levelname)s_p{}|%(asctime)s.%(msecs)03d|%(module)s'
            '|%(funcName)s|%(lineno)s|%(message)s'.format(
                multiprocess
            ), '%Y-%m-%dT%H:%M:%S'
        )
    else:
        chf = logging.Formatter(
            '%(levelname)s_p{}|%(module)s|%(funcName)s|%(lineno)s'
            '|%(message)s'.format(multiprocess)
        )
    ch.setFormatter(chf)
    self.close_handlers(logger)  # Ensure handlers are closed before adding new ones
    logger.addHandler(ch)
    logger.propagate = False
    self.log = logger
    self.stream = stream
    self.file_path = None
    self.level = level

Additional Notes
The added close_handlers method ensures that all file handlers are properly closed, preventing the ResourceWarning: unclosed file issue.

Hello! Would it be possible to provide remotior_sensus newest version on OSGeo4W? The latest version available on it is the 0.2.1-1. I use it to manage the QGIS installations in my lab. It would be nice to update it directly from OSGeo4W.

Hi Luca, this is a great effort from your part. I am trying to use the rs.band_classification function, I have this workflow so far.

if __name__ == '__main__':
    import remotior_sensus
    
    rs = remotior_sensus.Session(n_processes=16, available_ram=256* 16)
    cfg = rs.configurations
    
    # create BandSet
    catalog = rs.bandset_catalog()
    file_list = ['T17MQS_20230824T153621_B02.tif', 'T17MQS_20230824T153621_B03.tif', 'T17MQS_20230824T153621_B04.tif', 'T17MQS_20230824T153621_B05.tif', 'T17MQS_20230824T153621_B06.tif', 'T17MQS_20230824T153621_B08.tif', 'T17MQS_20230824T153621_B11.tif', 'T17MQS_20230824T153621_B12.tif', 'T17MQS_20230824T153621_B8A.tif']
    
    root_directory = 'data/'
    
    catalog.create_bandset(file_list, wavelengths=['Sentinel-2'], root_directory=root_directory)
    # set BandSet in SpectralCatalog
    signature_catalog_2 = rs.spectral_signatures_catalog(bandset=catalog.get(1))
    # import vector
    signature_catalog_2.import_vector(
        file_path=r'C:\Users\chelo\OneDrive\00_RACAR_FABIAN\land_use.gpkg',
        macroclass_field='macroclass', class_field='LandUseType',
        macroclass_name_field='macroclass', class_name_field='LandUseType',
        calculate_signature=True
    )
    temp = cfg.temp.temporary_file_path(
        name='class', name_suffix=cfg.tif_suffix
    )
    rs.band_classification(
        input_bands=catalog.get(1), output_path=temp,
        spectral_signatures=signature_catalog_2,
        signature_raster=False
    )

First I had a error, due to the fact that in the ogr could not read a band.tif file, this is the only situation that this does not work, because gdal and rasterio have no problem whit it. So I modify the function as follows, it worked after that.

# get CRS of a raster or vector
def get_crs(path):
    # try vector
    try:
        l_p = ogr.Open(path)
    except:
        l_p =  None
    # if raster
    if l_p is None:
        l_p = gdal.Open(path, gdal.GA_ReadOnly)
        if l_p is None:
            crs = None
        else:
            try:
                # check projections
                crs = l_p.GetProjection()
                crs = crs.replace(' ', '')
                if len(crs) == 0:
                    crs = None
            except Exception as err:
                crs = None
                cfg.logger.log.error(str(err))
    # if vector
    else:
        layer = l_p.GetLayer()
        # get projection
        crs = get_layer_crs(layer)
    cfg.logger.log.debug('path: %s; crs: %s' % (path, crs))
    return crs

After this part I have this error, in this function 'Gets x y arrays from rois' wich give me this error.

File "C:\Users\chelo\OneDrive\ALCANTARILLADO_PyQt5\00_MODULOS\pypiper\borrar.py", line 2873, in
rs.band_classification(
File "C:\Users\chelo\miniconda3\Lib\site-packages\remotior_sensus\tools\band_classification.py", line 1280, in band_classification
x_y_matrices = _collect_x_y_matrices(
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\chelo\miniconda3\Lib\site-packages\remotior_sensus\tools\band_classification.py", line 1522, in _collect_x_y_matrices
x_y_arrays = _get_x_y_arrays_from_rois(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\chelo\miniconda3\Lib\site-packages\remotior_sensus\tools\band_classification.py", line 1439, in _get_x_y_arrays_from_rois
matrix = np.stack(array_dictionary[s])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<array_function internals>", line 200, in stack
File "C:\Users\chelo\miniconda3\Lib\site-packages\numpy\core\shape_base.py", line 464, in stack
raise ValueError('all input arrays must have the same shape')
ValueError: all input arrays must have the same shape
forrtl: error (200): program aborting due to control-C event
Image PC Routine Line Source
libifcoremd.dll 00007FFA8673DF54 Unknown Unknown Unknown
KERNELBASE.dll 00007FFB048829E3 Unknown Unknown Unknown
KERNEL32.DLL 00007FFB05707344 Unknown Unknown Unknown
ntdll.dll 00007FFB071026B1 Unknown Unknown Unknown

Process finished with exit code 0

I have several warnings as this, but I think that is because of my type of features in the gpkg gile.

band classification [ 1%] [elapsed 1min23sec]:starting the classifier ⚙Warning 1: A geometry of type POLYGON is inserted into layer t20231010_201907039038_8290 of geometry type MULTIPOLYGON, which is not normally allowed by the GeoPackage specification, but the driver will however do it. To create a conformant GeoPackage, if using ogr2ogr, the -nlt option can be used to override the layer geometry type. This warning will no longer be emitted for this combination of layer and feature geometry type.

Glad to see this out, I used the plugin before so this is very useful for me as it let me do whole pipelines adding land use classification

Greetings,
Marcelo.