Comments (19)
Hi Mneuw,
sorry for running into this. You are doing nothing wrong. Just usual clash of dependencies, especially when you are working with gdal...there are some dependency issues between gdal and nodejs/npm.
Unfortunately I am not having too much time to look into this right now.
Could you check if removing the gdal related stuff in the 8th RUN solves this?
i.e. delete those 2 entries: libgdal-dev python3-gdal
gdal should later then be automatically installed via pip (so no additional entry needed).
Please let me know how that goes. And I hope that solves it.
Cheers
from opensartoolkit.
Finally got it to run:)
needed to modify the dockerfile quite a bit:
- updated to OTB Version 7.2.0 because wget couldn't find the old one? you think this could cause problems?
- removed libgdal-dev but needed python3-gdal
- added a RUN to add nodejs >= v.10 (was required for installation)
- added pip update > maybe this is already newest version and not necessary
- removed last line jupyter nbextension enable --py widgetsnbextension because of an error. are these extensions important?
FROM ubuntu:18.04
LABEL maintainer="Petr Sevcik, EOX"
LABEL OpenSARToolkit='0.10.1'
set work directory to home and download snap
WORKDIR /home/ost
copy the snap installation config file into the container
COPY snap7.varfile $HOME
update variables
ENV OTB_VERSION="7.2.0"
TBX_VERSION="7"
TBX_SUBVERSION="0"
ENV TBX="esa-snap_sentinel_unix_${TBX_VERSION}_${TBX_SUBVERSION}.sh"
SNAP_URL="http://step.esa.int/downloads/${TBX_VERSION}.${TBX_SUBVERSION}/installers"
OTB=OTB-${OTB_VERSION}-Linux64.run
HOME=/home/ost
PATH=$PATH:/home/ost/programs/snap/bin:/home/ost/programs/OTB-${OTB_VERSION}-Linux64/bin
install all dependencies
RUN groupadd -r ost &&
useradd -r -g ost ost &&
apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -yq
python3
python3-pip
git
python3-gdal
libspatialindex-dev
libgfortran3
wget
unzip
imagemagick
npm
curl
RUN apt-get update &&
curl -sL https://deb.nodesource.com/setup_10.x | bash - &&
apt-get -y install nodejs &&
ln -s /usr/bin/nodejs /usr/local/bin/node
RUN alias python=python3 &&
rm -rf /var/lib/apt/lists/* &&
python3 -m pip install -U pip &&
python3 -m pip install -U setuptools &&
python3 -m pip install jupyterlab &&
mkdir /home/ost/programs &&
wget $SNAP_URL/$TBX && \
chmod +x $TBX &&
./$TBX -q -varfile snap7.varfile &&
rm $TBX &&
rm snap7.varfile &&
cd /home/ost/programs &&
wget https://www.orfeo-toolbox.org/packages/${OTB} && \
chmod +x $OTB &&
./${OTB} &&
rm -f OTB-${OTB_VERSION}-Linux64.run
update snap to latest version
RUN /home/ost/programs/snap/bin/snap --nosplash --nogui --modules --update-all 2>&1 | while read -r line; do
echo "$line" &&
[ "$line" = "updates=0" ] && sleep 2 && pkill -TERM -f "snap/jre/bin/java";
done; exit 0
set usable memory to 12G
RUN echo "-Xmx36G" > /home/ost/programs/snap/bin/gpt.vmoptions
get OST and tutorials
RUN python3 -m pip install git+https://github.com/ESA-PhiLab/OpenSarToolkit.git &&
git clone https://github.com/ESA-PhiLab/OST_Notebooks &&
jupyter labextension install @jupyter-widgets/jupyterlab-manager
EXPOSE 8888
CMD jupyter lab --ip='0.0.0.0' --port=8888 --no-browser --allow-root
best
M
from opensartoolkit.
Hi,
responding to the numbers above:
- not at all. Orfeo is needed during mosaicking. So 7.2. is fine. it needs to ship with otbcli_Mosaic function (which it does)
2.ok - ok
- shouldn't do a harm
- ok, you might have visualization issues of the progress bar during download. but the download itself should still work
from opensartoolkit.
Thanks a lot, all clear now for me!
from opensartoolkit.
great, let us know if there are further issues and have a great christmas time.
from opensartoolkit.
ok!! I just tried the S1GRD_Batch-processing and encountered that the grds_to_ards-module doesn't work for the RTC-Gamma0 product type. For some reason it produces only NA-layers. Contrary it runs perfectly for the GTC-Gamma product.
And I was wondering if and how it is possible to change to out_projection for the ard-products e.g. to WGS 1984 UTM Zone 33N?
Anyway it's a great toolkit! wish you a pleasent christmas 2020!!
Best Martin
from opensartoolkit.
hmm, in the docker file there is still the update of snap . uncomment following lines and build again. There are some issues with the latest version of snap that I still need to fix (when I have time)
RUN /home/ost/programs/snap/bin/snap --nosplash --nogui --modules --update-all 2>&1 | while read -r line; do
echo "$line" &&
[ "$line" = "updates=0" ] && sleep 2 && pkill -TERM -f "snap/jre/bin/java";
done; exit 0
from opensartoolkit.
then try to use this notebook as template:
https://github.com/ESA-PhiLab/OST_Notebooks/blob/master/4c%20-%20Sentinel-1%20GRD%20Batch%20-%20Timescan.ipynb
you need to change aoi, start,end and project_dir of course
for me this works.
Check out also the other notebooks. Should not take you more than a day to go through, and will hopefully help to get the concept. Most trickiest parts are the refine inventory and the ard parameters. the rest is just code execution
happy christmas to you, too!
from opensartoolkit.
and yes, you can set the out projection by using the EPSG code. so Lat/Lon is 4326, UTM 33N is 32633
e.g. s1.ard_parameters['single_ARD']['dem']['out_projection'] = '32633'
from opensartoolkit.
Now after getting back to SNAP version 7 I'm running into the following error while executing grds_to_ards in your above shared notebook:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-26-b95cd587dd24> in <module>
3 timeseries=True,
4 timescan=True,
----> 5 overwrite=False
6 )
/usr/local/lib/python3.6/dist-packages/ost/Project.py in grds_to_ards(self, inventory_df, timeseries, timescan, mosaic, overwrite, max_workers, executor_type)
848 # time-series part
849 if timeseries or timescan:
--> 850 grd_batch.ards_to_timeseries(inventory_df, self.config_file)
851
852 if timescan:
/usr/local/lib/python3.6/dist-packages/ost/s1/grd_batch.py in ards_to_timeseries(inventory_df, config_file)
146
147 # create all extents
--> 148 _create_extents(inventory_df, config_file)
149
150 # update extents in case of ls_mask
/usr/local/lib/python3.6/dist-packages/ost/s1/grd_batch.py in _create_extents(inventory_df, config_file)
186 fargs=([str(config_file), ])
187 ):
--> 188 track, list_of_scenes, extent = task.result()
189 out_dict['track'].append(track)
190 out_dict['list_of_scenes'].append(list_of_scenes)
/usr/local/lib/python3.6/dist-packages/godale/_concurrent.py in result(self)
154 """Return func result or raise Exception."""
155 if self._exception:
--> 156 raise self._exception
157 else:
158 return self._result
AttributeError: 'DataFrame' object has no attribute 'crs'
I changed the crs of the inventory_pandas_dataframe o the same as the output projection with no effect.First results (ard products in snap-format) appear in the processing directory but timeseries and timescan products fails.
may you have an idea where this could come from? steps before while querring the S1-datasets I already encounter some crs-warnings:
` INFO (10:00:32): Created project directory at /home/ost/shared
INFO (10:00:32): Downloaded data will be stored in: /home/ost/shared/download.
INFO (10:00:32): Inventory files will be stored in: /home/ost/shared/inventory.
INFO (10:00:32): Processed data will be stored in: /home/ost/shared/processing.
INFO (10:00:32): Using /home/ost/shared/temp as directory for temporary files.
/usr/local/lib/python3.6/dist-packages/pyproj/crs/crs.py:53: FutureWarning: '+init=:' syntax is deprecated. ':
' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6
return _prepare_from_string(" ".join(pjargs))
/usr/local/lib/python3.6/dist-packages/pyproj/crs/crs.py:294: FutureWarning: '+init=:' syntax is deprecated. ':
' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6
projstring = _prepare_from_string(" ".join((projstring, projkwargs)))
/usr/local/lib/python3.6/dist-packages/geopandas/geodataframe.py:422: RuntimeWarning: Sequential read of iterator was interrupted. Resetting iterator. This can negatively impact the performance.
for feature in features_lst:
If you do not have a Copernicus Scihub user account go to: https://scihub.copernicus.eu and register
Your Copernicus Scihub Username: madduen
Your Copernicus Scihub Password: ·········
/usr/local/lib/python3.6/dist-packages/pyproj/crs/crs.py:53: FutureWarning: '+init=:' syntax is deprecated. ':' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6
return _prepare_from_string(" ".join(pjargs))
INFO (10:00:42): Writing inventory data to geopackage file: /home/ost/shared/inventory/full.inventory.gpkg
INFO (10:00:42): Coverage analysis for ASCENDING tracks in VV VH polarisation.
INFO (10:00:42): 5 frames for ASCENDING tracks in VV VH polarisation.
INFO (10:00:42): 5 frames remain after double entry removal
INFO (10:00:42): Excluding track 146
INFO (10:00:42): 2 frames remain after non-AOI overlap
INFO (10:00:42): All remaining tracks fully overlap the AOI. Not removing anything.
INFO (10:00:42): 2 frames remain after removal of non-full AOI crossing
/usr/local/lib/python3.6/dist-packages/pyproj/crs/crs.py:53: FutureWarning: '+init=:' syntax is deprecated. ':' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6
return _prepare_from_string(" ".join(pjargs))
/usr/local/lib/python3.6/dist-packages/ost/s1/refine_inventory.py:71: UserWarning: CRS mismatch between the CRS of left geometries and the CRS of right geometries.
Use to_crs()
to reproject one of the input geometries to match the CRS of the other.
Left CRS: EPSG:4326
Right CRS: +init=epsg:4326 +no_defs +type=crs
inventory_df, aoi_gdf, how='left', op='intersects'
INFO (10:00:43): Found 2 full coverage mosaics.
INFO (10:00:43): Coverage analysis for DESCENDING tracks in VV VH polarisation.
INFO (10:00:43): 6 frames for DESCENDING tracks in VV VH polarisation.
INFO (10:00:43): 6 frames remain after double entry removal
INFO (10:00:43): Excluding track 124
INFO (10:00:43): Excluding track 22
INFO (10:00:43): 3 frames remain after non-AOI overlap
INFO (10:00:43): All remaining tracks fully overlap the AOI. Not removing anything.
INFO (10:00:43): 3 frames remain after removal of non-full AOI crossing
/usr/local/lib/python3.6/dist-packages/ost/s1/refine_inventory.py:71: UserWarning: CRS mismatch between the CRS of left geometries and the CRS of right geometries.
Use to_crs()
to reproject one of the input geometries to match the CRS of the other.
Left CRS: EPSG:4326
Right CRS: +init=epsg:4326 +no_defs +type=crs
inventory_df, aoi_gdf, how='left', op='intersects'
INFO (10:00:43): Found 3 full coverage mosaics.`
from opensartoolkit.
Hi,
it is a problem that we do not fix the version of certain python packages. There have been huge changes to the pyproj lib. Almost all of the geolibs in python use this for projection related stuff, and the syntax changed. You could downgrade to Proj 4 somehow. But it will take me some time to fix all this, as at the moment I do not really have time for it.
Best,
Andreas
from opensartoolkit.
ok thanks for your answer! just let me know when there is an update. meanwhile I'll try on my own start.
Best, M
from opensartoolkit.
Hm, all steps are working now except the creation of the image bounds for the time-series processing. So I think we do not need huge changes. What does this actually? align the scences to same extent? and which script/function is responsible for the job?may you have another hint where I can start digging. sorry for interrupting.
Best,
Martin
from opensartoolkit.
Hi, this step is needed to calculate the common minimum bound of all images in the time-series, as their extent may vary (slightly).
I think it uses geopandas for that, which is based on pyproj, which again had those huge changes. So I assume something related to this. Here you can find an example of this problem (as many have this) https://gis.stackexchange.com/questions/348997/constant-future-warnings-with-new-pyproj
from opensartoolkit.
from opensartoolkit.
Yes, the apihub URL changed https://scihub.copernicus.eu/news/News00868
I still did not have the time do update the toolkit. I am working on a release with the new SNAP 8 and will include this.
from opensartoolkit.
from opensartoolkit.
Hi Martin,
so there is a solution I think without having to change the code.
You can set the base url in the search function of your class instance.
Let's say you initialized your Sentinel1Batch to s1 then:
s1.search(base_url='https://scihub.copernicus.eu/dhus/')
However, setting the base_url for DOWNLOAD from scihub is NOT possible. But you can use ASF for download, if you haven't done so yet. You can download 10 files concurrently and it keeps all of the archive, as opposed to scihub where you only have the last year as it is a rolling archive. It is much faster.
Hope that helps,
Andreas
from opensartoolkit.
from opensartoolkit.
Related Issues (20)
- apply linter on the lib
- publish on Conda ? HOT 2
- complete the community check list ? HOT 1
- remove wiki fro github HOT 1
- thank contributor using an Author file
- use comitizen to bump versions HOT 1
- stop importing gdal HOT 2
- reduce the number of dependencies
- docstring improvement
- create_ard fails with subset area HOT 2
- update readme HOT 3
- iso3 country code not working HOT 8
- Prettier class repr
- Autodoc skipping first argument. HOT 1
- "OutputComplex" deprecated
- Download size with MB HOT 9
- Problem creating ARD GeoTiff, GDAL TypeError "['to_tif'] = True" HOT 2
- Problem on ASF Login
- Bug when trying to download images using ONDA DIAS provider in the OST Tutorial I
- Unable to find gpt.exe when SNAP is installed for the current user only
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from opensartoolkit.