Cannot find .laads file within Docker container about force HOT 28 CLOSED

Hi hailihu,

well, this is unexpected. I wasn't aware that there isn't a user in a Docker container..

For downloading Sentinel-2 data (force-level1-sentinel2) I am using a similar mechanism (there needs to be a .scihub file in the home directory). I guess this won't work either when running in Docker..

Do you have a suggestion for a more generic solution? Placing such files in the home directory is actually a very common practice..

Best,
David

from force.

Hi David,

The simplest solution would be to, instead of exiting when there is no user, to proceed with /app/.laads file instead of /home/user/.laads. It is pretty common to have /app as workdir in docker containers.

Perhaps there is a way to set a user in a docker container such that getlogin_r() works, but I couldn't find it.

Cheers,
Haili

from force.

Hi Haili,

I guess there are several issues intermingled here.

First, is there really no user within Docker? Or is this just a deficiency of getlogin_r() ? What about force-level1-sentinel2 - this is a bash script and it expects a file $HOME/.scihub. Doesn't this work either?

Secondly. /app is a directory that is commonly not used on a regular Unix installation. However, in principle. I would not rule out to use a "if not here, then look there" method.

But in general, I think the credentials (both for ESA and LAADS) should not simply be copied into the container. They are private secrets. If the docker image is used by two persons (if that is how docker works), both should use their own credentials. I am not sure how to properly do this with Docker though (I am not a user). There seems to be a way to set up secrets. Any help here is appreciated.

@fegyi001 , do you have an opinion on this?

Cheers,
David

from force.

Hi David,

Thanks for your quick reply!

First, is there really no user within Docker? Or is this just a deficiency of getlogin_r()

Well both actually, there is normally no user within Docker, you could set one, but that doesn't get picked up by getlogin_r()

What about force-level1-sentinel2 - this is a bash script and it expects a file $HOME/.scihub. Doesn't this work either?

I haven't used that, but I assume that has the same problem.

But in general, I think the credentials (both for ESA and LAADS) should not simply be copied into the container

I am not copying the credentials in the container, but have my python application, that runs in docker, put the app-key in the file so that force-lut-modis can find it.

I am not sure if my workaround is the best method, so certainly it would be good to ask others.

Since I have your attention, could you have a look at this pull request: #23? That might actually be a more urgent issue. Thanks.

Cheers,
Haili

from force.

Yes, sorry, merged, and thanks for that fix!
Cheers,
David

from force.

Hey @davidfrantz, hey @hailihu, Docker implementor here.

It is true that the Docker image does not contain any credentials, and this is very deliberate. However, by using volume mounts you can add external files (such as .schihub credentials) anywhere on your Docker image since you run the container as root. Please note that you only have the root user.

I recommend doing it the following way:

docker run fegyi001/force -v /path/to/your/folder/where/credentials/can/be/found:/root/credentials force

In this way a folder from your machine will be mounted under /root/credentials or anywhere else.

Another approach is create another Dockerfile which starts from the FORCE Docker image and you can add any Docker logic you desire.

FROM fegyi001/force:latest

# Define where you wish to copy your credentials file later
ENV CREDENTIALS_DIR /root/credentials
# Create the desired folder if not exists
RUN mkdir -p $CREDENTIALS_DIR
# Go to this folder
WORKDIR $CREDENTIALS_DIR
# Copy the .scihub file here
COPY .scihub .
# Make it read only
RUN chmod 400 ./.scihub
# ...and any more logic you wish

Hope that helps!

from force.

By the way, I just uploaded the Docker images on Docker Hub for version v3.2.1, sorry for the delay.

from force.

Thanks @fegyi001, that's very helpful! However, it still requires some changes in modwvp-ll.c to ensure that force-lut-modis does not exit when it's run inside a docker (because user is unknown according to getlogin_r()) and to look for the credentials elsewhere in that case. For my particular use case, I did just that (updated modwvp-ll.c), compiled the code, and built a new Docker image, that works well.

from force.

Hi both,

thanks for clarifying the situation and recommending a proper way to handle this.

I have now introduced a fix with 8c1c320 in the develop branch. With the next release (likely in the 2nd half of July), this will go into the master branch as well.

The strategy is now like this:
I am looking for the user login. If this doesn't work, I am assuming we are in Docker, and then look in /app.

There, you can mount your local LAADS App Key. Does this make sense to both of you @hailihu @fegyi001 ?

Please let me know if this works (I will be off again for the next week though).

Cheers,
David

from force.

Hi @davidfrantz!

Yes, this should work.
But if this is the approach we follow from now on, before merging into master branch, please also do the same thing in force-level1-sentinel2.sh for .scihub credentials. It still looks only in the /home folder:

if [ ! -r $HOME/.scihub ]; then
  echo "Your ESA credentials must be placed in $HOME/.scihub"
  echo "    First line: User name" 
  echo "    Second line: Password, special characters might be problematic"
  exit
fi

H=$(head -n 2 $HOME/.scihub)
USER=$(echo $H | cut -d ' ' -f 1)
PW=$(echo $H | cut -d ' ' -f 2)
CRED="--user=$USER --password=$PW"
HUB="https://scihub.copernicus.eu/dhus/search"

And I guess the documentation should also be updated (at least in the Docker section).

(Minor stuff, but I found typos in the new code you added to the develop branch in file modwvp-ll.c, lines 53-55: "runnign" should be "running", I prefer writing "docker" capitalized as "Docker", and "retrieve user.." should only have one period as "retrieve user." I did not want to make a pull request just for them.)

from force.

Thanks @fegyi001 ,

I fixed the typos and I will care about the docs and level1-sentinel2.sh soon.

I am going to leave this issue open as a reminder until I fixed all of this.

from force.

Now I was thinking on this solution and I came up with another, more generic way. Perhaps it is not better, I'm not sure.
But the fact that we are adding code to FORCE just because it is used through Docker bothers me a bit, just not elegant. In Docker it is easy to set runtime environment variables, and it would be quite easy to have a predefined env variable that FORCE expects for credential location.

This could be called CREDENTIALS_DIR, and by default this could be /home/$USER, but it could be owervritten by Docker just like this:

docker run --env CREDENTIALS_DIR=/app/credentials fegyi001/force env

In the result you can see that the running Docker container now knows the $CREDENTIALS_DIR environment variable:

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=49c0d64ffae3
CREDENTIALS_DIR=/app/credentials
INSTALL_DIR=/opt/install/src
HOME=/root

So, my more generic idea is: use this $CREDENTIALS_DIR (or call it any other way) everywhere in FORCE where credentials files are used, and make it by default /home/$USER. If anyone uses FORCE with Docker overwriting the default value will be easy.

from force.

Yes, I would agree with @fegyi001 that using an environment variable is a more generic solution. Nice idea.

from force.

Hey,

yes, I would prefer this approach as well. This is a really nice idea. I would probably go with a more specific variable like $FORCE_CREDENTIALS to reduce potential conflicts.

However, I am still thinking on how to solve this practically.

Assuming we have such a variable, overwriting for Docker seems to be easy. The same for non-Docker, each user could set the variable on its own, e.g. in ~/.bashrc or ~/.profile.

The most important thing is how to initially define this variable, and this should work for all users on the system.

In the Makefile, I could probably add something like

ENVFILE=/etc/profile.d/force-credentials.sh

If not installed by an admin, but by a user, or if a different login shell is used, this would need to be changed to this or some other shell-specific file:

ENVFILE=/home/xxx/.profile

Then, the install target will write sth like this into $ENVFILE

FORCE_CREDENTIALS=$HOME

Each user can then decide to overwrite this env variable if necessary.

Do I miss something?

from force.

@davidfrantz I'm not sure if I get it right, but at first this seems a bit complicated to me. Plus, the Docker image does contain a compiled FORCE (that's the main benefit of it), the user will not perform any compiling after defining his/her credentials, therefore I would put nothing in the Makefile. Anyone who downloads & uses the dockerized version already has FORCE compiled and ready for work.

What if FORCE code would simply check whether a $FORCE_CREDENTIALS env variable exists or not whenever it tries to resolve a credentials file location. If there is $FORCE_CREDENTIALS, and it has value, then use it, if not, use the default, hard-coded value (e.g. /home/$USER). This would still be generic, because although there is an if/else statement in the code, nothing is Docker specific.

from force.

Yes, I was just thinking that. You could use getenv() and if it returns NULL, use the default location.

from force.

yes, seems like I overthought this... Sounds like a good idea, and I will implement this soon.

Thanks for this discussion!

from force.

Hi both,

I just pushed a new develop version: 381e61a, which looks for the FORCE_CREDENTIALS environment variable, and if not present defaults back to the HOME directory. Can you test if this works for you?

The changes are in force-lut-modis and force-level1-sentinel2.

Cheers,
David

from force.

from force.

OK no problem,
have a nice vacation!

from force.

Hi David,

I ran your dev branch in a docker and it works fine, thanks! I only tested force-lut-modis, because I don't use force-level1-sentinel2, the code looks fine to me though.

Cheers,
Haili

from force.

Thanks Haili!

I am glad that we could solve this, and I really like this solution!

Cheers,
David

from force.

I tried to compile FORCE with Docker on develop branch, but it fails. A /develop folder seems to be necessary.

cp temp-bin/* /develop
cp: target '/develop' is not a directory
make: *** [install_] Error 1
Makefile:360: recipe for target 'install_' failed
The command '/bin/sh -c make -j7   && make install   && make clean' returned a non-zero code: 2

In the Makefile I see BINDIR=/develop but only on develop branch. If I create the /develop folder before compiling there is no error, but I cannot run force commands:

"exec: \"force\": executable file not found in $PATH": unknown.

If I go to the /develop folder I can run ./force from that, and it works, but this is not system-wide in this case. I'm sure you have your reasons to put the executables in a different folder (other than /usr/local/bin) for the develop branch, but I don't really understand why. @davidfrantz could you please clarify this a bit to me?

from force.

Hi @fegyi001 , welcome back!

Let me explain why this is: in our lab, we are using co-existing installations of the master and develop branch. We do this to have both a stable version that is only updated periodically, and a frequently changing version holding the latest features. The latest features often come with new parameters, thus old parameterfiles need to be updated.

We install the master to /usr/local/bin, where it can be found using the $PATH variable, and thus, executed like this: force.
We install the develop branch to the /develop directory (which is not a standard directory, but can be accessed from all users), which is deliberately not in $PATH. It needs to be executed with absolute path like this: /develop/force.

I guess for running this in Docker (assuming there is no master version installed, too), the simplest solution would be to change the BINDIR to /usr/local/bin.

I hope this explains it a bit ;)

from force.

@davidfrantz thanks for the clarification, I get it now. If you were using Docker in your lab, there would be no need for such workarounds though: you would simply create multiple Docker images with different names, e.g. my-force_master, my-force_develop etc. which would hold different snapshots of FORCE, one for the master branch, one for the develop branch etc. In this way you could test and run each versions separately without interfering each other on the same machine without any issues since Docker containers run isolated.

I see some risk in keeping different Makefiles for different branches (merging develop into master for example which is quite common in most Git workflows, but in this case would cause major issues), but I'm sure you have your own custom workflow for avoiding such problems.

from force.

@davidfrantz I tested the develop branch, and it seems to work. I mounted two volumes inside the Docker container from my host machine, one for the FORCE datacube (mounted as /data), one for the credentials (mounted as /credentials). I built the Docker image under the name my-force.

Here is the command:

docker run \
    -v c:\Users\padanyi-gulyasg\PROJECTS\github\force\force_data:/data \ 
    -v c:\Users\padanyi-gulyasg\PROJECTS\github\force\force_credentials:/credentials \
    --env FORCE_CREDENTIALS=/credentials \
    my-force ./force-level1-sentinel2 /data/Dagobah/S2L1C /data/Dagobah/S2L1C/zambia.txt \
    "25.43/-12.46,25.94/-12.46,25.94/-11.98,25.39/-11.99,25.43/-12.46" 2019-07-01 2019-07-31 0 50 dry

And here are the results for the dry run:

2020-07-22_08:16:23 - Found 13 S2A/B files. Downloading 13 files on this page.
13 Sentinel-2 A/B L1C files available
5.19094 GB data volume available

I also started it without dry run to see whether my .scihub credentials work, and it also looks fine and the Level-1 imagery appears in my file system:

from force.

Hi @fegyi001 ,

not sure if I agree with the previous comment ;) On a native Linux, I would rather consider the Docker solution as the workaround. Don't get me wrong, I really really like it, but I see most value on non-Linux systems. But, I only need to think about accounting for one line in the Makefile every couple of months (when I update the master), whereas running the docker commands require much more typing (each time you use it).

force-level1-sentinel2 /data/Dagobah/S2L1C /data/Dagobah/S2L1C/zambia.txt \
    "25.43/-12.46,25.94/-12.46,25.94/-11.98,25.39/-11.99,25.43/-12.46" 2019-07-01 2019-07-31 0 50 dry

versus

docker run \
    -v c:\Users\padanyi-gulyasg\PROJECTS\github\force\force_data:/data \ 
    -v c:\Users\padanyi-gulyasg\PROJECTS\github\force\force_credentials:/credentials \
    --env FORCE_CREDENTIALS=/credentials \
    my-force ./force-level1-sentinel2 /data/Dagobah/S2L1C /data/Dagobah/S2L1C/zambia.txt \
    "25.43/-12.46,25.94/-12.46,25.94/-11.98,25.39/-11.99,25.43/-12.46" 2019-07-01 2019-07-31 0 50 dry

However, I don't want to start a discussion on this. I guess everybody has an own opinion on this, and it probably also depends on the employed workflows and systems. In any case, it is good to have both options, so users can decide how to use FORCE! :)

from force.

Thanks both @hailihu and @fegyi001 for testing and helping to ressolve this issue!

I will close this now, and plan to merge the develop into master very soon!

Cheers,
David

from force.