Problem testing docker deployment about sparql-micro-service HOT 25 CLOSED

Hi Christian,

I've seen this already, most likely this is due to the logs directory being read-only for Docker.
Can you please set the 777 rights and retry? (https://github.com/frmichel/sparql-micro-service/tree/master/deployment/docker#check-application-logs).

from sparql-micro-service.

Hi Frank,

at least this changed the behavior. Now I run into trouble because the new corese image does not expose port 8081 but port 8080. I need to use this image because it fixes cve 2021 44228 (log4j2 problem). So need to investigate how to change either the listening port for corese or the sparql endpoint configuration for sparql ms.

thanks a lot for the quick response,

Christian

from sparql-micro-service.

the new corese image does not expose port 8081 but port 8080.
I need to use this image because it fixes cve 2021 44228 (log4j2 problem). So need to investigate how to change either the listening port for corese or the sparql endpoint configuration for sparql ms.

Ok, indeed Corese starts on port 8081, which is not modifiable. And the SPARQL µservice code expects it to be listening on that port.
But in fact you can simply remove the port mapping 8081:8081 from the docker-compose.yml, because Corese does not need to be accessible from outside (I just figured that out lately.

services:
  corese:
    image: frmichel/corese
    networks:
      - sms-net
    volumes:
      - "./logs:/usr/local/corese/log"
      - "./services:/services"
  mongo:
    ...

Retry like that and let me know.
I will update this at a later release, and I also plan to make all ports more easily changeable.

from sparql-micro-service.

Hi Frank

could you share your dockerfile that you use to create the image? In this case I could create my own image configuring the ports and test.

thanks, Christian

from sparql-micro-service.

Hi Christian,

The two Dockerfile's are in deployment/docker/corese and deployment/docker/sparql-micro-service.

Note that I made a more recent Docker of Corese here: https://hub.docker.com/r/wimmics/corese
You may have a look at it to mount the folder of Corese config file. In time, I intend to rely on this one to deploy the SPARQL micro-services.

Franck.

from sparql-micro-service.

Hi Frank

Thanks again for the answer. I wrote one yesterday but I missed to send it in the end. It was about the listening port of corese. Here it is for completnes:

I logged into the running container and checked which port was listening and got the following:

root@0944e94c1e88:/usr/local/corese# netstat -an | grep LISTEN
tcp        0      0 127.0.0.11:43305        0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN     
root@0944e94c1e88:/usr/local/corese#

In the sms.log file I find the error:

2022-01-31 16:42:29 ["t42u6gqm40q2o"] service ERROR: EasyRdf\Exception: Unable to connect to corese:8081 (Connection refused) in /var/www/html/sparql-ms/vendor/easyrdf/easyrdf/lib/Http/Client.php:440
Stack trace:
#0 /var/www/html/sparql-ms/vendor/easyrdf/easyrdf/lib/Sparql/Client.php(372): EasyRdf\Http\Client->request()
#1 /var/www/html/sparql-ms/vendor/easyrdf/easyrdf/lib/Sparql/Client.php(244): EasyRdf\Sparql\Client->executeQuery()
#2 /var/www/html/sparql-ms/vendor/easyrdf/easyrdf/lib/Sparql/Client.php(190): EasyRdf\Sparql\Client->request()
#3 /var/www/html/sparql-ms/src/sparqlms/service.php(247): EasyRdf\Sparql\Client->update()
#4 /var/www/html/sparql-ms/src/sparqlms/service.php(167): frmichel\sparqlms\queryWebAPIAndGenerateTriples()
#5 {main}

So I suppose that the corese image I use exposes port 8080. The same can be found at https://hub.docker.com/r/wimmics/corese where the run examples are 👍

docker run --name my-corese \
    -p 8080:8080 \
    -d wimmics/corese

best, Christian

from sparql-micro-service.

The image at wimmics/corese runs Corese on port 8080.

Whereas the image frmichel/corese runs Corese on port 8081. The latter is the one I initially built for the SPARQL micro-services, and this is the one that is started in the docker-compose file that I provide. And the image sparql-micro-service expects Corese to be on port 8081, not 8080. This is in file config.ini.

So you should be able to run the whole thing with this docker-compose where I removed the port mapping for Corese:

services:
  corese:
    image: frmichel/corese
    networks:
      - sms-net
    volumes:
      - "./logs:/usr/local/corese/log"
      - "./services:/services"
  mongo:
    image: mongo:3.6-jessie
    networks:
      - sms-net
  sparql-micro-service:
    image: frmichel/sparql-micro-service
    networks:
      - sms-net
    ports:
      - "80:80"
    volumes:
      - "./logs:/var/www/html/sparql-ms/logs"
      - "./services:/services"
networks:
  sms-net:
    driver: bridge

from sparql-micro-service.

Additional: I do not advise to run the SPARQL micro-service with with Corese image wimmics/corese. First because I did not test it. And also besause some features are configured in the frmichel/corese which are not in wimmics/corese : in particular the sparql-to-spin service. But this is needed only if you pass arguments in the graph pattern (as opposed to URL parameters).
If you want to try it though, you have two options:

• start Corese on port 8081 instead of 8080
• or update the config.ini file of the SPARQL micro-services image to query Corese on port 8080:

sparql_endpoint = http://localhost:8080/sparql
spin_endpoint = http://localhost:8080/service/sparql-to-spin

from sparql-micro-service.

Hi Frank,

sorry for bothering you with my problems but I continue having trouble running the docker deployment. As I said I need to use a more recent version of corese than provided by your frmichel/corese docker image. The reason is that I must not use software containing log4j older than version 2.16. This is the case in corese 4.3.0. Now I did the following:

I created a new corese docker image deploying corese-server-4.3.0.jar instead of 4.1.6d. Now I can connect to this image from the sparql-ms image but the query result cannot be parsed by the sparql-ms. The error I get is:

root@668717d683cf:/usr/local/corese/log# cat sms-2022-02-01.log 
2022-02-01 12:38:08 ["88b3l4kg0v925"] Context NOTICE: --------- Starting SPARQL micro-service --------
2022-02-01 12:38:08 ["88b3l4kg0v925"] service NOTICE: Query parameter (html special chars encoded) 'service': musicbrainz/getSongByName
2022-02-01 12:38:08 ["88b3l4kg0v925"] service NOTICE: Query parameter (html special chars encoded) 'querymode': sparql
2022-02-01 12:38:08 ["88b3l4kg0v925"] Utils NOTICE: Query HTTP header "Content-Type" undefined.
2022-02-01 12:38:08 ["88b3l4kg0v925"] Utils NOTICE: Query HTTP header "Accept": application/sparql-results+json
2022-02-01 12:38:08 ["88b3l4kg0v925"] service NOTICE: Client SPARQL query (html special chars encoded):
select * where {?s ?p ?o}
2022-02-01 12:38:08 ["88b3l4kg0v925"] service NOTICE: Will query the Web API with arguments: Array
(
    [name] => Love
)

2022-02-01 12:38:08 ["88b3l4kg0v925"] service NOTICE: Web API query string: http://beta.musicbrainz.org/ws/2/work/?fmt=json&query=name:Love
2022-02-01 12:38:10 ["88b3l4kg0v925"] service ERROR: EasyRdf\Exception: Failed to parse SPARQL JSON Query Results format in /var/www/html/sparql-ms/vendor/easyrdf/easyrdf/lib/Sparql/Result.php:369
Stack trace:
#0 /var/www/html/sparql-ms/vendor/easyrdf/easyrdf/lib/Sparql/Result.php(75): EasyRdf\Sparql\Result->parseJson()
#1 /var/www/html/sparql-ms/vendor/easyrdf/easyrdf/lib/Sparql/Client.php(388): EasyRdf\Sparql\Result->__construct()
#2 /var/www/html/sparql-ms/vendor/easyrdf/easyrdf/lib/Sparql/Client.php(255): EasyRdf\Sparql\Client->parseResponseToQuery()
#3 /var/www/html/sparql-ms/vendor/easyrdf/easyrdf/lib/Sparql/Client.php(190): EasyRdf\Sparql\Client->request()
#4 /var/www/html/sparql-ms/src/sparqlms/service.php(247): EasyRdf\Sparql\Client->update()
#5 /var/www/html/sparql-ms/src/sparqlms/service.php(167): frmichel\sparqlms\queryWebAPIAndGenerateTriples()
#6 {main}

2022-02-01 12:38:10 ["88b3l4kg0v925"] service NOTICE: Returning HTTP status 500.
[beta.musicbrainz.org.pretty.json.zip](https://github.com/frmichel/sparql-micro-service/files/7978316/beta.musicbrainz.org.pretty.json.zip)

2022-02-01 12:38:10 ["88b3l4kg0v925"] service NOTICE: --------- Done - SPARQL µS execution --------

The json result from the service seems to be correct when I call the URL (http://beta.musicbrainz.org/ws/2/work/?fmt=json&query=name:Love) from the log file manually in firefox. I can verify this using jq command. I attached the resulting json to this comment. Could you please have a look into this?

In case that I use your original setup everything works fine. But as I said the log4j2.11 used in this corese is outdated and I am not allowed to use this any more.

thanks, Christian
beta.musicbrainz.org.pretty.json.zip

from sparql-micro-service.

Ok this is unexpected indeed. You can get information from the log of Corese. Can you check file logs/kgram_server.log?

from sparql-micro-service.

To help you debug, you can also mount the SPARQL µs config.ini so that you can change the log level.
In the docker compose, set the volumes as follows:

    volumes:
      - "./logs:/var/www/html/sparql-ms/logs"
      - "./services:/services"
      - type: bind
        source: ./config/config.ini
        target: /var/www/html/sparql-ms/src/sparqlms/config.ini

Then copy this config.ini to the config directory and restart Docker.

;--- Log level. One of: DEBUG, INFO, NOTICE, WARNING, ERROR, CRITICAL, ALERT. Default: NOTICE
log_level = INFO

;--- Version of the code currently deployed (corresponds to the git tag)
version = 0.5.2

;--- URL at which SPARQL micro-services are deployed (mandatory). May be overridden by query string parameter with same name.
;    This parameter is mandatory when the service is configured with a ServiceDescription.ttl file
root_url = http://example.org/sparql-ms

;--- Set of paths where the SPARQL micro-services are deployed. Relative path is wrt. to src/sparqlms.
;    Multiple paths may be defined to deploy different sets of SPARQL micro-services.
services_paths[] = /services


; --- Write-enabled SPARQL endpoint to store the temporary graphs (mandatory)
sparql_endpoint = http://corese:8081/sparql

; --- SPARQL-to-SPIN translation service (optional)
spin_endpoint = http://corese:8081/service/sparql-to-spin


; --- Default mime type returned by the SPARQL service used in response as header Content-Type
;     in case no Accept header is provided in the query (mandatory)
default_mime_type = application/sparql-results+json,application/sparql-results+xml


; --- Default name spaces. These are automatially added to any SPARQL query (optional)
;     See other defaults name spaces in https://github.com/njh/easyrdf/blob/master/lib/RdfNamespace.php
namespace[dce]      = http://purl.org/dc/elements/1.1/
namespace[dct]      = http://purl.org/dc/terms/
namespace[dwc]      = http://rs.tdwg.org/dwc/terms/
namespace[dwciri]   = http://rs.tdwg.org/dwc/iri/
namespace[taxrefp]  = http://taxref.mnhn.fr/lod/property/
namespace[xt]       = http://ns.inria.fr/sparql-extension/
namespace[api]      = http://ns.inria.fr/sparql-micro-service/api#
namespace[sms]      = http://ns.inria.fr/sparql-micro-service#
namespace[ro]       = http://purl.obolibrary.org/obo/


; --- Input parameters of any SPARQL micro-service ---

; service: path to the actual service to call formatted as '<api name>/<service>' (mandatory)
parameter[] = service

; querymode: either 'sparql' in case of a SPARQL query or 'ld' for dereferencing a URI (mandatory)
parameter[] = querymode

; root_url: URL at which SPARQL micro-services are deployed (optional). Overrides parameter with same name above.
;           Example: http://sparql-micro-services.org/service
parameter[] = root_url


; --- Cache configuration ---

; Cache db enpoint: MongoDB connection string. Default: mongodb://localhost:27017
; See https://docs.mongodb.com/manual/reference/connection-string/
cache_endpoint = mongodb://mongo:27017

; Wether to use the cache db or not (optional). Default: false
use_cache = true

; Name of the MongoDB database used as cache (optional). Default: sparql_micro_service
cache_db_name = sparql_micro_service


; --- HTTP proxy configuration ---
;proxy.host =
;proxy.port =
;proxy.user =
;proxy.password =

from sparql-micro-service.

Hi Frank,

I will try out the things you suggested above. Here is the log file you requested.

best Christian
kgram_server.zip

from sparql-micro-service.

Thx Christian, I can't see any error in Corese's log. That means that from Corese point of view everything is ok. Try to get more logs from the SPARQL micro-service to figure out what's wrong.

from sparql-micro-service.

Hi Frank,

I need some more time for experiments but I suppose the error is related to the corese image. Testing with your setup frmichel/corese:latest and frmichel/sparql-micros-services:latest works with no problem.
Testing with my self created image for corese using corese 4.3.0 returns the error I wrote about. The only change I did in the corese Dockerfile is that I replaced the jar file name and in the next step renamed it to the old one since this name is hard coded in some startup file. Neverthless both sms- log files seem to be identical until line 1350 (_:b316 <)

_:b316 <http://ns.inria.fr/sparql-micro-service/api#id> "82b101c9-4eb7-469e-8609-9480ecbc8130" .
_:b316 <http://ns.inria.fr/sparql-micro-service/api#title> "Good Name" .

2022-02-01 18:10:23 ["3v5058oj2s7gd"] service INFO: Creating temporary graph: <http://localhost/service/api-graph-61f9697f217302.96610024>
2022-02-01 18:10:23 ["3v5058oj2s7gd"] service NOTICE: Executing SPARQL CONSTRUCT query from file: /services/musicbrainz/getSongByName/construct.sparql
2022-02-01 18:10:23 ["3v5058oj2s7gd"] service INFO: CONSTRUCT query:
CONSTRUCT {
    ?work schema:sameAs ?page;
        schema:name ?title;
    .
} WHERE {
    [] api:works ?work .
    ?work api:id ?id;
        api:title ?title;
        
        api:type ?type;
        api:score ?score;
    .

    FILTER (LCASE(?type) = "song" && xsd:integer(?score) > 90)

    BIND(IRI(concat("https://musicbrainz.org/work/", ?id)) as ?page)
}
2022-02-01 18:10:23 ["3v5058oj2s7gd"] service INFO: Adding result of the CONSTRUCT query into graph: <http://localhost/service/resp-graph-61f9697f212892.65404557>
2022-02-01 18:10:23 ["3v5058oj2s7gd"] service DEBUG: Creating temporary graph: <http://localhost/service/resp-graph-61f9697f212892.65404557> with INSERT DATA query:
@prefix schema: <http://schema.org/> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix sd: <http://www.w3.org/ns/sparql-service-description#> .
@prefix ma: <http://www.w3.org/ns/ma-ont#> .
@prefix api: <http://ns.inria.fr/sparql-micro-service/api#> .

INSERT DATA { 
  GRAPH <http://localhost/service/resp-graph-61f9697f212892.65404557> {
     
    
    _:b835 schema:name "Lemondrop" ;
    schema:sameAs <https://musicbrainz.org/work/978a59b7-ea1d-4577-9e6d-8470dbfe442b> .

In case of your image the resulting graph is constructed with no error. In case of the 4.3.0 image the following is returned here:

_:b316 <http://ns.inria.fr/sparql-micro-service/api#id> "82b101c9-4eb7-469e-8609-9480ecbc8130" .
_:b316 <http://ns.inria.fr/sparql-micro-service/api#title> "Good Name" .

2022-02-01 16:50:03 ["0kn167vnkd68b"] service INFO: Creating temporary graph: <http://localhost/service/api-graph-61f956ab890655.90405611>
2022-02-01 16:50:04 ["0kn167vnkd68b"] service ERROR: EasyRdf\Exception: Failed to parse SPARQL JSON Query Results format in /var/www/html/sparql-ms/vendor/easyrdf/easyrdf/lib/Sparql/Result.php:369
Stack trace:
#0 /var/www/html/sparql-ms/vendor/easyrdf/easyrdf/lib/Sparql/Result.php(75): EasyRdf\Sparql\Result->parseJson()
#1 /var/www/html/sparql-ms/vendor/easyrdf/easyrdf/lib/Sparql/Client.php(388): EasyRdf\Sparql\Result->__construct()
#2 /var/www/html/sparql-ms/vendor/easyrdf/easyrdf/lib/Sparql/Client.php(255): EasyRdf\Sparql\Client->parseResponseToQuery()
#3 /var/www/html/sparql-ms/vendor/easyrdf/easyrdf/lib/Sparql/Client.php(190): EasyRdf\Sparql\Client->request()
#4 /var/www/html/sparql-ms/src/sparqlms/service.php(247): EasyRdf\Sparql\Client->update()
#5 /var/www/html/sparql-ms/src/sparqlms/service.php(167): frmichel\sparqlms\queryWebAPIAndGenerateTriples()
#6 {main}

2022-02-01 16:50:04 ["0kn167vnkd68b"] service NOTICE: Returning HTTP status 500.

2022-02-01 16:50:04 ["0kn167vnkd68b"] service NOTICE: --------- Done - SPARQL µS execution --------

So I probably need to find a new qay to create a corese image containing log4j2.17

best, Christian

from sparql-micro-service.

Or the other option is that there is something wrong with Corese 4.3.0, which would be surprising but still. I'm saying this because the EasyRdf library fails to parse to JSON SPARQL result that comes from Corese. Or the EasyRdf lib does not the JSON that Corese returns, which would be surprising too.

I have on my todo list to do this update to Corese 4.3.0. Not in the coming days though.
If that is ok with you, you could start with the existing Corese image, despite the log4j version, until I provide a functional updated version?

from sparql-micro-service.

Hi Frank,

that sounds reasonable. In fact I am not able to track this down any further by my own. I did some tests with different versions of corese (4.3.0, 4.2.4, 4.2.3, 4.1.6, 4.1.5) and found that the only working image I can create from your Dockerfile in ~/deployment/docker/corese is with version 4.1.6 / 4.1.6d. The images with higher versions return either the parse error I reported before or in case 4.2.3 a null pointer exception generated by corese server.
For version 4.1.5 I get an empty response although it seems that result data is transferred correctly from corese to the sparql-micro-services.

My only question is whether you can tell me by when you plan to have the migration done since we plan to use sparql microservices in a project where we need it by end of this month in a public accessible deployment. If you need any further tests or logs please tell me and I will try to support you.

best, Christian

from sparql-micro-service.

Ok I'll try to work on this next week, then I'll be on vacation so I cannot guarantee that this will be done by end of February.
However, you can run everything without exposing Corese port 8081: Corese is invoked by the SPARQL µs container from the internal Docker network. This is safer, even though I understand your concern with log4j.
I'll keep you updated.

from sparql-micro-service.

Hi Frank,

Thanks for the response and the information that you plan to work on this the next week. Unfortunately not exposing the port is not solution for me. In the cloud environment where I have to work the docker images are scanned for this vulnerabilities.

best, Christian

from sparql-micro-service.

Hi Frank,

I am looking forward to your notifications regarding this topic. Meanwhile I created an interim jar file from corese-server-4.1.6d.jar and copied the newest log4j classes to this file replacing older log4j implementation. This works for me so far. So my time constraints can be resolved by using this image. As I wrote before, if you need further tests or logs please contact me.

best regards an thanks for your help,
Christian

from sparql-micro-service.

Ok awesome, at least that will give more time to fix the Corese issues. I'll keep you posted.

from sparql-micro-service.

Hi @kleegrewec, I've published a new branch for you: https://github.com/frmichel/sparql-micro-service/tree/v052_upgrade_corese/deployment/docker/build

Quite some changes:

• upgrade to Corese 4.3.1 that fixes the pb you encountered (conflict with the EasyRDF library)
• update root images for Corese and SPARQL µservice images, and upgrade mongodb
• add parameters in the docker-compose.yml in build: CORESEJAR and SMS_VERSION. This avoids changes in multiple files.
• mount config/sparql-micro-service.ini to the main .ini file of the SPARQL µservices, easier if you want to change the log level

Please follow the README, in particular you need to download the environment.zip to get the config/sparql-micro-service.ini before running Docker.

Limitation: I've detected another issue that makes it impossible to use SPARQL µservices configured with a service description graph. But if you use SPARQL µservices configured with a simple config.ini, that will work.

I've not published the new images to Docker hub, I let you rebuilt them. I'll republish them only when the last issue is fixed.

Franck.

from sparql-micro-service.

from sparql-micro-service.

Hi @frmichel Thanks a lot for your help. I found some time to build the images and to do some basic tests and everything workes as expected.

Thanks a lot again, Christian

from sparql-micro-service.

Hi @kleegrewec

We fixed the last issue I was mentioning. So now everything should work fine with the last images I've published:
frmichel/corese4sms:v4.3.1 and frmichel/sparql-micro-service:v0.5.3.

Get the updated docker-compose.yml and explanations here: https://github.com/frmichel/sparql-micro-service/tree/master/deployment/docker.

Franck.

from sparql-micro-service.

Hi @frmichel
Thanks again for the help and the notification. I have tested with my self created images and I was able to integrate a first example service. I will test the new images as soon as possible,

best, Christian

from sparql-micro-service.