Use the SPARQL Service Description of a SPARQL µS to automatically generate provenance information

The parameter cache_expires_after in config.ini file of the micro-service does not work: JSON documents are still retrieved from cache even if they should have expired.

Hi,

I installed and configured SPARQL Microservice using provided dockers in a server where proxy is active. In this server, I am able to communicate outside (ofc with proxy enabled), but when I run command:

curl --header "Accept: text/turtle" http://localhost:8083/ld/flickr/photo/31173091246

I am receiving the following message:

"Cannot query the Web API or transform its response to JSON-LD."

I also installed in a clean no-proxy environment both using dockers and from git repo, and it was working fine.

Could you please let me know whether SPARQL Microservice works behind proxy, and what could be the fix?

In addition, if I would like to create a docker image with SPARQL Microservice installed in it, what is the entry command that should run on docker container initialization apart from starting Apache?

Thank you in advance,
Lavdim

hi @frmichel
i just came across your project and noticed how similar it is to
https://github.com/SPARQL-Anything/sparql.anything.

e.g. look how i use SPARQL Anything to write SPARQL queries against the arXiv REST API:
https://github.com/justin2004/weblog/tree/master/dynamic_pagination_with_sparql_anything

i wonder if we want to think about working together?

It should be possible to pass several values for an argument, i.e. for a specific predicate in a client's SPARQL query.

Like tags for photos:

?photo 
  a schema:Photography;
  schema:keywords "tag1", "tag2";
  ...

A simple JSON file can be considered as a basic non-parameterized Web API.

Yet, it is not possible to define a SµS with no parameter. One has to define a fake parameter that will not be used in the Web API query string.

Hello,

I try to run the docker image but I encountered the following issues:

First, I tried to follow the instructions and run docker-compose up -d in the directory the .yml file is located (after cloning this repo) but I got the following error:
build path .../sparql-micro-service/docker/sms-corese either does not exist, is not accessible, or is not a valid URL
Then, since I only want the sparql-micro-service docker, I downloaded this docker using docker pull and ran this command:

docker run -i -t frmichel/sparql-micro-service

Although I the docker instance seemed to be running, I could not access it via localhost:81. I noticed that it was running using port 80 instead of 81. So I added the port mapping to my command:

docker run -i -t -p 81:80 frmichel/sparql-micro-service

After typing localhost:81 on the browser this time I noticed that the port 81 is now used by the docker but I get an access forbidden message. More specifically, I get the following error in the docker logs:

Cannot serve directory /var/www/html/: No matching DirectoryIndex (index.php,index.html) found, and server-generated directory index forbidden by Options directive

Is there something I am missing? Do I have to perform any other configurations?

Any help would be very much appreciated!

Best,

Constantina

Hello,

I have problems using sparql-micro-service with my own REST Service. The scenario is the following:

I have set up a REST service at localhost:8080. One of the API calls lists a number of assets as a json list. Each asset consists of a name, type and id.

Now I wanted to query the REST service using sparql-micro-services. Therefor I:

• created a service assets/assetList containing files config.ini, profile.jsonld and service.php. I edited the config.ini file according to theAPI request

----- file config.ini 8<----------------

; Service custom parameters
custom_parameter[] =

; Web API query string
api_query = "http://service:8080/SDSP-SwitchingService/SDSP/assetList"

; Maximum time (in seconds) to cache responses from the Web API. Default: 2592000 = 30 days
cache_expires_after = 2592000

; Add provenance information to the graph generated at each invocation. Default: false
add_provenance = false

----- end file 8<-------------------

Now I change the docker compose instructions so that the new service is published in the /var/www/html/sparql-ms/services and in the corese image in /sparql-micro-service/services.
In addition I add a new container to the deployment that is named service an that hosts the REST service.

I now start the deployment using docker-compose up and the containers all start with no error.

Now I use the following curl command to test the service:

curl --data-urlencode "query=SELECT * WHERE { ?s ?p ?o }" http://localhost/service/assets/assetList

The response is: Internal error: Cannot query the Web API or transform its response to JSON-LD.

I tried to access the REST service from inside the sparql-micro-service container and get the following result:

root@ba90376da655:/sparql-micro-service# curl http://service:8080/SDSP-SwitchingService/SDSP/assetList
{"assetList":[{"id":"0","name":"Trafo_0","type":"Trafo"},{"id":"1","name":"Trafo_1","type":"Trafo"},{"id":"2","name":"Trafo_2","type":"Trafo"},{"id":"3","name":"Trafo_3","type":"Trafo"},{"id":"4","name":"Trafo_4","type":"Trafo"},{"id":"5","name":"Trafo_5","type":"Trafo"},{"id":"6","name":"Trafo_6","type":"Trafo"},{"id":"7","name":"Trafo_7","type":"Trafo"},{"id":"8","name":"Trafo_8","type":"Trafo"},{"id":"9","name":"Trafo_9","type":"Trafo"}]}

Inside the log file of sms i see the same result but somehow the JSON-LD cannot be created. The error in the log file is:

Syntax error, malformed JSON.

but the json is valid.

Do you have any ideas what is going wrong here?

thanks, Christian

sms-2021-02-24.log

The micro-service does not work if the SPARQL query contains a SERVICE keyword.

The service defined with the following config.ini file does not succeed:

custom_parameter[] =
api_query =  "https://maps.googleapis.com/maps/api/directions/json?destination=Paris&mode=car&origin=Nice&waypoints=optimize:true%7Clyon&key=<your key>"

The log shows this error:

Stack trace:
#0 /var/www/sparql-ms/vendor/ml/json-ld/FileGetContentsLoader.php(143): ML\JsonLD\Processor::parse()
#1 /var/www/sparql-ms/vendor/ml/json-ld/JsonLD.php(142): ML\JsonLD\FileGetContentsLoader->loadDocument()
#2 /var/www/sparql-ms/vendor/ml/json-ld/JsonLD.php(411): ML\JsonLD\JsonLD::expand()
#3 /var/www/sparql-ms/src/common/Utils.php(224): ML\JsonLD\JsonLD::toRdf()
#4 /var/www/sparql-ms/src/sparqlms/service.php(244): frmichel\sparqlms\common\Utils::translateJsonToNQuads()
#5 /var/www/sparql-ms/src/sparqlms/service.php(171): frmichel\sparqlms\queryWebAPIAndGenerateTriples()
#6 {main}
2024-01-20 23:18:36 ["uuepvo98gla0j"] service ERROR: Exception: Cannot query the Web API or transform its response to JSON-LD. in /var/www/sparql-ms/src/common/Utils.php:237
Stack trace:
#0 /var/www/sparql-ms/src/sparqlms/service.php(244): frmichel\sparqlms\common\Utils::translateJsonToNQuads()
#1 /var/www/sparql-ms/src/sparqlms/service.php(171): frmichel\sparqlms\queryWebAPIAndGenerateTriples()
#2 {main}

Hi Frank,

I have a problem testing the docker based deployment as described in the docker directory (deployment section of the repository). Starting up the deployment using docker-compose up works fine.

I had to make two changes in the docker-compose.yml file which are:

• set the corese image to wimmics/corese since this has fixed the log4j2 issue
• set the port mapping to port 8080:8080 for corese

When I now execute the command from the readme file (curl --header "Accept: application/sparql-results+json" "http://localhost/service/musicbrainz/getSongByName?query=${SELECT}&name=Love") I get the following error:

sparql-micro-service_1 | [Mon Jan 31 12:14:37.528072 2022] [php7:error] [pid 20] [client 127.0.0.1:55406] PHP Fatal error: Uncaught Error: Call to a member function error() on null in /var/www/html/sparql-ms/src/sparqlms/service.php:204\nStack trace:\n#0 {main}\n thrown in /var/www/html/sparql-ms/src/sparqlms/service.php on line 204
sparql-micro-service_1 | 172.19.0.1 - - [31/Jan/2022:12:14:37 +0000] "GET /service/musicbrainz/getSongByName?query=select%20*%20where%20%7B%3Fs%20%3Fp%20%3Fo%7D&name=Love HTTP/1.1" 500 378 "-" "curl/7.68.0"
sparql-micro-service_1 | 127.0.0.1 - - [31/Jan/2022:12:14:37 +0000] "GET /sparql-ms/src/sparqlms/service.php?root_url=http://localhost/service&querymode=sparql&service=musicbrainz/getSongByName&query=select%20*%20where%20%7B%3Fs%20%3Fp%20%3Fo%7D&name=Love HTTP/1.1" 500 378 "-" "curl/7.68.0"
^CGracefully stopping... (press Ctrl+C again to force)

Any idea what is going wrong here?

thanks, Christian

Hi,

I am trying to run the following query for combining data from a Jena Fuseki Endpoint with data from a SPARQL Endpoint created on top of a JSON Web Service, e.g.:

http://api.worldweatheronline.com/premium/v1/past-weather.ashx?key=[appKEY]&q=Berlin&format=json&date=25-11-2019&enddate=25-11-2019&tp=1

PREFIX ex:    <https://example.org/> 
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX rdf:  <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
 
 SELECT * WHERE  {
             ?subj a ex:City;
                      rdfs:label ?cityName.
 
 SERVICE <http://endpoint1:3030/sparql-ms/worldweather/getWeatherByCity_sd/> {		
        SELECT * WHERE {  
                                           
            ?s ?p ?o;
                schema:name ?cityName;
                schema:startDate "2019-11-14";
                schema:endDate "2019-11-14";
                schema:hasReportHeader ?headerReport.
       }
 }}

Basically, the variable ?cityName is bound with a number of cities, therefore the query should be executed in an iterative way, i.e. for each value of the ?cityName variable there should be initiated a call to http://endpoint1:3030/worldweather/getWeatherByCity_sd/.

I have seen a possibility provided in the script manual_config_example/service.php to perform specific actions, but I am not sure how to make this in the best way.

Best,
Lavdim

The installation procedure documents the Apache rewriting rules for SPARQL querying and URI dereferencing, but not for the HTML doc generation.

Hi,

could you please improve the documentation, in particular when creating a new SPARQL Endpoint, it should be obvious that the content defined in a Service Description file should be also added as a NAMEDGRAPH (the best with the same name as in the folder structure) in the Triple Store manually or by re-running the initial configuration file. Current, I am adding using the following SPARQL Insert query template:

PREFIXES

INSERT DATA {

GRAPH <http://localhost/sparql-ms/[ServiceName]/ServiceDescription> {

     Inner Content (the same what you would define in a ServiceDescription file)

}

Hi Frank,

When I create my sparql-ms deployment I am able to test the available services from localhost according to your documentation (https://github.com/frmichel/sparql-micro-service/blob/master/doc/04-install.md#test-sparql-querying) but when I try to query from a distant host I get a 404 URL not found as a response. To test from remote I simply replaced the "localhost" in the example test by the target host.

Looking into kgram.log file I find the following for the remote query:

10.0.7.10 - - [28/Jul/2022:15:33:38 +0000] "GET /service/siguard-pdp/getAllChannels?query=select%20*%20where%20%7B%3Fs%20%3Fp%20%3Fo%7D HTTP/1.1" 404 456 "-" "curl/7.58.0"

And for the localhost based query:

[Thu Jul 28 15:34:27.796260 2022] [php7:notice] [pid 17] [client 127.0.0.1:47892] PHP Notice:  Undefined index: custom_parameter in /var/www/html/sparql-ms/src/common/Configuration.php on line 81
[Thu Jul 28 15:34:27.796320 2022] [php7:warn] [pid 17] [client 127.0.0.1:47892] PHP Warning:  sizeof(): Parameter must be an array or an object that implements Countable in /var/www/html/sparql-ms/src/common/Configuration.php on line 81
[Thu Jul 28 15:34:27.814082 2022] [php7:warn] [pid 17] [client 127.0.0.1:47892] PHP Warning:  Invalid argument supplied for foreach() in /var/www/html/sparql-ms/src/common/Utils.php on line 315
127.0.0.1 - - [28/Jul/2022:15:34:27 +0000] "GET /sparql-ms/src/sparqlms/service.php?root_url=http://localhost/service&querymode=sparql&service=siguard-pdp/getAllChannels&query=select%20*%20where%20%7B%3Fs%20%3Fp%20%3Fo%7D HTTP/1.1" 200 9795 "-" "curl/7.58.0"
172.19.0.1 - - [28/Jul/2022:15:34:27 +0000] "GET /service/siguard-pdp/getAllChannels?query=select%20*%20where%20%7B%3Fs%20%3Fp%20%3Fo%7D HTTP/1.1" 200 9783 "-" "curl/7.58.0"

Do I need to send my query to a different URL to get things working?

Any help will be appreciated,

thanks, Christian

This issue was initially discussed in issue #12.

When a SPARQL micro-service receives multiple values for one argument, then they are passed to the Web API in a single invocation as a comma-separated list of values.

This does not fit with some use cases where the Web API should be invoked multiple times with one value each time.

For a SPARQL µservice configured with a config.ini (parameter passing as arguments on the http query string), multiple values should be passed as comma-separated:

flickr/getPhotosByTags?tags=sky,blue&query=...

This does not work: only one value is received "sky,value".

Some Web APIs do pause a hard limit to the number of results they can return.
As a result, getting all results is carried out through a paging mechanism, requiring multiple calls to the Web API.

Could it be possible to configure a SPARQL micro-service to say that paging is required, in a generic way?

'http_header' config param is implemented in the config.ini configuration mode.
See EoL example.
Need for the same possibility in the Service Description graph.

Using the syntax {param} to re-inject parameters of the SPARQL µs into the generated graph.
Service configured with config.ini.

2 parameters are declared:

custom_parameter[] = param1
custom_parameter[] = param2

Re-injecting param1 works fine.
But re-injecting param2 replaces it with the concatenation of the values of param1 and param2, e.g.:
{param2} will be replaced with "{value1}", "{value2}"

File construct.sparql is optional, yet if it is not present the service will not generate any triple.

I have implemented several sparql microservices which work fine in case I access them directly using curl: e.g.

adminuser@dev-sag-jumphost:~/bin$ curl -vvvv  --header "Content-Type:application/x-www-form-urlencoded" --data-urlencode 'query=prefix sim:<http://ontology.siemens.com/si/measurement/> SELECT * WHERE { ?measurement a sim:Measurement ; sim:time_stamp "22.05.10T10:00:00.000" ; ?p ?o. } ' http://dev-sag-sms.ensure2.siemens.cloud/service/siguard-pdp-sd/getAllChannelsByTimestamp
*   Trying 10.0.7.181...
* TCP_NODELAY set
* Connected to dev-sag-sms.ensure2.siemens.cloud (10.0.7.181) port 80 (#0)
> POST /service/siguard-pdp-sd/getAllChannelsByTimestamp HTTP/1.1
> Host: dev-sag-sms.ensure2.siemens.cloud
> User-Agent: curl/7.58.0
> Accept: */*
> Content-Type:application/x-www-form-urlencoded
> Content-Length: 241
>
* upload completely sent off: 241 out of 241 bytes
< HTTP/1.1 200 OK
< Date: Wed, 24 Aug 2022 08:21:30 GMT
< Server: Apache/2.4.52 (Debian)
< X-Powered-By: PHP/7.4.27
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Access-Control-Allow-Origin: *
< Content-Type: application/sparql-results+xml
< Set-Cookie: PHPSESSID=46snsnrqnfs42g6pbqhn0sn8s5; path=/
< Transfer-Encoding: chunked
<
<?xml version="1.0" ?>
<sparql xmlns='http://www.w3.org/2005/sparql-results#'>
<head>
<variable name='measurement'/>
<variable name='p'/>
<variable name='o'/>
</head>
<results>
<result>
<binding name='measurement'><uri>http://ontology.siemens.com/si/measurement/meas_1012_22.05.10T10:00:00.000</uri></binding>
<binding name='p'><uri>http://ontology.siemens.com/si/measurement/has_angle</uri></binding>
<binding name='o'><literal datatype='http://www.w3.org/2001/XMLSchema#double'>0.00929626</literal></binding>
</result>
...

But when I use federated queries to a blazegraph sparql endpoint using a service clause the query fails with the error:

Response=Erroneous request: SPARQL query with HTTP POST method but unexpected 'Content-Type': application/x-www-form-urlencoded; charset=UTF-8

It seems that the blazegraph adds the charset information to the call to service but there seem to be no way to stop this. I can verify the behavior if I add the charset info to my direct query:

adminuser@dev-sag-jumphost:~/bin$ curl -vvvv  --header "Content-Type:application/x-www-form-urlencoded;charset=utf-8" --data-urlencode 'query=prefix sim:<http://ontology.siemens.com/si/measurement/> SELECT * WHERE { ?measurement a sim:Measurement ; sim:time_stamp "22.05.10T10:00:00.000" ; ?p ?o. } ' http://dev-sag-sms.ensure2.siemens.cloud/service/siguard-pdp-sd/getAllChannelsByTimestamp                                                                                       *   Trying 10.0.7.181...
* TCP_NODELAY set
* Connected to dev-sag-sms.ensure2.siemens.cloud (10.0.7.181) port 80 (#0)
> POST /service/siguard-pdp-sd/getAllChannelsByTimestamp HTTP/1.1
> Host: dev-sag-sms.ensure2.siemens.cloud
> User-Agent: curl/7.58.0
> Accept: */*
> Content-Type:application/x-www-form-urlencoded;charset=utf-8
> Content-Length: 241
>
* upload completely sent off: 241 out of 241 bytes
< HTTP/1.1 400 Bad Request
< Date: Wed, 24 Aug 2022 08:51:43 GMT
< Server: Apache/2.4.52 (Debian)
< X-Powered-By: PHP/7.4.27
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Access-Control-Allow-Origin: *
< Content-Length: 133
< Content-Type: text/html; charset=UTF-8
< Set-Cookie: PHPSESSID=o8nm7ek0uobl9tl5ma5rkpdb96; path=/
< Connection: close
<
Erroneous request: SPARQL query with HTTP POST method but unexpected 'Content-Type': application/x-www-form-urlencoded;charset=utf-8

So I have the question whether there is a way to configure the microservices in a way that they accept content-type information with charset info attached.

I added two logfiles to this issue where I added tha query command in the top.

1. sms-2022-08-24.log from a successfully operated direct service call.
2. sms-2022-08-24.service.log from a failed call using federated queries.
   sms-2022-08-24.log
   sms-2022-08-24.service.log

thanks, Christian