iudx / iudx-catalogue Goto Github PK

An Open Source implementation of India Urban Data Exchange (IUDX) Catalogue Service and Search APIs using Vert.x, an event driven and non-blocking high performance reactive framework, for identifying assets and data resources in Smart Cities.

Java 2.07% Shell 0.16% HTML 14.69% CSS 41.89% JavaScript 41.19%

iudx-catalogue's People

Contributors

Stargazers

Watchers

Forkers

swaminathanvasanth svaderia arundprabhu dhivinshaji jishnujayakumar code-akki pct960 umeshpacholi arun-babu rbccps-iisc rnaaz ankita-3091 shwetams inetkenya

iudx-catalogue's Issues

Docker log files need to be cleared

After 60+ days of up-time, docker log file reached 4 Gigabytes, which caused the apiserver to not respond for requests.

Omitting attribute filter causes NULL exception

Catalogue search on tags

Search with tags that does not exist returns an empty result.

Eg. https://catalogue.iudx.org.in/cat/search/attribute?tags=flood will return empty array as there are no flood sensors.
The issue is when we search with https://catalogue.iudx.org.in/cat/search/attribute?tags=(pollution,flood), where there are items with pollution tags also returns an empty array.

Update APIs to v1

Creating and deleting an item does not throw an error when the item-type does not exist

IUDX Client Certificate Site not working

The site ca.iudx.org.in does not work, no documentation available on obtaining certificates for testing.

Querying for values which has a space in it

A URL should not have space. Browsers encode spaces into %20. We need to handle the conversion from %20 to space.

POST throws exception

The response is written twice.

java.lang.IllegalStateException: Response has already been written
	at io.vertx.core.http.impl.HttpServerResponseImpl.checkValid(HttpServerResponseImpl.java:544)
	at io.vertx.core.http.impl.HttpServerResponseImpl.end0(HttpServerResponseImpl.java:400)
	at io.vertx.core.http.impl.HttpServerResponseImpl.end(HttpServerResponseImpl.java:339)
	at iudx.catalogue.apiserver.APIServerVerticle.createItems(APIServerVerticle.java:259)
	at io.vertx.ext.web.impl.RouteImpl.handleContext(RouteImpl.java:225)
	at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:120)
	at io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:133)
	at io.vertx.ext.web.handler.impl.BodyHandlerImpl$BHandler.doEnd(BodyHandlerImpl.java:231)
	at io.vertx.ext.web.handler.impl.BodyHandlerImpl$BHandler.end(BodyHandlerImpl.java:211)
	at io.vertx.ext.web.handler.impl.BodyHandlerImpl.lambda$handle$0(BodyHandlerImpl.java:74)
	at io.vertx.core.http.impl.HttpServerRequestImpl.handleEnd(HttpServerRequestImpl.java:417)
	at io.vertx.core.http.impl.Http1xServerConnection.handleEnd(Http1xServerConnection.java:482)
	at io.vertx.core.http.impl.Http1xServerConnection.handleContent(Http1xServerConnection.java:477)
	at io.vertx.core.http.impl.Http1xServerConnection.processMessage(Http1xServerConnection.java:458)
	at io.vertx.core.http.impl.Http1xServerConnection.handleMessage(Http1xServerConnection.java:144)
	at io.vertx.core.http.impl.HttpServerImpl$ServerHandlerWithWebSockets.handleMessage(HttpServerImpl.java:728)
	at io.vertx.core.http.impl.HttpServerImpl$ServerHandlerWithWebSockets.handleMessage(HttpServerImpl.java:619)
	at io.vertx.core.net.impl.VertxHandler.lambda$channelRead$1(VertxHandler.java:146)
	at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:337)
	at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

POST does not work when the skip_validation header is invalid.

Handle the case when skip_validation is an invalid boolean value.

Wrong status code whenever Internal Server Error occur.

The application is showing 400 instead of 500 in case of internal server error.

Build Failure due to digestAsHex

iudx-catalogue/src/main/java/iudx/catalogue/apiserver/APIServerVerticle.java:[279,53] no suitable method found for digestAsHex(java.lang.String[])

Search-attribute throws error when no query is given

Update test-cases to test APIs using REST Assured

Case sensitive and insensitive search on tags

Searching for a particular tag is case sensitive. We get results when searched for tag=Water, but not for tag=water.

request to /cat/items/id/<UUID> is giving error

SEVERE: Failed to handleMessage. address: 1
java.lang.IllegalStateException: Response has already been written
	at io.vertx.core.http.impl.HttpServerResponseImpl.checkValid(HttpServerResponseImpl.java:544)
	at io.vertx.core.http.impl.HttpServerResponseImpl.end0(HttpServerResponseImpl.java:400)
	at io.vertx.core.http.impl.HttpServerResponseImpl.end(HttpServerResponseImpl.java:318)
	at io.vertx.core.http.impl.HttpServerResponseImpl.end(HttpServerResponseImpl.java:307)
	at iudx.catalogue.apiserver.APIServerVerticle.lambda$get_items$7(APIServerVerticle.java:422)
	at io.vertx.core.eventbus.impl.EventBusImpl.lambda$convertHandler$1(EventBusImpl.java:349)
	at io.vertx.core.eventbus.impl.HandlerRegistration.deliver(HandlerRegistration.java:223)
	at io.vertx.core.eventbus.impl.HandlerRegistration.handle(HandlerRegistration.java:200)
	at io.vertx.core.eventbus.impl.EventBusImpl.lambda$deliverToHandler$3(EventBusImpl.java:533)
	at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:339)
	at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

Feb 19, 2019 5:21:18 PM io.vertx.core.impl.ContextImpl
SEVERE: Unhandled exception
java.lang.IllegalStateException: Response has already been written
	at io.vertx.core.http.impl.HttpServerResponseImpl.checkValid(HttpServerResponseImpl.java:544)
	at io.vertx.core.http.impl.HttpServerResponseImpl.end0(HttpServerResponseImpl.java:400)
	at io.vertx.core.http.impl.HttpServerResponseImpl.end(HttpServerResponseImpl.java:318)
	at io.vertx.core.http.impl.HttpServerResponseImpl.end(HttpServerResponseImpl.java:307)
	at iudx.catalogue.apiserver.APIServerVerticle.lambda$get_items$7(APIServerVerticle.java:422)
	at io.vertx.core.eventbus.impl.EventBusImpl.lambda$convertHandler$1(EventBusImpl.java:349)
	at io.vertx.core.eventbus.impl.HandlerRegistration.deliver(HandlerRegistration.java:223)
	at io.vertx.core.eventbus.impl.HandlerRegistration.handle(HandlerRegistration.java:200)
	at io.vertx.core.eventbus.impl.EventBusImpl.lambda$deliverToHandler$3(EventBusImpl.java:533)
	at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:339)
	at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

Location query has to be changed according to the API documentation

Or query is not applicable on location
Its rigid structure has to be changed.

Return a response if an item could not be inserted into the database

The cause of failure has to be mentioned.

re-install script giving an error on Ubuntu 18.04

Command used:
sh re-install

Error:

re-install: 5: [: =: unexpected operator
re-install: 22: [: =: unexpected operator
Invalid argument passed.

Change UUID to id

decodeCertificate raises the following error

java.lang.ArrayIndexOutOfBoundsException: Index 6 out of bounds for length 6
	at iudx.catalogue.apiserver.APIServerVerticle.decodeCertificate(APIServerVerticle.java:238)
	at iudx.catalogue.apiserver.APIServerVerticle.create(APIServerVerticle.java:335)
	at io.vertx.ext.web.impl.RouteImpl.handleContext(RouteImpl.java:225)
	at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:120)
	at io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:133)
	at io.vertx.ext.web.handler.impl.BodyHandlerImpl$BHandler.doEnd(BodyHandlerImpl.java:231)
	at io.vertx.ext.web.handler.impl.BodyHandlerImpl$BHandler.end(BodyHandlerImpl.java:211)
	at io.vertx.ext.web.handler.impl.BodyHandlerImpl.lambda$handle$0(BodyHandlerImpl.java:74)
	at io.vertx.core.http.impl.HttpServerRequestImpl.handleEnd(HttpServerRequestImpl.java:417)
	at io.vertx.core.http.impl.Http1xServerConnection.handleEnd(Http1xServerConnection.java:482)
	at io.vertx.core.http.impl.Http1xServerConnection.handleContent(Http1xServerConnection.java:477)
	at io.vertx.core.http.impl.Http1xServerConnection.processMessage(Http1xServerConnection.java:458)
	at io.vertx.core.http.impl.Http1xServerConnection.handleMessage(Http1xServerConnection.java:144)
	at io.vertx.core.http.impl.HttpServerImpl$ServerHandlerWithWebSockets.handleMessage(HttpServerImpl.java:728)
	at io.vertx.core.http.impl.HttpServerImpl$ServerHandlerWithWebSockets.handleMessage(HttpServerImpl.java:619)
	at io.vertx.core.net.impl.VertxHandler.lambda$channelRead$1(VertxHandler.java:146)
	at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:337)
	at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)

get_item function not working properly

On doing following query
curl -ik -XGET 'https://localhost:8443/cat/search/attribute?owner=rbccps&tags=(etoilet,sanitation)&attributeFilter=(id,latitude,longitude)'
It gives an exception.

SEVERE: Failed to handleMessage. address: database
java.util.NoSuchElementException
	at java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:721)
	at java.util.LinkedHashMap$LinkedEntryIterator.next(LinkedHashMap.java:752)
	at java.util.LinkedHashMap$LinkedEntryIterator.next(LinkedHashMap.java:750)
	at io.vertx.core.json.JsonObject$Iter.next(JsonObject.java:970)
	at io.vertx.core.json.JsonObject$Iter.next(JsonObject.java:955)
	at iudx.catalogue.database.MongoDB.search_attribute(MongoDB.java:60)
	at iudx.catalogue.database.DatabaseVerticle.validateRequest(DatabaseVerticle.java:101)
	at iudx.catalogue.database.DatabaseVerticle.lambda$start$0(DatabaseVerticle.java:40)
	at io.vertx.core.eventbus.impl.HandlerRegistration.deliver(HandlerRegistration.java:223)
	at io.vertx.core.eventbus.impl.HandlerRegistration.handle(HandlerRegistration.java:200)
	at io.vertx.core.eventbus.impl.EventBusImpl.lambda$deliverToHandler$3(EventBusImpl.java:533)
	at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:339)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

Feb 11, 2019 2:38:51 PM io.vertx.core.impl.ContextImpl
SEVERE: Unhandled exception
java.util.NoSuchElementException
	at java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:721)
	at java.util.LinkedHashMap$LinkedEntryIterator.next(LinkedHashMap.java:752)
	at java.util.LinkedHashMap$LinkedEntryIterator.next(LinkedHashMap.java:750)
	at io.vertx.core.json.JsonObject$Iter.next(JsonObject.java:970)
	at io.vertx.core.json.JsonObject$Iter.next(JsonObject.java:955)
	at iudx.catalogue.database.MongoDB.search_attribute(MongoDB.java:60)
	at iudx.catalogue.database.DatabaseVerticle.validateRequest(DatabaseVerticle.java:101)
	at iudx.catalogue.database.DatabaseVerticle.lambda$start$0(DatabaseVerticle.java:40)
	at io.vertx.core.eventbus.impl.HandlerRegistration.deliver(HandlerRegistration.java:223)
	at io.vertx.core.eventbus.impl.HandlerRegistration.handle(HandlerRegistration.java:200)
	at io.vertx.core.eventbus.impl.EventBusImpl.lambda$deliverToHandler$3(EventBusImpl.java:533)
	at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:339)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

Use Basic HTTP authentication instead of custom header

Refer this for more info.

Search_attribute throws an exception if no query and attributeFilter is mentioned

Migrate the response body from TEXT to JSON

Delete Item API responds with 200 OK even when the item does not exist

Search with invalid tag throws an error

SEVERE: Unhandled exception
java.lang.IllegalArgumentException: state should be: writes is not an empty list
	at com.mongodb.assertions.Assertions.isTrueArgument(Assertions.java:99)
	at com.mongodb.operation.MixedBulkWriteOperation.<init>(MixedBulkWriteOperation.java:106)
	at com.mongodb.async.client.MongoCollectionImpl.executeBulkWrite(MongoCollectionImpl.java:490)
	at com.mongodb.async.client.MongoCollectionImpl.bulkWrite(MongoCollectionImpl.java:421)
	at io.vertx.ext.mongo.impl.MongoClientImpl.bulkWriteWithOptions(MongoClientImpl.java:488)
	at io.vertx.ext.mongo.impl.MongoClientImpl.bulkWrite(MongoClientImpl.java:476)
	at iudx.catalogue.database.MongoDB.lambda$updateNoOfHits$3(MongoDB.java:109)
	at io.vertx.ext.mongo.impl.MongoClientImpl.lambda$null$12(MongoClientImpl.java:795)
	at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:339)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

Error on creating new resource-item

Creating new resource-item responds with 400.

Catalogue delete does not remove item in MongoDB

Catalogue delete does not remove item in MongoDB
Status response is 204, which states item is removed

Deleting a valid entry

Delete to cat with UUID is successful but gives response code 500

Issue with adding elements with '$' in key field

OR query is not working in search-attribute when keys=tags

Internal Server Error due to asynchronous read and write operations.

The status of the messageHandler is checked before the mongo operations are executed. Thus, the DatabaseVerticle returns failure message leading to internal server error in APIServerVerticle.

Mongo find is not working

Unable to disable the display of multiple fields. Since we are not displaying two fields: _id and _tags, the search is failing.

Assign a default value to skip_validation

Add 'Content type: Json' in header

Avoid use of hardcoded domains in config.properties file

As I understand it, the config.properties file has a list of currently active catalogue domains. Since the catalogue will eventually be a multi-tenant module, the above design will require all the webservers to be restarted every time a new city comes up or an existing city disassociates from IUDX.

Every item has internal DB ID in response

Doing a GET request with RestAssured or cURL gives the following exception on the server

I don't understand what this exception means. Once this occurs server is not responding at all.
On @ADP01 's laptop doing a GET request is throwing a different exception on the server side but the server responds back. This two might be correlated.

io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f6c6973742f636174616c6f6775652f6974656d2d747970657320485454502f312e310d0a4163636570743a202a2f2a0d0a436f6e74656e742d4c656e6774683a20300d0a486f73743a206c6f63616c686f73743a383434330d0a436f6e6e656374696f6e3a204b6565702d416c6976650d0a557365722d4167656e743a204170616368652d48747470436c69656e742f342e352e3720284a6176612f312e382e305f323031290d0a4163636570742d456e636f64696e673a20677a69702c6465666c6174650d0a0d0a
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f6c6973742f636174616c6f6775652f6974656d2d747970657320485454502f312e310d0a4163636570743a202a2f2a0d0a436f6e74656e742d4c656e6774683a20300d0a486f73743a206c6f63616c686f73743a383434330d0a436f6e6e656374696f6e3a204b6565702d416c6976650d0a557365722d4167656e743a204170616368652d48747470436c69656e742f342e352e3720284a6176612f312e382e305f323031290d0a4163636570742d456e636f64696e673a20677a69702c6465666c6174650d0a0d0a
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1138)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1194)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
	... 16 more

Search query displays deprecated items

In-valid schema search doesn't return any response

Test scenario : Search for a schema 'ID' that doesn't exits.
Request : https://localhost:8443/cat/schemas/id/abc
Response : null

SEVERE: Unhandled exception
java.lang.NullPointerException
        at iudx.catalogue.database.MongoDB.decode_schema(MongoDB.java:152)
        at iudx.catalogue.database.MongoDB.lambda$read_schema$1(MongoDB.java:171)
        at io.vertx.ext.mongo.impl.MongoClientImpl.lambda$null$12(MongoClientImpl.java:795)
        at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:339)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:834)