atlasoflivingaustralia / ala-hub Goto Github PK

View Code? Open in Web Editor NEW

2.0 11.0 20.0 5.78 MB

ALA version of the biocache-hubs webapp

Home Page: https://biocache.ala.org.au

Groovy 24.59% CSS 62.18% JavaScript 1.07% Shell 8.06% Batchfile 4.10%

ala-product-biocache biocache-hub ala-systems

ala-hub's People

Contributors

Stargazers

Watchers

ala-hub's Issues

Reduce the default set of facets

The current default list is too large and leads to timeouts / poor query performance

fq=undefined used in a server side query from ala-hub to biocache-service

There may be a grails bug in ala-hub that caused a query from prod-bhub-b5-1 to prod-bws-b5-2 to contain fq=undefined, causing an exception in the log file in prod-bws-b5-2.

This is the sequence on prod-bws-b5-2, including the query that caused the exception, which indicates it originated from ala-hub itself, not directly from a browser to biocache-service as many queries are:

==> /var/log/tomcat7/biocache-service.log <==
2018-08-24 09:09:11,554 [http-bio-8080-exec-7330] DEBUG au.org.ala.biocache.util.SearchUtils  (SearchUtils.java:351) - requestParams = q=lsid:http://id.biodiversity.org.au/node/apni/2916755&fq=undefined&start=0&pageSize=20&sort=first_loaded_date&dir=desc&qc=&facets=taxon_name&facets=family&facets=common_name&facets=species_group&facets=state&facets=sensitive&facets=month&facets=occurrence_decade_i&facets=basis_of_record&facets=multimedia&facets=occurrence_status&facets=institution_uid&facets=data_resource_uid&flimit=10&facet=true
2018-08-24 09:09:11,554 [http-bio-8080-exec-7330] DEBUG au.org.ala.biocache.web.OccurrenceController  (OccurrenceController.java:796) - occurrence search params = q=lsid:http://id.biodiversity.org.au/node/apni/2916755&fq=undefined&start=0&pageSize=20&sort=first_loaded_date&dir=desc&qc=&facets=taxon_name&facets=family&facets=common_name&facets=species_group&facets=state&facets=sensitive&facets=month&facets=occurrence_decade_i&facets=basis_of_record&facets=multimedia&facets=occurrence_status&facets=institution_uid&facets=data_resource_uid&flimit=10&facet=true extra params = {}
2018-08-24 09:09:11,554 [http-bio-8080-exec-7330] DEBUG au.org.ala.biocache.util.QueryFormatUtils  (QueryFormatUtils.java:272) - term query: lsid:http
2018-08-24 09:09:11,554 [http-bio-8080-exec-7330] DEBUG au.org.ala.biocache.util.QueryFormatUtils  (QueryFormatUtils.java:273) - groups: lsid|http
2018-08-24 09:09:11,554 [http-bio-8080-exec-7330] DEBUG au.org.ala.biocache.util.QueryFormatUtils  (QueryFormatUtils.java:486) - pre-processing lsid:http://id.biodiversity.org.au/node/apni/2916755
2018-08-24 09:09:11,555 [http-bio-8080-exec-7330] DEBUG au.org.ala.biocache.util.QueryFormatUtils  (QueryFormatUtils.java:499) - lsid = http://id.biodiversity.org.au/node/apni/2916755
2018-08-24 09:09:11,555 [http-bio-8080-exec-7330] INFO au.org.ala.biocache.service.SpeciesLookupRestService  (SpeciesLookupRestService.java:396) - Requesting: https://bie.ala.org.au/ws/guid/batch?q=undefined
2018-08-24 09:09:11,674 [http-bio-8080-exec-7330] DEBUG au.org.ala.biocache.dao.SearchDAOImpl  (SearchDAOImpl.java:2587) - Solr query: qt=standard&facet=true&facet.field=taxon_name&facet.field=family&facet.field=common_name&facet.field=species_group&facet.field=state&facet.field=sensitive&facet.field=month&facet.field=occurrence_decade_i&facet.field=basis_of_record&facet.field=multimedia&facet.field=occurrence_status&facet.field=institution_uid&facet.field=data_resource_uid&f.taxon_name.facet.sort=index&f.family.facet.sort=index&f.common_name.facet.sort=index&f.species_group.facet.sort=index&f.state.facet.sort=index&f.sensitive.facet.sort=count&f.month.facet.sort=index&f.occurrence_decade_i.facet.sort=index&f.basis_of_record.facet.sort=index&f.multimedia.facet.sort=count&f.occurrence_status.facet.sort=index&f.institution_uid.facet.sort=count&f.data_resource_uid.facet.sort=count&facet.mincount=1&facet.limit=10&facet.sort=count&rows=20&start=0&facet.range.other=before&facet.range.other=after&q=lft:[576655+TO+576655]&fq=text:undefined&facet.missing=true&sort=first_loaded_date+desc
2018-08-24 09:09:11,674 [http-bio-8080-exec-7330] DEBUG au.org.ala.biocache.dao.SearchDAOImpl  (SearchDAOImpl.java:3859) - SOLR query:qt=standard&facet=true&facet.field=taxon_name&facet.field=family&facet.field=common_name&facet.field=species_group&facet.field=state&facet.field=sensitive&facet.field=month&facet.field=occurrence_decade_i&facet.field=basis_of_record&facet.field=multimedia&facet.field=occurrence_status&facet.field=institution_uid&facet.field=data_resource_uid&f.taxon_name.facet.sort=index&f.family.facet.sort=index&f.common_name.facet.sort=index&f.species_group.facet.sort=index&f.state.facet.sort=index&f.sensitive.facet.sort=count&f.month.facet.sort=index&f.occurrence_decade_i.facet.sort=index&f.basis_of_record.facet.sort=index&f.multimedia.facet.sort=count&f.occurrence_status.facet.sort=index&f.institution_uid.facet.sort=count&f.data_resource_uid.facet.sort=count&facet.mincount=1&facet.limit=10&facet.sort=count&rows=20&start=0&facet.range.other=before&facet.range.other=after&q=lft:[576655+TO+576655]&fq=text:undefined&facet.missing=true&sort=first_loaded_date+desc
2018-08-24 09:09:11,682 [http-bio-8080-exec-7330] DEBUG au.org.ala.biocache.dao.SearchDAOImpl  (SearchDAOImpl.java:2591) - qtime:6
2018-08-24 09:09:11,682 [http-bio-8080-exec-7330] DEBUG au.org.ala.biocache.dao.SearchDAOImpl  (SearchDAOImpl.java:2595) - Matched records: 0
2018-08-24 09:09:11,683 [http-bio-8080-exec-7330] DEBUG au.org.ala.biocache.dao.SearchDAOImpl  (SearchDAOImpl.java:2669) - sortField post-split: first_loaded_date|desc
2018-08-24 09:09:11,683 [http-bio-8080-exec-7330] INFO au.org.ala.biocache.dao.SearchDAOImpl  (SearchDAOImpl.java:692) - spatial search query: lft:[576655 TO 576655]

==> /var/log/tomcat7/catalina.out <==
org.springframework.http.converter.HttpMessageNotWritableException: Could not write JSON: Null key for a Map not allowed in JSON (use a converting NullKeySerializer?) (through reference chain: au.org.ala.biocache.dto.SearchResultDTO["activeFacetMap"]->java.util.HashMap["null"]); nested exception is com.fasterxml.jackson.databind.JsonMappingException: Null key for a Map not allowed in JSON (use a converting NullKeySerializer?) (through reference chain: au.org.ala.biocache.dto.SearchResultDTO["activeFacetMap"]->java.util.HashMap["null"])
	at org.springframework.http.converter.json.MappingJackson2HttpMessageConverter.writeInternal(MappingJackson2HttpMessageConverter.java:207)
	at org.springframework.http.converter.AbstractHttpMessageConverter.write(AbstractHttpMessageConverter.java:180)
	at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:168)
	at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:113)
	at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.handleReturnValue(RequestResponseBodyMethodProcessor.java:188)
	at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:69)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:122)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:743)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:672)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:82)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:933)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:951)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:842)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:827)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at au.org.ala.biocache.web.SimpleCORSFilter.doFilter(SimpleCORSFilter.java:33)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at au.org.ala.biocache.web.LoggerConfigFilter.doFilter(LoggerConfigFilter.java:26)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129)
	at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:106)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:221)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Null key for a Map not allowed in JSON (use a converting NullKeySerializer?) (through reference chain: au.org.ala.biocache.dto.SearchResultDTO["activeFacetMap"]->java.util.HashMap["null"])
	at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:284)
	at com.fasterxml.jackson.databind.SerializerProvider.mappingException(SerializerProvider.java:1251)
	at com.fasterxml.jackson.databind.SerializerProvider.reportMappingProblem(SerializerProvider.java:1145)
	at com.fasterxml.jackson.databind.ser.impl.FailingSerializer.serialize(FailingSerializer.java:35)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeOptionalFields(MapSerializer.java:777)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:635)
	at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)
	at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:727)
	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:719)
	at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
	at com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2655)
	at org.springframework.http.converter.json.MappingJackson2HttpMessageConverter.writeInternal(MappingJackson2HttpMessageConverter.java:204)
	... 50 more

==> /var/log/nginx/access.log <==
172.30.0.162 - - [24/Aug/2018:09:09:11 +1000] "GET /ws/occurrences/search?q=lsid%3Ahttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2916755&fq=undefined&start=0&pageSize=20&sort=first_loaded_date&dir=desc&qc=&facets=taxon_name%2Cfamily%2Ccommon_name%2Cspecies_group%2Cstate%2Csensitive%2Cmonth%2Coccurrence_decade_i%2Cbasis_of_record%2Cmultimedia%2Coccurrence_status%2Cinstitution_uid%2Cdata_resource_uid&flimit=10 HTTP/1.1" 500 520 "-" "Java/1.8.0_181" "52.64.45.52"

Remove obsolete fields in facets of all hubs

There are a number of field mappings which have been changed. Those fields in facets may cause failure. All changes should be applied all hubs , e.g biocache-hub, ala-hub

After redeployment, the clients may need to clear related cookies.

Identified:
Remove min_elevation_d_rng

Only about 10 facets showing up in the interface at one time

There seems to be a limit of 10 or 11 facets showing up in the interface at one time, even when the default 33 or less are selected. The list that do show up appear to be consistently ordered, and removing facets that are showing up makes other facets lower down in the ordering show up.

/admin needs to be replaced with a system that can replicate commands across different redundant nodes

The current /admin path is not able to replicate commands redundantly across different instances of ala-hub. The current plan for redundancy for the website is to redundantly deploy ala-hub across different servers and have a load balancer dispatching instructions to any of the servers that are available. This is an issue for /admin as it only sets/resets local state on the server that it is located on.

We need a way of dispatching commands to all of the servers that are currently redundantly handling queries for biocache.ala.org.au, which may include the removal of the /admin path and replacing it with a dedicated host app.

Support api keys for /alaAdmin/ operations so they can be scripted

See previous issue (#14) that was fixed with an nginx configuration change so individual servers respond to direct requests

Default set of facets (33) may be higher than the limit (32)

The default set of facets on biocache-clustered (33) may be higher than the newly imposed limit of 32.

This is based on clicking on "Customise filters" on a search results page and then making no changes and click "Update", which results in the following message:

Grails 4 upgrade

Add custom i18n value for "state"

Override the i18n value for the "state" field in biocache-hubs from State / Province to States & Territories.

See screenshot:

Grails 6

Logging in from a search results page fails

Logging in from a search results page fails with the following error in the logs:

2018-06-22 08:31:50.819 ERROR --- [io-8080-exec-10] o.s.boot.web.support.ErrorPageFilter     : Forwarding to error page from request [/occurrences/search] due to exception [org.jasig.cas.client.validation.TicketValidationException: Ticket 'ST-4760-re-4SDIrbfRySbU1Dcji-ASNpfgauth.ala.org.au' does not match supplied service. The original service was 'https://biocache-clustered.ala.org.au/occurrences/search?taxa=' and the supplied service was 'https://biocache-clustered.ala.org.au/occurrences/search'.]

javax.servlet.ServletException: org.jasig.cas.client.validation.TicketValidationException: Ticket 'ST-4760-re-4SDIrbfRySbU1Dcji-ASNpfgauth.ala.org.au' does not match supplied service. The original service was 'https://biocache-clustered.ala.org.au/occurrences/search?taxa=' and the supplied service was 'https://biocache-clustered.ala.org.au/occurrences/search'.
	at org.jasig.cas.client.validation.AbstractTicketValidationFilter.doFilter(AbstractTicketValidationFilter.java:227)
	at au.org.ala.cas.client.UriFilter.doFilter(UriFilter.java:196)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.jasig.cas.client.authentication.AuthenticationFilter.doFilter(AuthenticationFilter.java:164)
	at au.org.ala.cas.client.UriFilter.doFilter(UriFilter.java:196)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.jasig.cas.client.session.SingleSignOutFilter.doFilter(SingleSignOutFilter.java:97)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.grails.web.filters.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:67)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:117)
	at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:61)
	at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:92)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:110)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:221)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.jasig.cas.client.validation.TicketValidationException: Ticket 'ST-4760-re-4SDIrbfRySbU1Dcji-ASNpfgauth.ala.org.au' does not match supplied service. The original service was 'https://biocache-clustered.ala.org.au/occurrences/search?taxa=' and the supplied service was 'https://biocache-clustered.ala.org.au/occurrences/search'.
	at org.jasig.cas.client.validation.Cas20ServiceTicketValidator.parseResponseFromServer(Cas20ServiceTicketValidator.java:84)
	at org.jasig.cas.client.validation.AbstractUrlBasedTicketValidator.validate(AbstractUrlBasedTicketValidator.java:201)
	at org.jasig.cas.client.validation.AbstractTicketValidationFilter.doFilter(AbstractTicketValidationFilter.java:204)
	... 48 common frames omitted

Not sure where the best place for this issue is as I don't know where the relevant links are created, but the login link should possibly be https://auth.ala.org.au/cas/login?service=https://biocache-clustered.ala.org.au/occurrences/search instead of https://auth.ala.org.au/cas/login?service=https://biocache-clustered.ala.org.au/occurrences/search?taxa%3D

Assertion failed: failed to delete /data/ala-hub/cache/js/eya-images

The following exception is being thrown on biocache-b3:

Feb 01, 2017 10:42:13 PM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [default] in context with path [] threw exception [Filter execution threw an exception] with root cause
Assertion failed: 

assert f.delete()
       | |
       | false
       /data/ala-hub/cache/js/eya-images

	at org.codehaus.groovy.runtime.InvokerHelper.assertFailed(InvokerHelper.java:399)
	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.assertFailed(ScriptBytecodeAdapter.java:655)
	at org.grails.plugin.resource.ResourceProcessor.makeFileForURI(ResourceProcessor.groovy:489)
	at org.grails.plugin.resource.ResourceProcessor$makeFileForURI$11.call(Unknown Source)
	at org.grails.plugin.resource.ResourceMeta.beginPrepare(ResourceMeta.groovy:257)
	at org.grails.plugin.resource.ResourceMeta$beginPrepare$0.call(Unknown Source)
	at org.grails.plugin.resource.ResourceProcessor.prepareResource(ResourceProcessor.groovy:545)
	at sun.reflect.GeneratedMethodAccessor165.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1086)
	at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1110)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:910)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1031)
	at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1110)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:910)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:145)
	at org.grails.plugin.resource.ResourceProcessor$_getResourceMetaForURI_closure11.doCall(ResourceProcessor.groovy:432)
	at sun.reflect.GeneratedMethodAccessor205.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:272)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:64)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
	at org.grails.plugin.resource.util.ResourceMetaStore.getOrCreateAdHocResource(ResourceMetaStore.groovy:98)
	at org.grails.plugin.resource.util.ResourceMetaStore$getOrCreateAdHocResource$3.call(Unknown Source)
	at org.grails.plugin.resource.ResourceProcessor.getResourceMetaForURI(ResourceProcessor.groovy:412)
	at org.grails.plugin.resource.ResourceProcessor.getResourceMetaForURI(ResourceProcessor.groovy)
	at org.grails.plugin.resource.ResourceProcessor$getResourceMetaForURI$30.callCurrent(Unknown Source)
	at org.grails.plugin.resource.ResourceProcessor.processModernResource(ResourceProcessor.groovy:300)
	at org.grails.plugin.resource.ResourceProcessor$processModernResource$28.call(Unknown Source)
	at org.grails.plugin.resource.ProcessingFilter.doFilter(ProcessingFilter.groovy:39)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:106)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:343)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:260)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
	at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:193)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

Transfer of the various messages.properties files takes 2 to 3 seconds of walltime

Search result page loads have a component of time totalling about 2 to 3 seconds where there are sporadic requests for a range of messages.properties/messages.en.properties/messages_en_US.properties files. If this process were streamlined it would increase the usability of the website.

CAS tickets are not accessible on different load balanced nodes

The current setup for CAS tickets seems to be that they are not accessible across different load balanced nodes, resulting in non-terminating HTTP redirect chains while attempting to get tickets.

The stack trace on the load balanced instance of ala-hub that is attempting to handle the query is:

Feb 13, 2017 11:50:51 PM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [default] in context with path [] threw exception [org.jasig.cas.client.validation.TicketValidationException: 
		Ticket 'ST-77972-ypMSQbCicCHZmwiSnBuV-auth.ala.org.au' not recognized
	] with root cause
org.jasig.cas.client.validation.TicketValidationException: 
		Ticket 'ST-77972-ypMSQbCicCHZmwiSnBuV-auth.ala.org.au' not recognized
	
	at org.jasig.cas.client.validation.Cas20ServiceTicketValidator.parseResponseFromServer(Cas20ServiceTicketValidator.java:73)
	at org.jasig.cas.client.validation.AbstractUrlBasedTicketValidator.validate(AbstractUrlBasedTicketValidator.java:197)
	at org.jasig.cas.client.validation.AbstractTicketValidationFilter.doFilter(AbstractTicketValidationFilter.java:164)
	at au.org.ala.cas.client.UriFilter.doFilter(UriFilter.java:197)
	at org.jasig.cas.client.session.SingleSignOutFilter.doFilter(SingleSignOutFilter.java:65)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

This, along with #14 is blocking work to improve the availability of the biocache.ala.org.au website.

Init integration test

Facets are not reliably loaded

Facets are not reliably loaded in some cases when reloading the application. This results in an admin having to manually trigger a cache flush using the admin interface. When the Facets are not loaded, there should be a retry sequence implemented (on a sufficient multi-second or multi-minute delay) to reattempt loading. The Facets are one of the main UI elements that users rely on to browse through ALA data and hence need to be reliably implemented.

The static field pattern used by FacetThemes is tracked separately in: AtlasOfLivingAustralia/biocache-service#137

Change the facet grouping

It would be great to get the new facet grouping in the ALA and OZCAM hubs (see facet_config_ALA.txt file in an email from Alison on 5th March). The changes in the file follow a review by HISCOM and FCIG, and we think they new groupings will make more sense to users.
Thanks.

java.lang.NoSuchMethodError: java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView

It is possible that one of the dependencies of ala-hub were compiled using Java-8, because the following stack trace is indicative of compilation with -source 1.7 -target 1.7 being used with a JDK8. There is a workaround (change type of variable from ConcurrentHashMap to either Map or ConcurrentMap) if anyone is aware of where the change needs to be made or what binary was compiled, as the stack trace doesn't indicate where the failure actually occurred.

java.lang.NoSuchMethodError: java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView;

The standard method of compilation is through travis, which is configured to compile using jdk7 and hence should not generate this error, but it is possible that something was compiled and uploaded to Nexus manually. An alternative to debugging this is to switch from the out of date Java-7 to the maintained Java-8.

Logging when deploying ala-hub as ROOT.war using the ansible scripts in ala-install:

Jan 25, 2017 4:37:36 AM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive /var/lib/tomcat7/webapps-biocache.ala.org.au/ROOT.war
Jan 25, 2017 4:37:36 AM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/var/lib/tomcat7/webapps-biocache.ala.org.au/ROOT/WEB-INF/lib/javax.el-api-2.2.4.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
Jan 25, 2017 4:37:38 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: http://java.sun.com/jsp/jstl/core is already defined
Jan 25, 2017 4:37:38 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: http://java.sun.com/jsp/jstl/fmt is already defined
Jan 25, 2017 4:37:38 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: http://www.springframework.org/tags is already defined
Jan 25, 2017 4:37:38 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: http://www.springframework.org/tags/form is already defined
Jan 25, 2017 4:37:38 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Jan 25, 2017 4:37:38 AM org.apache.catalina.core.ApplicationContext log
INFO: No Spring WebApplicationInitializer types detected on classpath
[ala-hub] Including default configuration file: /data/ala-hub/config/ala-hub-config.properties
[ala-hub] (*) grails.config.locations = [file:/data/ala-hub/config/ala-hub-config.properties]
Jan 25, 2017 4:37:39 AM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
Jan 25, 2017 4:37:39 AM org.apache.catalina.core.StandardContext listenerStart
SEVERE: Exception sending context initialized event to listener instance of class org.codehaus.groovy.grails.web.context.GrailsContextLoaderListener
org.springframework.beans.factory.access.BootstrapException: Error executing bootstraps; nested exception is java.lang.NoSuchMethodError: java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView;
	at org.codehaus.groovy.grails.web.context.GrailsContextLoader.initWebApplicationContext(GrailsContextLoader.java:125)
	at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:112)
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5068)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5584)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
	at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1091)
	at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1980)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoSuchMethodError: java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView;
	... 5 more

Jan 25, 2017 4:37:39 AM org.apache.catalina.core.StandardContext startInternal
SEVERE: One or more listeners failed to start. Full details will be found in the appropriate container log file
Jan 25, 2017 4:37:39 AM org.apache.catalina.core.StandardContext startInternal
SEVERE: Context [] startup failed due to previous errors

Incorrect coordinates in production biocache are not being shown to admins

The correct coordinates for some records that were incorrectly processed by biocache-store are only being shown to logged in (possibly ROLE_ADMIN?) users:

https://biocache.ala.org.au/occurrences/2889ba55-73d8-4262-bbf2-ca4a3e967899

Correct, when logged in:

Incorrect, when logged out:

Logged in users should see the full information instead of having some of it hidden by their privilege.

EDIT: To clarify, the database was/is corrupted, but because I was logged in as an admin it appeared correct to me. I am not currently being shown the corrupted fields when logged in unless I know to look at the "Original versus processed", which shows the "Original" are not corrupted, but the "Processed" are corrupted. Data corruption is a serious issue that should not be hidden from administrators

Missing supportEmail configuration

Support email should be configurable in the error page:

MaxMind GeoIP data file missing

The MaxMind GeoIP data file appears to be missing when installing ala-hub via Ansible:

Jan 30, 2017 4:38:12 AM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive /var/lib/tomcat7/webapps-biocache.ala.org.au/ROOT.war
Jan 30, 2017 4:38:13 AM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/var/lib/tomcat7/webapps-biocache.ala.org.au/ROOT/WEB-INF/lib/javax.el-api-2.2.4.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
Jan 30, 2017 4:38:15 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: http://www.springframework.org/tags is already defined
Jan 30, 2017 4:38:15 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: http://www.springframework.org/tags/form is already defined
Jan 30, 2017 4:38:15 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: http://java.sun.com/jsp/jstl/core is already defined
Jan 30, 2017 4:38:15 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: http://java.sun.com/jsp/jstl/fmt is already defined
Jan 30, 2017 4:38:15 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Jan 30, 2017 4:38:15 AM org.apache.catalina.core.ApplicationContext log
INFO: No Spring WebApplicationInitializer types detected on classpath
[ala-hub] Including default configuration file: /data/ala-hub/config/ala-hub-config.properties
[ala-hub] (*) grails.config.locations = [file:/data/ala-hub/config/ala-hub-config.properties]
Jan 30, 2017 4:38:16 AM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
[ala-hub] Including default configuration file: /data/ala-hub/config/ala-hub-config.properties
[ala-hub] (*) grails.config.locations = [file:/data/ala-hub/config/ala-hub-config.properties]
grails.resources.work.dir = /data/ala-hub/cache

Configuring MaxMind GeoIP ...
... finished configuring MaxMind GeoIP

Jan 30, 2017 4:38:22 AM org.apache.catalina.core.StandardContext listenerStart
SEVERE: Exception sending context initialized event to listener instance of class org.codehaus.groovy.grails.web.context.GrailsContextLoaderListener
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'geoLookupService': Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.maxmind.geoip.LookupService]: Constructor threw exception; nested exception is java.io.FileNotFoundException: /data/ala-hub/config/GeoLiteCity.dat (No such file or directory)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.maxmind.geoip.LookupService]: Constructor threw exception; nested exception is java.io.FileNotFoundException: /data/ala-hub/config/GeoLiteCity.dat (No such file or directory)
	... 5 more
Caused by: java.io.FileNotFoundException: /data/ala-hub/config/GeoLiteCity.dat (No such file or directory)
	at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
	at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
	at com.maxmind.geoip.LookupService.<init>(LookupService.java:282)
	at com.maxmind.geoip.LookupService.<init>(LookupService.java:264)
	... 5 more

Jan 30, 2017 4:38:22 AM org.apache.catalina.core.StandardContext startInternal
SEVERE: One or more listeners failed to start. Full details will be found in the appropriate container log file
Jan 30, 2017 4:38:22 AM org.apache.catalina.core.StandardContext startInternal
SEVERE: Context [] startup failed due to previous errors
Jan 30, 2017 4:38:22 AM org.apache.catalina.core.ApplicationContext log
INFO: Closing Spring root WebApplicationContext
Jan 30, 2017 4:38:22 AM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deployment of web application archive /var/lib/tomcat7/webapps-biocache.ala.org.au/ROOT.war has finished in 9,987 ms

Button to try new functionality

Button on the production biocache inviting users to try new functionality:

e.g. Try the new Data Quality profiles

configurable so doesn’t require a release to use

On/off 

Text 

Target

What styling – colour, size of button and font?

Before enabling the invitation button the new functionality needs:

Functional testing – it should not have known bugs 

Help material available 

Feedback channel clear – how to provide feedback

Text for DQ filters: "try the new search interface with automatic filtering based on data quality metrics".

Biocache Map Tab's "View in Spatial Portal" not working

Clicking on the button "View in spatial portal" in the Map tab of biocache doesn't send data to Spatial Portal. SP open with no species layer. E.g., http://biocache.ala.org.au/occurrences/search?q=qid:1504050926107&fq=longitude:[115.009765625%20TO%20116.15234375]&fq=latitude:[-29.739521269703%20TO%20-28.744987541822997]#tab_mapView produces no data in SP.

atlasoflivingaustralia / ala-hub Goto Github PK

ala-hub's People

Contributors

Stargazers

Watchers

Forkers

ala-hub's Issues

Recommend Projects

Recommend Topics

Recommend Org