Opengrok web application isn't accessible during reindex about opengrok HOT 8 OPEN

Sounds like a bug. There is probably something in the setConfiguration() code path that acquires a lock used by the request processing.

One way to avoid this is to use the per project workflow which avoids uploading the configuration to the webapp.

from opengrok.

@vladak looks like with 1.13.2 this behavior is a little different. The webapp allows the source to be browsed although the search, history and any other functionality which requires some sort of querying doesn't respond until indexing is done.

Is it expected for Sending configuration to: http://localhost:8080/source to take so long (~5minutes for a ~3.5GB project even when there are no source changes)? It's also confusing since the configuration is only 8KB @ /opengrok/etc/configuration.xml.

My indexing command is:
opengrok-indexer -a /opengrok/dist/lib/opengrok.jar -- -s /opengrok/src -d /opengrok/data -H -P -S -G -W /opengrok/etc/configuration.xml -U "http://localhost:8080/source

I also couldn't get per project setup to work without the same issue either, could you elaborate on how the setup would look?

from opengrok.

The configuration processing in the webapp might be held by invalidating repositories. Could you take a look at the webapp logs while the configuration upload is in progress to see if that is the case ? Ideally also capture the thread stacks at that point using jstack.

The point of per project workflow is that per project reindex does not send the complete configuration at the end of the reindex:

It merely pokes the webapp to bump the index reader for given project.

from opengrok.

I continuously get the same log across multiple threads almost exactly 1 second apart FINEST [http-nio-8080-exec-<tid>] org.opengrok.web.api.v1.filter.IncomingFilter.filter allow request to status/<same-uuid-with-every-request> based on localhost IP address.

When searching a query I see FINEST [http-nio-8080-exec-15] org.opengrok.web.api.v1.suggester.parser.SuggesterQueryDataParser.processQuery Processing suggester query: hello at 5 and jstack suggests this is due to the Suggester.rebuild() holding a lock and the search uses SuggesterServiceImpl.getSuggestions and SuggesterServiceImpl.onSearch() waiting for that lock during indexing.

EDIT: Disabling suggester works fine, sending config finishes in sub second time. Looks like suggester should be temporarily turned off during it's rebuilding to avoid this denial of service.

from opengrok.

It may be, and this is purely a speculation at this point, that at the time the configuration is sent to the webapp, there is already a sequence of suggester rebuild jobs waiting in the queue and it somehow blocks the request. The rebuild() method holds a R/W lock for writing so this would block any suggester read requests. The just released 1.13.3 improves suggester rebuild request handling so this might provide some relief, however does not solve the problem.

from opengrok.

I see, I can reproduce it 100% of the time, so this waiting for rebuild thread is either triggering all the time or indexing somehow spawns it and takes a very very long time to do so as well.

Probably interesting to look into how we can optimize suggester rebuilds especially when we have history information and we can granularly build it. Or maybe disable the feature entirely when rebuild is on.

EDIT: 1.13.3 has the same issue and reproduces the same way. The performance of the rebuilds seems to be similar wallclock time.

from opengrok.

Could you take the stack trace snapshot using jstack on the Tomcat process when this issue occurs and there is a pending request to the webapp that is stuck ? There should be a better way to fix this than disabling suggester.

from opengrok.