TestPerFieldDocValuesFormat.testThreads2 fails with java.lang.ArrayIndexOutOfBoundsException about lucene HOT 8 CLOSED

@jpountz I will do that. But, keep the testing that provides the parallelism to ensure we are covered when things are enabled in the future

from lucene.

git bisect points to this commit: b940511

from lucene.

Intra merge concurrency is causing a race condition here it seems? I can debug where.

from lucene.

Reverting point value parallelism fixes this bug. This tells me that with how merging point values from multiple threads is busted. I will see if there is a quick fix. If there isn't I vote we do not merge points in parallel.

from lucene.

Ah, looking at the clone code for merge state @jpountz

    for (int i = 0; i < storedFieldsReaders.length; ++i) {
      if (storedFieldsReaders[i] != null) {
        storedFieldsReaders[i] = storedFieldsReaders[i].getMergeInstance();
      }
      if (termVectorsReaders[i] != null) {
        termVectorsReaders[i] = termVectorsReaders[i].getMergeInstance();
      }
      if (normsProducers[i] != null) {
        normsProducers[i] = normsProducers[i].getMergeInstance();
      }
      if (docValuesProducers[i] != null) {
        docValuesProducers[i] = docValuesProducers[i].getMergeInstance();
      }
      if (fieldsProducers[i] != null) {
        fieldsProducers[i] = fieldsProducers[i].getMergeInstance();
      }
      if (pointsReaders[i] != null) {
        pointsReaders[i] = pointsReaders[i].getMergeInstance();
      }
      if (knnVectorsReaders[i] != null) {
        knnVectorsReaders[i] = knnVectorsReaders[i].getMergeInstance();
      }
    }

This assumes getMergeInstance() does something. But points simply returns this.

I am thinking that sharing a reader between threads is always a bad idea.

from lucene.

Wow, thanks for finding this, it's indeed broken. I'll look into it.

from lucene.

@jpountz given the feature freeze of 9.12, what do you think of disabling intra-merge parallelism for everything :/ and we enable it one at a time for things in the future as wrinkles are worked out?

from lucene.

Agreed, this sounds safer with 9.12 around the corner.

from lucene.