Performance issues with large datasets about tntsearch HOT 6 OPEN

from tntsearch.

Thanks for the reply.

I am using a simple dataset with 50000+ company names. I am only using a custom tokenizer.

from tntsearch.

from tntsearch.

Hi,

This is my tokenizer

`<?php

namespace Search;

use TeamTNT\TNTSearch\Support\AbstractTokenizer;
use TeamTNT\TNTSearch\Support\TokenizerInterface;

class Tokenizer extends AbstractTokenizer implements TokenizerInterface
{
static protected $pattern = '/[^\p{L}\-\p{N}]+/u';

public function tokenize($text, $stopwords = [])
{
    if ($text === null) {
        return [];
    }

    $text = mb_strtolower($text, 'UTF-8');
    $text = str_replace(['-', '_', '~'], [' ', ' ', '-'], $text);
    $text = strip_tags($text);
    $split = preg_split($this->getPattern(), $text, -1, PREG_SPLIT_NO_EMPTY);

    return array_diff($split, $stopwords);
}

}
`

My query is super simpel

$indexer->query('SELECT id,nameFROMcompanies');

from tntsearch.

I'm using this package with a result set of 1.5 million records. Indexing from scratch takes ~5 minutes.

from tntsearch.

Thats crazy...

There is something weird anyways.

Indexing 10.000 rows takes like 20 seconds on my server.
But when i index 100.000 rows with alike data (so not different form of data or length) it takes about 30 minutes.
Even updating the index is very slow instead of complete reindex.

Could it be something with the size of the index file?

from tntsearch.