simple, single string body search exec'd as unneeded boolean OR? about dovecot-fts-flatcurve HOT 5 CLOSED

I can verify that this is happening, but this does not appear to be a flatcurve issue. It is, indeed, the snowball filter that causes two terms (the original term and the stemmed term) to be passed into flatcurve as an "OR" search.

Additional search terms I tested: "running" and "smashes". I see these queries:

May 23 22:51:29 imap(user)<139><ZS5ErrXftrJ/AAAB>: Debug: fts-flatcurve(imaptest): Query (body:running* OR body:run*) matches=0 uids=
May 23 22:51:29 imap(user)<139><ZS5ErrXftrJ/AAAB>: Debug: fts-flatcurve(imaptest): Query (body:smash* OR body:smashes*) matches=0 uids=

I will verify with the team whether this is the expected behavior of snowball.

Flatcurve has no way of knowing that the terms are related to each other. And it makes little sense for flatcurve to manually optimize these queries by filtering redundant search terms - that's quite complicated code, and Xapian index queries are plenty fast that this is not going to make any difference. So if this is truly the expected behavior of snowball, the current behavior is fine.

from dovecot-fts-flatcurve.

What are your Dovecot fts_filters settings?

It sort of looks like there is something weird going on with snowball filtering (or related). Since the first term is the stemmed version of the original query - without the English suffix ("ing"). https://doc.dovecot.org/settings/plugin/fts-plugin/#plugin_setting-fts-fts_filters

from dovecot-fts-flatcurve.

hi @slusarz

What are your Dovecot fts_filters settings?

currently, similar to the settings I'd been using with my prior fts_solr install,

...
mail_plugins = virtual acl fts fts_flatcurve
plugin {
	fts = flatcurve
	fts_enforced = yes

	fts_autoindex = yes
	fts_autoindex_max_recent_msgs = 999
	fts_autoindex_exclude  = \Junk
	fts_autoindex_exclude2 = \Trash

	fts_filters = normalizer-icu lowercase snowball stopwords
	fts_filters_en = lowercase snowball english-possessive stopwords

	fts_languages = en es de fr it pt
	fts_language_config = /usr/share/libexttextcat/fpdb.conf
	fts_tokenizers = generic email-address
	fts_tokenizer_generic = algorithm=simple

	# BUG: wait for dovecot 2.3.19 ...
	#  https://github.com/slusarz/dovecot-fts-flatcurve/issues/22
	# fts_header_excludes = *
	# fts_header_includes = From To Cc Bcc Subject Message-ID
}
...

from dovecot-fts-flatcurve.

@slusarz

fwiw, while troubleshooting tika searching, I exec'd a body search for a term "mairzy".

& noted in logs:

Debug: fts-flatcurve(INBOX): Query (body:mairzy* OR body:mairzi*) matches=0 uids=

notice that OR'd query is not just a redundant truncation, but an unwanted variant, namely:

mairzy*

, which would return desired results,

vs

mairzi*

, which would not

from dovecot-fts-flatcurve.

So I think you answered your own question - the snowball filter does potentially provide independent terms that WILL result in different search results.

So maybe the snowball filter could be improved to not pass these terms to the FTS backend if they are redundant. But regardless, the snowball filter is not a flatcurve component, it's a core component, so this is the wrong place to be discussing that improvement. So I'm going to close this ticket (since, even with the redundant queries, flatcurve is returning the correct results).

from dovecot-fts-flatcurve.