Multi word searches or "quoted search" about django-elasticsearch-dsl-drf HOT 18 CLOSED

@mrchrisadams:

Yep. You've got it all right!

from django-elasticsearch-dsl-drf.

@hlodver:

Elastic operator options are currently not supported.

I'm adding it to the TODO list.

I can't give a certain indication about when would it "arrive" in the package though.

from django-elasticsearch-dsl-drf.

For the reference:

https://www.elastic.co/guide/en/elasticsearch/guide/current/match-multi-word.html

from django-elasticsearch-dsl-drf.

Hi there, I'm new to ElasticSearch, and elasticsearch-dsl but this looks like something I'll need on existing project too.

From what I can tell, the search behaviour is implemented in SearchFilterBackend, and below is the code that carries out the query when you pass params to search (I think):

https://github.com/barseghyanartur/django-elasticsearch-dsl-drf/blob/master/src/django_elasticsearch_dsl_drf/filter_backends/search.py#L214-L216

def filter_queryset(self, request, queryset, view):
       # edited for brevity
        __queries = self.construct_search(request, view) + \
            self.construct_nested_search(request, view)

        if __queries:
            queryset = queryset.query('bool', should=__queries)

       return queryset

To make this support multi word searches, my thinking is that we'd need a way to use the must operator here, like so, instead of the should one:

if __queries:
            queryset = queryset.query('bool', must=__queries)
       return queryset

Would this be the case?

It wasn't obvious to me where the logic was that lets us use the pipe | operators for searching on specific field, and so on - if someone can let me know where this is, I'm up for having a go, as this multi word search would be incredibly handy.

from django-elasticsearch-dsl-drf.

@mrchrisadams:

Look, Elastic has a lot of options for that.

All listed here.

You're probably most interested in the following:

• match_phrase
• multi_match
• query_string
• simple_query_string

I must confess, that I didn't have much time to spend on the search filter backend, however, it needs major improvements in order to support all these things. I think finally all of them need to be implemented (as separate backends).

I'll try to start with simple_query_string.

Contributions are greatly appreciated. If you're willing to spend some time on this, we can discuss how to divide the work.

from django-elasticsearch-dsl-drf.

@mrchrisadams:

FYI, before query_string and simple_query_string are implemented, there's a need to find an alternative value separator here.

Update (2018-07-15)

Separators have been changed. Next release will be focused on new search backends.

from django-elasticsearch-dsl-drf.

Oh sweet, thanks @barseghyanartur - the simple_query_string looks like it does much more than what I need, so I don't mind having a go at one of these.

It looks like match_phrase only works on a single field, so I don't think it would work so well with how you specific the search fields in the Viewset, but multi_match does allow for phrase matching, across multiple fields.

https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-multi-match-query.html#type-phrase

From what I can see the different filters work by having their own reserved param, like ?search in the SearchFilter case, or in the case of the filter, listening for all the non reserved params, then checking for varous filters to add like __gt or __contains and so on.

I'm wondering if having an extra ?phrase_search param that carries out a phrase search on the fields provided might work, and support the same modifiers as the current search.

So you might do

search/?phrase_search=Day%20trips

Which would end up as this kind of query:

{
  "query": {
    "multi_match": {
        "query": "Day Trips",
        "type":"phrase",
        "fields": [ "all_of_the", "search_fields", "defined_in_the_viewset"]
    }
  }
}

To search all the normal search fields,

search/?phrase_search=specific_field|Day%20trips

Would end up with:

{
  "query": {
    "multi_match": {
        "query": "Day Trips",
        "type":"phrase",
        "fields": [ "all_of_the", "search_fields", "defined_in_the_viewset"]
    }
  }
}

I haven't looked into how elasticsearch-dsl treats MultiMatch queries yet, or whether it lets you mix them with other search or filter querie. If this is possible, I'll update here.

Also, I'm not fussy about the final name, phrase_search is just for these examples here.

Anyway, thanks for the response. I've switched on github notifcations again, so I'll actually get emails when you respond now…

from django-elasticsearch-dsl-drf.

@mrchrisadams:

Working branch for this is search_backends.

The first steps are done. The old single Search Filter Backend (SearchFilterBackend) is to be replaced with new style Search Filter Backends (at the moment, the django_elasticsearch_dsl_drf.filter_backends.search.compound.CompoundSearchFilterBackend resembles all the functionality of the SearchFilterBackend).

Differences:

(1) Pluggable query backends.

class CompoundSearchFilterBackend(BaseSearchFilterBackend):
    """Compound search backend."""

    query_backends = [
        MatchQueryBackend,
        NestedQueryBackend,
    ]

(2) Configurabe matching options. Default is should, but can be easily changed to must in your custom search query backend.

You can pretty much base your code on this functionality, as it will be genrally available soon. Tests are passing.

You could for instance, declare match_all Query Backend and plug it into your Search Filter Backend. I will myself work on that further, but any help is welcome.

from django-elasticsearch-dsl-drf.

Awesome, thanks for @barseghyanartur - I'll look into this tomorrow.

Are you proposing match_all query backend be something like a backend that has must as the default, instead of the should in the current one?

from django-elasticsearch-dsl-drf.

@mrchrisadams:

• match_all as a Query Backend: Yes.
• must instead of should: No, since it's not necessary. Instead, you can configure it in Search Filter Backend.

Note, that all Query Backends make use of the same request query param name within Search Filter Backend they are part of. At least, this is how it should be. So, you might even have multiple Search Filter Backends (and thus - various request query params) per ViewSet.

At least, this is how it's going to be on the package level. If you have too many things to customize, you should better make your custom Search Filter Backend and fine-tune queries there.

from django-elasticsearch-dsl-drf.

@mrchrisadams:

Initial version of Multi Match search backend is ready (without tests yet, as WIP).

Sample view

class BookMultiMatchSearchFilterBackendDocumentViewSet(
    BookDocumentViewSet
):

    filter_backends = [
        # ...
        OrderingFilterBackend,
        DefaultOrderingFilterBackend,
        MultiMatchSearchFilterBackend,
        # ...
    ]

    search_fields = {
        'title': {'boost': 4},
        'summary': {'boost': 2},
        'description': None,
    }
    ordering = ('_score', 'id', 'title', 'price',)

Sample request

Sample request: http://localhost:8000/search/books-multi-match-search-backend/?search_multi_match=debitis%20enim

Generated query

{
  "from": 0,
  "query": {
    "multi_match": {
      "query": "debitis enim",
      "fields": [
        "summary^2",
        "description",
        "title^4"
      ]
    }
  },
  "size": 38,
  "sort": [
    "_score",
    "id",
    "title",
    "price"
  ]
}

Fine-tune query options

This would search for a phrase.

class BookMultiMatchSearchFilterBackendDocumentViewSet(DocumentViewSet):

    # ...

    multi_match_options = {
        'type': 'phrase'
    }

Working example

Sample view

Documentation (WIP)

https://github.com/barseghyanartur/django-elasticsearch-dsl-drf/blob/search-backends/search_backends.rst

from django-elasticsearch-dsl-drf.

wow @barseghyanartur this is really cool! I had a go at setting up a test harness yesterday, with the vagrant box, but I think I had something misconfigured, as I couldn't get tests to pass locally on my laptop.

I'm going to have another go today to see if I can add some tests for the multi match stuff, and if I can't get them running, I'll file an issue explaining what I had trouble with.

Thanks again for this 👍

from django-elasticsearch-dsl-drf.

I don't recommend using Elastic 2.x.
Better use Elastic 5.x or 6.x. There's a doc here install_elasticsearch.rst (in the root). There are docker containers ready to use.

from django-elasticsearch-dsl-drf.

Okay, @barseghyanartur, I've got the test suite running here with python runtests.py and elastic search 5.x now.

Thanks for the pointers.

So, to set up a test for this, I'm assuming I'd create a separate TestMultiSearch class, and in the setup, add the new BookMultiMatchSearchFilterBackendDocumentViewSet view, and MultiMatchSearchFilterBackend

I can largely follow what's going on in the _search_by_field method in your test class, so I'm assuming I'd do something similar, but with search_multi_match as the search param.

Can you let me know if this is about right? If so, I'll have a look at it tonight

from django-elasticsearch-dsl-drf.

Once there are tests for what's implemented at the moment, I'll make a new release (0.12). Thus, with introduction of new-style Search Filter Backends, Multi Match Search Filter Backend and Compound Search Filter Backend (as well as a couple of other intermediate fixes).

Implementation of match_phrase, query_string and simple_query_string Search Filter Backends is moved to 0.13 milestone.

from django-elasticsearch-dsl-drf.

0.12 is released.

from django-elasticsearch-dsl-drf.

The simple_query_string implementation landed in the search_backends branch. Tests are passing. Documentation available. Comments are welcome.

from django-elasticsearch-dsl-drf.

simple_query_string search filter backend is implemented in 0.13.

The issue is considered to be solved. Implementation of match_phrase, match_phrase_prefix, common_terms and query_string backends shall still be implemented, however, I hope that someone (not me) would pick the task.

from django-elasticsearch-dsl-drf.