barseghyanartur / django-elasticsearch-dsl-drf Goto Github PK
View Code? Open in Web Editor NEWIntegrate Elasticsearch DSL with Django REST framework.
Home Page: https://pypi.python.org/pypi/django-elasticsearch-dsl-drf
Integrate Elasticsearch DSL with Django REST framework.
Home Page: https://pypi.python.org/pypi/django-elasticsearch-dsl-drf
I don't know why using Django signal in this way failed to sync elasticsearch with my QuestionSummary manytomany object
:
My models:
class Summary(models.Model):
url = models.CharField(max_length=255, unique=True)
created_at = models.DateTimeField(auto_now_add=True)
html_text = models.TextField(blank=True, unique=True)
url_local_path = models.CharField(max_length=255)
validated = models.BooleanField(default=False)
cover_image = models.CharField(max_length=255)
tag_category = models.ManyToManyField('TagCategory', default=[])
user_profiles = models.ManyToManyField('UserProfile', through='UserProfileSummary')
questions = models.ManyToManyField('Question', through='QuestionSummary', through_fields=('summary', 'question'),)
def __str__(self):
return self.html_text
class Meta:
verbose_name_plural = "Summaries"
@property
def null_field_indexing(self):
"""null_field for indexing.
Used in Elasticsearch indexing/tests of `isnull` functional filter.
"""
return None
class Question(models.Model):
title = models.CharField(max_length=255, unique=True)
created_at = models.DateTimeField(auto_now_add=True)
score = models.FloatField(default=0)
def __str__(self):
return self.title
class QuestionSummary(models.Model):
"""Manytomany table"""
# ToDo: field modified_by (and tracking the history) and created_by
is_summary_author = models.BooleanField(default=False)
is_question_author = models.BooleanField(default=False)
last_modified_date = models.DateTimeField(auto_now_add=True)
question = models.ForeignKey('Question', on_delete=models.CASCADE)
summary = models.ForeignKey('Summary', on_delete=models.CASCADE)
main_question = models.BooleanField(default=False)
class Meta:
verbose_name_plural = "Question summaries"
def __str__(self):
return "{} --- {}".format(
self.question.title,
self.summary.url
)
Signal.py
from django.db.models.signals import post_save, post_delete, m2m_changed
from django.dispatch import receiver
from django_elasticsearch_dsl.registries import registry
__all__ = (
'update_document',
'delete_document',
)
@receiver(post_save)
def update_document(sender, **kwargs):
app_label = sender._meta.app_label
model_name = sender._meta.model_name
instance = kwargs['instance']
print("EDGAR app_label ", app_label)
if app_label == 'meshine_api':
if model_name == 'summary':
print("EDGAR model_name ", model_name)
instances = instance.questions.all()
for _instance in instances:
registry.update(_instance)
I want that when the Summary
object get update, the QuestionSummary
object on elasticsearch get automatically update too. But now nothing happen, QuestionSummary
never get update when Summary does.
I've also tried to use m2m_changed
in signal.py:
`m2m_changed.connect(update_document, sender=models.Summary.questions.through)`
But it does nothing!
It's like django-elasticsearch-dsl-drf doesn't support ManyToMany Through object
(questions = models.ManyToManyField('Question', through='QuestionSummary', through_fields=('summary', 'question'),))
. Does it ?
Please what's wrong ?
Document definition:
# State history
state_history = fields.ListField(
fields.ObjectField(
attr='state_history_indexing',
multi=True,
properties={
'timestamp': fields.DateField(
),
'state': StringField(
analyzer=html_strip_lowercase,
fields={
'raw': KeywordField(),
}
),
}
)
)
The following structure (indexed in Elastic):
"state_history": [
{
"timestamp": "2017-10-18T11:51:04.037489+00:00",
"state": "On hold gezet"
}
,
{
"timestamp": "2017-10-18T11:40:20.648071+00:00",
"state": "Meer info nodig"
}
,
{
"timestamp": "2017-10-18T11:08:03.100521+00:00",
"state": "Meer info nodig"
}
,
{
"timestamp": "2017-10-18T11:05:26.640457+00:00",
"state": "Meer info nodig"
}
],
Is shown as follows in the REST framework:
"state_history": {
"state": "timestamp"
}
When I use pagination through pagination_class = LimitOffsetPagination
, it works very well, but the count value never change. For example if http://127.0.0.1:8000/search/books/
returns 10 results, count will be 10 and then if I use http://127.0.0.1:8000/search/books/?limit=1
, it will return only one result
but count doesn't change at all, we still have count=10
.
Is is normal ?
Add post_filter
option to search and filtering backends.
Work in progress in branch post-filter-support.
from django_elasticsearch_dsl_drf.filter_backends import (
DefaultOrderingFilterBackend,
FacetedSearchFilterBackend,
FilteringFilterBackend,
GeoSpatialFilteringFilterBackend,
GeoSpatialOrderingFilterBackend,
NestedFilteringFilterBackend,
OrderingFilterBackend,
PostFilterFilteringFilterBackend,
SearchFilterBackend,
SuggesterFilterBackend,
)
from django_elasticsearch_dsl_drf.pagination import LimitOffsetPagination
from django_elasticsearch_dsl_drf.viewsets import DocumentViewSet
from ..backends import NestedContinentsBackend
from ..documents import AddressDocument
from ..serializers import AddressDocumentSerializer
class AddressDocumentViewSet(DocumentViewSet):
"""The AddressDocument view."""
document = AddressDocument
serializer_class = AddressDocumentSerializer
lookup_field = 'id'
filter_backends = [
FacetedSearchFilterBackend,
FilteringFilterBackend,
OrderingFilterBackend,
SearchFilterBackend,
GeoSpatialFilteringFilterBackend,
GeoSpatialOrderingFilterBackend,
NestedContinentsBackend,
NestedFilteringFilterBackend,
PostFilterFilteringFilterBackend,
DefaultOrderingFilterBackend,
SuggesterFilterBackend,
]
pagination_class = LimitOffsetPagination
# Define filtering fields
filter_fields = {
'id': None,
'city': 'city.name.raw',
'country': 'city.country.name.raw',
}
post_filter_fields = {
'city_pf': 'city.name.raw',
'country_pf': 'city.country.name.raw',
}
Sample requests:
Filter
http://localhost:8000/search/addresses/?city=Yerevan
Post filter
http://localhost:8000/search/addresses/?city_pf=Yerevan
Make path a field option param, instead of leaving it to the dictionary key, which is used to identify the specific field.
Dictionary key is the path
in Elasticseach and also the GET param name.
Type 1
search_nested_fields = {
'country': ['name'],
'country.city': ['name'],
}
Type 2
search_nested_fields = {
'country': [{'name': {'boost': 2}}],
'country.city': [{'name': {'boost': 2}}],
}
Dictionary key is the GET param name. The path
option stands for the path
in Elasticsearch.
Type 1
search_nested_fields = {
'country': {
'path': 'country',
'fields': ['name'],
},
'city': {
'path': 'country.city',
'fields': ['name'],
},
}
Type 2
search_nested_fields = {
'country': {
'path': 'country',
'fields': [{'name': {'boost': 2}}]
},
'city': {
'path': 'country.city',
'fields': [{'name': {'boost': 2}}]
},
}
Hi guys,
Great project :)
I am however in a bit of a pickle. Ive setup ElasticSearch, django-elasticsearch-dsl-drf and django-elasticsearch-dsl very well in my project and it mostly seems to work great. Ive tried a lot of things and I am completely stumped so I am hoping someone here might have an idea whats going on :)
Whenever I execute one of those query :
http://localhost:8000/search/books/
http://localhost:8001/search/books/?ids=54|55|56
http://localhost:8001/search/books/?summary__contains=photography
http://localhost:8001/search/books/?tags__contains=ython
http://localhost:8001/search/books/?state=published
http://localhost:8001/search/books/?pages__gt=10&pages__lt=30
I get 0 results while my database contains the appropriate data and I've performed: ./manage.py search_index --create -f
and ./manage.py search_index --populate -f
Here is my python stack :
Django==2.0.1
django-elasticsearch-dsl==0.4.4
django-elasticsearch-dsl-drf==0.6.3
django-nine==0.1.13
django-ormex==0.2
djangorestframework==3.7.7
drf-extensions==0.3.1
elasticsearch==5.5.2
elasticsearch-dsl==5.4.0
factory-boy==2.10.0
Faker==0.8.11
ipaddress==1.0.19
Pillow==5.0.0
psycopg2==2.7.3.2
python-dateutil==2.6.1
pytz==2017.3
PyYAML==3.12
six==1.11.0
text-unidecode==1.1
urllib3==1.22
I used a ubuntu 16.0.4 virtualenv with elasticsearch 5.5.3
Thanks for your help
Make it easier to customize pagination classes
Hi,
I have this error using suggestion for a nullable field:
Thanks for your help.
It should be possible to choose whether the suggester backend shall be aware of current queryset or not.
If it doesn't fit into a single filter backend, make another suggester backend and name it QuerySetAwareSuggesterFilterBackend
.
Perhaps use https://www.elastic.co/guide/en/elasticsearch/reference/current/suggester-context.html
Otherwise, document edge_gram
analyzer usage (including performance comparison) with use of prefix
filter.
Further reading:
I just setup transaction within my Django view. I just noticed that every time there is a rollback
, the data is save and sync through Elasticsearch while unsave
on my database.
How can I apply transaction in this package ?
Add coreapi
and coreschema
integration.
Add support for global aggregations.
WIP in branch
class BookDocumentViewSet(BaseDocumentViewSet):
# ...
faceted_search_fields = {
'publisher_global': {
'field': 'publisher.raw',
'enabled': True,
'type': 'global',
},
'publisher': {
'field': 'publisher.raw',
'enabled': True,
},
}
Present in both filtering/common
and filtering/geo_spatial
Implement faceting/aggregations.
GET Query ?countries=1149361&countries=661882
I want this filter to always be an AND lookup (term), and never an OR lookup (terms), so I set the valid lookups to only "term"
filter_fields = {
'countries': {
'field': 'countries',
'lookups': LOOKUP_FILTER_TERM,
},
}
However when there's no lookup specified in the url and there's multiple values it always defaults to a "terms" filter without checking if it's allowed for that field.
How can I do a search like Kibana does with quotes and people are used to use with google?
/api/v1/search/?search="Day%20trips"
I could not find anything in the documentation
Implement the following search filter backends (read the thread from top to the bottom for discussions).
I am aware of #46, but downgrading elasticsearch
didn't help.
I get this error following http://django-elasticsearch-dsl-drf.readthedocs.io/en/0.10/quick_start.html.
ERROR 2018-07-11 12:11:08,036 exception Internal Server Error: /api/v2/search/
Traceback (most recent call last):
File "/home/my_username/.virtualenvs/my_venv_name/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
response = get_response(request)
File "/home/my_username/.virtualenvs/my_venv_name/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/home/my_username/.virtualenvs/my_venv_name/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/my_username/.virtualenvs/my_venv_name/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
return view_func(*args, **kwargs)
File "/home/my_username/.virtualenvs/my_venv_name/lib/python3.6/site-packages/rest_framework/viewsets.py", line 90, in view
return self.dispatch(request, *args, **kwargs)
File "/home/my_username/.virtualenvs/my_venv_name/lib/python3.6/site-packages/rest_framework/views.py", line 489, in dispatch
response = self.handle_exception(exc)
File "/home/my_username/.virtualenvs/my_venv_name/lib/python3.6/site-packages/rest_framework/views.py", line 449, in handle_exception
self.raise_uncaught_exception(exc)
File "/home/my_username/.virtualenvs/my_venv_name/lib/python3.6/site-packages/rest_framework/views.py", line 477, in dispatch
self.initial(request, *args, **kwargs)
File "/home/my_username/.virtualenvs/my_venv_name/lib/python3.6/site-packages/rest_framework/views.py", line 395, in initial
self.check_permissions(request)
File "/home/my_username/.virtualenvs/my_venv_name/lib/python3.6/site-packages/rest_framework/views.py", line 328, in check_permissions
if not permission.has_permission(request, self):
File "/home/my_username/.virtualenvs/my_venv_name/lib/python3.6/site-packages/rest_framework/permissions.py", line 133, in has_permission
perms = self.get_required_permissions(request.method, queryset.model)
AttributeError: 'Search' object has no attribute 'model'
Packages I have installed
Django==1.11.10
django-elasticsearch-dsl==0.5.0
django-elasticsearch-dsl-drf==0.10
djangorestframework==3.6.4
elasticsearch==6.3.0
elasticsearch-dsl==6.1.0
Implement faceted search filtering.
Hi,
I have field:
provider = fields.NestedField(
properties={
'id': fields.IntegerField(),
'name': fields.TextField(boost=0.8)
}
)
In views I have:
filter_fields = {
'provider_id': 'provider.id'
}
When I filtering over provider_id it's genereting:
{
"query": {
"bool": {
"must": [{"match": {"provider.id": "1"}}]
}
}
}
It always return 0 elements.
How to achive to get filtered elements?
Please is it possible to display the _score within my json results and filtering my search thereby ?
Affected versions of Django: 1.10.
Works fine on Django versions: 1.11, 2.0
Fix in any case.
Traceback:
File "/home/user/.virtualenvs/myproject/lib/python3.5/site-packages/django/core/handlers/exception.py" in inner
39. response = get_response(request)
File "/home/user/.virtualenvs/myproject/lib/python3.5/site-packages/django/core/handlers/base.py" in _get_response
187. response = self.process_exception_by_middleware(e, request)
File "/home/user/.virtualenvs/myproject/lib/python3.5/site-packages/django/core/handlers/base.py" in _get_response
185. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/user/.virtualenvs/myproject/lib/python3.5/site-packages/django/views/decorators/csrf.py" in wrapped_view
58. return view_func(*args, **kwargs)
File "/home/user/.virtualenvs/myproject/lib/python3.5/site-packages/rest_framework/viewsets.py" in view
83. return self.dispatch(request, *args, **kwargs)
File "/home/user/.virtualenvs/myproject/lib/python3.5/site-packages/rest_framework/views.py" in dispatch
477. response = self.handle_exception(exc)
File "/home/user/.virtualenvs/myproject/lib/python3.5/site-packages/rest_framework/views.py" in handle_exception
437. self.raise_uncaught_exception(exc)
File "/home/user/.virtualenvs/myproject/lib/python3.5/site-packages/rest_framework/views.py" in dispatch
474. response = handler(request, *args, **kwargs)
File "/home/user/.virtualenvs/myproject/lib/python3.5/site-packages/django_elasticsearch_dsl_drf/viewsets.py" in functional_suggest
72. queryset = self.filter_queryset(self.get_queryset())
File "/home/user/.virtualenvs/myproject/lib/python3.5/site-packages/rest_framework/generics.py" in filter_queryset
151. queryset = backend().filter_queryset(self.request, queryset, self)
File "/home/user/.virtualenvs/myproject/lib/python3.5/site-packages/django_elasticsearch_dsl_drf/filter_backends/suggester/functional.py" in filter_queryset
502. raise ValidationError()
Exception Type: TypeError at /search/books/functional_suggest/
Exception Value: __init__() missing 1 required positional argument: 'detail'
Hi,
It would be nice to have a way to pass highlight options, maybe as a new highlight_fields
in viewsets (used by SearchFilterBackend
) or just add an attribute pass all sorts of cusom kwargs to django dsl's queries (including but not limited to highlights
).
No matter how you decide it should be done, the second part of the problem is to figure out a way to pass the highlights to serializers. Any idea ?
I am willing to implement it.
Introduce detail-view and list-view more-like-this functionality.
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-mlt-query.html
Add support for nested aggregations.
Some references:
This is on-going issue. At the moment, there's a temporary solution for that (by making a custom filtering backend). See the documentation and examples:
https://github.com/elastic/elasticsearch-dsl-py/blob/master/elasticsearch_dsl/faceted_search.py#L161
elasticsearch-dsl
version 6.x supports nested aggregations, however, since this library aims to support 2.x (not so important anymore, might as well be deprecated) and 5.x, the total switch to 6.x functionality can't be made (simple), unless we copy some of the elasticsearch-dsl
functionality locally or have it supported for Elasticsearch 6.x only. Help here is appreciated. If you need this badly, either see my remarks above (2018-06-27, regarding custom nested aggregations backend) or consider making a contribution (I will assist you).
Add boost support in the search filter backend
Implement nested queries
search_nested_fields
functionality fully solves is.I just noticed that wildcard search is case sensitive. How could we make it case insensitive ?
pip install django-elasticsearch-dsl-drf
will install django-elasticsearch-dsl-0.4.4 and elasticsearch-dsl-6.1.0 which are incompatible.
For example the String class doesn't exist anymore in elasticsearch 6.
I fetched the example project to run it in my local, but no elasticsearch query work. I all time get this kind of error whenever I run an elasticsearch query :
RequestError at /search/authors/ TransportError(400, 'search_phase_execution_exception', 'No mapping found for [name.raw] in order to sort on')
Where could I get wrong ?
Thx
How to implement "multi_match" query with phrase_prefix type.
I want to implement search for keywords.
Also,
let me know how to handle date?
While creating Date field i am also getting timestamp.
Django 2.0 support. Although package works fine with Django 2.0, tests fail (since example project on which the tests are dependant) is not yet compatible with Django 2.0.
hi~ I used "_source" in elasticsearch to show some certain fields in the searching results like this.
GET /_search
{
"query": { "match_all": {}},
"_source": [ "title", "created" ]
}
so, I wonder how to use "_source" with django-elasticsearch-dsl-drf๏ผneed someone help~
BTW, I've tried http://localhost:8080/search/books/?_source=author, but the results return with all fields defined in serializers.
I am facing this error while calling Viewset for book document.
http://localhost:8000/search/books/
Error:
Request Method: | GET
-- | --
http://localhost:8000/search/books/
1.11.1
**AttributeError
'Search' object has no attribute 'model'**
/home/hasher/elastic_search_django-drf/venv/local/lib/python2.7/site-packages/rest_framework/permissions.py in has_permission, line 133
/home/hasher/elastic_search_django-drf/venv/bin/python
2.7.12
['/home/hasher/elastic_search_django-drf/es-dsl-drf-demo', '/home/hasher/elastic_search_django-drf', '/home/hasher/elastic_search_django-drf/venv/lib/python2.7', '/home/hasher/elastic_search_django-drf/venv/lib/python2.7/plat-x86_64-linux-gnu', '/home/hasher/elastic_search_django-drf/venv/lib/python2.7/lib-tk', '/home/hasher/elastic_search_django-drf/venv/lib/python2.7/lib-old', '/home/hasher/elastic_search_django-drf/venv/lib/python2.7/lib-dynload', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/home/hasher/elastic_search_django-drf/venv/local/lib/python2.7/site-packages', '/home/hasher/elastic_search_django-drf/venv/lib/python2.7/site-packages']
Mon, 16 Apr 2018 07:53:13 +0000
I am using:
certifi==2018.1.18
chardet==3.0.4
coreapi==2.3.3
coreschema==0.0.4
coverage==4.4.1
dj-database-url==0.4.2
dj-email-url==0.0.10
Django==1.11.1
django-coverage-plugin==1.4.2
django-elasticsearch-dsl==0.4.4
django-elasticsearch-dsl-drf==0.7.1
django-filter==1.0.2
django-nine==0.1.13
django-rest-swagger==2.1.2
djangorestframework==3.6.3
drf-extensions==0.3.1
elasticsearch==5.5.2
elasticsearch-dsl==5.1.0
gunicorn==19.7.1
idna==2.6
ipaddress==1.0.21
itypes==1.1.0
Jinja2==2.10
Logentries==0.17
MarkupSafe==1.0
openapi-codec==1.3.2
python-dateutil==2.7.2
pytz==2018.4
requests==2.18.4
simplejson==3.13.2
six==1.11.0
uritemplate==3.0.0
urllib3==1.22
whitenoise==3.3.0
also,
Here are my models:
book.py
from django.conf import settings
from django.db import models
from .tag import *
from .author import *
from six import python_2_unicode_compatible
BOOK_PUBLISHING_STATUS_PUBLISHED = 'published'
BOOK_PUBLISHING_STATUS_NOT_PUBLISHED = 'not_published'
BOOK_PUBLISHING_STATUS_IN_PROGRESS = 'in_progress'
BOOK_PUBLISHING_STATUS_CANCELLED = 'cancelled'
BOOK_PUBLISHING_STATUS_REJECTED = 'rejected'
BOOK_PUBLISHING_STATUS_CHOICES = (
(BOOK_PUBLISHING_STATUS_PUBLISHED, "Published"),
(BOOK_PUBLISHING_STATUS_NOT_PUBLISHED, "Not published"),
(BOOK_PUBLISHING_STATUS_IN_PROGRESS, "In progress"),
(BOOK_PUBLISHING_STATUS_CANCELLED, "Cancelled"),
(BOOK_PUBLISHING_STATUS_REJECTED, "Rejected"),
)
BOOK_PUBLISHING_STATUS_DEFAULT = BOOK_PUBLISHING_STATUS_PUBLISHED
@python_2_unicode_compatible
class Publisher(models.Model):
name = models.CharField(max_length=30)
address = models.CharField(max_length=50)
city = models.CharField(max_length=60)
state_province = models.CharField(max_length=30)
country = models.CharField(max_length=50)
website = models.URLField()
latitude = models.DecimalField(null=True,
blank=True,
decimal_places=15,
max_digits=19,
default=0)
longitude = models.DecimalField(null=True,
blank=True,
decimal_places=15,
max_digits=19,
default=0)
class Meta(object):
ordering = ["id"]
def __str__(self):
return self.name
@property
def location_field_indexing(self):
return {
'lat': self.latitude,
'lon': self.longitude,
}
@python_2_unicode_compatible
class Book(models.Model):
title = models.CharField(max_length=100)
description = models.TextField(null=True, blank=True)
summary = models.TextField(null=True, blank=True)
authors = models.ManyToManyField(Author, related_name='books')
publisher = models.ForeignKey(Publisher, related_name='books')
publication_date = models.DateField()
state = models.CharField(max_length=100,
choices=BOOK_PUBLISHING_STATUS_CHOICES,
default=BOOK_PUBLISHING_STATUS_DEFAULT)
isbn = models.CharField(max_length=100, unique=True)
price = models.DecimalField(max_digits=10, decimal_places=2)
pages = models.PositiveIntegerField(default=200)
stock_count = models.PositiveIntegerField(default=30)
tags = models.ManyToManyField(Tag,
related_name='books',
blank=True)
class Meta(object):
"""Meta options."""
ordering = ["isbn"]
def __str__(self):
return self.title
# The only publisher information we're going to need in our document
# is the publisher name. Since publisher isn't a required field,
# we define a properly on a model level to avoid indexing errors on
# non-existing relation.
@property
def publisher_indexing(self):
if self.publisher is not None:
return self.publisher.name
# As of tags, again, we only need a flat list of tag names, on which
# we can filter. Therefore, we define a properly on a model level,
# which will return a JSON dumped list of tags relevant to the
# current book model object.
@property
def tags_indexing(self):
return [tag.title for tag in self.tags.all()]
here is my document:
documents/book.py
from django_elasticsearch_dsl import DocType, Index, fields
from elasticsearch_dsl import analyzer
from ..models import *
# Name of the Elasticsearch index
INDEX = Index('books')
# See Elasticsearch Indices API reference for available settings
INDEX.settings(
number_of_shards=1,
number_of_replicas=1
)
html_strip = analyzer(
'html_strip',
tokenizer="standard",
filter=["standard", "lowercase", "stop", "snowball"],
char_filter=["html_strip"]
)
@INDEX.doc_type
class BookDocument(DocType):
"""Book Elasticsearch document."""
id = fields.IntegerField(attr='id')
title = fields.StringField(
analyzer=html_strip,
fields={
'raw': fields.StringField(analyzer='keyword'),
}
)
description = fields.StringField(
analyzer=html_strip,
fields={
'raw': fields.StringField(analyzer='keyword'),
}
)
summary = fields.StringField(
analyzer=html_strip,
fields={
'raw': fields.StringField(analyzer='keyword'),
}
)
publisher = fields.StringField(
attr='publisher_indexing',
analyzer=html_strip,
fields={
'raw': fields.StringField(analyzer='keyword'),
}
)
publication_date = fields.DateField()
state = fields.StringField(
analyzer=html_strip,
fields={
'raw': fields.StringField(analyzer='keyword'),
}
)
isbn = fields.StringField(
analyzer=html_strip,
fields={
'raw': fields.StringField(analyzer='keyword'),
}
)
price = fields.FloatField()
pages = fields.IntegerField()
stock_count = fields.IntegerField()
tags = fields.StringField(
attr='tags_indexing',
analyzer=html_strip,
fields={
'raw': fields.StringField(analyzer='keyword', multi=True),
'suggest': fields.CompletionField(multi=True),
},
multi=True
)
class Meta(object):
"""Meta options."""
model = Book # The model associate with this DocType
here is my serializer:
I am using BookDocumentSimpleSerializer however its showing same error for other serializers too.
serializers/book.py
from rest_framework import serializers
from django_elasticsearch_dsl_drf.serializers import DocumentSerializer
from ..documents import BookDocument
class BookDocumentSerializer1(serializers.Serializer):
"""Serializer for the Book document."""
id = serializers.IntegerField(read_only=True)
title = serializers.CharField(read_only=True)
description = serializers.CharField(read_only=True)
summary = serializers.CharField(read_only=True)
publisher = serializers.CharField(read_only=True)
publication_date = serializers.DateField(read_only=True)
state = serializers.CharField(read_only=True)
isbn = serializers.CharField(read_only=True)
price = serializers.FloatField(read_only=True)
pages = serializers.IntegerField(read_only=True)
stock_count = serializers.IntegerField(read_only=True)
tags = serializers.SerializerMethodField()
# Used in testing of `isnull` functional filter.
null_field = serializers.CharField(read_only=True,
required=False,
allow_blank=True)
class Meta(object):
"""Meta options."""
fields = (
'id',
'title',
'description',
'summary',
'publisher',
'publication_date',
'state',
'isbn',
'price',
'pages',
'stock_count',
'tags',
'null_field', # Used in testing of `isnull` functional filter.
)
read_only_fields = fields
def create(self, validated_data):
"""Create.
Do nothing.
:param validated_data:
:return:
"""
def update(self, instance, validated_data):
"""Update.
Do nothing.
:param instance:
:param validated_data:
:return:
"""
def get_tags(self, obj):
"""Get tags."""
if obj.tags:
return list(obj.tags)
else:
return []
class BookDocumentSerializer2(serializers.Serializer):
"""Serializer for the Book document."""
id = serializers.IntegerField(read_only=True)
title = serializers.CharField(read_only=True)
description = serializers.CharField(read_only=True)
summary = serializers.CharField(read_only=True)
publisher = serializers.CharField(read_only=True)
publication_date = serializers.DateField(read_only=True)
state = serializers.CharField(read_only=True)
isbn = serializers.CharField(read_only=True)
price = serializers.FloatField(read_only=True)
pages = serializers.IntegerField(read_only=True)
stock_count = serializers.IntegerField(read_only=True)
tags = serializers.SerializerMethodField()
class Meta(object):
"""Meta options."""
# List the serializer fields. Note, that the order of the fields
# is preserved in the ViewSet.
fields = (
'id',
'title',
'description',
'summary',
'publisher',
'publication_date',
'state',
'isbn',
'price',
'pages',
'stock_count',
'tags',
)
def get_tags(self, obj):
"""Get tags."""
if obj.tags:
return list(obj.tags)
else:
return []
class BookDocumentSimpleSerializer(DocumentSerializer):
"""Serializer for the Book document."""
# tags = serializers.SerializerMethodField()
# authors = serializers.SerializerMethodField()
highlight = serializers.SerializerMethodField()
class Meta(object):
"""Meta options."""
document = BookDocument
fields = (
'id',
'title',
'description',
'summary',
'authors',
'publisher',
'publication_date',
'state',
'isbn',
'price',
'pages',
'stock_count',
'tags',
'highlight', # Used in highlight tests
'null_field', # Used in testing of `isnull` functional filter.
)
def get_highlight(self, obj):
if hasattr(obj.meta, 'highlight'):
return obj.meta.highlight.__dict__['_d_']
return {}
here is my viewset:
viewsets/book.py
LOOKUP_FILTER_TERMS,
LOOKUP_FILTER_RANGE,
LOOKUP_FILTER_PREFIX,
LOOKUP_FILTER_WILDCARD,
LOOKUP_QUERY_IN,
LOOKUP_QUERY_GT,
LOOKUP_QUERY_GTE,
LOOKUP_QUERY_LT,
LOOKUP_QUERY_LTE,
LOOKUP_QUERY_EXCLUDE,
)
from django_elasticsearch_dsl_drf.filter_backends import (
FilteringFilterBackend,
IdsFilterBackend,
OrderingFilterBackend,
DefaultOrderingFilterBackend,
SearchFilterBackend,
)
from django_elasticsearch_dsl_drf.views import BaseDocumentViewSet
from ..documents import BookDocument
from ..serializers import BookDocumentSimpleSerializer
class BookDocumentView(BaseDocumentViewSet):
"""The BookDocument view."""
document = BookDocument
serializer_class = BookDocumentSimpleSerializer
lookup_field = 'id'
filter_backends = [
FilteringFilterBackend,
IdsFilterBackend,
OrderingFilterBackend,
DefaultOrderingFilterBackend,
SearchFilterBackend,
]
# Define search fields
search_fields = (
'title',
'description',
'summary',
)
# Define filter fields
filter_fields = {
'id': {
'field': 'id',
# Note, that we limit the lookups of id field in this example,
# to `range`, `in`, `gt`, `gte`, `lt` and `lte` filters.
'lookups': [
LOOKUP_FILTER_RANGE,
LOOKUP_QUERY_IN,
LOOKUP_QUERY_GT,
LOOKUP_QUERY_GTE,
LOOKUP_QUERY_LT,
LOOKUP_QUERY_LTE,
],
},
'title': 'title.raw',
'publisher': 'publisher.raw',
'publication_date': 'publication_date',
'state': 'state.raw',
'isbn': 'isbn.raw',
'price': {
'field': 'price.raw',
# Note, that we limit the lookups of `price` field in this
# example, to `range`, `gt`, `gte`, `lt` and `lte` filters.
'lookups': [
LOOKUP_FILTER_RANGE,
LOOKUP_QUERY_GT,
LOOKUP_QUERY_GTE,
LOOKUP_QUERY_LT,
LOOKUP_QUERY_LTE,
],
},
'pages': {
'field': 'pages',
# Note, that we limit the lookups of `pages` field in this
# example, to `range`, `gt`, `gte`, `lt` and `lte` filters.
'lookups': [
LOOKUP_FILTER_RANGE,
LOOKUP_QUERY_GT,
LOOKUP_QUERY_GTE,
LOOKUP_QUERY_LT,
LOOKUP_QUERY_LTE,
],
},
'stock_count': {
'field': 'stock_count',
# Note, that we limit the lookups of `stock_count` field in
# this example, to `range`, `gt`, `gte`, `lt` and `lte`
# filters.
'lookups': [
LOOKUP_FILTER_RANGE,
LOOKUP_QUERY_GT,
LOOKUP_QUERY_GTE,
LOOKUP_QUERY_LT,
LOOKUP_QUERY_LTE,
],
},
'tags': {
'field': 'tags',
# Note, that we limit the lookups of `tags` field in
# this example, to `terms, `prefix`, `wildcard`, `in` and
# `exclude` filters.
'lookups': [
LOOKUP_FILTER_TERMS,
LOOKUP_FILTER_PREFIX,
LOOKUP_FILTER_WILDCARD,
LOOKUP_QUERY_IN,
LOOKUP_QUERY_EXCLUDE,
],
},
'tags.raw': {
'field': 'tags.raw',
# Note, that we limit the lookups of `tags.raw` field in
# this example, to `terms, `prefix`, `wildcard`, `in` and
# `exclude` filters.
'lookups': [
LOOKUP_FILTER_TERMS,
LOOKUP_FILTER_PREFIX,
LOOKUP_FILTER_WILDCARD,
LOOKUP_QUERY_IN,
LOOKUP_QUERY_EXCLUDE,
],
},
}
# Define ordering fields
ordering_fields = {
'id': 'id',
'title': 'title.raw',
'price': 'price.raw',
'state': 'state.raw',
'publication_date': 'publication_date',
}
# Specify default ordering
ordering = ('id', 'title', 'price',)
here is my urls.py:
from rest_framework_extensions.routers import ExtendedDefaultRouter
from .viewsets import BookDocumentView
from .views import titleQuery
router = ExtendedDefaultRouter()
books = router.register(r'books',
BookDocumentView,
base_name='bookdocument')
urlpatterns = [
url(r'^', include(router.urls)),
]
I have tried rebuilding all indexes too but it doesn't worked after that too.
Hi~ I ran into some problem when I made a custom search query. Need Your Help! In this case, I need the creator field in the search results according to my own logic as creator_indexing below.
By the way, creator is a foreign key in topic model.
Run command like this: python manage.py search_index --rebuild
Error is :
Traceback (most recent call last):
File "manage.py", line 22, in <module>
execute_from_command_line(sys.argv)
File "/virtualenv/env/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
utility.execute()
File "/virtualenv/env/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/virtualenv/env/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/virtualenv/env/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
output = self.handle(*args, **options)
File "/virtualenv/env/lib/python2.7/site-packages/django_elasticsearch_dsl/management/commands/search_index.py", line 134, in handle
self._rebuild(models, options)
File "/virtualenv/env/lib/python2.7/site-packages/django_elasticsearch_dsl/management/commands/search_index.py", line 115, in _rebuild
self._populate(models, options)
File "/virtualenv/env/lib/python2.7/site-packages/django_elasticsearch_dsl/management/commands/search_index.py", line 92, in _populate
doc().update(qs)
File "/virtualenv/env/lib/python2.7/site-packages/django_elasticsearch_dsl/documents.py", line 231, in update
self._get_actions(object_list, action), **kwargs
File "/virtualenv/env/lib/python2.7/site-packages/django_elasticsearch_dsl/documents.py", line 191, in bulk
return bulk(client=self.connection, actions=actions, **kwargs)
File "/virtualenv/env/lib/python2.7/site-packages/elasticsearch/helpers/__init__.py", line 257, in bulk
for ok, item in streaming_bulk(client, actions, **kwargs):
File "/virtualenv/env/lib/python2.7/site-packages/elasticsearch/helpers/__init__.py", line 180, in streaming_bulk
client.transport.serializer):
File "/virtualenv/env/lib/python2.7/site-packages/elasticsearch/helpers/__init__.py", line 58, in _chunk_actions
for action, data in actions:
File "/virtualenv/env/lib/python2.7/site-packages/django_elasticsearch_dsl/documents.py", line 214, in _get_actions
yield self._prepare_action(object_instance, action)
File "/virtualenv/env/lib/python2.7/site-packages/django_elasticsearch_dsl/documents.py", line 200, in _prepare_action
self.prepare(object_instance) if action != 'delete' else None
File "/virtualenv/env/lib/python2.7/site-packages/django_elasticsearch_dsl/documents.py", line 167, in prepare
instance, self._related_instance_to_ignore
File "/virtualenv/env/lib/python2.7/site-packages/django_elasticsearch_dsl/fields.py", line 121, in get_value_from_instance
instance, field_value_to_ignore
File "/virtualenv/env/lib/python2.7/site-packages/django_elasticsearch_dsl/fields.py", line 70, in get_value_from_instance
"{!r}".format(attr, instance)
django_elasticsearch_dsl.exceptions.VariableLookupError: Failed lookup for key [creator_indexing] in <Topic: This is FIFA World Cup!>
[topic Document Part]
creator = fields.ObjectField(
attr='creator_indexing',
properties={
'auth_user': fields.ObjectField(
properties={
'id': fields.IntegerField(),
'username': fields.StringField(analyzer='keyword'),
}
),
'nickname': fields.StringField(
analyzer=html_strip,
fields={
'raw': fields.StringField(analyzer='keyword'),
}
),
'avatar': fields.StringField(
analyzer=html_strip,
fields={
'raw': fields.StringField(analyzer='keyword'),
}
),
'gender': fields.IntegerField(),
'chataddr': fields.StringField(
analyzer=html_strip,
fields={
'raw': fields.StringField(analyzer='keyword'),
}
),
}
)
[topic Model Part]
@property
def creator_indexing(self):
"""
creator for indexing.
"""
creator_dict = {
'auth_user': {
'id': self.creator.auth_user.id,
'username': 'xxxxxxx',
},
'nickname': 'yyyyyyyy',
'avatar': '/static/img/m.png',
'gender': self.creator.gender,
'chataddr': 'chat_dddxxx222',
}
if gender == 1:
creator_dict['avatar'] = '/static/img/f.png
wrapper = dict_to_obj(creator_dict)
return wrapper
The queryset.count() should be done after executing the query.
Since what's going to happen most of the time it's that you get a single result, I don't see why it's necesary to avoid doing the query when you're going to get more than one result, when you can check this later and avoid doing the count query.
Great job on this package, btw.
I've had a look at DocumentSerializer and can't find any serialization for elasticsearch-dsl's ObjectField representation of Object
I've had a look at the implementation to see if I could come up with a quick solution but I'm not readily seeing anything.
Let me know if there's an easy way to do this or if it needs some work then point me in the right direction and I'll try to submit a PR.
When I declare something like this
filter_fields = {
'foobar': {
'field': 'foobar',
'lookups': [
LOOKUP1,
LOOKUP2,
# etc ..
],
},
}
My lookups seem to be ignored, what is used instead is the content of ALL_LOOKUP_FILTERS_AND_QUERIES
here.
It is due to this line, maybe it should be lookups
instead of filters
.
I can fix it if you confirm it is actually a bug.
Similar to Django-filter's MultipleChoiceFilter providing a conjoined=True
property.
Reference https://django-filter.readthedocs.io/en/master/ref/filters.html#multiplechoicefilter
I need compatibility with django rest framework query-string behaviour, which allows me to do ?field=value&field=value2
and have it return only results where field has both value
AND value2
.
Hi,
Just curious: when will there be support for ES 6.x? I have already based my latest application on it.
Kind regards,
Bas
This is an enhancement suggestion. In https://github.com/barseghyanartur/django-elasticsearch-dsl-drf/blob/master/basic_usage_examples.rst
In the example query http://127.0.0.1:8080/search/publisher/?city__in=yerevan|amsterdam
I was thinking http://127.0.0.1:8080/search/publisher/?city__in=yerevan,amsterdam
feels more readable to me ! do you see any conflicts if we use " , " instead of | .
In the example http://127.0.0.1:8080/search/publisher/?search=name|reilly&search=city|london
how about http://127.0.0.1:8080/search/publisher/?search=name:reilly&search=city:london
I'm not familiar in elasticsearch, but in solr we use field:value
patterns in search, Im assuming that syntax would be more user friendly and already established syntax.
context of pipe in both the places feels bit confusing for me. If you think we can change that without breaking any features. I can pitch-in to fix that .
UPDATE:
I can see comma as a seperator doesn't work in this scenario http://localhost:8000/search/publishers/?location__geo_polygon=40,-70|30,-80|20,-90
After building document I am running
python manage.py search_index --create -f
python manage.py search_index --populate -f
python manage.py search_index --rebuild -f
commands but it is not showing any output on screen commands simply runs and exits from command and doesn't shows any output on screen. Beside rebuild only shows an output for choosing yes or no.
IT SHOWS :
Are you sure you want to delete the '' indexes? [n/Y]:
I am using,
certifi==2018.1.18
chardet==3.0.4
coreapi==2.3.3
coreschema==0.0.4
coverage==4.4.1
dj-database-url==0.4.2
dj-email-url==0.0.10
Django==1.11.1
django-coverage-plugin==1.4.2
django-elasticsearch-dsl==0.4.4
django-elasticsearch-dsl-drf==0.7.1
django-filter==1.0.2
django-nine==0.1.13
django-rest-swagger==2.1.2
djangorestframework==3.6.3
elasticsearch==5.5.2
elasticsearch-dsl==5.1.0
gunicorn==19.7.1
idna==2.6
ipaddress==1.0.21
itypes==1.1.0
Jinja2==2.10
Logentries==0.17
MarkupSafe==1.0
openapi-codec==1.3.2
python-dateutil==2.7.2
pytz==2018.4
requests==2.18.4
simplejson==3.13.2
six==1.11.0
uritemplate==3.0.0
urllib3==1.22
whitenoise==3.3.0
these versions also i have turned on elasticsearch 5.5.2 server on localhost 9200 and added these configs in my settings.py.
Help me if you have any idea why am I facing this issue.
Goal
Implement query_string
and simple_query_string
search backends.
Problem
At the moment, in the constants
module the folliwing separators are defined:
# Lookup separator
SEPARATOR_LOOKUP_FILTER = '__'
# Lookup filter value separator. To be used for `terms` and `range` filters
# lookups.
SEPARATOR_LOOKUP_VALUE = '|'
# Lookup filter value complex separator. To be used with geo-spatial features.
SEPARATOR_LOOKUP_COMPLEX_VALUE = ':'
SEPARATOR_LOOKUP_COMPLEX_MULTIPLE_VALUE = ','
Alternative values for SEPARATOR_LOOKUP_VALUE
and SEPARATOR_LOOKUP_COMPLEX_VALUE
shall be found, since they clash with own separators of Elastic.
Proposal
SEPARATOR_LOOKUP_NAME
shall most likely become :
SEPARATOR_LOOKUP_VALUE
shall most likely be deprecated in favour of SEPARATOR_LOOKUP_COMPLEX_VALUE
SEPARATOR_LOOKUP_COMPLEX_VALUE
shall become __
References
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html
Things seem to work
In branch change-separators.
Sample requests
How it was:
http://localhost:8000/api/users/?age__range=16|67|2.0
What will it likely become:
http://localhost:8000/api/users/?age__range=16__67__2.0
Another example how it was:
http://localhost:8000/api/articles/?location__geo_polygon=40.73,-74.1|40.01,-71.12|_name,myname|validation_method,IGNORE_MALFORMED;type,indexed
What will it likely become:
http://localhost:8000/api/articles/?location__geo_polygon=40.73,-74.1__40.01,-71.12___name,myname__validation_method,IGNORE_MALFORMED;type,indexed
The latter is certainly less readable, but who cares as long as it works?
lib/python3.6/site-packages/django_elasticsearch_dsl_drf/pagination.py", line 13, in <module>
from rest_framework.pagination import _get_count
ImportError: cannot import name '_get_count'
Implement geo-spatial search/filtering/ordering filter backend.
It appears a bug was introduced in this commit b397c49
line 119 - field, value = __values
needs to be outside the if statement.
Errors : with field is used before assignment
regards
--Tolu
Implement aggregations backend (rethink faceted search backend).
As continuation of the #4
Elasticsearch supports two types of geo data:
- geo_point fields, which support lat/lon pairs.
- geo_shape fields, which support points, lines, circles, polygons,
multi-polygons etc.
The queries in this group are:
- geo_shape query: Find document with geo-shapes which either intersect,
are contained by, or do not intersect with the specified geo-shape.
+ geo_bounding_box query: Finds documents with geo-points that fall into
the specified rectangle.
+ geo_distance query: Finds document with geo-points within the specified
distance of a central point.
* geo_distance_range query: Like the geo_distance query, but the range
starts at a specified distance from the central point.
+ geo_polygon query: Find documents with geo-points within the specified
polygon.
Where:
- : Not yet implemented
* : Won't be implemented. We're not going to support deprecated
features (such as geo-distance-range).
+ : Already implemented.
All other features (marked as -) are meant to be supported.
I have gone through the documentation and was implementing django-elasticsearch-dsl-drf on my project.
I have created documents but
./manage.py search_index command is not working.
It's showing
"""""""""""""
Unknown command: 'search_index'
Type 'manage.py help' for usage.
""""""""""""
in output console.
PS: I have already installed rest framework, elasticsearch-dsl and django-elasticsearch-dsl-drf
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.