Course search not working about devstack HOT 10 CLOSED

Fixing this will most likely require us to update the edx/edx-search package. A version supporting ES 1.x has not yet been published; although, the code is completed. Additionally, the package will need to be updated on LMS.

from devstack.

I did some experimenting today, and I have no clue how this ever made it to production. The simple act of creating the index is not documented anywhere (specifically http://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/latest/configuration/edx_search.html)! After creating the indices manually, and upgrading to edx-search 1.0.1, I was unable to reindex the courses.

curl -XPUT 'edx.devstack.elasticsearch:9200/courseware_index'
./manage.py cms reindex_course --all --settings=devstack_docker --setup

I have disabled this feature for the time being: https://github.com/edx/edx-platform/pull/14651. While I would love to fix it, doing so should be done by someone who knows the magic incantations to make it work!

from devstack.

@clintonb edx-search is only used on openedx installs. As such, it's relatively unmaintained. The person who wrote it no longer works here. I think Andy or Robert are the closer you have to a SME if you want thumbs on that PR.

from devstack.

@andy-armstrong @robrap I am merging https://github.com/edx/edx-platform/pull/14651 to get us moving along. Please advise on fixing this if it is a priority.

from devstack.

Thanks for fixing this, and for the heads up, @clintonb. As @jibsheet said, course-search is not something that anyone has looked at since Marty left. It is on @marcotuts's radar to try to enable at some point in the future, but there's no timeline. @robrap has been doing the necessary work to upgrade the ElasticSearch usage so he may have more to say.

from devstack.

@clintonb I'm not really clear on what makes this issue appear. edx/edx-search already has a 1.0.1 release version which supports 1.x ES, but it is not yet in Production. This is the PR that upgrades edx-platform: https://github.com/edx/edx-platform/pull/14463/files.

When we tested on a sandbox, course search seemed to work on that sandbox. @dianakhuang did the configuration, so should could explain more if needed.

Is this a devstack specific issue? I don't think we have tested it on devstack yet, and it would be good to know if we have a problem coming soon. However, one solution would be to disable this by default on all devstacks, and either the community can help resolve, or let it wait until we (edX) makes course search a priority.

from devstack.

@robrap this issue is specific to Docker-based devstack. I haven't tried with Vagrant-based devstack. I suspect it's a configuration error, but there isn't enough documentation around this functionality to figure out where the error is.

@dianakhuang if you can share what values need to be set where, please do. No rush, however, as fixing this is not a high priority for anyone (just a nice-to-have).

from devstack.

@dianakhuang is out. I think she just used the Read the Docs page you had posted:
http://edx.readthedocs.io/projects/edx-installing-configuring-and-running/en/latest/configuration/edx_search.html#supported-flags

from devstack.

I did some digging this morning, and discovered that the index is installed at the moment the SearchEngine wrapper class is instantiated: https://github.com/edx/edx-search/blob/master/search/elastic.py#L275-L276.

The paver devstack task is responsible for calling a management command that attempts to reindex the courses, and subsequently creates the index: https://github.com/edx/edx-platform/blob/d3b873c7e9ec85247b901bf988e295dbfc816903/pavelib/servers.py#L141-L141.

Additional settings: https://github.com/edx/edx-platform/blob/480a3ca60b30f9a4970094c9e84316fce8c5b631/cms/envs/aws.py#L452-L454.

Ensuring the port is an integer will solve the connection issues; however, that is not easily done with Ansible. See https://github.com/edx/configuration/pull/3887 for details.

The simplest solution may be to pass the combined host and port (e.g Elasticsearch(host='edx.devstack.elasticsearch:9200') instead of Elasticsearch(host='edx.devstack.elasticsearch' port=9200)). This format should be supported: https://elasticsearch-py.readthedocs.io/en/master/api.html#elasticsearch.

from devstack.

https://openedx.atlassian.net/browse/PLAT-1668

from devstack.