This project covers serch use cases on Harry Potter text databases, with a focus on python integrations.

Create a index where each document is a Harry Potter character with their attributes. This index can them be used to create customized search queries to identify subsets of characters with particular properties.

This example project covers the basic introductory concepts of elasticsearch and kibana.

Introduce the python client to communicate with the Elasticsearch engine via code. Create an index from the first Harry Potter movie script to use fore more complex, natural language queries. Use Hugging Face models to add Sentiment Analysis and Embeddings for Semantic Search. Combine multiple models for hybrid search; compare to the native functionality of ELSER (knn Search).

Build a simple Flask APP as a User Interface for search Introduce a new index to store historical searches as they are ran - we can use this for observability & tracking. Separate some helper_functions that we can reuse.

• HP characters index & search
• HP characters index - python client interface for search
• HP sentiment analysis on movie subtitles
• Embeddings and semantic search with ELSER
• Python-DSL client
• Web APP with Flask
• Observability & Monitoring

Requirements Installation of Elasticsearch (either local or on cloud) see docs

For python environment, recommend to set up a virtual environment see docs. Requirements: pandas.

Python notebook for some essential data cleaning with pandas dataframes.

Instructinos for adding data to the elastic cluster.

Short intro to Dashboards and visualizations in Kibana.

Short intro to Discover and KQL.

Working with Console / dev tools, intro to data types in elastic.

Building requests and intro to queries.

Working with the python client to build an index and mapping, bulk ingest documents, and run queries.

Use the Eland client to import models from Hugging Face and run Sentiment Analysis on the data

Create embeddings for semantic (natural language) search

Compare with the ELSER model built by Elastic

See blog for our Advent Calendar here

Using Flask for a simple user interface allowing users to search (for a live demo). Added historical query tracking in a new index to later use for observability.