Code Monkey home page Code Monkey logo

gatsby-plugin-meilisearch's Introduction

Meilisearch Gatsby

Gatsby plugin Meilisearch

Bors enabled Tests License


A plugin to index your Gatsby content to Meilisearch based on graphQL queries


Table of Contents

πŸ“– Documentation

To understand Meilisearch and how it works, see the Meilisearch's documentation.

To understand Gatsby and how it works, see Gatsby's documentation.

⚑ Supercharge your Meilisearch experience

Say goodbye to server deployment and manual updates with Meilisearch Cloud. Get started with a 14-day free trial! No credit card required.

πŸ”§ Installation

Inside your Gatsby app, add the package:

With npm:

npm install gatsby-plugin-meilisearch

With yarn:

yarn add gatsby-plugin-meilisearch

πŸƒβ€β™€οΈ Run Meilisearch

There are many easy ways to download and run a Meilisearch instance.

For example, if you use Docker:

docker pull getmeili/meilisearch:latest # Fetch the latest version of Meilisearch image from Docker Hub
docker run -it --rm -p 7700:7700 getmeili/meilisearch:latest meilisearch --master-key=masterKey

With this command, your Meilisearch instance host is http://localhost:7700 and your master key is masterKey

πŸš€ Run Gatsby

If you don't have a running Gatsby, you can either launch the [playground present in this project)(./playground/README.md) or create a Gatsby project.

Run your app if it is not running yet:

gatsby develop

Now that your Gatsby app is running you have access to the following URLs:

  • http://localhost:8000/ URL of your web app.
  • http://localhost:8000/___graphql: URL to the GraphiQL tool where you can build graphQL queries on the playground and request them.

🎬 Getting started

Now you should have a running Gatsby app with gatsby-plugin-meilisearch installed and a running Meilisearch instance.

Let's configure our plugin to make it work! In this example, we will fetch every page's URL of our Gatsby website, and index them to Meilisearch.

To make the plugin work, open the gatsby-config.js configuration file located at the root of your Gatsby project. All the configuration takes place in that file.

βš™οΈ Configure your plugin options

πŸ”‘ Add your Meilisearch credentials

First, you need to add your Meilisearch credentials.

The credentials are composed of:

⚠️ Keys with permissions other than search should never be used on your front end. For searching, use the Default Search Key key available on the key route or create a custom API key with only search rights.

Add the credentials the following way in your gatsby-config.js file:

{
  plugins: [
    {
      resolve: 'gatsby-plugin-meilisearch',
      options: {
        host: 'http://localhost:7700',
        apiKey: 'masterKey',
      },
    },
  ]
}

See this section if you don't know what your credentials are.

☝️ Fill in the indexes field

The next step is to define which data we want to add in Meilisearch and how. This happens in the indexes field.

The indexes field is an array that can be composed of multiple index objects. Each index object contains the following information:

indexUid: The name of the index in which the data is added.

Let's define the index uid to pages_url. On build, the pages_url index is created inside Meilisearch.

indexUid: 'pages_url'

if pages_url already existed, it is deleted and recreated on build

query: GraphQL query fetching the data to add in Meilisearch

Let's provide the graphQL query that retrieves the URL's of the pages of our application.

query: `
  query MyQuery {
    allSitePage {
      nodes {
        path
      }
    }
  }
`,

After executing this query, we receive a data object containing the following:

{
  data: {
    allSitePage: {
      nodes: [
        {
          path: '/404/'
        },
        {
          path: '/404.html'
        },
        {
          path: '/'
        }
      ]
    }
  }
}

transformer: Transform the data fetched to a format compatible to Meilisearch.

Now that we have fetched the data with the query field, it is not yet ready to be sent to Meilisearch.

Using a transformer function, we can transform the fetched data to a compatible format.

The first problem of the fetched data is that the documents to send to Meilisearch are nested, while they should be at the root in an array. So the content of nodes should be at the root.

{
  data: {
    allSitePages: {
     nodes: [
       {
        'path': '/404/'
      },
     ]
    }
  }
}

should become:

[
  {
    'path': '/404/'
  },
  {
    'path': '/'
  },
]

The second problem is that each document in Meilisearch requires an unique indentifier called primary key.

Thus every document needs a unique field called id. For example:

{
  'id': 1
  'path': '/404/'
},

To do so, we need to use the transformer method to create the final compatible array of objects:

{
  transformer: data =>
    data.allSitePage.nodes.map((node, index) => ({
      id: index,
      ...node,
    })),
}

In this function, we map on data.allSitePage.nodes in order to return an array of objects that can be indexed by Meilisearch. We add an id field as Meilisearch needs it for the indexation. As we don't have any field here that can be used as an id, we use the index of the current element in the array.

If you want to learn more about these options (indexUid, query and transformer) see indexes options

πŸŽ‰ Complete configuration

After filling in those fields, your Meilisearch configuration should look like this:

plugins: [
  {
    resolve: 'gatsby-plugin-meilisearch',
    options: {
      host: 'http://localhost:7700',
      apiKey: 'masterKey',
      indexes: [
        {
          indexUid: 'pages_url',
          transformer: (data) =>
            data.allSitePage.nodes.map((node, index) => ({
              id: index,
              ...node,
            })),
          query: `
            query MyQuery {
              allSitePage {
                nodes {
                  path
                }
              }
            }
          `,
        },
      ],
    },
  },
];

πŸ₯ Build your project

The gatsby-plugin-meilisearch fetches and adds your data to Meilisearch on your Gatsby build.

gatsby build

After the build, a message in your terminal confirms that your content was successfully indexed:

success gatsby-plugin-meilisearch - x.xxxs - Documents added to Meilisearch

πŸͺ„ Integrate search components

If you need tools to integrate a search experience on your app, we have tools that might help you:

  • docs-searchbar: a tool to display a searchbar on your website
  • meilisearch-react: a React UI library that lets you quickly build a search interface in your front-end application

πŸ›Ό Usage

In the gatsby-config.js file, the Meilisearch plugin accepts the following options:

host (required)

The host field is the address where your Meilisearch instance is running. gatsby-plugin-meilisearch needs it in order to communicate with your Meilisearch instance, and send your data to it.

apiKey (optional)

The apiKey field contains the API key if the Meilisearch instance is password protected.

skipIndexing (optional)

This option allows you to build your website without indexing to Meilisearch. Default to false

batchSize (optional)

The number of documents that should be included in each batch. Default to 1000

settings (optional)

If you want to pass settings to your Meilisearch instance, you can do it here. Read more about Meilisearch settings

indexes (required)

The indexes field is an array of objects, each of them represents how to add data to a specific index

You can have one or multiple index objects in indexes, which can be useful if you want to index different content in different indexes (or multiple different data to the same index).

Each index object should contain the following fields:

indexUid (required)

This is the name of your Meilisearch index. This is a required field as it is where the retrieved data is added inside Meilisearch. For example if your indexUid is pages_url, your content will be indexed inside the pages_url index in Meilisearch. If you provide an index name that already exists, the index will be deleted and recreated.

Example:

indexUid: 'pages_url'

You can learn more about indexes on our documentation.

query (required)

This is the graphQL query that will be executed in order to retrieve your data. Your query can be very specific depending on the plugins you're using. If you're not sure about your query, you can use the GraphiQL tool (http://localhost:8000/\_\_\_graphql) provided by Gatsby on development mode to help you build it.

Example:

query: `
  query MyQuery {
    allSitePage {
      nodes {
        path
      }
    }
  }
`,

You can also check our playground's configuration file to have an example of a graphQL query using the gatsby-plugin-mdx plugin.

transformer (required)

This is a function that transforms the fetched data before sending it to Meilisearch.

After executing the graphQL query, a data object is received with a structure that can differ from one project to another, depending on the query you provided. As Meilisearch requires a unique identifier at the root of each document and it should avoid nested objects, you will need to transform your data object accordingly. The transformer function is the correct place to do so.

Example:

transformer: data =>
  data.allSitePage.nodes.map((node, index) => ({
    id: index,
    ...node,
  })),

Without using the transformer function, the data will look like this:

{
  data: {
    allSitePage: {
      nodes: [
        {
          path: '/404/'
        },
        {
          path: '/404.html'
        },
        {
          path: '/'
        }
      ]
    }
  }
}

After using the transformer function as in the above example, the data will look like this, and will be ready for indexation:

[
  {
    id: 0,
    path: '/404/',
  },
  {
    id: 1,
    path: '/404.html',
  },
  {
    id: 2,
    path: '/',
  },
];

If you want to learn more about Meilisearch's documents structure, you can do so in our documentation.

Full usage example:

{
  resolve: 'gatsby-plugin-meilisearch',
  options: {
    host: 'http://localhost:7700',
    apiKey: 'masterKey',
    skipIndexing: false,
    batchSize: 1000,
    options: {
      settings: {
        searchableAttributes: ["*"],
      },
    },
    indexes: [
    {
      indexUid: 'my_index',
      transformer: () => {},
      query: ``
    },
  ],
}

πŸ€– Compatibility with Meilisearch and Gatsby

Supported Gatsby versions:

  • Gastby v4.3.x

(This plugin may work with the older Gatsby versions, but these are not tested nor officially supported at this time.)

Supported Meilisearch versions:

This package guarantees compatibility with version v1.x of Meilisearch, but some features may not be present. Please check the issues for more info.

Node / NPM versions:

  • NodeJS >= 14.15.X && <= 16.X
  • NPM >= 6.x

We recommend always using the latest version of Gatsby to start your new projects.

βš™οΈ Development Workflow and Contributing

Any new contribution is more than welcome in this project!

If you want to know more about the development workflow or want to contribute, please visit our contributing guidelines for detailed instructions!

gatsby-plugin-meilisearch's People

Contributors

bidoubiwa avatar bors[bot] avatar brunoocasali avatar curquiza avatar denke8 avatar dependabot[bot] avatar mdubus avatar meili-bors[bot] avatar meili-bot avatar tommasoamici avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

gatsby-plugin-meilisearch's Issues

Current example in the README is not much of use

Other

Current example on README create index generated pages by path, this is less useful, why?

  • Users does not want to search queries that match the slug, they want the pages where content matches with their queries.

I know what is shown in README is a dummy example, but it is worth to have real life example like product search. Example using Medusa would be great.

Allow multi-indexing

Transform the indexes option into an array, and allow multi-indexing to be done.
Make tests to make sure multi-indexing is correctly done

Cannot read properties of undefined (reading 'length')

I tried to do an Indexes using gatsby-plugin-meilisearch with a yarn build, but I got the following error
Anyone else with the same error?

Error


 ERROR 

Cannot read properties of undefined (reading 'length')

success gatsby-plugin-meilisearch - 0.054s - Failed to index to Meilisearch

package.json:

{
  "name": "test",
  "version": "1.0.0",
  "private": true,
  "description": "test",
  "author": "tukimi",
  "keywords": [
    "gatsby"
  ],
  "scripts": {
    "dev": "gatsby develop",
    "start": "gatsby develop",
    "build": "gatsby build",
    "serve": "gatsby serve",
    "clean": "gatsby clean"
  },
  "dependencies": {
    "@hcaptcha/react-hcaptcha": "^1.4.1",
    "@headlessui/react": "^1.6.5",
    "axios": "^0.27.2",
    "daisyui": "^2.17.0",
    "gatsby": "^4.17.1",
    "gatsby-plugin-image": "^2.17.0",
    "gatsby-plugin-manifest": "^4.17.0",
    "gatsby-plugin-meilisearch": "^0.1.2",
    "gatsby-plugin-sharp": "^4.17.0",
    "gatsby-plugin-sitemap": "^5.17.0",
    "gatsby-remark-component": "^1.1.3",
    "gatsby-remark-images": "^6.17.0",
    "gatsby-remark-relative-images": "^2.0.2",
    "gatsby-source-filesystem": "^4.17.0",
    "gatsby-transformer-remark": "^5.17.0",
    "gatsby-transformer-sharp": "^4.17.0",
    "react": "^18.1.0",
    "react-dom": "^18.1.0",
    "react-helmet": "^6.1.0",
    "rehype-react": "^7.1.1"
  },
  "devDependencies": {
    "autoprefixer": "^10.4.7",
    "gatsby-plugin-postcss": "^5.17.0",
    "postcss": "^8.4.14",
    "tailwindcss": "^3.1.4"
  }
}

gatsby-config.js

module.exports = {
  siteMetadata: {
    title: `test`,
    siteUrl: `https://example.com`
  },
  plugins: [
    "gatsby-plugin-image",
    "gatsby-plugin-sitemap",
    "gatsby-plugin-postcss",
    {
      resolve: 'gatsby-plugin-manifest',
      options: {
        "icon": "./src/images/symbol.png"
      }
    },
    "gatsby-plugin-sharp",
    {
      resolve: "gatsby-transformer-remark",
      options: {
        plugins: [
          "gatsby-remark-relative-images",
          {
            resolve: "gatsby-remark-images",
            options: {
              maxWidth: 840,
              linkImagesToOriginal: false,
            }
          },
          {
            resolve: "gatsby-remark-component",
            options: { components: ["alert", "warning", "info"] }
          }
        ]
      }
    },
    "gatsby-transformer-sharp",
    {
      resolve: 'gatsby-source-filesystem',
      options: {
        "name": "images",
        "path": "./src/images/"
      },
      __key: "images"
    },
    {
      resolve: 'gatsby-source-filesystem',
      options: {
        "name": "pages",
        "path": "./src/pages/"
      },
      __key: "pages"
    },
    {
      resolve: 'gatsby-source-filesystem',
      options: {
        "name": "article",
        "path": `${__dirname}/article/`
      }
    },
    {
      resolve: 'gatsby-plugin-meilisearch',
      options: {
        host: 'http://localhost:7700',
        apiKey: 'masterKey',
        indexes: [
          {
            indexUid: 'testIndex',
            transformer: (data) => {
              data.allMarkdownRemark.edges.map(({ node }) => {
                return {
                  id: node.id,
                  title: node.frontmatter.title,
                  content: node.rawMarkdownBody,
                  slug: '/articles' + node.fields.slug,
                }
              })
            },
            query: `
            query IndexQuery {
              allMarkdownRemark {
                edges {
                  node {
                    id
                    rawMarkdownBody
                    frontmatter {
                      title
                    }
                    fields {
                      slug
                    }
                  }
                }
              }
            }
            `,
          },
        ],
      },
    }
  ],
  trailingSlash: "never"
};

Should the plugin become compatible with Gatsby v5 ?

Other

Currently the plugin is compatible with Gatsby v4. Migrating to v5 requires more than changing the version. It implies a whole migration.

Also, once the plugin is compatible v5, it will probably not be compatible v4 anymore. Do we want that?

This decision depends on the needs of our users. If this issue receives enough upvotes, a migration to v5 might be considered

Add error code at the end of the error message

Other

In order to have clear error messages, we should display the error message returned by MeiliSearch but also the error code. It could be of the form ${error.message} (${error.code})

Create Playground

Create a playground inside the project, with the following:

  • gatsby project with blog articles
  • Searchbar with Instant MeiliSearch to test that the indexation was successful

Also make sure that eslint & prettier works well with the playground

Create MVP

The plugin will have the following prototype:

module.exports = {
  siteMetadata: {
    title: "My First Gatsby Site",
  },
  plugins: [
    {
      resolve: `gatsby-plugin-meilisearch`,
      options: {
        host: 'http://127.0.0.1:7700',
        apiKey: 'masterKey',
        skipIndexing: true,
        queries: {
          indexUid: 'MyBlog',
          query: `
            query MyQuery {
              allMdx {
                edges {
                  node {
                    frontmatter {
                      title
                    }
                    tableOfContents
                  }
                }
              }
            }
          `,
        },
      },
    },
  ],
};

The MVP will be ready for release after doing the following:

The following also has to be done (during the development process):

Custom clients / proxy settings are missing.

Description
I work behind proxy server and I can't configure plugin to work with it. In meilisearch-js library you can specify custom client for MeiliSearch. This gatsby plugin doesn't have httpClient parameter for custom clients.

Expected behavior
Be able to specify custom client/proxy.

Current behavior
No such parameter.

Bind playground with the plugin

Bind the playground with the plugin (empty by default). We have to make sure that editing the plugin files refreshes the playground

Also make sure that eslint & prettier works well with the plugin

Add Cypress

Add Cypress and do the following:

  • Enable watch mode
  • Run the playground and make sure its title is present
  • Make sure that the search bar is present
  • Update CONTRIBUTING.md

error timeout of 5000ms has exceeded on process 3 when waiting a task to be resolved.

Hi - we are using this plugin (which is fantastic) with Gatsby however we've started getting this issue only in CI (Gitlab). If I run the build locally on my Mac it is fine which would suggest some kind of resourcing issue?

success Building static HTML for pages - 39.268s - 2714/2714 69.12/s
error timeout of 5000ms has exceeded on process 3 when waiting a task to be resolved.
success gatsby-plugin-meilisearch - 9.715s - Failed to index to Meilisearch
success onPostBuild - 10.556s

A similar issue (meilisearch/strapi-plugin-meilisearch#337) talks about auto-batching and changing the timeout. Auto-batching is apparently enabled now and there doesn't seem to be any option for changing the timeout with the plugin. I've tried different batchSize settings to no avail. We have approx 2,700 documents to index so not a massive amount.

We run the docker version of Meilisearch (v029.2) in ECS Fargate.

"gatsby": "4.14.1",
"gatsby-plugin-meilisearch": "^0.2.1",

Is there a way to change the timeout or can you advise how I can troubleshoot?

Many thanks.

Send the query result to MeiliSearch

Send the query result to MeiliSearch:

  • retrieve options to get the host, API key, and indexName
  • use MeiliSearch JS to index the results of the query previously made

Bug: Dependency on Meilisearch API Header : x-meili-api-key not found

Current behavious

image

When trying to build the playground project, we get the following error of The X-MEILI-API-KEY header is missing.

Expected Behavior

Should be able to build the Gatsby project and index to Meilisearch

Steps To Reproduce

In playground directory,
Run : npm install
npm run build

Anything else?

I used a cloud.meilisearch.com instance with the version of Meilisearch as v0.24.0.

No response

Test job ran in CI is stuck on waiting for the web server to run

Other

For a reason we ignore, the test in the CI started to fail on the 12 July 2022. Here is the last test that was successful.

This is the test command that used to work before the 12th:

"concurrently --kill-others -s first \"yarn playground:start\" \"wait-port http://localhost:8000/___graphql && jest --testPathPattern=tests\"",

Using the concurrently tool the following is happening

  • Kill any server that are still running: --kill-others
  • Run the start command of the web server -s first "yarn playground:start"
  • Run wait-port that waits on the playground to serve localhost:8000/__graphql
  • Run the tests once the above route is served.

This used to work perfectly.

For some reason that we are still trying to figure out, the CI is stuck after the start of the playground (see failing CI).

[0] You can now view playground in the browser.
[0] β €
[0]   http://localhost:8000/
[0] β €
[0] View GraphiQL, an in-browser IDE, to explore your site's data and schema
[0] β €
[0]   http://localhost:8000/___graphql
[0] β €
[0] Note that the development build is not optimized.
[0] To create a production build, use gatsby build
[0] β €
[0] success Building development bundle - 17.638s
[0] success Writing page-data.json files to public directory - 0.076s - 9/9 118.[61](https://github.com/meilisearch/gatsby-plugin-meilisearch/runs/7397680361?check_suite_focus=true#step:7:62)/s 
## STUCK HERE
Error: The operation was canceled.

This part Error: The operation was canceled. was triggered by me clicking on cancel workflow. If I do not click on it, the CI ran for hours.

A possibility is that wait-for is listening for localhost:8000 to be served but it never happens. Because it does not happen, the tests are not triggered as well.

The best theory that we have now is that github changed the way they handle the github actions. Unfortunately I was not able to find any information on the subject.

Temporary solution

We removed the concurrently part and the wait-for with a more straightforward approach

  • Start the playground in the background
  • Wait for 30 seconds
  • Run the tests

This solution is absolutely not something we would like to keep.
Any help on finding out the issue is greatly appreciated πŸ™

Add the `skipIndexing` option

Retrieve the skipIndexing (true/false) from the options, and use it to determine if the indexation has to be done or not.
This can be usefull for example if a user is in development mode, and doesn't want to index the changes he just made.

Create chore of project

Initialize the project by adding the following:

  • package.json
  • gitignore
  • .editorconfig
  • bors
  • dependabot
  • release-drafter
  • eslint
  • prettier
  • licence file
  • readme
  • issue template
  • test.yml to test the linting

Allow multi queries on same index

After #15, the config file will look like this. You will be able to create multiple indexes with each their own graphQL query.

{
  "options" : {
    host: '...'
    // ...
    indexes: [
      {
        indexUid: 'my_blog'
        // ...
        query: ` ... `
      }
    ]
  }
}

This implies that for every index created in MeiliSearch you can provide one graphQL query.
As this seems a bit limitated since you can have multiple page structures but you'd like to add them all in the same index, we want to change query to queries. That would accept an array of queries.

So instead of for example:

query: `
  query MyQuery {
    allMdx {
      // ....  
    }
  }
`,

It becomes:

queries: [`
  query MyQuery {
    allMdx {
      // ....  
    }
  }
`],

This changes the code here:

// Prepare data for indexation
const transformedData = await queries.transformer(data)

Where you will have to iterate on the queries to fetch all data.

Remove support for node 12 and ensure support for node 18

Node version compatibilities should be changed in the tests and the readme.

See related issue.

Previous ensured compatibilities:

  • Node 12
  • Node 13
  • Node 14

New compatibilities to ensure:

  • Node 14
  • Node 16
  • Node 18

TODO:

  • Update requirements/compatibilities in README.md.
  • Update node versions tested in CI tests.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.