Code Monkey home page Code Monkey logo

laravel-scout-postgres's Introduction

PostgreSQL Full Text Search Engine for Laravel Scout

Latest Version on Packagist Software License tests StyleCI Total Downloads License

This package makes it easy to use native PostgreSQL Full Text Search capabilities with Laravel Scout.

If you find this package usefull, please consider bying me a coffee.

Buy Me a Coffee at ko-fi.com

Contents

Installation

You can install the package via composer:

composer require pmatseykanets/laravel-scout-postgres

Laravel

If you're using Laravel < 5.5 or if you have package auto-discovery turned off you have to manually register the service provider:

// config/app.php
'providers' => [
    ...
    ScoutEngines\Postgres\PostgresEngineServiceProvider::class,
],

Lumen

Scout service provider uses config_path helper that is not included in Lumen. To fix this include the following snippet either directly in bootstrap.app or in your autoloaded helpers file i.e. app/helpers.php.

if (! function_exists('config_path')) {
    /**
     * Get the configuration path.
     *
     * @param  string  $path
     * @return string
     */
    function config_path($path = '')
    {
        return app()->basePath() . '/config'.($path ? DIRECTORY_SEPARATOR.$path : $path);
    }
}

Create the scout.php config file in app/config folder with the following contents

<?php

return [
    'driver' => env('SCOUT_DRIVER', 'pgsql'),
    'prefix' => env('SCOUT_PREFIX', ''),
    'queue' => false,
    'pgsql' => [
        'connection' => 'pgsql',
        'maintain_index' => true,
        'config' => 'english',
    ],
];

Register service providers:

// bootstrap/app.php
$app->register(Laravel\Scout\ScoutServiceProvider::class);
$app->configure('scout');
$app->register(ScoutEngines\Postgres\PostgresEngineServiceProvider::class);

Configuration

Configuring the Engine

Specify the database connection that should be used to access indexed documents in the Laravel Scout configuration file config/scout.php:

// config/scout.php
...
'pgsql' => [
    // Connection to use. See config/database.php
    'connection' => env('DB_CONNECTION', 'pgsql'),
    // You may want to update index documents directly in PostgreSQL (i.e. via triggers).
    // In this case you can set this value to false.
    'maintain_index' => true,
    // You can explicitly specify what PostgreSQL text search config to use by scout.
    // Use \dF in psql to see all available configurations in your database.
    'config' => 'english',
    // You may set the default querying method
    // Possible values: plainquery, phrasequery, tsquery
    // plainquery is used if this option is omitted.
    'search_using' => 'tsquery'
],
...

Configuring PostgreSQL

Make sure that an appropriate default text search configuration is set globbaly (in postgresql.conf), for a particular database (ALTER DATABASE ... SET default_text_search_config TO ...) or alternatively set default_text_search_config in each session.

To check the current value

SHOW default_text_search_config;

Prepare the Schema

By default the engine expects that parsed documents (model data) are stored in the same table as the Model in a column searchable of type tsvector. You'd need to create this column and an index in your schema. You can choose between GIN and GiST indexes in PostgreSQL.

class CreatePostsTable extends Migration
{
    public function up()
    {
        Schema::create('posts', function (Blueprint $table) {
            $table->increments('id');
            $table->text('title');
            $table->text('content')->nullable();
            $table->integer('user_id');
            $table->timestamps();
        });

        DB::statement('ALTER TABLE posts ADD searchable tsvector NULL');
        DB::statement('CREATE INDEX posts_searchable_index ON posts USING GIN (searchable)');
        // Or alternatively
        // DB::statement('CREATE INDEX posts_searchable_index ON posts USING GIST (searchable)');
    }

    public function down()
    {
        Schema::drop('posts');
    }
}

Configuring Searchable Data

In addition to Model's attributes you can bring other any other data to the index document. I.e. a list of Tags for a Post.

public function toSearchableArray()
{
    return [
        'title' => $this->title,
        'content' => $this->content,
        'author' => $this->user->name,
        'tags' => $this->tags->pluck('tag')->implode(' '),
    ];
}

Configuring the Model

You may fine tune the engine behavior for a particular Model by implemeting searchableOptions() in your Model.

class Post extends Model
{
    use Searchable;

    // ...
    public function searchableOptions()
    {
        return [
            // You may wish to change the default name of the column
            // that holds parsed documents
            'column' => 'indexable',
            // You may want to store the index outside of the Model table
            // In that case let the engine know by setting this parameter to true.
            'external' => true,
            // If you don't want scout to maintain the index for you
            // You can turn it off either for a Model or globally
            'maintain_index' => true,
            // Ranking groups that will be assigned to fields
            // when document is being parsed.
            // Available groups: A, B, C and D.
            'rank' => [
                'fields' => [
                    'title' => 'A',
                    'content' => 'B',
                    'author' => 'D',
                    'tags' => 'C',
                ],
                // Ranking weights for searches.
                // [D-weight, C-weight, B-weight, A-weight].
                // Default [0.1, 0.2, 0.4, 1.0].
                'weights' => [0.1, 0.2, 0.4, 1.0],
                // Ranking function [ts_rank | ts_rank_cd]. Default ts_rank.
                'function' => 'ts_rank_cd',
                // Normalization index. Default 0.
                'normalization' => 32,
            ],
            // You can explicitly specify a PostgreSQL text search configuration for the model.
            // Use \dF in psql to see all available configurationsin your database.
            'config' => 'simple',
        ];
    }
}
...

If you decide to keep your Model's index outside of the Model's table you can let engine know that you want to push additional fields in the index table that you can then use to filter the result set by applying where() with the Scout Builder. In this case you'd need to implement searchableAdditionalArray() on your Model. Of course the schema for the external table should include these additional columns.

public function searchableAdditionalArray()
{
    return [
        'user_id' => $this->user_id,
    ];
}

You may want to make your searchable column hidden so it's not standing in your way

protected $hidden = [
    'searchable',
];

Usage

// plainto_tsquery()
$posts = App\Post::search('cat rat')
    ->usingPlainQuery()->get()

// phraseto_tsquery()
$posts = App\Post::search('cat rat')
    ->usingPhraseQuery()->get()

// to_tsquery()
$posts = App\Post::search('fat & (cat | rat)')
    ->usingTsQuery()->get()

// websearch_to_tsquery()
// uses web search syntax 
$posts = App\Post::search('"sad cat" or "fat rat" -mouse')
    ->usingWebSearchQuery()->get()

// DIY using a callback
use ScoutEngines\Postgres\TsQuery\ToTsQuery;

$results = App\Post::search('fat & (cat | rat)', function ($builder, $config) {
    return new ToTsQuery($builder->query, $config);
})->get();

Please see the official documentation on how to use Laravel Scout.

Testing

composer test

Security

If you discover any security related issues, please email [email protected] instead of using the issue tracker.

Changelog

Please see CHANGELOG for more information what has changed recently.

Contributing

Please see CONTRIBUTING for details.

Credits

License

The MIT License (MIT). Please see License File for more information.

laravel-scout-postgres's People

Contributors

cpgo avatar gregmartyn avatar keithbrink avatar pmatseykanets avatar roquie avatar tortuetorche avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

laravel-scout-postgres's Issues

Getting all columns even hidden

I am getting all columns even hidden by just running below:

$model->search($query)->get() or $model->search($query)->paginate()

As you can see in below screenshot, there is hidden columns set but in attributes you can see password, searchable and remember_token

Screenshot 2020-09-21 at 11 05 57

Am I missing something?

Index new records only

I've run php artisan scout:import "App\Models\MyModel" which creates indexes for all the records in my table. Unfortunately, an external process is inserting new records into the table. The searchable column is null and those records are not searchable. I'm in the process of creating an artisan command to index the records that have searchable equal to null. Would a scout:import-new (or other signature) be welcome as a feature to index new records?

Breaking HTTP Tests

Hi,
It appears that when I run my HTTP Tests on my project, the package is breaking the test

SQLSTATE[HY000]: General error: 1 near "USING": syntax error (SQL: CREATE INDEX posts_searchable_index ON posts USING GIN (searchable))"

I followed the doc and it works very well but in my HTTP test 🤷
Maybe someone has stumble on the same issue

specify vector language on each row

Hi,

in the context of a multilingual table, would it be possible to specify a language for the ts_vector for each row working on create and update ? Any way to add this in the model ?

Thanks !

Class 'ScoutEngines\Postgres\PostgresEngineServiceProvider' not found


            throw new Exception('The bootstrap/cache directory must be present and writable.');
        }
 
        $this->files->put(
            $this->manifestPath, '<?php return '.var_export($manifest, true).';'
        );
 
        return array_merge(['when' => []], $manifest);
    }
 
    /**
     * Create a new provider instance.
     *
     * @param  string  $provider
     * @return \Illuminate\Support\ServiceProvider
     */
    public function createProvider($provider)
    {
        return new $provider($this->app);
    }
}
 
Arguments
"Class 'ScoutEngines\Postgres\PostgresEngineServiceProvider' not found"

No support for Laravel Scout 5.0

Using version ^2.3 for pmatseykanets/laravel-scout-postgres
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Can only install one of: laravel/scout[4.0.x-dev, v5.0.3].
    - Can only install one of: laravel/scout[v5.0.3, 4.0.x-dev].
    - Can only install one of: laravel/scout[4.0.x-dev, v5.0.3].
    - pmatseykanets/laravel-scout-postgres v2.3.0 requires laravel/scout ~4.0 -> satisfiable by laravel/scout[4.0.x-dev].
    - Installation request for pmatseykanets/laravel-scout-postgres ^2.3 -> satisfiable by pmatseykanets/laravel-scout-postgres[v2.3.0].
    - Installation request for laravel/scout (locked at v5.0.3, required as ^5.0) -> satisfiable by laravel/scout[v5.0.3].

Update?

Is this package going anywhere? Will there be any update on pull requests?

Scout Import fails with eager loaded relationships

First - Thank you for making a PostgreSql extension for Scout! I was able to resolve the issue, but still reporting it in case it's something in Scout or the driver. If I need to bring this over to Scout, let me know and I can do that.

The following code in my model was causing a PostgreSql error. This code automatically loads my relationships when I pull back a record:

Model Relationships Causing Error

protected $with = [
    'category',
    'attachments',
];

Here's the error I would get when attempting a Scout Import using the Command Line command:

PostgreSql Error

SQLSTATE[HY093]: Invalid parameter number: parameter was not defined (SQL: select to_tsvector(COALESCE(english, get_current_ts_config()), 177) || to_tsvector(COALESCE(english, get_current_ts_config()), 2017-05-25 09:46:03) || to_tsvector(COALESCE(english, get_current_ts_config()), 2017-05-25 09:46:03) || to_tsvector(COALESCE(english, get_current_ts_config()), ) || to_tsvector(COALESCE(english, get_current_ts_config()), ) || to_tsvector(COALESCE(english, get_current_ts_config()), ) || to_tsvector(COALESCE(english, get_current_ts_config()), 1) || to_tsvector(COALESCE(english, get_current_ts_config()), Videos) || to_tsvector(COALESCE(english, get_current_ts_config()), 23) || to_tsvector(COALESCE(english, get_current_ts_config()), <pTest Article Content</p>) || to_tsvector(COALESCE(english, get_current_ts_config()), ) || to_tsvector(COALESCE(english, get_current_ts_config()), 23) || to_tsvector(COALESCE(2017-05-23 16:26:58, get_current_ts_config()), 2017-05-23 16:26:58) || to_tsvector(COALESCE(, get_current_ts_config()), ) AS tsvector limit 1)

After commenting out the above code from my model I was able to import successfully:

Success

#> php artisan scout:import "App\Data\Models\HelpCenter\Article" 

Imported [App\Data\Models\HelpCenter\Article] models up to ID: 235
All [App\Data\Models\HelpCenter\Article] records have been imported.

Scout 7 on Laravel 7.x

Scout 7.x is technically available in Laravel 7.x. Can the Scout requirements for the 7.0.0 tag include Scout 7.x and 8.x? I have a setup atm that has v7.2.1 of Lumen and 7.2.1 of Scout. But I can't install the 7.0.0 tag of this package due to it requiring Scout 8.0

Can't use in when

I'm try use search in when but this throw exception BadMethodCallException .

   $campaign = Campaign::when($search, function ($q, $search) {
        $q->search($search);
    })->get();

No support for Laravel Scout 6.x

composer require pmatseykanets/laravel-scout-postgres
Using version ^3.1 for pmatseykanets/laravel-scout-postgres
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Can only install one of: laravel/scout[5.0.x-dev, v6.0.0].
    - Can only install one of: laravel/scout[v6.0.0, 5.0.x-dev].
    - Can only install one of: laravel/scout[5.0.x-dev, v6.0.0].
    - pmatseykanets/laravel-scout-postgres v3.1.0 requires laravel/scout ~5.0 -> satisfiable by laravel/scout[5.0.x-dev].
    - Installation request for pmatseykanets/laravel-scout-postgres ^3.1 -> satisfiable by pmatseykanets/laravel-scout-postgres[v3.1.0].
    - Installation request for laravel/scout (locked at v6.0.0, required as ^6.0) -> satisfiable by laravel/scout[v6.0.0].


Installation failed, reverting ./composer.json to its original content.

Unique Violation

I'm receiving a unique violation, we are queueing the the scout processes with 5 concurrent workers. Does this look like it is caused by a race condition of some sort? Maybe two jobs of the same model are being executed concurrently? How many simultaneous jobs would you recommend?

Doctrine\DBAL\Driver\PDOException:
SQLSTATE[23505]: Unique violation: 7 ERROR:  duplicate key value violates unique constraint "users_pkey"
DETAIL:  Key (id)=(136778) already exists.
SQLSTATE[23505]: Unique violation: 7 ERROR:  duplicate key value violates unique constraint "users_pkey"
DETAIL:  Key (id)=(136778) already exists. (SQL: insert into "users" ("searchable", "id") values ('136778':1 'username':2A,3A, 136778))

Searching model with Soft Delete

I'm using dev-master version of laravel-scout-postgres.
Both withTrashed and onlyTrashed method are not properly handled.
Scout Builder has __soft_deleted in wheres attribute when its config soft_delete set to true. This will throw an Exception because no such column in the table.

Current PostgresEngine only set $query->whereNull($builder->model->getDeletedAtColumn()); if the model use SoftDeletes.

I modify the engine to deal with this problem.

        // Handle soft deletes
        if (!$this->isExternalIndex($builder->model)) {
            if ($this->usesSoftDeletes($builder->model) && isset($builder->wheres['__soft_deleted'])) {
                if ($builder->wheres['__soft_deleted']) {
                    $query->whereNotNull($builder->model->getDeletedAtColumn());
                } else {
                    $query->whereNull($builder->model->getDeletedAtColumn());
                }
                unset($builder->wheres['__soft_deleted']);
            }
        }

        // Apply where clauses that were set on the builder instance if any
        foreach ($builder->wheres as $key => $value) {
            $query->where($key, $value);
            $bindings->push($value);
        }

        /* Deleted
        // If parsed documents are being stored in the model's table
        if (! $this->isExternalIndex($builder->model)) {
            // and the model uses soft deletes we need to exclude trashed rows
            if ($this->usesSoftDeletes($builder->model)) {
                $query->whereNull($builder->model->getDeletedAtColumn());
            }
        }
        */

Scout 9

Helo, when scout 9.x will be support ?

Use with Lumen

I'd love to use this package with Laravel Lumen. When running the migration I get:

lumen.ERROR: Symfony\Component\Debug\Exception\FatalThrowableError: Call to undefined function ScoutEngines\Postgres\resolve() in customer/Api/vendor/pmatseykanets/laravel-scout-postgres/src/PostgresEngineServiceProvider.php:12

If i changed to this I got the migrate working:

    $this->app->configure('scout');
    $this->app->singleton(EngineManager::class, function ($app) {
        return (new EngineManager($app))->extend('pgsql', function ($app) {
			return new PostgresEngine($this->app['db'], config('scout.pgsql', []));
        });
    });

Is it possible to use with Lumen?

Laravel 6 support

Hey.

Trying to upgrade to Laravel 6 but your package isn't compatible. Any plans to update it?

 Problem 1
    - Conclusion: don't install laravel/framework v6.0.3
    - Conclusion: don't install laravel/framework v6.0.2
    - Conclusion: don't install laravel/framework v6.0.1
    - Conclusion: don't install laravel/framework v6.0.0
    - pmatseykanets/laravel-scout-postgres v5.0.0 requires illuminate/support ~5.4 -> satisfiable by laravel/framework[v5.8.34, 5.5.x-dev, 5.6.x-dev, 5.7.x-dev], illuminate/support[5.4.x-dev, 5.5.x-dev, 5.6.x-dev, 5.7.17, 5.7.18, 5.7.19, 5.7.x-dev, 5.8.x-dev, v5.4.0, v5.4.13, v5.4.17, v5.4.19, v5.4.27, v5.4.36, v5.4.9, v5.5.0, v5.5.16, v5.5.17, v5.5.2, v5.5.28, v5.5.33, v5.5.34, v5.5.35, v5.5.36, v5.5.37, v5.5.39, v5.5.40, v5.5.41, v5.5.43, v5.5.44, v5.6.0, v5.6.1, v5.6.10, v5.6.11, v5.6.12, v5.6.13, v5.6.14, v5.6.15, v5.6.16, v5.6.17, v5.6.19, v5.6.2, v5.6.20, v5.6.21, v5.6.22, v5.6.23, v5.6.24, v5.6.25, v5.6.26, v5.6.27, v5.6.28, v5.6.29, v5.6.3, v5.6.30, v5.6.31, v5.6.32, v5.6.33, v5.6.34, v5.6.35, v5.6.36, v5.6.37, v5.6.38, v5.6.39, v5.6.4, v5.6.5, v5.6.6, v5.6.7, v5.6.8, v5.6.9, v5.7.0, v5.7.1, v5.7.10, v5.7.11, v5.7.15, v5.7.2, v5.7.20, v5.7.21, v5.7.22, v5.7.23, v5.7.26, v5.7.27, v5.7.28, v5.7.3, v5.7.4, v5.7.5, v5.7.6, v5.7.7, v5.7.8, v5.7.9, v5.8.0, v5.8.11, v5.8.12, v5.8.14, v5.8.15, v5.8.17, v5.8.18, v5.8.19, v5.8.2, v5.8.20, v5.8.22, v5.8.24, v5.8.27, v5.8.28, v5.8.29, v5.8.3, v5.8.30, v5.8.31, v5.8.32, v5.8.33, v5.8.34, v5.8.35, v5.8.4, v5.8.8, v5.8.9].
... and more ...

Search part of word

Hello,
Is it possible to enable search for part of word?
For example search for "Postgre" and find "PostgreSQL".

Kind regards,
Salvis.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.