At the moment the queries are created as shown in the images below.
By dispatching an event that will create the queries, the bundle would become more flexible and it would for example allow to implement fuzzy queries, or anything more advanced.

Another way to allow more flexibility is to extract the search method of the adapter into two stages, creating the query prepareSearch and running it executeQuery. Extending the massive_search.adapter would be much easier that way.

When you clone the repo and try to run only the Zend Lucene tests (vendor/bin/behat --suite=zend_lucene) you end up with many same errors:

There is no search mapping for object with class "Massive\Bundle\SearchBundle\Tests\Resources\TestBundle\Entity\Product" (Massive\Bundle\SearchBundle\Search\Exception\MetadataNotFoundException)

This happens because MassiveSearchExtension on line 123 checks whether the massive-search exists, which doesn't. Also, directory Entity doesn't exist.

The traviscl tests run fine because they're executed in order where vendor/bin/phpunit --coverage-text is run first. As a side effect it creates also massive-search and Entity directories and doesn't remove them when it finishes (which it probably should).

To see how it fails run:

$ git clone https://github.com/massiveart/MassiveSearchBundle.git
$ cd MassiveSearchBundle
$ composer install
$ vendor/bin/behat --suite=zend_lucene

To execute the tests properly run:

$ git clone https://github.com/massiveart/MassiveSearchBundle.git
$ cd MassiveSearchBundle
$ composer install
$ mkdir Tests/Resources/TestBundle/Resources/config/massive-search
$ mkdir Tests/Resources/TestBundle/Entity
$ vendor/bin/behat --suite=zend_lucene

I scratched my head around why my search isn't filtered by locale. I'm using DoctrineExtensions Translatable extension and I made a custom reindex provider because I see this is used in ReindexCommand to extract translations for each object.

Well, I didn't expect that calling $this->searchManager->index($object, $locale); ignores $locale because index() method takes only one argument. So, this entire block of code is in fact useless:

$locales = [null];
if ($provider instanceof LocalizedReindexProviderInterface) {
  $locales = $provider->getLocalesForObject($object);
}
try {
  foreach ($locales as $locale) {
    if (null !== $locale) {
      $object = $provider->translateObject($object, $locale);
    }
    $this->searchManager->index($object, $locale);
  }

This all has no effect on entitie's indexed locale because real locale is always extracted from the entity in ObjectToDocumentConverter.

I actually ended up using deprecated call to entity repository that returns all objects that I need to index even though it's probably supposed to throw an error with trigger_error() which is silenced with @ so I doesn't.

I think the entire thing with creating custom providers is useful but now it's too tightly coupled with the entity structure. It has to contain its locale somewhere which is exactly how DoctrineExtensions Translatable extension doesn't work. The LocalizedReindexProviderInterface could work for me when reindexing even though I'd appreciate if providers could check whether they support particular entity class.
Maybe If I could use the same LocalizedReindexProviderInterface provider in IndexListener to grab all translations for particular entity and index them instead of relying on locale property? That would help me a lot.

In order to support multiple languages, one option would be to have a listener which modifies the name of the index name depending on the locale.

The unit tests for the adapters are quite basic and should be extended with pagination and sorting.

The Document and the QueryHit are two differnet classes. One is about storing the data, the other is, maybe, about the search results.

Currently the document has the methods getTitle, getDescription and getUrl. Which I think would better belong to QueryHit.

But having hard-methods in QueryHit is quite limiting to extensiblility, so they should instead be registered dynamic fields.

Hey,

i want to use this bundle outside of Symfony - in that case for ZendFramework.
Is there an easy way to work without the whole symfony mvc in use?

regards thx

In order to display search results and link to documents we need to map these properties.

If you define adapter as elastic, and do not define adaptres: elastic then you get an undefined index error,

Apply the CS fixer and fix all the headers (should not read part of the Sulu CMS)

There is no XSD file for the search mapping defined in XML. It should definitely be there, because the IDE can support the developer better, and there is an additional validation.

Hi there. i'm trying to create new adapter, in this case to make it work with Algolia. i've created the services but problem is when i set the new adapter in the config like

# MassiveSearch Configuration
massive_search:
    adapter: algolia

When i do this i get an error like

In EnumNode.php line 46:
                                                                                                                              
  The value "algolia" is not allowed for path "massive_search.adapter". Permissible values: "zend_lucene", "elastic", "test"  
                                                                                                                             

so, is there any way to add another adapter or to modify that list of possible values?

Thanks in advance.

The sorting of the fields doesn't work as expected. Seems like the mapping of the indexed fields should look like followed:

"type" :    "string",
"index":    "not_analyzed"

Further information can be found on the elasticsearch documentation:
https://www.elastic.co/guide/en/elasticsearch/guide/current/mapping-intro.html
https://www.elastic.co/guide/en/elasticsearch/guide/current/_sorting.html

massive:search:reindex would be a better command name than massive:search:index:rebuild.

Also it makes for a better namespace in Search\ReIndex\..

Note this is a BC breaking change.

The metadata is currently not cached.

In version 8 of elasticsearch/elasticsearch the namespace changed from Elasticsearch to Elastic\Elasticsearch (https://github.com/elastic/elasticsearch-php/blob/main/BREAKING_CHANGES.md). Integrating this bundle into a Symfony 6 project I've noticed that there is no way to change massive_search.search.adapter.elastic.client.class (https://github.com/massiveart/MassiveSearchBundle/blob/2.8/Resources/config/adapter_elastic.xml#L7).

The issue is that in loads the adapter_elastic.xml before checking the existence of that class and loading the xml will always reset it:
https://github.com/massiveart/MassiveSearchBundle/blob/2.8/DependencyInjection/MassiveSearchExtension.php#L79-L81

When i want to extend the e.g. SuluContactBundle:Account Entity with SuluContactExtensionBundle:Account by overriding the massive-search/Account.xml file, it does not have any impact.

The override attempt:
sulu/SuluContactExtensionBundle#18

Resulting from the following discussion:
sulu/sulu#1349

If other indexes exist in Elastic search and a global search is performed, then these external items will also be found.

Add dispatching of a PRE_DEINDEX event in SearchManager::deindex to giv a possibility to react on deindexing an object.

Please update the site to the latest GIT version https://massivesearchbundle.readthedocs.io/en/latest/

I had a hard time to figure how the Elastic Search configuration is because the latest github Docpage is not used for your doc site linked from the readme. Lost some hours because of you :P Glad I asked Alex about it!

Hello,
I use https://github.com/sulu/sulu witch implement the massive-search with ZendLucene adapter with load-balancing.
The indexes directory was shared between servers.

Thanks for help

Here the stack trace :
ZendSearch\Lucene\Exception\InvalidFileFormatException:
Wrong TermInfoIndexFile file format

at vendor/handcraftedinthealps/zendsearch/library/ZendSearch/Lucene/Index/DictionaryLoader.php:52
at ZendSearch\Lucene\Index\DictionaryLoader::load('�' . "\0" . '' . "\0" . '' . "\0" . '' . "\0" . '' . "\0" . '' . "\0" . '' . "\0" . '��_8s.fdx' . "\0" . '' . "\0" . '' . "\0" . '' . "\0" . '' . "\0" . '' . "\0" . '' . "\0" . '' . "\0" . '�_8s.fdt' . "\0" . '' . "\0" . '')
(vendor/handcraftedinthealps/zendsearch/library/ZendSearch/Lucene/Index/SegmentInfo.php:774)
at ZendSearch\Lucene\Index\SegmentInfo->_loadDictionaryIndex()
(vendor/handcraftedinthealps/zendsearch/library/ZendSearch/Lucene/Index/SegmentInfo.php:803)
at ZendSearch\Lucene\Index\SegmentInfo->getTermInfo(object(Term))
(vendor/handcraftedinthealps/zendsearch/library/ZendSearch/Lucene/Index.php:898)
at ZendSearch\Lucene\Index->hasTerm(object(Term))
(vendor/handcraftedinthealps/zendsearch/library/ZendSearch/Lucene/Search/Query/Preprocessing/Term.php:120)
at ZendSearch\Lucene\Search\Query\Preprocessing\Term->rewrite(object(Index))
(vendor/handcraftedinthealps/zendsearch/library/ZendSearch/Lucene/Search/Query/Boolean.php:132)
at ZendSearch\Lucene\Search\Query\Boolean->rewrite(object(Index))
(vendor/handcraftedinthealps/zendsearch/library/ZendSearch/Lucene/Index.php:669)
at ZendSearch\Lucene\Index->find(object(Boolean))
(vendor/massive/search-bundle/Search/Adapter/ZendLuceneAdapter.php:390)
at Massive\Bundle\SearchBundle\Search\Adapter\ZendLuceneAdapter->removeExisting(object(Index), object(Document))
(vendor/massive/search-bundle/Search/Adapter/ZendLuceneAdapter.php:113)
at Massive\Bundle\SearchBundle\Search\Adapter\ZendLuceneAdapter->index(object(Document), 'massive_media-fr-i18n')
(vendor/massive/search-bundle/Search/SearchManager.php:178)
at Massive\Bundle\SearchBundle\Search\SearchManager->index(object(FileVersionMeta))
(vendor/massive/search-bundle/Search/EventListener/IndexListener.php:39)
at Massive\Bundle\SearchBundle\Search\EventListener\IndexListener->onIndex(object(IndexEvent), 'massive_search.index', object(TraceableEventDispatcher))
(vendor/symfony/event-dispatcher/Debug/WrappedListener.php:117)
at Symfony\Component\EventDispatcher\Debug\WrappedListener->__invoke(object(IndexEvent), 'massive_search.index', object(TraceableEventDispatcher))
(vendor/symfony/event-dispatcher/EventDispatcher.php:230)
at Symfony\Component\EventDispatcher\EventDispatcher->callListeners(array(object(WrappedListener)), 'massive_search.index', object(IndexEvent))
(vendor/symfony/event-dispatcher/EventDispatcher.php:59)
at Symfony\Component\EventDispatcher\EventDispatcher->dispatch(object(IndexEvent), 'massive_search.index')
(vendor/symfony/event-dispatcher/Debug/TraceableEventDispatcher.php:154)
at Symfony\Component\EventDispatcher\Debug\TraceableEventDispatcher->dispatch(object(IndexEvent), 'massive_search.index')
(vendor/massive/search-bundle/Search/EventSubscriber/DoctrineOrmSubscriber.php:77)
at Massive\Bundle\SearchBundle\Search\EventSubscriber\DoctrineOrmSubscriber->indexEntity(object(FileVersionMeta))
(vendor/massive/search-bundle/Search/EventSubscriber/DoctrineOrmSubscriber.php:51)
at Massive\Bundle\SearchBundle\Search\EventSubscriber\DoctrineOrmSubscriber->postPersist(object(LifecycleEventArgs))
(vendor/symfony/doctrine-bridge/ContainerAwareEventManager.php:68)
at Symfony\Bridge\Doctrine\ContainerAwareEventManager->dispatchEvent('postPersist', object(LifecycleEventArgs))
(vendor/doctrine/orm/lib/Doctrine/ORM/Event/ListenersInvoker.php:99)
at Doctrine\ORM\Event\ListenersInvoker->invoke(object(ClassMetadata), 'postPersist', object(FileVersionMeta), object(LifecycleEventArgs), 4)
(vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:1150)
at Doctrine\ORM\UnitOfWork->executeInserts(object(ClassMetadata))
(vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:415)
at Doctrine\ORM\UnitOfWork->commit(null)
(vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php:388)
at Doctrine\ORM\EntityManager->flush(null)
(var/cache/admin/dev/ContainerYLhkVOa/EntityManager_9a5be93.php:136)
at ContainerYLhkVOa\EntityManager_9a5be93->flush()
(vendor/sulu/sulu/src/Sulu/Bundle/MediaBundle/Media/Manager/MediaManager.php:576)
at Sulu\Bundle\MediaBundle\Media\Manager\MediaManager->createMedia(array('locale' => 'fr', 'collection' => '20', 'contentLanguages' => array(), 'publishLanguages' => array(), 'title' => 'transition énergétique', 'formats' => array(), 'id' => null, 'storageOptions' => array('segment' => '05', 'fileName' => 'transition-energetique.svg'), 'name' => 'transition énergétique.svg', 'size' => 2700, 'mimeType' => 'image/svg', 'properties' => array('width' => 72, 'height' => 73), 'type' => array('id' => 2)), object(User))
(vendor/sulu/sulu/src/Sulu/Bundle/MediaBundle/Media/Manager/MediaManager.php:527)
at Sulu\Bundle\MediaBundle\Media\Manager\MediaManager->buildData(object(UploadedFile), array('locale' => 'fr', 'collection' => '20', 'contentLanguages' => array(), 'publishLanguages' => array(), 'title' => 'transition énergétique', 'formats' => array(), 'id' => null, 'storageOptions' => array('segment' => '05', 'fileName' => 'transition-energetique.svg'), 'name' => 'transition énergétique.svg', 'size' => 2700, 'mimeType' => 'image/svg', 'properties' => array('width' => 72, 'height' => 73), 'type' => array('id' => 2)), object(User))
(vendor/sulu/sulu/src/Sulu/Bundle/MediaBundle/Media/Manager/MediaManager.php:322)
at Sulu\Bundle\MediaBundle\Media\Manager\MediaManager->save(object(UploadedFile), array('locale' => 'fr', 'collection' => '20', 'contentLanguages' => array(), 'publishLanguages' => array(), 'title' => 'transition énergétique', 'formats' => array(), 'id' => null), 1)
(vendor/sulu/sulu/src/Sulu/Bundle/MediaBundle/Controller/MediaController.php:533)
at Sulu\Bundle\MediaBundle\Controller\MediaController->saveEntity(null, object(Request))
(vendor/sulu/sulu/src/Sulu/Bundle/MediaBundle/Controller/MediaController.php:414)
at Sulu\Bundle\MediaBundle\Controller\MediaController->postAction(object(Request))
(vendor/symfony/http-kernel/HttpKernel.php:152)
at Symfony\Component\HttpKernel\HttpKernel->handleRaw(object(Request), 1)
(vendor/symfony/http-kernel/HttpKernel.php:74)
at Symfony\Component\HttpKernel\HttpKernel->handle(object(Request), 1, true)
(vendor/symfony/http-kernel/Kernel.php:202)
at Symfony\Component\HttpKernel\Kernel->handle(object(Request))
(public/index.php:66)

The latest Elastic Search release supported by elastic/[email protected] [1] is Elastic Search 2.4.x to which all support ended on 28.02.2018 [2].

It looks like the elastic feature set used by MassiveSearchBundle is fully supported by Elastic Search 6 and therefore elastic/elasticsearch-php should be upgraded to the 6.0 branch.

Links:

[1] https://github.com/elastic/elasticsearch-php/tree/master#version-matrix
[2] https://www.elastic.co/support/eol

Hi!

I am trying to add a new field to the Document object by a custom factory:

My factory class:

namespace SearchBundle;

use Massive\Bundle\SearchBundle\Search\Factory as BaseFactory;

class Factory extends BaseFactory
{

    public function createDocument()
    {
        $document = new SearchBundle\Document();
        return $document;
    }

}

My Document class:

namespace SearchBundle;

use Massive\Bundle\SearchBundle\Search\Document as BaseDocument;

class Document extends BaseDocument {

    protected $editUrl;

    public function getEditUrl()
    {
        return $this->editUrl;
    }

    public function setEditUrl($editUrl)
    {
        $this->editUrl = $editUrl;
    }

    public function jsonSerialize()
    {
        return [
            'id' => $this->id,
            'title' => $this->title,
            'description' => $this->description,
            'class' => $this->class,
            'url' => $this->url,
            'image_url' => $this->imageUrl,
            'locale' => $this->locale,
            'edit_url' => $this->editUrl,
        ];
    }

My config:

massive_search:
    persistence:
        doctrine_orm:
            enabled: true
    adapter: elastic
    metadata:
        prefix: %database_name%
    services:
        factory: factory_service

services:
    factory_service:
        class: SearchBundle\Factory

In the onPreIndex event I call this

$document->setEditUrl('test-edit-url')

But as I see the index still does not include the edit_url field.

Did i miss something?

Thanks!

Instead of unStored, stored, text, etc. Use store and index.

Appplies to the Index::INDEX_* constants.

Add command to remove unused indexes.

It should remove indexes which are managed by massive search including localized indexes.

I use Sulu for a couple of weeks now and I tried to index an entity of my project in App\Entity (App\Entity\Commune).
I put the Commune.xml in src/Resources/config/massive-search/Commune.xml
But when I execute php bin/console massive:search:reindex, nothing happens.

It seems that the explored paths for metadatas are only from bundle :
$container->getParameter('kernel.bundles')

And so when I dump the paths form getBundleMappingPaths (DependencyInjection/MassiveSearchExtension.php), I only get this :

array:3 [
"Sulu\Bundle\ContactBundle\Entity" => "/Users/me/Documents/WebProjets/myproject/vendor/sulu/sulu/src/Sulu/Bundle/ContactBundle/Resources/config/massive-search"
"Sulu\Bundle\MediaBundle\Entity" => "/Users/me/Documents/WebProjets/myproject/vendor/sulu/sulu/src/Sulu/Bundle/MediaBundle/Resources/config/massive-search"
"Sulu\Bundle\CategoryBundle\Entity" => "/Users/me/Documents/WebProjets/myproject/vendor/sulu/sulu/src/Sulu/Bundle/CategoryBundle/Resources/config/massive-search"
]

Is there a way to add AppBundle or is it a limitation which obliges to use a bundle to index entities ?
Thank you very much :)

• It is currently possible to create mappings without indexes
• Context is not taken into account by anything -- it simply creates additional indexes but searching is not context aware.
• Category is not context aware (always uses _default context).

If you have 2 projects on the same server and reindex both at the same time the first project will lock the second project because the same directory (sys temp dir) is used:

In use with Sulu v2.3.3, i tried to add an additional, non-searchable field to an index by listening to the PreIndexEvent and adding the field manually using Massive\Bundle\SearchBundle\Search\Factory\Factory like so:

        $document->addField($this->factory->createField(
            'my_additional_field',
            $value,
            Field::TYPE_STRING,
            true,
            false
        ));

Although the $indexed-param of $factory::createField() is set to false, the additionally created field is still searchable within the index.

According to feedback on this topic in the Sulu Slack Channel, the root of the problem is to be found in ElasticSearchAdapter::prepareSearch(SearchQuery $searchQuery), as the query_string-array should contain an additional fields array containing the names of all fields where $indexed is true.

https://github.com/massiveart/MassiveSearchBundle/blob/develop/Search/Adapter/ZendLuceneAdapter.php#L92

When executing the index rebuild command on ~ 10.000 Contacts:
app/console massive:search:index:rebuild

a OutOfMemoryException accurs:
[Symfony\Component\Debug\Exception\OutOfMemoryException]
Error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 65552 bytes)

Before making the next release we should review the documentation and ensure that it is accurate.

I'm playing around with this nice bundle and noticed that the image-fields of my search-index were empty even if I defined them. After digging through the code, it seems to me, that the mapping is missing.

With the following additions it works as expected:

diff --git a/Search/Metadata/Driver/XmlDriver.php b/Search/Metadata/Driver/XmlDriver.php
index bfc3f76..0e7f044 100644
--- a/Search/Metadata/Driver/XmlDriver.php
+++ b/Search/Metadata/Driver/XmlDriver.php
@@ -118,6 +118,7 @@ class XmlDriver extends AbstractFileDriver implements DriverInterface
             $indexMetadata->setTitleField($mapping['title']);
             $indexMetadata->setUrlField($mapping['url']);
             $indexMetadata->setDescriptionField($mapping['description']);
+            $indexMetadata->setImageUrlField($mapping['image']);

             foreach ($mapping['fields'] as $fieldName => $fieldData) {
                 $indexMetadata->addFieldMapping($fieldName, $fieldData);
@@ -148,6 +149,9 @@ class XmlDriver extends AbstractFileDriver implements DriverInterface
         $urlField = $this->getMapping($mapping, 'url');
         $indexMapping['url'] = $urlField;

+        $imageField = $this->getMapping($mapping, 'image');
+        $indexMapping['image'] = $imageField;
+
         $descriptionField = $this->getMapping($mapping, 'description');
         $indexMapping['description'] = $descriptionField;

Am I missing something or is this feature really missing in current master?

Currently an event is fired instructing listeners to reindex, all of the responsiblity is on the "client" side, including logging progress, filtering etc.

It would be better to have the "clients" simply provide the entities which need to be reindexed. This however presents some issues -

• each client has its own way of finding entities (different query builder implementations for example).
• some entities/documents/objects may be localized.

Perhaps the best option would be to have a ReindexProvider class:

class ReindexProviderInterface
{
    public function provide($offset, $limit, $classFqn = null)
    {
    }
}

In use with Sulu-Standard

• Create a date property in a page template
• Create another page template with Teaser Selection
• Add a tag to it for Search
• Define the tag type as date
• Reindex

Removing type="date" gets rid of this issue.

When I try to run massive:search:reindex while I don't have write permissions to the directory where I store indices I get confusing exception not helping me at all.

[ZendSearch\Lucene\Exception\RuntimeException]

Currently localized index names are inferred from existing indexes, however if the indexes do not exist then no indexes are inferreed.

Adapters tend to perform global searches when no indexes are given.

This needs to be fixed.

I just tried to install your promising package on a symfony 3.0.4 installation and sadly failed because of the dependency on "symfony/property-access ~2.4". Are there any plans to support sf3 in the near future?

Thanks for your awesome work!

Introduce support for offest and max results in order to support pagination.

I noticed a difference between the docs source and the rendered docs. For example, directories for the mapping:

https://github.com/massiveart/MassiveSearchBundle/blob/2.8/Resources/docs/mapping.rst#full-example
https://massivesearchbundle.readthedocs.io/en/latest/mapping.html#full-example

Source: "This file MUST be located in config/massive-search/YourBundle/Resources/config/massive-search"
Docs: "This file MUST be located in YourBundle/Resources/config/massive-search"

Clicking at "edit in Github" results in a 404 too. Looks like an error to me.

Support document load event.

Use case

Document URL generators listen to a massive_search.document_post_load event (for example) and can do things such as setting the URL for a document after it has been loaded from the search index.

There would be two:

• StaticRouteUrlDocumentListener: Generate routes using the Symfony routing system (e.g. for products)
  • Will need some sort of class map in configuration, and maybe something to do with the ExpressionLangugae to map properties to route parameters (??)
  • Or we could create a decorator per class
• StructureRouteUrlDocumentListener: Set the URL for the search result based on the current portal, etc.
• SymfonyCmfRoutingDocumentListener: Set the URL based on classes that implement the CMF RouteReferrersInterface for example..