Scout does not update the index for entry set to status disabled, does not delete on entry delete

[twosixcode]

I have sync set to true and my criteria look like this

->criteria(function` (\craft\elements\db\EntryQuery $query) {
    // Sections to be indexed, indexed across all siteIds ('*')
    return $query->section('videos, etc, etc')->siteId('*')->status(['live', 'disabled']);
})

Because I am indexing all sites, I can not rely on Entry ID as Object ID (because the same entry exists in multiple languages). If I do not use a unique Object ID I end up overwriting the record with whatever language was most recently saved. So I have a transformer that creates a unique Object ID by appending the locale slug to the entry id like this 'objectID' => $entry->id .'-'. $entry->locale, This allows me to have all of my records in the same Algolia index.

Scout works perfectly, firing on entry save when entries have a status of enabled. However if I set an entry status to disabled Scout does not fire on entry save and my Algolia index is not updated.

Similarly, if I Delete an entry from Craft, the index item is not removed from Algolia (regardless of the current status of the entry).

It appears the unique Object ID approach I am using is causing Scout not to remove the item from the index when disabled or deleted. If I use Entry Id as ObjectId the deleted or disabled entries are removed as expected.

Is there a way around this (other than separating every Craft Site into a different index)? Or relying on periodic batch indexing instead of sync on entry save? For example a Scout Event that I could hook into with a small custom Module where I could transform the ObjectId so that removal of an item from the index will find a match and succeed?

[timkelty]

For example a Scout Event that I could hook into with a small custom Module where I could transform the ObjectId so that removal of an item from the index will find a match and succeed?

Possibly, but it also sounds like something we should account for out of the box. PRs are welcome, but I'll try and look at it this week with other updates.

[epapada]

I have a similar issue, but also with enabledForSite status. I have multiple indices (one for each language - it is a commerce site) and I cannot figure out how to remove all products from all indices when a product is disabled, or remove only a product from a specific index when I enabledForSite is false.

I do this to go through each site and create the indices

foreach (Craft::$app->getSites()->getAllSites() as $site) {
      $productIndices[] =
            \rias\scout\ScoutIndex::create(getenv('ENVIRONMENT').'_products_'.$site->language)
            ->elementType(\craft\commerce\elements\Product::class)
            ->criteria(function (\craft\commerce\elements\db\ProductQuery $query) use ($site) {
                return $query->siteId($site->id);
            })
            ->transformer(function (\craft\commerce\elements\Product $product) {
                return (new \modules\myshopmodule\services\MyshopModuleService)->transformProductData($product);
            });
    }
    return $productIndices;
}

this way when I disable a product - it is not removed or only removed from one index. Also it only changes the status for only the index where I disabled the product.

What I am trying to achieve is this:
Remove products from all indices when a product is disabled.
Remove product from the specific index when a product has enabledForSite false.

Is this doable?
Cheers.

[tehtrav]

The way I'm getting around this is I query for all entries regardless of their status and then filter them out on the front end with forced facets

[tehtrav]

The way I'm getting around this is I query for all entries regardless of their status and then filter them out on the front end with forced facets

(Nvm, this works for disabled but not delete)

[twosixcode]

@timkelty Hi, hope you are well. I am wondering if you can suggest a workaround for this issue in the meantime.

Is there a way around this (other than separating every Craft Site into a different index)? Or relying on periodic batch indexing instead of sync on entry save? For example a Scout Event that I could hook into with a small custom Module where I could transform the ObjectId so that removal of an item from the index will find a match and succeed?

[timkelty]

@twosixcode going to take a look at this this afternoon. Stay tuned

[timkelty]

@twosixcode can you try using this branch in your composer.json?

"rias/craft-scout": "dev-97-multisite-single-index as 2.3.2",

I don't think it is going to solve your delete issue, but it may at least reindex your items with the updated status when you disable.

[twosixcode]

@timkelty I just tested dev-97-multisite-single-index and Scout updates the index on entry save regardless of the entry's status now. That's great, thank you.

A similar issue I encountered is that re-saving of an Expired entry (or setting an entry to Expired) does not cause the entry to be dropped from the Algolia index. I assume this is a similar issue and is related to me using the custom objectId. Any chance that could be addressed in this branch?

Are there any plans for deletion to work in this scenario? If not, I'll need to go ahead and pursue a workaround (do you have a suggestion -- ie, a Scout event hook I should use where I can override the objectId to match my situation during delete?) or just move to having a separate index per site.

Thanks for your help! I appreciate it.

[timkelty]

@twosixcode can you post your whole scout.php file? or at least the full index we're talking about?

[timkelty]

@twosixcode try something like this as a temporary workaround:

->splitElementsOn(['anyField'])
->transformer(function (\craft\elements\Entry $entry) {
    return [
        'objectID' => $entry->id .'-'. $entry->locale,
        'distinctID' => $entry->id,
    ];
})
->indexSettings(
    \rias\scout\IndexSettings::create()
        ->attributesForFaceting(['distinctID'])
)

I know you're not actually trying to split elements, but it may trick Scout into deleting things by the distinctID instead.

[twosixcode]

Thanks, I tried adding distinctID and the indexSettings you suggested. Deleting in Craft still did not drop the record from Algolia. $entry-id is not really distinct across all sites, though (all versions of a given entry across each site have the same entry id, which is why I was using 'objectID' => $entry->id .'-'. $entry->locale in the first place).

Here are the relevant bits from my scout.php file
`'indices' => [
\rias\scout\ScoutIndex::create(getenv('ALGOLIA_MAIN_INDEX'))
// Scout uses this by default, so this is optional
->elementType(\craft\elements\Entry::class)

// If you don't define a siteId, the primary site is used
->criteria(function (\craft\elements\db\EntryQuery $query) {

    // Sections to be indexed, indexed across all siteIds ('*')
    return $query->section('a, bunch, of, sections, here')->siteId('*')->status(['live', 'disabled', 'expired']);
})

/*
 * The element gets passed into the transform function, you can omit this
 * and Scout will use the \rias\scout\ElementTransformer class instead
*/
->transformer(function (\craft\elements\Entry $entry) {

    // Transform entry data in custom module
    return Craft::$app->getModule('algolia-transform-module')->transform($entry);

})`

And in the transformer module, the relevant bit is what you have already noted, but here is a snippet from the transformer

// Set the universal data attributes
$return = [
    'title' => $entry->title,
    'id' => $entry->id,
    'objectID' => $entry->id .'-'. $entry->locale,
    'distinctID' => $entry->id,
    'slug' => $entry->slug,
    'url' => $entry->url,
    'uri' => '/' . $entry->uri,
    'section' => $entry->getSection()->handle,
    'locale' => $entry->locale,
    'status' => $entry->status,
    'expiryDate' => is_null($entry->expiryDate) ? 4102444799 : $entry->expiryDate->getTimestamp(),
    '_tags' => $entry->getSection()->handle,
];

I am deliberately indexing disabled and expired entries to workaround the issue that cropped up with Scout not dropping records on entry save of disabled and expired entries when my objectId is not the same as the craft entry id (the impetus for creating this issue). I'm now filtering those out on the front end of the site with algolia instantsearch. So really the only outstanding challenge I have to work around is deletion of entries on the craft side not deleting in algolia.

[timkelty]

I don't see splitElementsOn…Did you try it with ->splitElementsOn(['anyField']) like in my example?

$entry-id is not really distinct across all sites

I realize this, but when scout deletes things, if you happen to be using splitElementsOn, it deletes things by the distinctID (entry ID in this case), which would be what you want.

I am deliberately indexing disabled and expired entries

I think you realize this, but with expired in the status as you have it, Scout shouldn't de-index on save of an expired entry, as it will match the query.

[twosixcode]

I don't see splitElementsOn…Did you try it with ->splitElementsOn(['anyField']) like in my example?

I did, yes. Sorry, I pasted in my current config file. But I did try your suggestion, verbatim. Maybe I need to try it again. It sounds like you expect that to work.

I think you realize this, but with expired in the status as you have it, Scout shouldn't de-index on save of an expired entry, as it will match the query.

Yes, definitely. I only recently added that, as a workaround. Totally understand what you're saying there. But even without that in there Scout does not drop an expired entry from the index. All told, it's really not a problem for us since we can easily just index disabled and expired entries and then filter them out on the front end with Instantsearch. But I'd love to find a solution for Deletion.

Thanks for the continued help!

[timkelty]

Yeah try deleting, with splitElementsOn in place, as well as all the other stuff. Maybe try splitElementsOn with an actual field handle you have.

At any rate, it would just be a hack – working on fixing this ASAP.

[timkelty]

@twosixcode otherwise, adding something like this should serve as a workaround for the deletion bit:

Event::on(
    \craft\services\Elements::class,
    \craft\services\Elements::EVENT_AFTER_DELETE_ELEMENT,
    function (\craft\events\ElementEvent $event) {
        $element = $event->element;
        $index = Craft::$container->get(\Algolia\AlgoliaSearch\SearchClient::class)->initIndex('yourIndexName');
        $objectIDs = [
          $element->id . '-' . 'en',
          $element->id . '-' . 'de',
          $element->id . '-' . 'etc',
        ];

        $index->deleteObjects($objectIDs);
    }
);

[twosixcode]

@timkelty Just confirming that we used something similar to the workaround you posted above to ensure deletion. Thanks!

[weotch]

I was thinking of submitting a PR that replaces this bit:

https://github.com/riasvdv/craft-scout/blob/e36d5c7cdcc3380d1de53fc818e77b182ebf9976/src/jobs/ImportIndex.php#L25-L35

... with replaceAllObjects(). See any issues with this?

[timkelty]

@weotch that seems like it would be problematic for large indexes, where batching is required due to the request size limits.

[weotch]

What if replaceAllObjects() is conditionally used based on the presence of some config option (like use_replace_all)? In the docs for this option we'd say like:

Use Algolia's replaceAllObjects() which will remove entries that were previously synced to Algolia who no longer match your criteria(). Note, this disables batching (which could cause issues with large indices) and will increase your Algolia "operations" usage.

[twitcher07]

I have a site using Craft version 4.3.3 and Scout plugin version 3.0.0 and changing the status of entries from enabled to disabled seems to work for me as long as splitElementsOn() is not enabled. Is this a related bug? or is something not right with my setup?

Here is a simplified example of my index setup in scout.php with splitting enabled that doesn't work with deleting entries from the Algolia index:

    return [
        'indices' => [
            \rias\scout\ScoutIndex::create($allIndexName)
            // Scout uses this by default, so this is optional
            ->elementType(\craft\elements\Entry::class)
            // If you don't define a siteId, the primary site is used
            ->criteria(function (\craft\elements\db\EntryQuery $query) {
                return  $query
                        ->section([
                            'pages', 
                            'newsAndInsights', 
                            'people', 
                            'practices', 
                            'services', 
                            'resources'
                        ]);
            })
            /*
             * The element gets passed into the transform function, you can omit this
             * and Scout will use the \rias\scout\ElementTransformer class instead
            */
            ->transformer(function (\craft\elements\Entry $entry) {

                $pagesIndexName             = 'Pages';
                $practicesServicesIndexName = 'Practices/Services';
                $peopleIndexName            = 'People';
                $newsIndexName              = 'News & Insights';
                $resourcesIndexName         = 'Resources';

                if ($entry->section->handle == 'pages'):

                    return [
                        'title' => $entry->title,
                        'permalink' => $entry->url,
                        'description' => $entry->contentSummary,
                        'defaultSortBy' => $entry->title,
                        'index' => $pagesIndexName
                    ];

                elseif ($entry->section->handle == 'newsAndInsights'):
                
                    return [
                        'title' => $entry->title,
                        'description' => $entry->contentSummary,
                        'article_content' => ['some', 'content', 'from', 'a', 'matrix'],
                        'post_date' => array(
                            'lvl0' => $entry->postDate->format('Y'),
                            'lvl1' => $entry->postDate->format('Y > F')
                        ),
                        'post_date_timestamp' => $entry->postDate->getTimestamp(),
                        'post_date_formatted' => $entry->postDate->format('m.d.Y'),
                        'permalink' => $entry->url,
                        'defaultSortBy' => $entry->postDate->getTimestamp(),
                        'index' => $newsIndexName
                    ];

                elseif ($entry->section->handle == 'people'):

                    // Change data so we can filter out later by if the person is
                    // a redirected entry type.
                    if ($entry->type->handle == 'redirected') {

                        return [
                            'name' => $entry->title,
                            'entryType' => $entry->type->handle,
                            'index' => $peopleIndexName
                        ];

                    }

                    return [
                        'objectID' => $entry->id,
                        'entryType' => $entry->type->handle,
                        'name' => $entry->title,
                        'last_name' => $entry->peopleLastName,
                        'first_name' => $entry->peopleFirstName . (!empty($entry->peopleMiddleName) ? ' ' . $entry->peopleMiddleName : ''),
                        'position' => $entry->peoplePosition2->label,
                        'email' => $entry->peopleEmail,
                        'level' => $entry->peopleTitle3->exists() ? $entry->peopleTitle3->one()->title : null,
                        'image' => $entry->peopleHeadshot->exists() ? $entry->peopleHeadshot->one()->optimize5x4->src() : null,
                        'permalink' => $entry->url,
                        'defaultSortBy' => $entry->peopleLastName . ' ' . $entry->peopleFirstName . (!empty($entry->peopleMiddleName) ? ' ' . $entry->peopleMiddleName : ''), 
                        'index' => $peopleIndexName
                    ];

                elseif ($entry->section->handle == 'practices' || $entry->section->handle == 'services'):

                    return [
                        'title' => $entry->title,
                        'status' => $entry->status,
                        'permalink' => $entry->url,
                        'description' => $entry->contentSummary,
                        'defaultSortBy' => $entry->title,
                        'index' => $practicesServicesIndexName
                    ];

                elseif ($entry->section->handle == 'resources'):

                    // Remove from index if there isn't a related asset.
                    if (!$entry->resourceAsset1->exists()) {
                        return [];
                    }

                    return [
                        'objectID' => $entry->id,
                        'title' => $entry->title,
                        'description' => $entry->contentSummary,
                        'permalink' => $entry->resourceAsset1->exists() ? $entry->resourceAsset1->one()->url : null,
                        'defaultSortBy' => $entry->title,
                        'index' => $resourcesIndexName
                    ];

                endif;

            })
            ->splitElementsOn([
                'article_content'
            ])
            /*
             * You can use this to define index settings that get synced when you call
             * the ./craft scout/settings/update console command. This way you can
             * keep your index settings in source control. The IndexSettings
             * object provides autocompletion for all Algolia's settings
            */
            ->indexSettings()
        ]
    ];

[MangoMarcus]

I'm having the same problem when unpublishing an entry doesn't remove it from the algolia index.

I understand that I could use ->status() in the criteria and then filter by a status facet on the front-end as per tehtrav's comment, but I'm a little weary about potentially exposing unpublished content to exploits on the front end.

Is there a recommended workaround?

I think the problem is that the MakeSearchableJob bails here if it can't find the element (which it won't do, assuming you don't have ->status(null) in your criteria).

I wonder if somewhere down the chain if should add a DeIndex job, something like this psuedo-code

$el = $index->criteria->id($id)->siteId($siteId)

if $el
     Queue the MakeSearchable as normal
     ...
else
     Check for unpublished element
     $el = $index->criteria->id($id)->siteId($siteId)->status(null)
    
     if $el
         Queue the DeIndex job to remove the element from the index
         ...

That raises another question - if an entry is updated and no longer matches the criteria, is it it removed from the index? My tests don't show that it is.

Eg. if I have a lightswitch field, showInSearch

    ->criteria(fn(EntryQuery $query) => $query->showInSearch(true))

And I toggle the field then save the element, the element isn't removed from the index.

This seems like a fairly big bug, unless I've overlooked something or misunderstood the usage?

[janhenckens]

Hey @MangoMarcus

Thanks for bumping this issue. I'm looking at a similar bug in #281, and your pseudo-code could be a good starter for a fix.

I gather you're seeing the issue regardless of wether your using splitElementsOn or not?

[MangoMarcus]

Hey @janhenckens , thanks for the speedy response, I've subscribed for that bug now.

I'm not using splitElementsOn, I haven't tested it to be honest but will give it a go when time allows 🙂

[mathg]

I am having the same issue here. When an element is deleted, it is not removed from Algolia automatically. I had to add a afterDelete hook to make sure they are deleted.

Event::on(Entry::class, Element::EVENT_AFTER_DELETE, function (Event $e) {
    $entry = $e->sender;
    $objectIDs = [$entry->id];

    // On delete, remove entry from all indexes
    $sites = Craft::$app->sites->getAllSites();
    foreach($sites as $site) {
        $index = Craft::$container->get(\Algolia\AlgoliaSearch\SearchClient::class)->initIndex(App::env('ALGOLIA_INDEX_PREFIX') . 'default_' . $site->handle);
        $index->deleteObjects($objectIDs);
    }
});

Still having an issue with records that have splitElementsOn as the deleteObjects is not working on them because their objectID is 1234_0, 1234_1, etc.

EDIT : If the element is splitElementsOn, it's possible to delete them that way

$index = Craft::$container->get(\Algolia\AlgoliaSearch\SearchClient::class)->initIndex(App::env('ALGOLIA_INDEX_PREFIX') . 'event_' . $handle);
$index->deleteBy([
    'filters' => 'distinctID:' . implode(' OR distinctID:', $objectIDs),
]);

[janhenckens]

@mathg Could you try the fix I mentioned here and report back if that fixes things? #281 (comment)

[low]

@janhenckens Hijacking this thread to add some findings.

The above custom deleteBy filter only works for string values. The distinctID is a numeric value if we're using the standard element ID, so the correct syntax would need to be 'filters' => 'distinctID = 123'.

It would be great if Scout could check this in some way and use the correct filter syntax accordingly.

[janhenckens]

Hey @low

Scout doesn't currently track objectIds and assumes the default elementId (which obviously isn't always correct).

I don't think I want to add the tracking for that in the Craft database, so maybe a clean example in the docs in case someone is using anything other then elementId could be a work around?

[GaryReckard]

I'm running into this issue now, for the same reason @twosixcode was, I'm appending the site handle on to the end of my objectID.

In the update() method in AlgoliaEngine, the objects get transformed, and therefore get my custom objectID:

craft-scout/src/engines/AlgoliaEngine.php

Line 48 in e59aa31

$objects = $this->transformElements($elements);

I'm curious if there's a reason that step is excluded from the delete() method.

If this bit in delete():

craft-scout/src/engines/AlgoliaEngine.php

Lines 66 to 72 in e59aa31

	$objectIds = $elements->map(function($object) {
	if ($object instanceof Element) {
	return $object->id;
	}
	return $object['distinctID'] ?? $object['objectID'];
	})->unique()->values()->all();

was replaced with

        $objects = $this->transformElements($elements);

        $objectIds = collect($objects)->map(function ($object) {
            return $object['distinctID'] ?? $object['objectID'];
        })->unique()->values()->all();

it has my custom objectID, and successfully deletes my element from my index when I disable it.

Perhaps it is because transforming could potentially be expensive, with things like image transforms, etc, and normally wouldn't be necessary when deleting?

[harry2909]

Getting this too with a pretty simple scout setup. It looks like deleting entries or setting them to disabled doesn't remove them from the index. What is the workaround for this? I tried with split elements and this didn't seem to work.

ScoutIndex::create($env . '_people')
            ->elementType(Entry::class)
            ->criteria(fn(EntryQuery $query) => $query
                ->hidePerson(false)
                ->section('people')
            )
            ->transformer(function (Entry $person) use ($imageTransformService) {
                if ($person->hidePerson) {
                    return [];
                }
                /** @var craft\elements\Asset|null $image */
                $image = $person->profileImage->eagerly()->one();
                $sizes = [
                    'xs' => ['width' => 300, 'height' => 300],
                ];
                $nameParts = explode(' ', $person->title);
                // Parse the surname from title
                return [
                    'objectID' => $person->id . '-' . $person->site->id,
                    'title' => $person->title,
                    /*
                     * This is used in the algolia search ranking settings to order the results alphabetically by surname
                     * @see https://dashboard.algolia.com/apps/POGHSC6W6T/explorer/configuration/staging_people/ranking-and-sorting
                     */
                    'surname' => end($nameParts) ?: null,
                    'jobRoleGroup' => (int)$person->relatedJobRoleGroup->select('peopleSearchRanking')->eagerly()->scalar() ?: 99999,
                    'jobRole' => $person->jobRole,
                    'offices' => $person->relatedOffices->select('title')->eagerly()->column(),
                    'services' => array_merge(
                        $person->relatedBusinessServices->select('title')->eagerly()->column(),
                        $person->relatedPersonalServices->select('title')->eagerly()->column(),
                    ),
                    'url' => $person->url,
                    'image' => $image ? $imageTransformService->getFrontendImageData($image, $sizes) : null,
                ];
            })
            ->indexSettings(
                IndexSettings::create()
                    ->searchableAttributes(['title'])
                    ->attributesForFaceting([
                        'searchable(services)',
                    ])
                    ->removeWordsIfNoResults('allOptional')
                    ->hitsPerPage(12)
                    ->minWordSizefor1Typo(4)
            ),

Any ideas? Thanks!

[GaryReckard]

Hey @harry2909

I'm not sure if this is the best approach, but here is how I got around this issue.

I set up a couple event handlers

        Event::on(Entry::class, Element::EVENT_AFTER_SAVE, new AfterSaveEntryHandler);
        Event::on(Entry::class, Element::EVENT_AFTER_DELETE, new AfterDeleteEntryHandler);

That AfterDeleteEntryHandler look like this:

<?php

namespace modules\algoliamodule\handlers;

use modules\algoliamodule\AlgoliaModule;
use yii\base\Event;

class AfterDeleteEntryHandler
{

    public function __invoke(Event $event)
    {
        $element = $event->sender;

        if ($element instanceof \craft\elements\Entry) {
            $objectId = $element->id . '_' . $element->site->handle;
            AlgoliaModule::getInstance()->algolia->deleteObjects([$objectId]);
        }

    }
}

... basically just manually deleting the object (using my customized objectId) when an entry is deleted...

And the AfterSaveEntryHandler looks like this:

<?php

namespace modules\algoliamodule\handlers;

use craft\elements\Entry;
use craft\events\ModelEvent;
use craft\helpers\App;
use craft\helpers\ElementHelper;
use modules\algoliamodule\AlgoliaModule;
use nystudio107\seomatic\Seomatic;

class AfterSaveEntryHandler
{

    public function __invoke(ModelEvent $event)
    {
        $entry = $event->sender;

        if (!filter_var(App::env('ALGOLIA_SYNC_ENABLED'), FILTER_VALIDATE_BOOLEAN)) {
            return;
        }

        /** @var Entry $entry */
        if (ElementHelper::isDraftOrRevision($entry)) {
            return;
        }

        $objectId = $entry->id . '_' . $entry->site->handle;

        if(!$entry->enabledForSite) {
            AlgoliaModule::getInstance()->algolia->deleteObjects([$objectId]);
            return;
        }
        if(!$entry->enabled) {
            AlgoliaModule::getInstance()->algolia->deleteObjects([$objectId]);
            return;
        }
        if(!in_array($entry->status, ['live','enabled'])) {
            AlgoliaModule::getInstance()->algolia->deleteObjects([$objectId]);
            return;
        }
        if($entry->archived) {
            AlgoliaModule::getInstance()->algolia->deleteObjects([$objectId]);
            return;
        }

        // if the SEOMatic Robots setting is to not index, remove this from index
        if(!empty($entry->uri)){
            Seomatic::$plugin->helper->loadMetadataForUri($entry->uri);

            // If SEOMatic settings are that this entry should not be indexed... do not index
            if(in_array(Seomatic::$plugin->metaContainers->metaGlobalVars->robots, ['none','noindex'])){
                AlgoliaModule::getInstance()->algolia->deleteObjects([$objectId]);
            }
        }

    }
}

which manually removes the entry from Algolia (again, using my custom $objectId format, whenever the entry is saved and is disabled, archived, or set to noindex in SEOMatic.

Seems to be working for me. I do see multiple deletes come in to Algolia, one still for the plain numeric $objectId and one for my customized $objectId coming from these event handlers.

Hope that is helpful!

[harry2909]

Thanks @GaryReckard I'll give that a try!