Disabling Scout sync during Feed Me Import

[mgburns]

I'm trying to figure out how I can temporarily disable Algolia sync during Feed Me imports.

I know it can be done manually via the settings page if we omit it from config/scout.php but we're hiding those in prod so I'm trying to figure out a way to do it programmatically. I see that Feed Me fires events before / after feed processing but I'm not sure what I would do on the Scout end to make this work.

[timkelty]

@mgburns Yep, agreed this is annoying. Adding to my list!

[lenvanessen]

@timkelty would you accept a PR for this? Need this as well. Most likely will go with the same approach as Blitz (a batch mode flag)

[janhenckens]

Hey @lenvanessen! Sure, happy to review a PR for this if you can write it up 👍

[joepagan]

Just chipping in that if your indices are limited to just commerce elements run on the import you could set:

scout.php config:

'sync' => false,

feedme event

Event::on(
  Process::class,
  Process::EVENT_AFTER_PROCESS_FEED,
    function(FeedProcessEvent $event) {
      try {
        Craft::$app->runAction('scout/index/refresh');
      } catch (InvalidRouteException | Exception $e) {
        Craft::error($e->getMessage(), __METHOD__);
        throw new Exception($e->getMessage());
      }
    }
);

[jamesmacwhite]

Has anyone made any progress in turning off Algolia sync events during a FeedMe import or deferring until the end?

Disabling the sync outright prevents general entries that are updated by Control Panel users being synchronised automatically, so it's not an ideal solution, you can hook into various events with FeedMe to trigger imports instead, but I was wondering if anyone had thoughts of a solution while keeping the original sync option enabled.

[lenvanessen]

@jamesmacwhite I've done it using the a listener on the queue:

 /**
     * We prevent the jobs from Scout to be triggered during CP requests.
     * This way we don't bloat the Queue with syncing products individually, but we can
     * just trigger a bulk sync after the import has completed
     */
    private function _preventSyncJobs(): void
    {
        Event::on(
            Queue::class,
            Queue::EVENT_BEFORE_PUSH,
            function(PushEvent $event) {
                if($event->job instanceof MakeSearchable) {
                    $event->handled = true;
                }
            }
        );
    }
    ```
    
And then, i've bound this to only be triggered if it's a console request in my modules init:

if (Craft::$app->getRequest()->isConsoleRequest) {
$this->_preventSyncJobs();
}

[jamesmacwhite]

@lenvanessen Interesting solution, I'll take a look.

You can technically disable the queue jobs with queue => false, this obviously has performance implications, but if you run the queue with a worker that's separate you can do that without too many issues, but it does mean they are being triggered still individually on feed importing.

I am however a fan of triggering these at the end of a successful feed import, leveraging feed events, because it will probably also speed up the import process.

Do you still have sync enabled with this event before push code, so general entry saves in the CP aren't impacted.

That's my main concern, the feed imports are one thing, I still want the automatic sync behaviour to work when entries not imported via FeedMe are modified.

[lenvanessen]

@jamesmacwhite, there are a couple of solutions you can go with, for one, you can temporary change the .env. You can see an example in the SetupController from Craft, where they set certain variables.

So in your case, you could set the scout.php to:

'queue' => App::env('ENABLE_SCOUT_QUEUE') === 'true';

And then using the feed-me imports

Event::on(
  Process::class,
  Process::EVENT_BEFORE_PROCESS_FEED,
    function(FeedProcessEvent $event) {
        Craft::$app->getConfig()->setDotEnvVar('ENABLE_SCOUT_QUEUE', false);
    }
);

And once the import is done:

Event::on(
  Process::class,
  Process::EVENT_AFTER_PROCESS_FEED,
    function(FeedProcessEvent $event) {
        Craft::$app->getConfig()->setDotEnvVar('ENABLE_SCOUT_QUEUE', true);
    }
);

Or, if you don't want to touch the .env, you could set a mutex lock:

Event::on(
  Process::class,
  Process::EVENT_BEFORE_PROCESS_FEED,
    function(FeedProcessEvent $event) {
         Craft::$app->getMutex()->acquire('FEED_ME_PROCESSING', 15);
    }
);

And then make the code I previously shared like this:

  Event::on(
            Queue::class,
            Queue::EVENT_BEFORE_PUSH,
            function(PushEvent $event) {
                if($event->job instanceof MakeSearchable) {
                    $event->handled =  Craft::$app->getMutex()->isAcquired('FEED_ME_PROCESSING');
                }
            }
        );

And then after the import is done, release it

Event::on(
  Process::class,
  Process::EVENT_AFTER_PROCESS_FEED,
    function(FeedProcessEvent $event) {
          Craft::$app->getMutex()->release('FEED_ME_PROCESSING');
    }
);

[jamesmacwhite]

@lenvanessen Thanks. This is an interesting solution! My concern around using a mutex lock would be you could potentially trigger a mutex lock error if Feed Me processing would be happening to be occurring while another entry that's connected to Algolia events (not part of FeedMe) tries to be updated at the same time, but I'll have to do some testing.