Build: 2024022702

.gitignore

@@ -1,2 +1,3 @@
 .vscode
-.phpcs.xml
+.phpcs.xml
+.DS_Store

CHANGES.md

@@ -1,5 +1,11 @@
 # Changelog
 
+## 2024022202
+
+- Major Refactor
+- Better mlang support
+- Fixed various bugs after testing with production data
+
 ## 2024021700
 
--   Initial Release
+- Initial Release

README.md

@@ -1,26 +1,46 @@
 # Moodle Autostranslate Filter
 
-[![Latest Release](https://img.shields.io/github/v/release/jamfire/moodle-filter_autotranslate)](https://github.com/jamfire/moodle-filter_autotranslate/releases)
-[![Moodle Plugin CI](https://github.com/jamfire/moodle-filter_autotranslate/actions/workflows/moodle-ci.yml/badge.svg)](https://github.com/jamfire/moodle-filter_autotranslate/actions/workflows/moodle-ci.yml)
+[![Latest Release](https://img.shields.io/github/v/release/jamfire/moodle-filter_autotranslate)](https://github.com/jamfire/moodle-filter_autotranslate/releases) [![Moodle Plugin CI](https://github.com/jamfire/moodle-filter_autotranslate/actions/workflows/moodle-ci.yml/badge.svg)](https://github.com/jamfire/moodle-filter_autotranslate/actions/workflows/moodle-ci.yml) [![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=jamfire_moodle-filter_autotranslate&metric=alert_status)](https://sonarcloud.io/summary/new_code?id=jamfire_moodle-filter_autotranslate)
+
+## Requirements
+
+- PHP 8.2+
+- Moodle 4.2+
+
+## Limitations
+
+- Only supports 2 letter locale codes.
+- Limited multilang ({mlang}) support. This filter will process your existing translations in mlang. See the Multilang Support section for more information.
+- This filter only supports languages supported in DeepL.
 
 ## Installation
 
--   Unzip the plugin in the moodle .../filter/ directory.
+- Unzip the plugin in the moodle .../filter/ directory.
 
 This plugin is in an Alpha state and it is highly recommended that you backup your database before installing and enabling this plugin.
 
 ## Non-destructive translation
 
-This autotranslation filter does not alter the original text of any of your content on Moodle. It stores all source text and translations in a translation table. The filter works by retrieving the source text from the translation table after it has been saved for the first time. If in the event something goes arwy, disable the filter and all of your original text will be restored immediately.
+This autotranslation filter does not alter the original text of any of your content on Moodle. It stores all source and target text in a translation table. The filter works by retrieving the source text from the translation table after it has been saved for the first time. If in the event something goes arwy, disable the filter and all of your original text will be restored immediately.
 
 You need to consider your database performance and size when using this plugin. It will effectively double the size of your database if every page of your Moodle site is visited because all source text is being saved as a reference.
 
-## Multlang support
+## Multilang support
+
+This plugin provides limited support for [multilang](https://docs.moodle.org/403/en/Multi-language_content_filter) and [multilang2.](https://moodle.org/plugins/filter_multilang2) A custom parser has been written to find existing translations on your website and store those in the autotranslate table instead of fetching new translations from DeepL. You should enable the translator on each context where you have used multilang in the Autotranslate filter settings.
+
+### Database prepwork
+
+If you choose to leave the Multilang and Multilang 2 filters on, you need to do some database prepwork before enabling the Autotranslate filter. You should have a default site language set and know the 2 letter local code for that language. We will use en (English) as an example.
+
+Navigate to `/admin/tool/replace` and perform a search and replace across your entire database for any `{mlang en}` tags and replace them with `{mlang other}`. It goes without saying that you should backup your database before performing this step. This can also be done via the [CLI.](https://docs.moodle.org/403/en/Search_and_replace_tool) This step is needed for the Autotranslate filter to work correctly. The `other` lang code is needed so that text can be referenced for pages without a translation.
+
+### Multilang 2 formatting requirements
 
-This plugin provides limited support for [multilang](https://docs.moodle.org/403/en/Multi-language_content_filter) and [multilang2.](https://moodle.org/plugins/filter_multilang2). A custom parser has been written to find existing translations on your website and store those in the autotranslate table instead of fetching new translations from DeepL. This requires that you have only used a single {mlang} tag per translation in your content using the following format:
+This filter was developed against a production database where mlang had been wildly used. It is impossible to account for every variation of how someone might have used `{mlang}` tags and whether they used them properly. This filter requires that you have only used a single {mlang} tag per translation in your content using the following format:
 
 ```
-{mlang en}Hello{mlang}
+{mlang other}Hello{mlang}
 {mlang es}Hola{mlang}
 {mlang fr}Bonjour{mlang}
 ```
@@ -29,13 +49,13 @@ This filter does not support the following structure:
 
 ```
 Moodle
-{mlang en}Hello{mlang}
+{mlang other}Hello{mlang}
 {mlang es}Hola{mlang}
 {mlang fr}Bonjour{mlang}
-{mlang en}Goodbye{mlang}
+{mlang other}Goodbye{mlang}
 ```
 
-The word Moodle would be stripped out from your translation and you would end up with the words Hola, Bonjour, Goodbye in the translation table. The first Hello would be lost because of the duplicate "en" key.
+The word Moodle would be stripped out from your translation and you would end up with the words Hola, Bonjour, Goodbye in the translation table. The first Hello would be lost because of the duplicate "other" key.
 
 ## DeepL integration and plugin settings
 
@@ -47,8 +67,8 @@ You can signup for a free or pro version key of DeepL's [API.](https://www.deepl
 
 ## Enable the filter and move it to the top
 
--   Go to "Site Administration >> Plugins >> Filters >> Manage filters" and enable the plugin there.
--   It is recommended that you position the Autotranslate Filter at the top of your filter list and enable it on headings and content.
+- Go to "Site Administration >> Plugins >> Filters >> Manage filters" and enable the plugin there.
+- It is recommended that you position the Autotranslate Filter at the top of your filter list and enable it on headings and content.
 
 ## Autotranslation scheduled task
 

classes/autotranslate/translator.php

@@ -36,7 +36,7 @@ class translator {
     /**
      * @var \DeepL\Translator $translator DeepL Translator
      */
-    public \DeepL\Translator $translator;
+    public ?\DeepL\Translator $translator;
 
     /**
      * @var array $sourcelangs Supported Source Languages
@@ -66,11 +66,13 @@ public function __construct() {
         // Get the api key from settings.
         $authkey = get_config('filter_autotranslate', 'deeplapikey');
         if (!$authkey) {
-            throw new \moodle_exception('missingapikey', 'filter_autotranslate');
+            $this->translator = null;
+        } else {
+            // Load deepl translator.
+            $this->translator = new \DeepL\Translator($authkey);
         }
 
-        // Load deepl translator.
-        $this->translator = new \DeepL\Translator($authkey);
+        // Get the langs supported by the site.
         $this->langs = get_string_manager()->get_list_of_translations();
     }
 
@@ -134,13 +136,13 @@ public function getsupportedglossarylangs() {
      * Get DeepL Usage
      */
     public function getusage() {
-        return $this->translator->getUsage();
+        return $this->translator ? $this->translator->getUsage() : 0;
     }
 
     /**
      * Get Glossaries.
      */
     public function listglossaries() {
-        return $this->translator->listGlossaries();
+        return $this->translator ? $this->translator->listGlossaries() : [];
     }
 }

classes/output/glossary_form.php

@@ -60,7 +60,6 @@ public function definition() {
 
         // URL params for pagination.
         $this->urlparams = [];
-        $this->urlparams['source_lang'] = $sourcelang;
         $this->urlparams['target_lang'] = $targetlang;
         $this->urlparams['limit'] = $limit;
 

classes/output/glossary_page.php

@@ -36,7 +36,7 @@
  */
 class glossary_page implements renderable, templatable {
     /**
-     * @var string $this->sitelang Default Moodle Language
+     * @var string $sitelang Default Moodle Language
      */
     private string $sitelang;
 
@@ -46,12 +46,12 @@ class glossary_page implements renderable, templatable {
     private array $langs;
 
     /**
-     * @var string $source_lange Source language of the text
+     * @var string $sourcelang Source language of the text
      */
     private string $sourcelang;
 
     /**
-     * @var string $target_lang Target language of the text
+     * @var string $targetlang Target language of the text
      */
     private string $targetlang;
 
@@ -114,8 +114,7 @@ public function __construct() {
         // Qury params.
         $this->sitelang = get_config('core', 'lang', PARAM_NOTAGS);
         $this->langs = $translator->getsupportedglossarylangs();
-        $this->sourcelang = optional_param('source_lang', $this->sitelang, PARAM_NOTAGS);
-        $this->sourcelang = clean_param($this->sourcelang, PARAM_NOTAGS);
+        $this->sourcelang = $this->sitelang;
         $this->targetlang = optional_param('target_lang', $this->sitelang, PARAM_NOTAGS);
         $this->targetlang = clean_param($this->targetlang, PARAM_NOTAGS);
         $this->page = optional_param('page', 1, PARAM_INT);
@@ -135,7 +134,6 @@ public function __construct() {
 
         // Url params.
         $urlparams = [];
-        $urlparams['source_lang'] = $this->sourcelang;
         $urlparams['target_lang'] = $this->targetlang;
         $urlparams['limit'] = $this->limit;
         $urlparams['page'] = $this->page;

classes/output/manage_form.php

@@ -63,7 +63,6 @@ public function definition() {
 
         // URL params for pagination.
         $this->urlparams = [];
-        $this->urlparams['source_lang'] = $sourcelang;
         $this->urlparams['target_lang'] = $targetlang;
         $this->urlparams['limit'] = $limit;
         if ($instanceid) {

classes/output/manage_page.php

@@ -48,14 +48,13 @@ class manage_page implements renderable, templatable {
      */
     private array $langs;
 
-
     /**
-     * @var string $source_lange Source language of the text
+     * @var string $sourcelang Target language of the text
      */
     private string $sourcelang;
 
     /**
-     * @var string $target_lang Target language of the text
+     * @var string $targetlang Target language of the text
      */
     private string $targetlang;
 
@@ -126,8 +125,7 @@ public function __construct() {
         // Qury params.
         $this->sitelang = get_config('core', 'lang', PARAM_NOTAGS);
         $this->langs = get_string_manager()->get_list_of_translations();
-        $this->sourcelang = optional_param('source_lang', $this->sitelang, PARAM_NOTAGS);
-        $this->sourcelang = clean_param($this->sourcelang, PARAM_NOTAGS);
+        $this->sourcelang = $this->sitelang;
         $this->targetlang = optional_param('target_lang', $this->sitelang, PARAM_NOTAGS);
         $this->targetlang = clean_param($this->targetlang, PARAM_NOTAGS);
         $this->instanceid = optional_param('instanceid', null, PARAM_INT);
@@ -152,7 +150,6 @@ public function __construct() {
 
         // Url params.
         $urlparams = [];
-        $urlparams['source_lang'] = $this->sourcelang;
         $urlparams['target_lang'] = $this->targetlang;
         $urlparams['limit'] = $this->limit;
         $urlparams['page'] = $this->page;

classes/task/autotranslate_task.php

@@ -69,7 +69,7 @@ public function execute() {
         // Get 100 existing jobs.
         $jobs = $DB->get_records('filter_autotranslate_jobs', [
             'fetched' => '0',
-            'source_missing' => '0',
+            'source' => '1',
         ], null, '*', 0, $fetchlimit);
         $jobscount = count($jobs);
         mtrace("$jobscount jobs found...");
@@ -101,6 +101,7 @@ public function execute() {
                 // Get the translation.
                 $options = [];
                 $options['formality'] = 'prefer_more';
+                $options['tag_handling'] = 'html';
                 if ($glossaryid) {
                     $options['glossary'] = $glossaryid;
                 }
@@ -129,24 +130,24 @@ public function execute() {
                 }
 
                 // Update the job to fetched.
-                $jid = $DB->update_record(
+                $DB->set_field(
                     'filter_autotranslate_jobs',
+                    'modified_at',
+                    time(),
                     [
                         'id' => $job->id,
-                        'fetched' => "1",
                     ]
                 );
-
-                mtrace("completed job $job->id...");
-            } else if (!$sourcerecord) {
-                // Update the job to fetched.
-                $jid = $DB->update_record(
+                $jid = $DB->set_field(
                     'filter_autotranslate_jobs',
+                    'fetched',
+                    '1',
                     [
                         'id' => $job->id,
-                        'source_missing' => "1",
                     ]
                 );
+
+                mtrace("completed job $job->id...");
             }
         }
     }

classes/task/check_source_task.php

@@ -0,0 +1,110 @@
+<?php
+// This file is part of Moodle - http://moodle.org/
+//
+// Moodle is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+//
+// Moodle is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with Moodle.  If not, see <http://www.gnu.org/licenses/>.
+
+namespace filter_autotranslate\task;
+
+/**
+ * Autotranslate Jobs
+ *
+ * Checks against the job database for unfetched translations
+ *
+ * @package    filter_autotranslate
+ * @copyright  2024 Kaleb Heitzman <[email protected]>
+ * @license    http://www.gnu.org/copyleft/gpl.html GNU GPL v3 or later
+ */
+class check_source_task extends \core\task\scheduled_task {
+    /**
+     * Get name of the Check Source Task
+     */
+    public function get_name() {
+        return get_string('checksourcetask', 'filter_autotranslate');
+    }
+
+    /**
+     * Execute the Autotranslate Task
+     */
+    public function execute() {
+        global $DB;
+
+        // Get the site language.
+        $sitelang = get_config('core', 'lang');
+
+        // Get the fetch limit.
+        $fetchlimit = get_config('filter_autotranslate', 'fetchlimit');
+        if (!$fetchlimit) {
+            $fetchlimit = 200;
+        }
+
+        // Your task logic goes here.
+        mtrace("Executing autotranslation check source tasks...");
+
+        // Get 100 existing jobs.
+        $jobs = $DB->get_records('filter_autotranslate_jobs', [
+            'fetched' => '0',
+            'source' => '0',
+        ], null, '*');
+        $jobscount = count($jobs);
+        mtrace("$jobscount jobs found...");
+
+        // Iterate through the jos and add translations.
+        foreach ($jobs as $job) {
+            mtrace("checking $job->lang source for $job->hash key...");
+
+            // Get the source text.
+            $sourcerecord = $DB->get_record('filter_autotranslate', ['hash' => $job->hash, 'lang' => $sitelang], 'id');
+            if ($sourcerecord) {
+                // Update the job to fetched.
+                mtrace("source for $job->id found...");
+
+                // Set the source to 1.
+                $DB->set_field(
+                    'filter_autotranslate_jobs',
+                    'source',
+                    '1',
+                    [
+                        'id' => $job->id,
+                    ]
+                );
+
+                // Set the modified time.
+                $DB->set_field(
+                    'filter_autotranslate_jobs',
+                    'modified_at',
+                    time(),
+                    [
+                        'id' => $job->id,
+                    ]
+                );
+            } else {
+                // No source record found and we should
+                // see if the record should be deleted.
+                $currenttime = time();
+                $timediff = $currenttime - $job->modified_at;
+                $timelimit = 60 * 60 * 24 * 7; // 7 days
+
+                if ($timediff > $timelimit) {
+                    mtrace("check source job $job->id for $job->lang has expired, cleaning up...");
+                    $DB->delete_records(
+                        'filter_autotranslate_jobs',
+                        [
+                            'id' => $job->id,
+                        ]
+                    );
+                }
+            }
+        }
+    }
+}