CATL-1613: Add options to Mail Account settings to improve inbound mail processing

[i-grou]

Overview

According to new requirements we need to add an additional options to the Mail Account settings (see Mail Accounts page /civicrm/admin/mailSettings?action=add&reset=1) to change the email to activity processing behavior - when Email-to-Activity Processing value is selected for Used For? field we need to:

1. Show an additional checkbox field - Skip emails which do not have a Case ID or Case token (defaults to unticked).
   If checked: emails which do not have a valid case ID or case token should be moved into the ignored mail folder after processing and no activity should be created.
2. Show an additional checkbox field - Do not create new contacts when filing emails (defaults to unticked).
   If checked: when CiviCRM checks for a matching contact, if no matching contact is found it will not create one and the email is filed.
   So now administrator should be able to configure whether CiviCRM will create a new contact where a contact does not already exist within the system when filing inbound emails, so that he can ensure that new contacts are not created in the system when filing an email. The email should still be filed, even if no contacts are matched.

Email is considered to be related to cases if it's subject matches any of the following conditions:

• Starts with [<any string> #<any string>], e.g: [case #2cd8467], [policy initiative #bdf3896].
• Ends with Case ID <number>, e.g: Case ID 1298, case id 9.

Also:

• Label of new field is printed to the right of the checkbox, not to the left, because left side is thin and long labels would look weird there.
• Activity Status field label is also moved to the right for consistency. Thanks to this approach these fields would look more like Used For? dependent fields, and this would make them look more intuitive hence will improve UX.

Before

Without the new options above enabled CiviCRM will process emails as normal i.e:

1. Will file emails without a case ID or case hash in the subject line against a contact record.
2. Will always create new contacts if no matching contact found.

And Activity Status field label location is not changed.

After

With the new options above enabled CiviCRM:

1. Will skip any emails that do not have either the case ID or case hash in the subject line and will instead move those emails to the ignored folder.
2. Will not create new contacts if no matching contact found (from / to / cc / bcc), but email still will be filed.

Now form looks like this:

Technical Details

To make the Skip emails which do not have a Case ID or Case token checkbox work we:

1. Added new field to the civicrm_mail_settings db table - is_non_case_email_skipped. Respective upgrader is added.
2. Updated schema MailSettings.xml file to have new field.
3. Updated form related files - Form/MailSettings.php and MailSettings.tpl - to print new field and save the value on form submit. As it has long help text it was desided to move it to MailSettings.hlp file and use civicrm ajax help popup feature.
4. Updated EmailProcessor.php to handle new field during inbound email processing.
5. The CRM_Utils_Mail_CaseMail class is added to store isCaseEmail() method, which is used to check if email is related to cases or not before email processing (creating new activity). The regexp patterns are copied from CRM_Activity_BAO_Activity->create() method for consistency, as it also performs this check.
6. The caseEmailSubjectPatterns() hook is added to make it possible to change default email subject regexp patterns. It may be useful for example for CiviCase extension to add case type category names to patterns.
7. Unit tests were added to test processing (with is_non_case_email_skipped enabled) of 2 emails: with case hash in subject and without case hash.
8. Also as part of this PR formatting of MailSettings.tpl file is fixed.
9. Also as part of this PR the deprecated _civicrm_api3_deprecated_activity_buildmailparams() method was moved to CRM_Utils_Mail_ParamsBuilder class that was just created and now we can use CRM_Utils_Mail_ParamsBuilder->buildActivityParams() instead. This was done to get rid of some toxic code.

To make the Do not create new contacts when filing emails checkobx work we:

1. Added new field to the civicrm_mail_settings db table - is_contact_creation_disabled_if_no_match. Respective upgrader is added.
2. Updated DAO/MailSettings.php and schema MailSettings.xml files to have new field.
3. Updated form related files - Form/MailSettings.php and MailSettings.tpl - to print new field and save the value on form submit.
4. Updated EmailProcessor.php to handle new field during inbound email processing. This also involved updating the Incoming.php to make contact creation optional.
5. Unit tests were added to test processing with is_contact_creation_disabled_if_no_match enabled.

About the new CRM_Utils_Mail_CaseMail class

This class has a potential to become generic and widely used class, thats why:
• Email subject patterns are not just hardcoded, but are reusable, see getSubjectPatterns() method.
• The getSubjectPatterns() method is written keeping in mind CiviCase extension (thats why caseEmailSubjectPatterns() hook is added), which has case type categories, which names could be used in email subject, e.g: [case #hahoheh] Super email -> [policy initiative #hahoheh] Super email.

For now the most useful part of this class is isCaseEmail() method, but some other methods may be added later. For example:

1. Generate case hash - there is a code duplication in civicrm to generate case hash, i.e:
   $hash = substr(sha1(CIVICRM_SITE_KEY . $form->_caseId), 0, 7).
2. Generate case subject - there is a code duplication in civicrm to generate case related mail subject, i.e:
   $subject = "[case #$hash] $subject".

⚠ The email regexp patterns are copied from CRM_Activity_BAO_Activity->create() and now it makes sense to refactor that method to use CRM_Utils_Mail_CaseMail->getSubjectPatterns() to get patterns. But will skip it now as it's not in the scope of the task.

One more comment

I've noticed that civicrm stops processing of inbound email if it failed to connect to some mail account. This may cause some issues in case if there are several accounts created: if processing would be stopped on first account - then no other accounts would be processed as well.
I guess this should be added to the todo list to fix in the nearest future.