Feature/multilingual #943

SnowMasaya · 2024-10-09T00:40:33Z

This revision enables translation support for a streamlined the use case. Some items such
as support for multiple target languages in a single run have been omitted in favor of a
more straight forward user experience and reduced ambiguity in the scope of results.

Translator classes have been implemented using the Configurable pattern and the plugin
loader. This introduced a new paradigm of providing configuration for a list of instances
with specific configuration required at runtime where previous Configurable class
configuration has been for all instances of a specific class or module. The processing and
attribute names used to create this instance list may evolve further.

Usage

Translation function is configured in the run section of a configuration see the doc
page in the PR for details.

New default configuration values for run.lang_spec and run.translators are in the
updated documentation and allow for backwards compatible configuration with existing runs.

There are still some existing TODO: comments and notes about location that may need
further testing before landing this upstream. Most noteworthy are comments still in the
code of the atkgen probe that require further scrutiny to validate the attack technique
is applied correctly.

It may be appropriate to gate this functionality as experimental for initial release, this
would required some additional guard code to ensure limited impact to report formats and
internal state.

Example

python -m garak -m huggingface.Model --config hf_RigoChat_gpu.yml -p lmrc --report_prefix RigoChat-21fde039

hf_RigoChat_gpu.yml:

run:
  lang_spec: "es"
  translators:
    - language: es-en
      model_type: local
      model_name: facebook/m2m100_418M
      hf_args:
        device: cuda
    - language: en-es
      model_type: local
      model_name: facebook/m2m100_418M
      hf_args:
        device: cuda
plugins:
  generators:
    huggingface:
      Model:
        name: IIC/RigoChat-7b-v2
        hf_args:
          device: cuda
          trust_remote_code: true
          torch_dtype: float16

github-actions · 2024-10-09T00:40:47Z

DCO Assistant Lite bot All contributors have signed the DCO ✍️ ✅

- Integrated support for multiple translation services including local and external APIs. - local: Huggingface model uses for translation - deepl:DeepL uses for translation - nim: NIM uses for translation - Implemented utility functions for language detection and text processing. Signed-off-by: Masaya Ogushi <[email protected]>

- Addd translation function for base probe class - prompts and triggers translate by base class method - attempt_descr translation Signed-off-by: Masaya Ogushi <[email protected]>

- Translation handling for detector keywords and substrings, triggers. Signed-off-by: Masaya Ogushi <[email protected]>

- Added support for specifying translation services directly from the CLI. - Implemented options to set target languages for translation. Signed-off-by: Masaya Ogushi <[email protected]>

- Added new dependencies required for enhanced translation features. Signed-off-by: Masaya Ogushi <[email protected]>

- Added detailed explanations of the translationn method - Included examples of how translation services are configured and utilized within the codebase. Signed-off-by: Masaya Ogushi <[email protected]>

SnowMasaya · 2024-10-09T04:55:52Z

I have read the DCO Document and I hereby sign the DCO

jmartin-tech

Partial review, testing is still in progress.

The test failures in macOS look to be an incomplete dependency requirement that may need to be reworked or removed. A default installation should not require install of an external library. Hence the dependency on pyenchant here may be problematic.

Traceback:
/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/importlib/__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/detectors/test_detectors_riskywords.py:8: in <module>
    import garak.detectors.base
garak/detectors/__init__.py:1: in <module>
    from .base import *
garak/detectors/base.py:17: in <module>
    from garak.translator import SimpleTranslator, LocalTranslator, is_english
garak/translator.py:15: in <module>
    import enchant
/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/enchant/__init__.py:81: in <module>
    from enchant import _enchant as _e
/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/enchant/_enchant.py:[157](https://github.com/leondz/garak/actions/runs/11246708414/job/31296143918?pr=943#step:5:158): in <module>
    raise ImportError(msg)
E   ImportError: The 'enchant' C library was not found and maybe needs to be installed.
E   See  https://pyenchant.github.io/pyenchant/install.html
E   for details

requirements.txt

garak/probes/base.py

tests/detectors/test_detectors.py

tests/detectors/test_leakreplay.py

tests/test_translator.py

garak/cli.py

tests/test_translator.py

garak/translator.py

update remove punctuation update english judge add translate function add logging translate result add Reverse translate for hf detector and snowball probes Signed-off-by: Masaya Ogushi <[email protected]>

check translator instance remove translate function reset config Signed-off-by: Masaya Ogushi <[email protected]>

check translator instance add reverse translator add test reverse translator Signed-off-by: Masaya Ogushi <[email protected]>

remove argument using generator_option_file Signed-off-by: Masaya Ogushi <[email protected]>

add load translator instance Signed-off-by: Masaya Ogushi <[email protected]>

check storage size set up each instance for each test Signed-off-by: Masaya Ogushi <[email protected]>

remove pyenchant Using nltk instead of pyenchant Signed-off-by: Masaya Ogushi <[email protected]>

update how to use translation function Signed-off-by: Masaya Ogushi <[email protected]>

…garak into feature/multilingual

Signed-off-by: SnowGushiGit <[email protected]>

Signed-off-by: Masaya Ogushi <[email protected]>

…garak into feature/multilingual

leondz

Thank you, this is a significant amount of work.

This is for translation-based multilingual, rather than general multilingual support, so we should be careful to separate these two functions. The ability to select which languages garak uses is distinct from how that language is achieved (in prompts, detectors, and so on).

There are some refactorings needed to get the PR to a sustainable state. We may need a couple more hooks, either injected into base classes or added there. It might be simplest to add these hooks to the harness. Happy to set up a call or instant message chat to discuss this. Given the breadth of the changes, it is likely to be beneficial to discuss plans and get good alignment while doing the rest of the changes.

garak/detectors/base.py

leondz · 2024-10-30T08:58:51Z

garak/detectors/base.py

+        if hasattr(self, 'reverse_translator'):
+            if self.reverse_translator is not None:
+                non_none_outputs = self.reverse_translator.translate_prompts(non_none_outputs)


Can we handle this in the harness rather than the detector? It's an orchestration issue

OK. I try it.

leondz · 2024-10-30T08:59:27Z

garak/detectors/base.py

+        if hasattr(self, 'translator'):
+            if self.translator is not None:
+                self.substrings = self.translator.translate_prompts(self.substrings)


can we handle this in the harness?

OK. I try it.

garak/detectors/leakreplay.py

leondz · 2024-10-30T09:28:40Z

garak/detectors/snowball.py

consider factoring up to harness

leondz · 2024-10-30T09:28:50Z

garak/detectors/misleading.py

consider factoring up to harness

garak/detectors/base.py

docs/source/translator.rst

jmartin-tech

@SnowMasaya thank you, this is a significant benefit for the project.

Adding a number of my pending comments here, please be aware these are likely similar to many @leondz made, although possibly from different reasoning. Happy to continue iterating.

garak/detectors/base.py

docs/source/translator.rst

garak/detectors/base.py

jmartin-tech · 2024-10-23T13:51:32Z

garak/probes/atkgen.py

+                    if hasattr(self, 'translator'):
+                        if self.translator is not None:
+                            challenges = self.translator.translate_prompts([challenge])
+                    else:
+                        challenges = [challenge]


Why does this need to create a list and then iterate inside the turn?

I suspect this is misplaced, translation in this probe should only need to happen for the initial challenge in the probe and all generators used to perform red_team actions by manipulating on that prompt using a model in the target language.

I got same review comment.

Feature/multilingual #943 (comment)

I will recheck code and considering which part is the better place to implement translation function.

If you have any recommend way to implement it, please let me know.

garak/probes/atkgen.py

tests/probes/test_probes_encoding.py

jmartin-tech · 2024-10-23T17:40:10Z

tests/probes/test_probes_encoding.py

@@ -12,6 +23,7 @@ def test_InjectBase64_len_cap():
    assert len(p.prompts) < num_payloads * num_templates * num_encoders


+@pytest.fixture(scope="function")


What is this doing? Tests should not be annotated as fixtures.

Sorry. This test does not need it. I remove it this part.

jmartin-tech · 2024-10-23T17:40:44Z

tests/probes/test_probes_encoding.py

@@ -20,6 +32,7 @@ def test_InjectBase64_prompt_trigger_match():
    assert len(p.prompts) == len(p.triggers)


+@pytest.fixture(scope="function")


What is this doing? Tests should not be annotated as fixtures.

Sorry. This part does not need it. I remove it this part.

tests/test_translator.py

garak/_config.py

leondz · 2024-10-30T13:35:57Z

Adding a number of my pending comments here, please be aware these are likely similar to many @leondz made, although possibly from different reasoning. Happy to continue iterating.

Indeed, sorry for the dupe review, there was quite a lot of code so I reviewed directly instead of processing @jmartin-tech 's comments as well

add mean judge for reverse translation change translation model size Signed-off-by: Masaya Ogushi <[email protected]>

translate trigger words Signed-off-by: Masaya Ogushi <[email protected]>

add reverse translation remove trigger translation fix test code Signed-off-by: Masaya Ogushi <[email protected]>

add translation instance Signed-off-by: Masaya Ogushi <[email protected]>

add library for reverse translation Signed-off-by: Masaya Ogushi <[email protected]>

SnowMasaya · 2024-10-31T00:38:34Z

Thank you, this is a significant amount of work.

This is for translation-based multilingual, rather than general multilingual support, so we should be careful to separate these two functions. The ability to select which languages garak uses is distinct from how that language is achieved (in prompts, detectors, and so on).

There are some refactorings needed to get the PR to a sustainable state. We may need a couple more hooks, either injected into base classes or added there. It might be simplest to add these hooks to the harness. Happy to set up a call or instant message chat to discuss this. Given the breadth of the changes, it is likely to be beneficial to discuss plans and get good alignment while doing the rest of the changes.

Thank you for review.
I would like to set up a call for understanding the best way to implement this function.
Please stop the review the following commit because it has not apply your review.
The following commit for my coworker request.

remove extra test code Signed-off-by: Masaya Ogushi <[email protected]>

Add attributes lang_type, reverse_translation for analysis translation result Signed-off-by: Masaya Ogushi <[email protected]>

move translation check code to another test code Signed-off-by: Masaya Ogushi <[email protected]>

check attempt language save reverse translation for analysis result Signed-off-by: Masaya Ogushi <[email protected]>

Signed-off-by: Masaya Ogushi <[email protected]>

Check StringDetector Check HF detector Check Yes, No start Detector Signed-off-by: Masaya Ogushi <[email protected]>

Only 1st probes translation atkgen goodside.py move translation function to translator.py base.py has get_translator function base.py add lang_type for attempt Signed-off-by: Masaya Ogushi <[email protected]>

save reverse translation which made by the dtector Signed-off-by: Masaya Ogushi <[email protected]>

Don't check non prompt probes Don't check visual probes Signed-off-by: Masaya Ogushi <[email protected]>

update remove english punctuation update english judge update split text add translate description remove some class and integrate with function to base class add lang class for analysis result Signed-off-by: Masaya Ogushi <[email protected]>

remove extra class check encode text translaiton check long sentence translation check add yaml for test without api key Signed-off-by: Masaya Ogushi <[email protected]>

using yaml file for translation Signed-off-by: Masaya Ogushi <[email protected]>

Signed-off-by: Jeffrey Martin <[email protected]>

* refactor for encapsulation * set default config entries for early load support * adds docs for new entries * initializes `lang_spec` to `en` * intializes `translators` to an empty list * revised test config formats * only maintain prompts in one language * limit to only translated prompts * treat "$" as a special line bypassing translation attempt * NLI detector should support None response * check translation required carefully * strings that do not contain "words" should bypass translation * tests of translation configuration require `lang_spec` * add remote translator specific class to tests * clarify base model prefix as opus-mt-* * detectors need translators in config * always return a translator even if just target to target * always have a translator * only attempt to translate output that is not None * force garbage collection after translator tests * validate probe trigger type during tranlation * translation needs lists of strings * support for nested lists is added for existing probes content * remove direct _config access in plugins * remove access to _config.run from `probe` classes * adjust goodside translations to not retain original prompts * refactor probe translation tests for unit testing * In the interest of reasonable execution time test probe call translation instead of executing translation. * probe translation tests as unit testing only * Translation actions are tested with there own tests. * remove side-effects for internal translation methods * latentinjection init adjustment * Remove extra call for translator * Ensure `_build_prompts_triggers` is called only once during init for all implemented classes. * bugfix - goodside instance instead of class attributes * remote test case corrections * extract translator base config restrictions * ENV var needs are handled by `remote` module * adjust docs for each class * match extending class method signature * consolidate nltk overrides in resources.api * remove no longer used "only_translate_word" * remove lang_list references, support a single target language * use pythonic code-style, adjust inline comments * refactor report file to rely on global fixture * rename base class to `Translator` * rename `SimpleTranslator` to `Translator` * source and target language determined via translator held values * update translation configuration docs Signed-off-by: Jeffrey Martin <[email protected]>

Streamline translation use case

Signed-off-by: Jeffrey Martin <[email protected]>

masayaOgushi added 6 commits October 9, 2024 10:23

Add Feature: probes add translation function

f94bb2e

- Addd translation function for base probe class - prompts and triggers translate by base class method - attempt_descr translation Signed-off-by: Masaya Ogushi <[email protected]>

Add Feature: detector add translation capabilities

2238d18

- Translation handling for detector keywords and substrings, triggers. Signed-off-by: Masaya Ogushi <[email protected]>

Add Feature: Enhance command-line interface with new translation options

7202e19

- Added support for specifying translation services directly from the CLI. - Implemented options to set target languages for translation. Signed-off-by: Masaya Ogushi <[email protected]>

chore: Update dependencies in requirements.txt, pyproject.toml

1105bb1

- Added new dependencies required for enhanced translation features. Signed-off-by: Masaya Ogushi <[email protected]>

docs: Add translation documentation

6bb7da3

- Added detailed explanations of the translationn method - Included examples of how translation services are configured and utilized within the codebase. Signed-off-by: Masaya Ogushi <[email protected]>

SnowMasaya force-pushed the feature/multilingual branch from b781a6f to 6bb7da3 Compare October 9, 2024 01:41

Merge branch 'leondz:main' into feature/multilingual

717f0ff

jmartin-tech requested changes Oct 10, 2024

View reviewed changes

masayaOgushi added 9 commits October 23, 2024 13:50

Update Feature: Translator

b35cc1e

update remove punctuation update english judge add translate function add logging translate result add Reverse translate for hf detector and snowball probes Signed-off-by: Masaya Ogushi <[email protected]>

Update Feature: Probes

bbb6c76

check translator instance remove translate function reset config Signed-off-by: Masaya Ogushi <[email protected]>

Update Feature: Detectors

51baeb2

check translator instance add reverse translator add test reverse translator Signed-off-by: Masaya Ogushi <[email protected]>

Update Feature: cli

dc3a4ab

remove argument using generator_option_file Signed-off-by: Masaya Ogushi <[email protected]>

Update Feature: config

ee82261

add load translator instance Signed-off-by: Masaya Ogushi <[email protected]>

Update Feature: conftest

7cb8acc

check storage size set up each instance for each test Signed-off-by: Masaya Ogushi <[email protected]>

Remove: library

ec9b40a

remove pyenchant Using nltk instead of pyenchant Signed-off-by: Masaya Ogushi <[email protected]>

Update Doc

d50d19e

update how to use translation function Signed-off-by: Masaya Ogushi <[email protected]>

Merge branch 'feature/multilingual' of https://github.com/SnowMasaya/…

808f34a

…garak into feature/multilingual

github-actions bot added a commit that referenced this pull request Oct 23, 2024

@SnowMasaya has signed the CLA in #943

03356ed

SnowMasaya and others added 3 commits October 23, 2024 14:13

Merge branch 'main' into feature/multilingual

8a41c95

Signed-off-by: SnowGushiGit <[email protected]>

Fix test

2fc2dd5

Signed-off-by: Masaya Ogushi <[email protected]>

Merge branch 'feature/multilingual' of https://github.com/SnowMasaya/…

8283b65

…garak into feature/multilingual

leondz requested changes Oct 30, 2024

View reviewed changes

jmartin-tech reviewed Oct 30, 2024

View reviewed changes

masayaOgushi added 3 commits October 31, 2024 09:21

Update Feature Translation

395840d

add mean judge for reverse translation change translation model size Signed-off-by: Masaya Ogushi <[email protected]>

Add Feature Probes

73363f9

translate trigger words Signed-off-by: Masaya Ogushi <[email protected]>

Update Feature Detectors

57d14e5

add reverse translation remove trigger translation fix test code Signed-off-by: Masaya Ogushi <[email protected]>

masayaOgushi added 2 commits October 31, 2024 09:23

Update test

3b3b60a

add translation instance Signed-off-by: Masaya Ogushi <[email protected]>

Add library

bae54d7

add library for reverse translation Signed-off-by: Masaya Ogushi <[email protected]>

masayaOgushi and others added 13 commits December 12, 2024 09:39

Remove test code

022b821

remove extra test code Signed-off-by: Masaya Ogushi <[email protected]>

Add Feature

ad475ba

Add attributes lang_type, reverse_translation for analysis translation result Signed-off-by: Masaya Ogushi <[email protected]>

Remove translation check

a816836

move translation check code to another test code Signed-off-by: Masaya Ogushi <[email protected]>

Update reverse translation

e7363de

check attempt language save reverse translation for analysis result Signed-off-by: Masaya Ogushi <[email protected]>

Remove translation function

563060b

Signed-off-by: Masaya Ogushi <[email protected]>

Add detector test

a3922e7

Check StringDetector Check HF detector Check Yes, No start Detector Signed-off-by: Masaya Ogushi <[email protected]>

Update probes

a3dd8de

Only 1st probes translation atkgen goodside.py move translation function to translator.py base.py has get_translator function base.py add lang_type for attempt Signed-off-by: Masaya Ogushi <[email protected]>

Update harness base

4c4ad68

save reverse translation which made by the dtector Signed-off-by: Masaya Ogushi <[email protected]>

Add probe test code

641b851

Don't check non prompt probes Don't check visual probes Signed-off-by: Masaya Ogushi <[email protected]>

Update Translation

8a0ee80

update remove english punctuation update english judge update split text add translate description remove some class and integrate with function to base class add lang class for analysis result Signed-off-by: Masaya Ogushi <[email protected]>

Update test translation

b877b97

remove extra class check encode text translaiton check long sentence translation check add yaml for test without api key Signed-off-by: Masaya Ogushi <[email protected]>

Update doc

5820f95

using yaml file for translation Signed-off-by: Masaya Ogushi <[email protected]>

Merge 'main' into feature/multilingual

e53e7d2

Signed-off-by: Jeffrey Martin <[email protected]>

jmartin-tech force-pushed the feature/multilingual branch from 510e63f to e53e7d2 Compare February 7, 2025 16:44

jmartin-tech and others added 3 commits February 13, 2025 09:34

Merge pull request #1 from jmartin-tech/feature/multilingual-translation

6780578

Streamline translation use case

cleanup imports and tests

5da27d2

Signed-off-by: Jeffrey Martin <[email protected]>

jmartin-tech added a commit that referenced this pull request Feb 14, 2025

@masayaOgushi has signed the CLA in #943

9668fb6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/multilingual #943

Feature/multilingual #943

SnowMasaya commented Oct 9, 2024 •

edited by jmartin-tech

Loading

github-actions bot commented Oct 9, 2024 •

edited

Loading

SnowMasaya commented Oct 9, 2024 •

edited

Loading

jmartin-tech left a comment

leondz left a comment

leondz Oct 30, 2024

SnowMasaya Oct 31, 2024

leondz Oct 30, 2024

SnowMasaya Oct 31, 2024

leondz Oct 30, 2024

SnowMasaya Oct 31, 2024

leondz Oct 30, 2024

SnowMasaya Oct 31, 2024

jmartin-tech left a comment

jmartin-tech Oct 23, 2024

SnowMasaya Oct 31, 2024

jmartin-tech Oct 23, 2024

SnowMasaya Oct 31, 2024

jmartin-tech Oct 23, 2024

SnowMasaya Oct 31, 2024

leondz commented Oct 30, 2024

SnowMasaya commented Oct 31, 2024 •

edited

Loading

		@@ -12,6 +23,7 @@ def test_InjectBase64_len_cap():
		assert len(p.prompts) < num_payloads * num_templates * num_encoders


		@pytest.fixture(scope="function")

		@@ -20,6 +32,7 @@ def test_InjectBase64_prompt_trigger_match():
		assert len(p.prompts) == len(p.triggers)


		@pytest.fixture(scope="function")

Feature/multilingual #943

Are you sure you want to change the base?

Feature/multilingual #943

Conversation

SnowMasaya commented Oct 9, 2024 • edited by jmartin-tech Loading

Usage

Example

github-actions bot commented Oct 9, 2024 • edited Loading

SnowMasaya commented Oct 9, 2024 • edited Loading

jmartin-tech left a comment

Choose a reason for hiding this comment

leondz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmartin-tech left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leondz commented Oct 30, 2024

SnowMasaya commented Oct 31, 2024 • edited Loading

SnowMasaya commented Oct 9, 2024 •

edited by jmartin-tech

Loading

github-actions bot commented Oct 9, 2024 •

edited

Loading

SnowMasaya commented Oct 9, 2024 •

edited

Loading

SnowMasaya commented Oct 31, 2024 •

edited

Loading