New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ja] replace Japanese PII #49

Open

2 tasks

fujiki-1emon opened this issue Mar 13, 2023 · 0 comments

Contributor

fujiki-1emon commented Mar 13, 2023 •

edited

Loading

Background

Seems that we don't have to implement a lot of pre-processing to replace Japanese PII
- because there are already some PII pre-processing in language agnostic processing.
But we might need to implement additionally to replace some Japanese specific PII.

TODOs

replace Japanese phone number
- But we might be able to just re-use the Korean one
  - cf. https://github.com/EleutherAI/dps/blob/master/dps/spark/utils/korean_utils.py#L182
replace Japanese bank account number

fujiki-1emon changed the title ~~replace Japanese PII~~ Japanese pre-processing - replace Japanese PII

fujiki-1emon changed the title ~~Japanese pre-processing - replace Japanese PII~~ [ja] replace Japanese PII

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment