Adding Gujarati language vocab #1806
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1806 +/- ##
==========================================
+ Coverage 96.56% 96.59% +0.02%
==========================================
Files 164 165 +1
Lines 7895 7893 -2
==========================================
Hits 7624 7624
+ Misses 271 269 -2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Hi @sarjil77 👋,
Thanks for the PR 👍
The workflow to create a PR:
- update your repository fork (there is a button on the forked repo)
- local checkout to main and pull the latest changes
- create a feature branch
git checkout -b <FEAT_BRANCH_NAME> - Add, commit and push your code
- Go to the original docTR repo (here) and open a pull request
:)
There was a problem hiding this comment.
this file shouldn't be in the PR ^^
| "gujarati_consonants":"કખગઘઙચછજઝઞટઠડઢણતથદધનપફબભમયરલવશષસહષજ્ઞ", | ||
| "gujarati_vowels": "અઆઇઈઉઊઋએઐઓઔઅંઅઃ ", | ||
| "gujarati_digits":"૦૧૨૩૪૫૬૭૮૯", | ||
| "gujarati_diacritics":""" કકાકિકીકુકૂકૃકેકૈકોકૌકંકઃ ખખાખિખીખુખૂખૃખેખૈખોખૌખંખઃ ગગાગિગીગુગૂગૃગેગૈગોગૌગંગઃ ઘઘાઘિઘીઘુઘૂઘૃઘેઘૈઘોઘૌઘંઘઃ ઙઙાઙિઙીઙુઙૂઙૃઙેઙૈઙોઙૌઙંઙઃ ચચાચિચીચુચૂચૃચેચૈચોચૌચંચઃ |
There was a problem hiding this comment.
This looks a bit weird 😅
I have found the following: https://www.easygujaratityping.com/gujarati/alphabet
The entries shouldn't contain whitespaces or linebreaks
There was a problem hiding this comment.
Additional each character should be only one time in the vocab
| છછાછિછીછુછૂછૃછેછૈછોછૌછંછઃ જજાજિજીજુજુજૃજેજૈજોજૌજંજઃ ઝઝાઝિઝીઝુઝૂઝૃઝેઝૈઝોઝૌઝંઝઃ ઞઞાઞિઞીઞુઞૂઞૃઞેઞૈઞોઞૌઞંઞઃ ટટાટિટીટુટૂટૃટેટૈટોટૌટંટઃ ઠઠાઠિઠીઠુઠૂઠૃઠેઠૈઠોઠૌઠંઠઃ ડડાડિડીડુડૂડૃડેડૈડોડૌડંડઃ ઢઢાઢિઢીઢુઢૂઢૃઢેઢૈઢોઢૌઢંઢઃ ણણાણિણીણુણૂણૃણેણૈણોણૌણંણઃ | ||
| તતાતિતીતુતૂતૃતેતૈતોતૌતંતઃ થથાથિથીથુથૂથૃથીથૈથોથૌથંથઃ દદાદિદીદુદૂદૃદેદૈદોદૌદંદઃ ધધાધિધીધુધૂધૃધેધૈધોધૌધંધઃ નનાનિનીનુનૂનૃનેનૈનોનૌનંનઃ પપાપિપીપુપૂપૃપેપૈપોપૌપંપઃ ફફાફિફીફુફૂફૃફેફૈફોફૌફંફઃ બબાબિબીબુબૂબૃબેબૈબોબૌબંબઃ ભભાભિભીભુભૂભૃભેભૈભોભૌભંભઃ | ||
| મમામિમીમુમૂમૃમેમામોમાયમંમઃ યયાયિયીયુયુયૃયેયૈયોયૌયંયઃ રરારિરીરૂરૃરેરૈરોરૌરંરઃ લલાલિલીલુલૂલૃલેલૈલોલૌલંલઃ વવાવિવીવિવૂવૃવેવૈવોવૈવંવઃ શશાશિશીશુશૂશૃશેશૈશોશૌશંશઃ ષષાષિષીષુષૂષૃષેષૈષોષૌષંષઃ જ્ઞજ્ઞાજ્ઞિજ્ઞીજ્ઞુજ્ઞૂજ્ઞૃજ્ઞેજ્ઞૈજ્ઞોજ્ઞૌજ્ઞંજ્ઞઃ """, | ||
| "gujarati_punctuation":",.!?:;'()[]-_/|\✶૰૱`'", |
There was a problem hiding this comment.
here we should combine the current default punctuation entry with additional like:
"gujarati_punctuation": VOCABS["punctuation"] + "૰◌્◌઼ઽ◌ઁ◌ંઃ॥ૐ" + "૱"
There was a problem hiding this comment.
thanks felix for your suggestions :)
i know it looks weird but it is right, when we combine vowels and consonants we get the diacritics in gujarati language, so when we train and during inferencing, characters will be from diacritics. And it is complete vocabulary as per the best of my knowledge and no other alphabet will be from outside of this. I will make changes as you suggested no whitespaces, linebreaks and should be present only once, and also for punctuation as well.
There was a problem hiding this comment.
Sounds good thanks for the update :)
Here i am adding gujarati language vocabulary along with its vowels, consonants , digits and diacritics