Skip to content

Commit

Permalink
Merge pull request #11242 from keymanapp/fix/common/11220-emoji-strip
Browse files Browse the repository at this point in the history
fix(common): update emoji stripping for Unicode 15.1
  • Loading branch information
mcdurdin authored Apr 18, 2024
2 parents 7aa6b66 + db514e0 commit 66b9f7e
Show file tree
Hide file tree
Showing 8 changed files with 106 additions and 43 deletions.
74 changes: 37 additions & 37 deletions HISTORY.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@
## 17.0.307 beta 2024-04-12

* fix(common): specify title explicitly when opening PR with hub (#11173)
* refactor(web): better centralizes OSK layout internals to prepare for optimization efforts 🪠 (#11176)
* feat(web): VisualKeyboard layout-reflow optimization 🪠 (#11177)
* change(web): OSK optimization, improved responsiveness 🪠 (#11140)
* refactor(web): better centralizes OSK layout internals to prepare for optimization efforts (#11176)
* feat(web): VisualKeyboard layout-reflow optimization (#11177)
* change(web): OSK optimization, improved responsiveness (#11140)
* (#11216)

## 17.0.306 beta 2024-04-11
Expand All @@ -28,9 +28,9 @@
## 17.0.305 beta 2024-04-10

* (#11169)
* change(web): merges split async method in gesture engine 🪠 (#11142)
* change(web): merges split async method in gesture engine (#11142)
* fix(web): blocks nextLayer for keys quickly typed when multitapping to new layer when final tap is held (#11189)
* refactor(web): OSK spacebar-label updates now managed by layer object 🪠 (#11175)
* refactor(web): OSK spacebar-label updates now managed by layer object (#11175)

## 17.0.304 beta 2024-04-09

Expand All @@ -50,14 +50,14 @@

* feat(core): support modifiers=other (#11118)
* chore(core): dx better err message on embedded test vkeys (#11119)
* fix(web): key preview stickiness 🪠 (#10778)
* fix(web): early gesture-match abort when unable to extend existing gestures 🪠 (#10836)
* fix(web): infinite model-match replacement looping 🪠 (#10838)
* fix(web): proper gesture-match sequencing 🪠 (#10840)
* change(web): input-event sequentialization 🪠 (#10843)
* fix(web): proper linkage of sources to events 🪠 (#10960)
* fix(web): key preview stickiness (#10778)
* fix(web): early gesture-match abort when unable to extend existing gestures (#10836)
* fix(web): infinite model-match replacement looping (#10838)
* fix(web): proper gesture-match sequencing (#10840)
* change(web): input-event sequentialization (#10843)
* fix(web): proper linkage of sources to events (#10960)
* fix(developer): handle buffer boundaries in four cases (#11137)
* chore(linux): Build packages for next Ubuntu version separately :cherries: (#11153)
* chore(linux): Build packages for next Ubuntu version separately (#11153)
* fix(common): upgrade sentry-cli to 2.31.0 (#11151)
* fix(android/app): Track previous device orientation for SystemKeyboard (#11134)
* (#11129)
Expand Down Expand Up @@ -2036,8 +2036,8 @@
* fix(developer): package editor no longer loses RTL flag for LMs (#8607)
* chore(common): use mac /usr/bin/stat rather than homebrew version (#8925)
* chore(ios): replace fv cert (#8923)
* fix(core): Fix compilation if hotdoc is installed :cherries: (#8929)
* chore(linux): Move some files to keyman-config :cherries: (#8930)
* fix(core): Fix compilation if hotdoc is installed (#8929)
* chore(linux): Move some files to keyman-config (#8930)
* fix(core): Fix compiling with GCC 13 (#8932)

## 16.0.139 stable 2023-03-16
Expand Down Expand Up @@ -2274,7 +2274,7 @@
* fix(developer): URL parameters should be UTF-8 (#7631)
* fix(android/engine): Use IME package name if query permission denied (#7668)
* fix(linux): Fix keyboard icon in system tray (#7678)
* chore(linux): Update debian changelog :cherries: (#7682)
* chore(linux): Update debian changelog (#7682)
* chore(linux): Update Debian standards version (#7683)

## 16.0.100 beta 2022-11-10
Expand Down Expand Up @@ -2488,7 +2488,7 @@
## 16.0.70 alpha 2022-09-26

* fix(android): Add language name when installing default lexical-model (#7347)
* chore(common): Add 15.0 stable entries to HISTORY.md :cherries: (#7350)
* chore(common): Add 15.0 stable entries to HISTORY.md (#7350)

## 16.0.69 alpha 2022-09-21

Expand Down Expand Up @@ -2678,7 +2678,7 @@
## 16.0.37 alpha 2022-07-22

* fix(linux): Catch PermissionError exception (#6968)
* chore(linux): Update Debian changelog :cherries: (#6973)
* chore(linux): Update Debian changelog (#6973)
* chore(linux): Add support for Ubuntu 22.10 "Kinetic Kudu" (#6975)
* fix(developer): reduce timeouts if server shut down (#6943)
* fix(linux): Implement refresh after keyboard installation (#6956)
Expand Down Expand Up @@ -2871,12 +2871,12 @@
## 15.0.269 stable 2022-08-29

* chore(linux): Update debian changelog (#7040)
* feat(linux): Replace deprecated distutils :cherries: (#7052)
* feat(linux): Replace deprecated distutils (#7052)
* fix(developer): compiler emitting garbage for readonly groups (#7014)
* chore: Change platform advocates per discussion (#7114)
* fix(windows): remove saving and restoring context kbd options (#7107)
* fix(windows): Add invalidate context action to non-updatable parse (#7108)
* fix(android/engine): :cherries: Lower the max height for landscape orientation (#7128)
* fix(android/engine): Lower the max height for landscape orientation (#7128)

## 15.0.268 stable 2022-08-04

Expand All @@ -2893,10 +2893,10 @@
* fix(ios): ignore CFBundleShortVersionString (#6935)
* fix(web): context-only rule effects, set(&layer) from physical keystrokes ️ (#6949)
* chore(web): remove invalid warning msg (#6951)
* fix(common): Fix `delete` :cherries: (#6966)
* fix(linux): Another attempt at fixing postinst script :cherries: (#6961)
* fix(linux): Fix uninstallation when using fcitx5 :cherries: (#6964)
* fix(linux): Catch PermissionError exception :cherries: (#6969)
* fix(common): Fix `delete` (#6966)
* fix(linux): Another attempt at fixing postinst script (#6961)
* fix(linux): Fix uninstallation when using fcitx5 (#6964)
* fix(linux): Catch PermissionError exception (#6969)
* chore(linux): Update Debian changelog (#6972)
* fix(developer): kmdecomp virtual character key output (#6945)
* fix(developer): crash on exit when checking for updates (#6946)
Expand All @@ -2908,7 +2908,7 @@

## 15.0.266 stable 2022-07-08

* fix(linux): Fix post-install script :cherries: (#6895)
* fix(linux): Fix post-install script (#6895)
* fix(web): improve `console.error()` reporting (#6904)
* fix(web): ncaps rules not matching on touch (#6913)

Expand Down Expand Up @@ -3358,7 +3358,7 @@

## 15.0.183 alpha 2022-01-21

* chore(linux): Update changelogs for 14.0.284 :cherries: (#6132)
* chore(linux): Update changelogs for 14.0.284 (#6132)
* chore(linux): Revert workaround for Python bug (#6133)

## 15.0.182 alpha 2022-01-21
Expand Down Expand Up @@ -3477,12 +3477,12 @@

## 15.0.162 alpha 2021-12-05

* chore(linux): Update changelogs for 14.0.283 :package: :cherries: (#6008)
* chore(linux): Update changelogs for 14.0.283 (#6008)

## 15.0.161 alpha 2021-12-04

* chore(linux): Allow to specify debian revision :package: :cherries: (#5999)
* chore(linux): Remove lintian warning :package: :cherries: (#6000)
* chore(linux): Allow to specify debian revision (#5999)
* chore(linux): Remove lintian warning (#6000)

## 15.0.160 alpha 2021-12-03

Expand Down Expand Up @@ -3668,7 +3668,7 @@
* fix(android/engine): Remove unnecessary permissions from Manifest (#5752)
* feat(developer): touch layout testing (#5723)
* fix(web): popup positioning (#5742)
* chore(linux): Update changelog files for 14.0.282 :cherries: (#5794)
* chore(linux): Update changelog files for 14.0.282 (#5794)

## 15.0.124 alpha 2021-10-04

Expand Down Expand Up @@ -4484,8 +4484,8 @@

## 14.0.285 stable 2022-01-20

* fix(linux): Fix lintian errors :cherries: (#6107)
* fix(linux): Add workaround for Python bug :cherries: (#6125)
* fix(linux): Fix lintian errors (#6107)
* fix(linux): Add workaround for Python bug (#6125)
* fix(web): Use regex to determine display layer and functional layers (#6123)

## 14.0.284 stable 2022-01-11
Expand All @@ -4496,20 +4496,20 @@
* fix(android/engine): Fix font paths (#5990)
* chore(linux): Remove lintian warning (#5993)
* chore(linux): Allow to specify debian revision (#5998)
* chore(linux): Update changelogs for 14.0.283 :package: (#6007)
* fix(linux): fix release version number for Sentry reporting :cherries: (#6053)
* chore(linux): Update changelogs for 14.0.283 (#6007)
* fix(linux): fix release version number for Sentry reporting (#6053)
* chore(common): Check in crowdin strings for Spanish (Latin America) (#6060)
* fix(linux): fix release version number for Sentry reporting :cherries: (#6069)
* fix(linux): fix release version number for Sentry reporting (#6069)
* chore(android/samples): Add -no-daemon flag to KMSample2 build script (#6083)
* fix(linux): Fix attribute error :cherries: (#6087)
* fix(linux): Fix attribute error (#6087)

## 14.0.283 stable 2021-11-17

* chore(windows): fix broken links in help (#5766)
* chore(linux): copy Keyman for Linux 15 reference to 14 (#5764)
* fix(linux): Fix debian package script (#5772)
* chore(common): Enhance cherry-pick labeling (#5773)
* fix(common): Fix cherry-pick labeling (:cherries:) (#5783)
* fix(common): Fix cherry-pick labeling () (#5783)
* fix(linux): Don't crash displaying keyboard details (#5757)
* chore(linux): Update changelog files for 14.0.282 (#5793)
* fix(linux): Don't crash with non-keyboard package file (#5754)
Expand Down Expand Up @@ -6028,7 +6028,7 @@

## 14.0.81 alpha 2020-05-27

* refactor(resources): convert gosh into npm package 🙃 (#3159)
* refactor(resources): convert gosh into npm package (#3159)
* chore(common,web): use consistent TypeScript dep on all packages (#3158)
* chore(common/resources): add `common/models` to build trigger definitions (#3144)
* fix(common/resources): adds package-lock.json for gosh package (#3171)
Expand Down
5 changes: 2 additions & 3 deletions ios/tools/prepRelease.sh
Original file line number Diff line number Diff line change
Expand Up @@ -50,9 +50,8 @@ get_version_notes "ios" "${BUILD_NUMBER}" "$TIER" > $CHANGELOG_PATH
echo "* Minor fixes and performance improvements" >> $CHANGELOG_PATH
assertFileExists "${CHANGELOG_PATH}"

# Strip emoji to make Apple happy
emoji="\U1f300-\U1f5ff\U1f900-\U1f9ff\U1f600-\U1f64f\U1f680-\U1f6ff\U2600-\U26ff\U2700-\U27bf\U1f1e6-\U1f1ff\U1f191-\U1f251\U1f004\U1f0cf\U1f170-\U1f171\U1f17e-\U1f17f\U1f18e\U3030\U2b50\U2b55\U2934-\U2935\U2b05-\U2b07\U2b1b-\U2b1c\U3297\U3299\U303d\U00a9\U00ae\U2122\U23f3\U24c2\U23e9-\U23ef\U25b6\U23f8-\U23fa"
LC_ALL=UTF-8 sed -e "s/[$(printf $emoji)]//g" < "$CHANGELOG_PATH" > "$CHANGELOG_PATH.1"
# Strip emoji as App Store does not allow emoji in changelogs
node "$KEYMAN_ROOT/resources/tools/strip-emoji" < "$CHANGELOG_PATH" > "$CHANGELOG_PATH.1"
mv -f "$CHANGELOG_PATH.1" "$CHANGELOG_PATH"
assertFileExists "${CHANGELOG_PATH}"

Expand Down
29 changes: 29 additions & 0 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
"scripts": {},
"workspaces": [
"resources/gosh",
"resources/tools/strip-emoji",
"resources/build/version",
"core/include/ldml",
"developer/src/common/web/test-helpers",
Expand Down
1 change: 1 addition & 0 deletions resources/build/version/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
"dependencies": {
"@actions/core": "^1.9.1",
"@actions/github": "^2.1.0",
"emoji-regex": "^10.3.0",
"typescript": "^4.9.5",
"yargs": "^17.7.2"
},
Expand Down
7 changes: 4 additions & 3 deletions resources/build/version/src/reportHistory.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import { GitHub } from '@actions/github';

import { findLastHistoryPR, getAssociatedPR} from './graphql/queries.js';
import { spawnChild } from './util/spawnAwait.js';
import emojiRegex from 'emoji-regex';

const getPullRequestInformation = async (
octokit: GitHub, base: string
Expand Down Expand Up @@ -125,10 +126,10 @@ export const reportHistory = async (
} else {
let git_tag = 'Next Version', git_tag_data = 'Next Version';
const re = /#(\d+)/;
const emojiRE = /[\u{1f300}-\u{1f5ff}\u{1f900}-\u{1f9ff}\u{1f600}-\u{1f64f}\u{1f680}-\u{1f6ff}\u{2600}-\u{26ff}\u{2700}-\u{27bf}\u{1f1e6}-\u{1f1ff}\u{1f191}-\u{1f251}\u{1f004}\u{1f0cf}\u{1f170}-\u{1f171}\u{1f17e}-\u{1f17f}\u{1f18e}\u{3030}\u{2b50}\u{2b55}\u{2934}-\u{2935}\u{2b05}-\u{2b07}\u{2b1b}-\u{2b1c}\u{3297}\u{3299}\u{303d}\u{00a9}\u{00ae}\u{2122}\u{23f3}\u{24c2}\u{23e9}-\u{23ef}\u{25b6}\u{23f8}-\u{23fa}]/gu;
const emojiRE = emojiRegex();
for(const commit of new_commits) {
if(!useGitHubPRInfo) {
const git_pr_title = (await spawnChild('git', ['log', '--format=%b', '-n', '1', commit])).replace(emojiRE, ' ').trim();
const git_pr_title = (await spawnChild('git', ['log', '--format=%b', '-n', '1', commit])).replaceAll(emojiRE, ' ').trim();
if(git_pr_title.match(/^auto\:/)) continue;
const git_pr_data = (await spawnChild('git', ['log', '--format=%s', '-n', '1', commit])).trim();
const this_git_tag = (await spawnChild('git', ['tag', '--points-at', commit])).trim();
Expand Down Expand Up @@ -164,7 +165,7 @@ export const reportHistory = async (
if(pulls.find(p => p.number == pr.number) == undefined) {
pr.tag_data = git_tag_data;
pr.version = git_tag;
pr.title = pr.title.replace(emojiRE, ' ').trim();
pr.title = pr.title.replaceAll(emojiRE, ' ').trim();
pulls.push(pr);
}
}
Expand Down
18 changes: 18 additions & 0 deletions resources/tools/strip-emoji/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@

const emojiRegex = require('emoji-regex');

process.stdin.setEncoding('utf-8');

// We will concatenate all strings and assume we are not processing a huge file,
// so we don't break in the middle of a UTF-8 sequence or split emoji sequences
// in half.

let stream = '';
process.stdin.on('readable', () => {
const data = process.stdin.read();
if(data) {
stream += data;
} else {
process.stdout.write(stream.replaceAll(emojiRegex(), ''));
}
});
14 changes: 14 additions & 0 deletions resources/tools/strip-emoji/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"name": "stripemoji",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"author": "",
"license": "MIT",
"dependencies": {
"emoji-regex": "^10.3.0"
}
}

0 comments on commit 66b9f7e

Please sign in to comment.