Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Live2D with Lipsync (using audio file/link) #122

Open
wants to merge 47 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
99fd153
Added lip-sync module with audio
RaSan147 Oct 6, 2023
d378339
Automated report
RaSan147 Oct 6, 2023
3ff3a53
Update index.js links
RaSan147 Oct 6, 2023
8539e76
Update test.yml
RaSan147 Oct 12, 2023
cf0f4c4
Update test.yml
RaSan147 Oct 12, 2023
f53e25a
Update test.yml
RaSan147 Oct 12, 2023
7deba12
Fix Lip sync. Breaking change
RaSan147 Oct 12, 2023
06bfbd3
Automated report
RaSan147 Oct 12, 2023
1e89254
Update README.md and added video
RaSan147 Oct 12, 2023
34831e8
Merge branch 'master' into master
RaSan147 Dec 7, 2023
2d0c512
Fix type error on ci test
RaSan147 Dec 7, 2023
2617eae
clear dist, using workflow to get output files
RaSan147 Dec 7, 2023
de764a1
removed Version number
RaSan147 Dec 7, 2023
2877f47
Now supports other b64 audio
RaSan147 Dec 7, 2023
25ddef6
Remove audio url validation
RaSan147 Dec 7, 2023
b4a394a
rename speakUp -> speak
RaSan147 Dec 7, 2023
6f3a779
rename resetMotions -> stopMotions
RaSan147 Dec 7, 2023
e878d79
return false when audio play fails
RaSan147 Dec 7, 2023
031ca86
Added crossorigin for speak voice source
RaSan147 Dec 13, 2023
ae1c923
Place missing CrossOrigin arg
RaSan147 Dec 13, 2023
c0cb7f5
Update MotionManager.ts
RaSan147 Dec 14, 2023
c8eca3f
Update readme as per PR
RaSan147 Dec 14, 2023
7f8ec7a
Check any base64 data for audio
RaSan147 Dec 14, 2023
06d7255
Remove autoplay attr from audio
RaSan147 Dec 14, 2023
96a6f82
Remove cache blocker.
RaSan147 Dec 14, 2023
9fc5a21
remove wav only blob condition
RaSan147 Dec 30, 2023
b005f73
Merge branch 'guansss:master' into for_PR
RaSan147 Dec 30, 2023
ff1a588
RAN `npm run lint:fix`
RaSan147 Dec 30, 2023
0495f5b
Merge branch 'for_PR' of https://github.com/RaSan147/pixi-live2d-disp…
RaSan147 Dec 30, 2023
1cd47d6
fix startmotion, getting error
RaSan147 Feb 9, 2024
65f31e2
All test passed. Everything OK
RaSan147 Feb 9, 2024
92af59b
revert: undo unexpected changes
guansss Feb 28, 2024
b00b64b
fix: model.speak() not returning true on success
guansss Feb 28, 2024
b511d74
feat: extract lip sync code to LipSync class
guansss Mar 18, 2024
12e9530
test: fix broken test procedure
guansss Mar 18, 2024
ee42229
test: add lip sync test
guansss Mar 18, 2024
6f3cd68
chore: fix failing npm install
guansss Mar 18, 2024
40cd3f3
feat: skip lip sync analyzer when no audio is playing
guansss Apr 1, 2024
638053e
fix: model.motion() parameters should be optional
guansss Apr 8, 2024
9a9d20f
feat: default lip sync id for cubism 4 models
guansss Apr 8, 2024
2ecdb53
feat: make options.crossOrigin also work for audios
guansss Apr 8, 2024
9cb5c38
test: add aborted lip sync test
guansss Apr 8, 2024
f53509b
test: make lip sync test more stable
guansss Apr 14, 2024
d69a771
test: make lip sync test more stable (for real)
guansss Apr 14, 2024
94a7ea1
feat: merge lipSync.analyze() into lipSync.getValue()
guansss Apr 15, 2024
87476a4
test: temporarily disable parallelism to fix random failures
guansss Apr 15, 2024
718215d
revert: remove updateFacialEmotion method
guansss Apr 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
test: add lip sync test
  • Loading branch information
guansss committed Mar 18, 2024
commit ee422294c32553529bf134d7de17e2ffe132f16a
2 changes: 1 addition & 1 deletion test/env.ts
Original file line number Diff line number Diff line change
@@ -196,7 +196,7 @@ export const test = baseTest.extend<CustomContext>({
onload,
onerror,
);
};
} as typeof originalCreateXHR;
});

const loaderMock: CustomContext["loaderMock"] = {
55 changes: 55 additions & 0 deletions test/features/sound.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
import { config } from "@/config";
import { SoundManager } from "@/cubism-common";
import { Cubism4InternalModel, Cubism4ModelSettings } from "@/cubism4";
import { afterEach, beforeEach, expect } from "vitest";
import soundUrl from "../assets/shizuku/sounds/tapBody_00.mp3";
import { TEST_MODEL4, test } from "../env";
import { delay } from "../utils";

beforeEach(() => {
config.sound = true;
});

afterEach(() => {
SoundManager.destroy();
});

test("lip sync", async () => {
const mouthParam = "ParamMouthOpenY";
const model = new Cubism4InternalModel(
await TEST_MODEL4.coreModel(),
new Cubism4ModelSettings(TEST_MODEL4.modelJsonWithUrl),
{ idleMotionGroup: "nonExistent" },
);

expect(model.coreModel.getParameterValueById(mouthParam)).toBe(0);

await expect(model.lipSync.play(soundUrl, { volume: 1 })).resolves.toBe(undefined);

const audio = model.lipSync.currentAudio!;
expect(audio).toBeTruthy();

let prevTime = 0;

async function seekAndGetParam(time: number) {
audio.currentTime = time / 1000;
await delay(100);

let value = NaN;

model.once("beforeModelUpdate", () => {
value = model.coreModel.getParameterValueById(mouthParam);
});
model.update(time - prevTime, time);

prevTime = time;

return value;
}

const timeOffsetBeforeWave = 100;
const timeOffsetDuringWave = 560;

await expect(seekAndGetParam(timeOffsetBeforeWave)).resolves.toBe(0);
await expect(seekAndGetParam(timeOffsetDuringWave)).resolves.toBeGreaterThan(0);
});
24 changes: 24 additions & 0 deletions test/load-cores.ts
Original file line number Diff line number Diff line change
@@ -1,10 +1,34 @@
declare const __CUBISM2_CORE_SOURCE__: string;
declare const __CUBISM4_CORE_SOURCE__: string;

const excludedLogs = [
`Live2D %s`,
`profile : Desktop`,
`[PROFILE_NAME]`,
`[USE_ADJUST_TRANSLATION]`,
`[USE_CACHED_POLYGON_IMAGE]`,
`[EXPAND_W]`,
`Live2D Cubism SDK Core Version`,
`[CSM][I]`,
];

const log = console.log;
console.log = function (...args) {
const firstArg = args[0];

if (typeof firstArg === "string" && excludedLogs.some((log) => firstArg.includes(log))) {
return;
}

log(...args);
};

(() => {
__CUBISM2_CORE_SOURCE__;
})();
(() => {
__CUBISM4_CORE_SOURCE__;
window.Live2DCubismCore = Live2DCubismCore;
})();

console.log = log;
Loading