Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add OneDNN or DirectML support #2303

Open
thewh1teagle opened this issue Jul 15, 2024 · 9 comments
Open

Add OneDNN or DirectML support #2303

thewh1teagle opened this issue Jul 15, 2024 · 9 comments

Comments

@thewh1teagle
Copy link
Contributor

thewh1teagle commented Jul 15, 2024

Currently the best results we can get with whisper.cpp is with Cuda (Nvidia) or CoreML (macOS).

On Windows there's only OpenBlas and it works slow, maybe 2 times of the duration of the audio (amd ryzen 5 4500u, medium model).
When using ctranslate2 on the same machine it works 2-3 times faster than the audio duration on CPU only!

Since recently whisper.cpp removed support for OpenCL, I think that it's important having good alternative to Windows users with Intel / AMD CPUs / TPUs.

There's few different options that can be added:
oneDNN-ExecutionProvider.html
DirectML-ExecutionProvider.html

In addition ctranslate2 uses ruy

Related: ggerganov/ggml#406 (comment)

@thewh1teagle thewh1teagle changed the title Add OneDNN support Add OneDNN or DirectML support Jul 15, 2024
@WilliamTambellini
Copy link
Contributor

+1 for oneDNN

@WilliamTambellini
Copy link
Contributor

ggerganov/ggml#855

@WilliamTambellini
Copy link
Contributor

cf @rfsaliev

@thewh1teagle
Copy link
Contributor Author

Update:
meanwhile I'm sticking to release v1.6.2 which still have OpenCL support.
Otherwise as I said the speed is too much slow and not usable (2-5x time more than the audio duration).

Now with OpenCL it takes 40s to transcribe 47s audio on the same normal TPU hardware (amd ryzen5 4500u)

By the way there was weird issues with OpenCL that prevent it from work. the solution I found is to set CMAKE_BUILD_TYPE to RelWithDebInfo

@Osiris-Team
Copy link

Any updates on this?
I am trying to run this with GPU acceleration on Windows with AMD GPU.

@thewh1teagle
Copy link
Contributor Author

@Osiris-Team

Use vulkan it works fast with amd glu on all platforms. You can try with the app vibe

@Osiris-Team
Copy link

Osiris-Team commented Oct 16, 2024

@thewh1teagle thx that looks great, giving it a try!
Can I select a custom model since I already got one downloaded?
I assume its running whisper.cpp under the hood and thus maybe has the same format (ggml)? - It does.

@Osiris-Team
Copy link

@thewh1teagle There is a good minute at the start where it says "Transcribing - 0%", thought it wasn't working (no cpu/gpu/io activity), maybe adding some more detailed logging in whatever is happening at the beginning helps here.

@thewh1teagle
Copy link
Contributor Author

@Osiris-Team

You can take a look in the docs of the repository in debug.md.
Also you can open new issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants