Some features of whisper.cpp was removed while combining llama.cpp branch. Suggest for sepereate branch for llama.cpp and whisper.cpp

[Pi-tool]

Some of features like ggml_graph_plan function was removed from ggml library when combining llama.cpp branch, it seems not using the previous branch from ggml with whisper.cpp is fully supported.

Is it possible to make ggml library can be used for both llama.cpp and whisper.cpp, and both can invoke the function from ggml library instead of using the library on the repository in both projects?

My suggestion is devide two branches into llama.cpp and whisper.cpp, and each project can invoke from different branch. Then every year or half a year, we can evaluate the pull requests, and see which changement should be merged into the main stream of master(or other stable branch), and which should be discarded.I would like to volenteer to do the maintainence job if nessersary.

[alinemati-uwm]

same here !

[kouhkanamir]

Issue: Compilation error in whisper.cpp and llama.cpp when integrating with ggml.

Details: I'm encountering several compilation errors when building whisper.cpp with ggml. Below is a summary of the issues:

use of undeclared identifier 'ggml_graph_plan' - The code suggests using ggml_graph_plan, but only ggml_graph_import is declared in ggml.h.
cannot initialize a parameter of type 'const char *' with an lvalue of type 'struct ggml_cgraph *'.
use of undeclared identifier 'ggml_backend_is_cpu' - Suggested identifier ggml_backend_is_blas is not appropriate here.
use of undeclared identifier 'ggml_backend_cpu_set_n_threads' - Similar issue, suggested ggml_backend_blas_set_n_threads.
use of undeclared identifier 'ggml_set_f32', ggml_get_f32_nd, and others not declared in the ggml library.

Attempt to build the project.
System information:
macOS

compile error:

[  4%] Built target build_info
[ 31%] Built target ggml
[ 46%] Built target llama
[ 64%] Built target common
[ 68%] Built target llama_project
[ 71%] Building CXX object external/whisper.cpp/src/CMakeFiles/whisper.dir/whisper.cpp.o
/Users/amir/Workspace/Python/embedding_python_nomic/external/whisper.cpp/src/whisper.cpp:176:30: error: use of undeclared identifier 'ggml_graph_plan'; did you mean 'ggml_graph_import'?
    struct ggml_cplan plan = ggml_graph_plan(graph, n_threads, nullptr);
                             ^~~~~~~~~~~~~~~
                             ggml_graph_import
/Users/amir/Workspace/Python/embedding_python_nomic/external/llama.cpp/ggml/src/../include/ggml.h:2029:35: note: 'ggml_graph_import' declared here
    GGML_API struct ggml_cgraph * ggml_graph_import(const char * fname, struct ggml_context ** ctx_data, struct ggml_context ** ctx_eval);
                                  ^
/Users/amir/Workspace/Python/embedding_python_nomic/external/whisper.cpp/src/whisper.cpp:176:46: error: cannot initialize a parameter of type 'const char *' with an lvalue of type 'struct ggml_cgraph *'
    struct ggml_cplan plan = ggml_graph_plan(graph, n_threads, nullptr);
                                             ^~~~~
/Users/amir/Workspace/Python/embedding_python_nomic/external/llama.cpp/ggml/src/../include/ggml.h:2029:66: note: passing argument to parameter 'fname' here
    GGML_API struct ggml_cgraph * ggml_graph_import(const char * fname, struct ggml_context ** ctx_data, struct ggml_context ** ctx_eval);
                                                                 ^
/Users/amir/Workspace/Python/embedding_python_nomic/external/whisper.cpp/src/whisper.cpp:176:23: error: variable has incomplete type 'struct ggml_cplan'
    struct ggml_cplan plan = ggml_graph_plan(graph, n_threads, nullptr);
                      ^
/Users/amir/Workspace/Python/embedding_python_nomic/external/whisper.cpp/src/whisper.cpp:176:12: note: forward declaration of 'ggml_cplan'
    struct ggml_cplan plan = ggml_graph_plan(graph, n_threads, nullptr);
           ^
/Users/amir/Workspace/Python/embedding_python_nomic/external/whisper.cpp/src/whisper.cpp:196:13: error: use of undeclared identifier 'ggml_backend_is_cpu'; did you mean 'ggml_backend_is_blas'?
        if (ggml_backend_is_cpu(backend)) {
            ^~~~~~~~~~~~~~~~~~~
            ggml_backend_is_blas
/Users/amir/Workspace/Python/embedding_python_nomic/external/llama.cpp/ggml/src/../include/ggml-blas.h:14:15: note: 'ggml_backend_is_blas' declared here
GGML_API bool ggml_backend_is_blas(ggml_backend_t backend);
              ^
/Users/amir/Workspace/Python/embedding_python_nomic/external/whisper.cpp/src/whisper.cpp:197:13: error: use of undeclared identifier 'ggml_backend_cpu_set_n_threads'; did you mean 'ggml_backend_blas_set_n_threads'?
            ggml_backend_cpu_set_n_threads(backend, n_threads);
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            ggml_backend_blas_set_n_threads
/Users/amir/Workspace/Python/embedding_python_nomic/external/llama.cpp/ggml/src/../include/ggml-blas.h:18:15: note: 'ggml_backend_blas_set_n_threads' declared here
GGML_API void ggml_backend_blas_set_n_threads(ggml_backend_t backend_blas, int n_threads);
              ^
/Users/amir/Workspace/Python/embedding_python_nomic/external/whisper.cpp/src/whisper.cpp:1343:22: error: use of undeclared identifier 'ggml_backend_cpu_init'; did you mean 'ggml_backend_blas_init'?
    result.push_back(ggml_backend_cpu_init());
                     ^~~~~~~~~~~~~~~~~~~~~
                     ggml_backend_blas_init
/Users/amir/Workspace/Python/embedding_python_nomic/external/llama.cpp/ggml/src/../include/ggml-blas.h:12:25: note: 'ggml_backend_blas_init' declared here
GGML_API ggml_backend_t ggml_backend_blas_init(void);
                        ^
/Users/amir/Workspace/Python/embedding_python_nomic/external/whisper.cpp/src/whisper.cpp:4254:42: error: use of undeclared identifier 'ggml_cpu_has_neon'; did you mean 'ggml_cpu_has_cann'?
    s += "NEON = "      + std::to_string(ggml_cpu_has_neon())      + " | ";
                                         ^~~~~~~~~~~~~~~~~
                                         ggml_cpu_has_cann
/Users/amir/Workspace/Python/embedding_python_nomic/external/llama.cpp/ggml/src/../include/ggml.h:2413:18: note: 'ggml_cpu_has_cann' declared here
    GGML_API int ggml_cpu_has_cann       (void);
                 ^
/Users/amir/Workspace/Python/embedding_python_nomic/external/whisper.cpp/src/whisper.cpp:7113:12: error: use of undeclared identifier 'ggml_set_f32'; did you mean 'ggml_set_name'?
    cost = ggml_set_f32(cost, INFINITY);
           ^~~~~~~~~~~~
           ggml_set_name
/Users/amir/Workspace/Python/embedding_python_nomic/external/llama.cpp/ggml/src/../include/ggml.h:770:35: note: 'ggml_set_name' declared here
    GGML_API struct ggml_tensor * ggml_set_name   (      struct ggml_tensor * tensor, const char * name);
                                  ^
/Users/amir/Workspace/Python/embedding_python_nomic/external/whisper.cpp/src/whisper.cpp:7113:31: error: cannot initialize a parameter of type 'const char *' with an rvalue of type 'float'
    cost = ggml_set_f32(cost, INFINITY);
                              ^~~~~~~~
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.5.sdk/usr/include/math.h:69:21: note: expanded from macro 'INFINITY'
#define INFINITY    HUGE_VALF
                    ^~~~~~~~~
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.5.sdk/usr/include/math.h:59:28: note: expanded from macro 'HUGE_VALF'
#   define    HUGE_VALF    __builtin_huge_valf()
                           ^~~~~~~~~~~~~~~~~~~~~
/Users/amir/Workspace/Python/embedding_python_nomic/external/llama.cpp/ggml/src/../include/ggml.h:770:100: note: passing argument to parameter 'name' here
    GGML_API struct ggml_tensor * ggml_set_name   (      struct ggml_tensor * tensor, const char * name);
                                                                                                   ^
/Users/amir/Workspace/Python/embedding_python_nomic/external/whisper.cpp/src/whisper.cpp:7114:13: error: use of undeclared identifier 'ggml_set_f32'; did you mean 'ggml_set_name'?
    trace = ggml_set_f32(trace, -1);
            ^~~~~~~~~~~~
            ggml_set_name
/Users/amir/Workspace/Python/embedding_python_nomic/external/llama.cpp/ggml/src/../include/ggml.h:770:35: note: 'ggml_set_name' declared here
    GGML_API struct ggml_tensor * ggml_set_name   (      struct ggml_tensor * tensor, const char * name);
                                  ^
/Users/amir/Workspace/Python/embedding_python_nomic/external/whisper.cpp/src/whisper.cpp:7114:33: error: cannot initialize a parameter of type 'const char *' with an rvalue of type 'int'
    trace = ggml_set_f32(trace, -1);
                                ^~
/Users/amir/Workspace/Python/embedding_python_nomic/external/llama.cpp/ggml/src/../include/ggml.h:770:100: note: passing argument to parameter 'name' here
    GGML_API struct ggml_tensor * ggml_set_name   (      struct ggml_tensor * tensor, const char * name);
                                                                                                   ^
/Users/amir/Workspace/Python/embedding_python_nomic/external/whisper.cpp/src/whisper.cpp:7115:5: error: use of undeclared identifier 'ggml_set_f32_nd'
    ggml_set_f32_nd(cost, 0, 0, 0, 0, 0.0);
    ^
/Users/amir/Workspace/Python/embedding_python_nomic/external/whisper.cpp/src/whisper.cpp:7122:24: error: use of undeclared identifier 'ggml_get_f32_nd'; did you mean 'ggml_set_2d'?
            float c0 = ggml_get_f32_nd(cost, i - 1, j - 1, 0, 0);
                       ^~~~~~~~~~~~~~~
                       ggml_set_2d
/Users/amir/Workspace/Python/embedding_python_nomic/external/llama.cpp/ggml/src/../include/ggml.h:1194:35: note: 'ggml_set_2d' declared here
    GGML_API struct ggml_tensor * ggml_set_2d(
                                  ^
/Users/amir/Workspace/Python/embedding_python_nomic/external/whisper.cpp/src/whisper.cpp:7122:40: error: cannot initialize a parameter of type 'struct ggml_context *' with an lvalue of type 'struct ggml_tensor *'
            float c0 = ggml_get_f32_nd(cost, i - 1, j - 1, 0, 0);
                                       ^~~~
/Users/amir/Workspace/Python/embedding_python_nomic/external/llama.cpp/ggml/src/../include/ggml.h:1195:35: note: passing argument to parameter 'ctx' here
            struct ggml_context * ctx,
                                  ^
/Users/amir/Workspace/Python/embedding_python_nomic/external/whisper.cpp/src/whisper.cpp:7123:24: error: use of undeclared identifier 'ggml_get_f32_nd'; did you mean 'ggml_set_2d'?
            float c1 = ggml_get_f32_nd(cost, i - 1, j, 0, 0);
                       ^~~~~~~~~~~~~~~
                       ggml_set_2d
/Users/amir/Workspace/Python/embedding_python_nomic/external/llama.cpp/ggml/src/../include/ggml.h:1194:35: note: 'ggml_set_2d' declared here
    GGML_API struct ggml_tensor * ggml_set_2d(
                                  ^
/Users/amir/Workspace/Python/embedding_python_nomic/external/whisper.cpp/src/whisper.cpp:7123:40: error: cannot initialize a parameter of type 'struct ggml_context *' with an lvalue of type 'struct ggml_tensor *'
            float c1 = ggml_get_f32_nd(cost, i - 1, j, 0, 0);
                                       ^~~~
/Users/amir/Workspace/Python/embedding_python_nomic/external/llama.cpp/ggml/src/../include/ggml.h:1195:35: note: passing argument to parameter 'ctx' here
            struct ggml_context * ctx,
                                  ^
/Users/amir/Workspace/Python/embedding_python_nomic/external/whisper.cpp/src/whisper.cpp:7124:24: error: use of undeclared identifier 'ggml_get_f32_nd'; did you mean 'ggml_set_2d'?
            float c2 = ggml_get_f32_nd(cost, i, j - 1, 0, 0);
                       ^~~~~~~~~~~~~~~
                       ggml_set_2d
/Users/amir/Workspace/Python/embedding_python_nomic/external/llama.cpp/ggml/src/../include/ggml.h:1194:35: note: 'ggml_set_2d' declared here
    GGML_API struct ggml_tensor * ggml_set_2d(
                                  ^
/Users/amir/Workspace/Python/embedding_python_nomic/external/whisper.cpp/src/whisper.cpp:7124:40: error: cannot initialize a parameter of type 'struct ggml_context *' with an lvalue of type 'struct ggml_tensor *'
            float c2 = ggml_get_f32_nd(cost, i, j - 1, 0, 0);
                                       ^~~~
/Users/amir/Workspace/Python/embedding_python_nomic/external/llama.cpp/ggml/src/../include/ggml.h:1195:35: note: passing argument to parameter 'ctx' here
            struct ggml_context * ctx,
                                  ^
/Users/amir/Workspace/Python/embedding_python_nomic/external/whisper.cpp/src/whisper.cpp:7139:17: error: use of undeclared identifier 'ggml_get_f32_nd'; did you mean 'ggml_set_2d'?
            c = ggml_get_f32_nd(x, i - 1, j - 1, 0, 0) + c;
                ^~~~~~~~~~~~~~~
                ggml_set_2d
/Users/amir/Workspace/Python/embedding_python_nomic/external/llama.cpp/ggml/src/../include/ggml.h:1194:35: note: 'ggml_set_2d' declared here
    GGML_API struct ggml_tensor * ggml_set_2d(
                                  ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
make[2]: *** [external/whisper.cpp/src/CMakeFiles/whisper.dir/whisper.cpp.o] Error 1
make[1]: *** [external/whisper.cpp/src/CMakeFiles/whisper.dir/all] Error 2
make: *** [all] Error 2

[TXTware]

This is not only for swift package, but also for cmake. It is important for us to lavarage a consist library between llama.cpp and whisper.cpp