How to build milvus and run it distributed?

[alanrivers]

Hi there,

I would like to do some customized modifactions on milvus, build milvus myself and run it distributed. My expectation is to obtain a customized milvus image after build process and I may use milvus-operator to start the system. I wonder if I need to do some modifications on milvus-operator as it seems related to an official docker repository.

Besides that, when I follow milvus/DEVELOPMENT.md's instructions and have a try, I come across problems during build process.

BTW, I'm working on Apple M3 with 16GB memory. Go version: go1.23.0 darwin/arm64; Cmake version: 3.30.2; llvm version: 18.1.8.

Building milvus with Docker
My steps:

./scripts/devcontainer.sh up
docker exec -ti milvus_builder_1 bash
make milvus

And I got error:

-- ---------------------------------------------------------------------
-- MILVUS version:                                 2.0
--
-- Build configuration summary:
--   Generator: Unix Makefiles
--   Build type: Release
--   Source directory: /go/src/github.com/milvus-io/milvus/internal/core
--   Compile commands: /go/src/github.com/milvus-io/milvus/cmake_build/compile_commands.json
--
-- Milvus Build Option options:
--   Build GPU version                                                      OFF   [default] [MILVUS_GPU_VERSION]
--
-- Thirdparty options:
--   Method to use for acquiring MILVUS's build dependencies                "BUNDLED" [default] [MILVUS_DEPENDENCY_SOURCE] [AUTO|BUNDLED|SYSTEM]
--   Use ccache when compiling (if available)                               ON        [default] [MILVUS_USE_CCACHE]
--   Show output from ExternalProjects rather than just logging to files    ON        [default] [MILVUS_VERBOSE_THIRDPARTY_BUILD]
--
-- Test and benchmark options:
--   Build the MILVUS googletest unit tests                                 OFF   [default] [MILVUS_BUILD_TESTS]
-- Knowhere repo: https://github.com/zilliztech/knowhere.git
-- Knowhere version: 38e4c88c
-- Building knowhere- from source
-- Release
-- Populating knowhere
-- Configuring done (0.1s)
-- Generating done (0.0s)
-- Build files have been written to: /go/src/github.com/milvus-io/milvus/cmake_build/3rdparty_download/knowhere-subbuild
No such file or directory
CMake Error: Generator: execution of make failed. Make command was: /usr/bin/gmake -f Makefile
CMake Error at /usr/local/share/cmake-3.27/Modules/FetchContent.cmake:1662 (message):
  Build step for knowhere failed: 1
Call Stack (most recent call first):
  /usr/local/share/cmake-3.27/Modules/FetchContent.cmake:1802:EVAL:2 (__FetchContent_directPopulate)
  /usr/local/share/cmake-3.27/Modules/FetchContent.cmake:1802 (cmake_language)
  thirdparty/knowhere/CMakeLists.txt:51 (FetchContent_Populate)


-- Configuring incomplete, errors occurred!
make[1]: *** [Makefile:1096: cmake_check_build_system] Error 1
make[1]: Leaving directory '/go/src/github.com/milvus-io/milvus/cmake_build'
make: *** [Makefile:257: build-cpp] Error 1

Buiilding milvus on a local OS/shell environment
My steps:

./scripts/install_deps.sh
make

And I got error:

In file included from /Users/alanrivers/.conan/data/c-ares/1.33.0/_/_/build/65d0c6387c314520223dd15a4a7aadd3107d2214/src/src/lib/ares_sysconfig_mac.c:64:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/SystemConfiguration.framework/Headers/SCNetworkConfiguration.h:31:
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/SystemConfiguration.framework/Headers/SystemConfiguration.h:126:
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/SystemConfiguration.framework/Headers/SCPreferences.h:196:6: error: unknown type name 'AuthorizationRef'
                                        AuthorizationRef        __nullable      authorization
                                        ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/SystemConfiguration.framework/Headers/SCPreferences.h:196:23: error: nullability specifier '_Nullable' cannot be applied to non-pointer type 'int'
                                        AuthorizationRef        __nullable      authorization
                                                                ^
<built-in>:337:20: note: expanded from here
#define __nullable _Nullable
                   ^
2 errors generated.
make[3]: *** [src/lib/CMakeFiles/c-ares.dir/ares_sysconfig_mac.c.o] Error 1
make[2]: *** [src/lib/CMakeFiles/c-ares.dir/all] Error 2
make[1]: *** [all] Error 2
c-ares/1.33.0:
c-ares/1.33.0: ERROR: Package '65d0c6387c314520223dd15a4a7aadd3107d2214' build failed
c-ares/1.33.0: WARN: Build folder /Users/alanrivers/.conan/data/c-ares/1.33.0/_/_/build/65d0c6387c314520223dd15a4a7aadd3107d2214/build/Release
ERROR: c-ares/1.33.0: Error in build() method, line 63
	cmake.build()
	ConanException: Error 2 while executing cmake --build "/Users/alanrivers/.conan/data/c-ares/1.33.0/_/_/build/65d0c6387c314520223dd15a4a7aadd3107d2214/build/Release" '--' '-j8'
conan install failed
make: *** [build-3rdparty] Error 1

[yhmo]

I have no M3, but looks like the conan is not installed properly: conan install failed
Conan is a key tool to build milvus, you can search more info in google or conan documents, this is not a milvus problem. Before you compile milvus, makesure conan is installed correctly on the machine.

[alanrivers]

I try to compile it on Windows 10 with Docker and after more than 10 hours I've finally got the binary file. And I am able to build milvus image based on that. Here's my steps:

./build/builder.sh make install;
LD_LIBRARY_PATH=./internal/core/output/lib:lib:$LD_LIBRARY_PATH;
./build/build_image.sh;

I wonder if there's any difference between systems of host machines even if I'm compiling with Docker. With my understanding I might only need docker and docker-compose installed on host machines.
And also, I check out the document and it seems that Helm Chart supports local image rather than milvus operator. I wonder if I can choose to use Helm Chart to run it distributed, after specifying the milvus image to be my local image in yaml file.

[Presburger]

@alanrivers hello, Could you tell me how many processor cores and how much memory your Windows container build environment can use?

[alanrivers]

My Windows laptop is with 8 logical cores and 16GB memory, which is basically the same as Apple M3.

[xiaofan-luan]

@alanrivers hello, Could you tell me how many processor cores and how much memory your Windows container build environment can use?

we probably need to fix the dev container first so at least there is a choice here

[mhsmith]

/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/SystemConfiguration.framework/Headers/SCPreferences.h:196:6: error: unknown type name 'AuthorizationRef'
                                        AuthorizationRef        __nullable      authorization

I got this error in the following situation:

• xcode-select was set to use my XCode installation, which was version 14.
• My XCode command line tools (installed separately) were version 16.

It was resolved by updating XCode to be version 16 as well.