Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DuckDB.jl fails precompile on windows machine #108

Open
drizk1 opened this issue Feb 11, 2025 · 9 comments
Open

DuckDB.jl fails precompile on windows machine #108

drizk1 opened this issue Feb 11, 2025 · 9 comments

Comments

@drizk1
Copy link
Member

drizk1 commented Feb 11, 2025

I am opening this issue here as well, just so folks can rapidly find the answer.

For some windows users, In Julia 1.11.2 and 1.11.3, I can't use Tidier because of an error with DuckDB.jl:

julia> using Tidier
Precompiling Tidier...
Info Given Tidier was explicitly requested, output will be shown live
ERROR: LoadError: could not load symbol "duckdb_vector_size":
The specified procedure could not be found.
Stacktrace:
 [1] duckdb_vector_size()
   @ DuckDB C:\Users\franc\.julia\packages\DuckDB\8Gvy6\src\api.jl:715
 [2] top-level scope
   @ C:\Users\franc\.julia\packages\DuckDB\8Gvy6\src\database.jl:100
 [3] include(mod::Module, _path::String)
   @ Base .\Base.jl:557
 [4] include(x::String)
   @ DuckDB C:\Users\franc\.julia\packages\DuckDB\8Gvy6\src\DuckDB.jl:1
 [5] top-level scope
   @ C:\Users\franc\.julia\packages\DuckDB\8Gvy6\src\DuckDB.jl:28
 [6] include
   @ .\Base.jl:557 [inlined]
 [7] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
   @ Base .\loading.jl:2881
 [8] top-level scope
   @ stdin:6
in expression starting at C:\Users\franc\.julia\packages\DuckDB\8Gvy6\src\database.jl:100
in expression starting at C:\Users\franc\.julia\packages\DuckDB\8Gvy6\src\DuckDB.jl:1
in expression starting at stdin:6
ERROR: LoadError: Failed to precompile DuckDB [d2f5444f-75bc-4fdf-ac35-56f514c445e1] to "C:\\Users\\franc\\.julia\\compiled\\v1.11\\DuckDB\\jl_EB27.tmp".
Stacktrace:
  [1] error(s::String)
    @ Base .\error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool; flags::Cmd, cacheflags::Base.CacheFlags, reasons::Dict{String, Int64}, loadable_exts::Nothing)
    @ Base .\loading.jl:3174
  [3] (::Base.var"#1110#1111"{Base.PkgId})()
    @ Base .\loading.jl:2579
  [4] mkpidlock(f::Base.var"#1110#1111"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
    @ FileWatching.Pidfile C:\Users\franc\.julia\juliaup\julia-1.11.3+0.x64.w64.mingw32\share\julia\stdlib\v1.11\FileWatching\src\pidfile.jl:95
  [5] #mkpidlock#6
    @ C:\Users\franc\.julia\juliaup\julia-1.11.3+0.x64.w64.mingw32\share\julia\stdlib\v1.11\FileWatching\src\pidfile.jl:90 [inlined]
  [6] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
    @ FileWatching.Pidfile C:\Users\franc\.julia\juliaup\julia-1.11.3+0.x64.w64.mingw32\share\julia\stdlib\v1.11\FileWatching\src\pidfile.jl:116
  [7] #invokelatest#2
    @ .\essentials.jl:1057 [inlined]
  [8] invokelatest
    @ .\essentials.jl:1052 [inlined]
  [9] maybe_cachefile_lock(f::Base.var"#1110#1111"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
    @ Base .\loading.jl:3698
 [10] maybe_cachefile_lock
    @ .\loading.jl:3695 [inlined]
 [11] _require(pkg::Base.PkgId, env::String)
    @ Base .\loading.jl:2565
 [12] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base .\loading.jl:2388
 [13] #invoke_in_world#3
    @ .\essentials.jl:1089 [inlined]
 [14] invoke_in_world
    @ .\essentials.jl:1086 [inlined]
 [15] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base .\loading.jl:2375
 [16] macro expansion
    @ .\loading.jl:2314 [inlined]
 [17] macro expansion
    @ .\lock.jl:273 [inlined]
 [18] __require(into::Module, mod::Symbol)
    @ Base .\loading.jl:2271
 [19] #invoke_in_world#3
    @ .\essentials.jl:1089 [inlined]
 [20] invoke_in_world
    @ .\essentials.jl:1086 [inlined]
 [21] require(into::Module, mod::Symbol)
    @ Base .\loading.jl:2260
 [22] include
    @ .\Base.jl:557 [inlined]
 [23] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
    @ Base .\loading.jl:2881
 [24] top-level scope
    @ stdin:6
in expression starting at C:\Users\franc\.julia\packages\TidierDB\J2fSt\src\TidierDB.jl:1
in expression starting at stdin:6
ERROR: LoadError: Failed to precompile TidierDB [86993f9b-bbba-4084-97c5-ee15961ad48b] to "C:\\Users\\franc\\.julia\\compiled\\v1.11\\TidierDB\\jl_E74F.tmp".
Stacktrace:
  [1] error(s::String)
    @ Base .\error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool; flags::Cmd, cacheflags::Base.CacheFlags, reasons::Dict{String, Int64}, loadable_exts::Nothing)
    @ Base .\loading.jl:3174
  [3] (::Base.var"#1110#1111"{Base.PkgId})()
    @ Base .\loading.jl:2579
  [4] mkpidlock(f::Base.var"#1110#1111"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
    @ FileWatching.Pidfile C:\Users\franc\.julia\juliaup\julia-1.11.3+0.x64.w64.mingw32\share\julia\stdlib\v1.11\FileWatching\src\pidfile.jl:95
  [5] #mkpidlock#6
    @ C:\Users\franc\.julia\juliaup\julia-1.11.3+0.x64.w64.mingw32\share\julia\stdlib\v1.11\FileWatching\src\pidfile.jl:90 [inlined]
  [6] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
    @ FileWatching.Pidfile C:\Users\franc\.julia\juliaup\julia-1.11.3+0.x64.w64.mingw32\share\julia\stdlib\v1.11\FileWatching\src\pidfile.jl:116
  [7] #invokelatest#2
    @ .\essentials.jl:1057 [inlined]
  [8] invokelatest
    @ .\essentials.jl:1052 [inlined]
  [9] maybe_cachefile_lock(f::Base.var"#1110#1111"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
    @ Base .\loading.jl:3698
 [10] maybe_cachefile_lock
    @ .\loading.jl:3695 [inlined]
 [11] _require(pkg::Base.PkgId, env::String)
    @ Base .\loading.jl:2565
 [12] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base .\loading.jl:2388
 [13] #invoke_in_world#3
    @ .\essentials.jl:1089 [inlined]
 [14] invoke_in_world
    @ .\essentials.jl:1086 [inlined]
 [15] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base .\loading.jl:2375
 [16] macro expansion
    @ .\loading.jl:2314 [inlined]
 [17] macro expansion
    @ .\lock.jl:273 [inlined]
 [18] __require(into::Module, mod::Symbol)
    @ Base .\loading.jl:2271
 [19] #invoke_in_world#3
    @ .\essentials.jl:1089 [inlined]
 [20] invoke_in_world
    @ .\essentials.jl:1086 [inlined]
 [21] require(into::Module, mod::Symbol)
    @ Base .\loading.jl:2260
 [22] include
    @ .\Base.jl:557 [inlined]
 [23] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
    @ Base .\loading.jl:2881
 [24] top-level scope
    @ stdin:6
in expression starting at C:\Users\franc\.julia\packages\Tidier\owMbd\src\Tidier.jl:1
in expression starting at stdin:6
  ✗ DuckDB
  ✗ TidierDB
  ✗ Tidier
  0 dependencies successfully precompiled in 24 seconds. 389 already precompiled.

ERROR: The following 1 direct dependency failed to precompile:

Tidier

Failed to precompile Tidier [f0413319-3358-4bb0-8e7c-0c83523a93bd] to "C:\\Users\\franc\\.julia\\compiled\\v1.11\\Tidier\\jl_C1DB.tmp".
ERROR: LoadError: could not load symbol "duckdb_vector_size":
The specified procedure could not be found.
Stacktrace:
 [1] duckdb_vector_size()
   @ DuckDB C:\Users\franc\.julia\packages\DuckDB\8Gvy6\src\api.jl:715
 [2] top-level scope
   @ C:\Users\franc\.julia\packages\DuckDB\8Gvy6\src\database.jl:100
 [3] include(mod::Module, _path::String)
   @ Base .\Base.jl:557
 [4] include(x::String)
   @ DuckDB C:\Users\franc\.julia\packages\DuckDB\8Gvy6\src\DuckDB.jl:1
 [5] top-level scope
   @ C:\Users\franc\.julia\packages\DuckDB\8Gvy6\src\DuckDB.jl:28
 [6] include
   @ .\Base.jl:557 [inlined]
 [7] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
   @ Base .\loading.jl:2881
 [8] top-level scope
   @ stdin:6
in expression starting at C:\Users\franc\.julia\packages\DuckDB\8Gvy6\src\database.jl:100
in expression starting at C:\Users\franc\.julia\packages\DuckDB\8Gvy6\src\DuckDB.jl:1
in expression starting at stdin:6
ERROR: LoadError: Failed to precompile DuckDB [d2f5444f-75bc-4fdf-ac35-56f514c445e1] to "C:\\Users\\franc\\.julia\\compiled\\v1.11\\DuckDB\\jl_EB27.tmp".
Stacktrace:
  [1] error(s::String)
    @ Base .\error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool; flags::Cmd, cacheflags::Base.CacheFlags, reasons::Dict{String, Int64}, loadable_exts::Nothing)
    @ Base .\loading.jl:3174
  [3] (::Base.var"#1110#1111"{Base.PkgId})()
    @ Base .\loading.jl:2579
  [4] mkpidlock(f::Base.var"#1110#1111"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
    @ FileWatching.Pidfile C:\Users\franc\.julia\juliaup\julia-1.11.3+0.x64.w64.mingw32\share\julia\stdlib\v1.11\FileWatching\src\pidfile.jl:95
  [5] #mkpidlock#6
    @ C:\Users\franc\.julia\juliaup\julia-1.11.3+0.x64.w64.mingw32\share\julia\stdlib\v1.11\FileWatching\src\pidfile.jl:90 [inlined]
  [6] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
    @ FileWatching.Pidfile C:\Users\franc\.julia\juliaup\julia-1.11.3+0.x64.w64.mingw32\share\julia\stdlib\v1.11\FileWatching\src\pidfile.jl:116
  [7] #invokelatest#2
    @ .\essentials.jl:1057 [inlined]
  [8] invokelatest
    @ .\essentials.jl:1052 [inlined]
  [9] maybe_cachefile_lock(f::Base.var"#1110#1111"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
    @ Base .\loading.jl:3698
 [10] maybe_cachefile_lock
    @ .\loading.jl:3695 [inlined]
 [11] _require(pkg::Base.PkgId, env::String)
    @ Base .\loading.jl:2565
 [12] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base .\loading.jl:2388
 [13] #invoke_in_world#3
    @ .\essentials.jl:1089 [inlined]
 [14] invoke_in_world
    @ .\essentials.jl:1086 [inlined]
 [15] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base .\loading.jl:2375
 [16] macro expansion
    @ .\loading.jl:2314 [inlined]
 [17] macro expansion
    @ .\lock.jl:273 [inlined]
 [18] __require(into::Module, mod::Symbol)
    @ Base .\loading.jl:2271
 [19] #invoke_in_world#3
    @ .\essentials.jl:1089 [inlined]
 [20] invoke_in_world
    @ .\essentials.jl:1086 [inlined]
 [21] require(into::Module, mod::Symbol)
    @ Base .\loading.jl:2260
 [22] include
    @ .\Base.jl:557 [inlined]
 [23] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
    @ Base .\loading.jl:2881
 [24] top-level scope
    @ stdin:6
in expression starting at C:\Users\franc\.julia\packages\TidierDB\J2fSt\src\TidierDB.jl:1
in expression starting at stdin:6
ERROR: LoadError: Failed to precompile TidierDB [86993f9b-bbba-4084-97c5-ee15961ad48b] to "C:\\Users\\franc\\.julia\\compiled\\v1.11\\TidierDB\\jl_E74F.tmp".
Stacktrace:
  [1] error(s::String)
    @ Base .\error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool; flags::Cmd, cacheflags::Base.CacheFlags, reasons::Dict{String, Int64}, loadable_exts::Nothing)
    @ Base .\loading.jl:3174
  [3] (::Base.var"#1110#1111"{Base.PkgId})()
    @ Base .\loading.jl:2579
  [4] mkpidlock(f::Base.var"#1110#1111"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
    @ FileWatching.Pidfile C:\Users\franc\.julia\juliaup\julia-1.11.3+0.x64.w64.mingw32\share\julia\stdlib\v1.11\FileWatching\src\pidfile.jl:95
  [5] #mkpidlock#6
    @ C:\Users\franc\.julia\juliaup\julia-1.11.3+0.x64.w64.mingw32\share\julia\stdlib\v1.11\FileWatching\src\pidfile.jl:90 [inlined]
  [6] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
    @ FileWatching.Pidfile C:\Users\franc\.julia\juliaup\julia-1.11.3+0.x64.w64.mingw32\share\julia\stdlib\v1.11\FileWatching\src\pidfile.jl:116
  [7] #invokelatest#2
    @ .\essentials.jl:1057 [inlined]
  [8] invokelatest
    @ .\essentials.jl:1052 [inlined]
  [9] maybe_cachefile_lock(f::Base.var"#1110#1111"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
    @ Base .\loading.jl:3698
 [10] maybe_cachefile_lock
    @ .\loading.jl:3695 [inlined]
 [11] _require(pkg::Base.PkgId, env::String)
    @ Base .\loading.jl:2565
 [12] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base .\loading.jl:2388
 [13] #invoke_in_world#3
    @ .\essentials.jl:1089 [inlined]
 [14] invoke_in_world
    @ .\essentials.jl:1086 [inlined]
 [15] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base .\loading.jl:2375
 [16] macro expansion
    @ .\loading.jl:2314 [inlined]
 [17] macro expansion
    @ .\lock.jl:273 [inlined]
 [18] __require(into::Module, mod::Symbol)
    @ Base .\loading.jl:2271
 [19] #invoke_in_world#3
    @ .\essentials.jl:1089 [inlined]
 [20] invoke_in_world
    @ .\essentials.jl:1086 [inlined]
 [21] require(into::Module, mod::Symbol)
    @ Base .\loading.jl:2260
 [22] include
    @ .\Base.jl:557 [inlined]
 [23] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
    @ Base .\loading.jl:2881
 [24] top-level scope
    @ stdin:6
in expression starting at C:\Users\franc\.julia\packages\Tidier\owMbd\src\Tidier.jl:1
in expression starting at stdin:
@drizk1
Copy link
Member Author

drizk1 commented Feb 11, 2025

Temporary solution until it is fixed, which is being worked on

This is an ongoing issue with DuckDB.jl on windows machines.

There is a discourse conversation here and a PR

I will reach out and see if there is away to get around this. In the meantime, you may have to individually use the Tidier Pkgs you want ie

using TidierData, TidierPlots, TidierDates

You might be able to get a temp fix based on that discourse conversation by installing the binary from the webpage and setting

ENV["JULIA_DUCKDB_LIBRARY"] = raw"C:\path\to\duckdb.dll"

@abelsiqueira
Copy link

Hi, just wanted to link to the duckdb issue with some investigation on this: duckdb/duckdb#13911
My colleague @suvayu (we're not from DuckDB) is looking into this, but we are currently stuck. If anybody with more Windows DLL knowledge finds this and has any idea, it will be much appreciated.

@drizk1
Copy link
Member Author

drizk1 commented Feb 19, 2025 •
edited
Loading

I have no windows experience/computer.

I've read around all the posts I've found. It seems fixable considering the workaround seems to work.

Has the workaround worked for you and your colleagues @abelsiqueira

@abelsiqueira
Copy link

We also don't use Windows, but all our partners in this project (and their server) use Windows machines. And we run CI on Windows as well.

We fixed the DuckDB (and JLL) versions to an earlier version (https://github.com/TulipaEnergy/TulipaEnergyModel.jl/blob/91619f7529ec7c350b1928dfab7307c1c20f4469/Project.toml#L25), until the issue is resolved.

Fixing the path is not an option. We don't want to request our users to manually do anything extra. The good thing about JLL is that it handles this automatically (when it works 😉)

@kdpsingh
Copy link
Member

Thanks @abelsiqueira! @drizk1, could we similarly pin to the older version and DuckDB_jll, or do we rely on newer DuckDB features?

@drizk1
Copy link
Member Author

drizk1 commented Feb 20, 2025 •
edited
Loading

We can def pinto the older version.

Would lose access to UDFs and some of the 1.2 updated windowing abilities but a reasonable sacrifice w prob no noticeable changes

@drizk1
Copy link
Member Author

drizk1 commented Feb 21, 2025

@abelsiqueira thanks for the tip.

one question since you seem well versed and i can test this myself
will this work ? or does DuckDB_jll being 1.1 cause the issue and it needs to be 1.0? I noticed your link is 1.0 but says it supports 1.1 as well

DuckDB = "1.0"
DuckDB_jll = "1.0, 1.1"

@abelsiqueira
Copy link

It needs to be 1.0.
That comment is indeed not explaining anything. The problem is DuckDB_jll 1.1.0 (and later 1.0.1) is the broken package. At the time, DuckDB 1.0 was not restricting DuckDB_jll to 1.0. So we had initially only restricted DuckDB to 1.0 and it was not enough, so we also had to restrict DuckDB_jll to 1.0 because "# DuckDB 1.0.0 still allows DuckDB_jll 1.1.0".
We expected it to be fixed soon, but it's probably not a priority since Julia has tier 2 support from DuckDB (https://duckdb.org/docs/clients/overview.html).

@drizk1
Copy link
Member Author

drizk1 commented Feb 22, 2025

This is a great temp solution to avoid issues for the user. I do hope it's gets fixed soon. Some of the PRs look encouraging.

Thanks again for pointing this out

@digital-carver digital-carver mentioned this issue Feb 24, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@abelsiqueira @kdpsingh @drizk1