Linker multiple definition issue with FATFS and VFS (IDFGH-13239)

[AntoineSX]

Answers checklist.

• I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

v5.3-rc1

Operating System used.

Linux

How did you build your project?

VS Code IDE

If you are using Windows, please specify command line type.

None

What is the expected behavior?

Calling opendir, closedir, mkdir works in v5.2.2. I use std::filesystem to call std::filesystem::exists, std::filesystem::remove and std::filesystem::file_size with no issues.

What is the actual behavior?

With v5.3 you get a multiple definition error at link time with newlib's implementations of opendir, closedir, mkdir and so on.

Steps to reproduce.

1. Make a component with fatfs in the component's REQUIRES
2. call any of the vfs io functions.
3. build
   ...

Build or installation Logs.

/opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/bin/ld: /opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/lib/esp32s3/no-rtti/libstdc++.a(fs_ops.o):(.literal._ZNSt10filesystem12current_pathB5cxx11ERSt10error_code+0x8): warning: pathconf is not implemented and will always fail
/opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/bin/ld: /opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/lib/esp32s3/no-rtti/libstdc++.a(fs_ops.o):(.literal._ZNSt10filesystem11permissionsERKNS_7__cxx114pathENS_5permsENS_12perm_optionsERSt10error_code+0xc): warning: chmod is not implemented and will always fail
/opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/bin/ld: /opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/lib/esp32s3/no-rtti/libstdc++.a(fs_dir.o):(.literal._ZNKSt10filesystem7__cxx114_Dir7currentEv[_ZNKSt10filesystem7__cxx114_Dir7currentEv]+0x4): warning: dirfd is not implemented and will always fail
/opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/bin/ld: esp-idf/mbedtls/mbedtls/library/libmbedx509.a(x509_crt.c.obj):(.literal.mbedtls_x509_crt_parse_path+0x38): warning: readdir is not implemented and will always fail
/opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/bin/ld: /opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/lib/esp32s3/no-rtti/libstdc++.a(fs_ops.o):(.literal._ZNSt10filesystem12current_pathERKNS_7__cxx114pathERSt10error_code+0x0): warning: chdir is not implemented and will always fail
/opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/bin/ld: /opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/lib/esp32s3/no-rtti/libstdc++.a(fs_ops.o):(.literal._ZNSt10filesystem12current_pathB5cxx11ERSt10error_code+0x14): warning: getcwd is not implemented and will always fail
/opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/bin/ld: /opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/lib/esp32s3/no-rtti/libstdc++.a(fs_ops.o):(.literal._ZN12_GLOBAL__N_110create_dirERKNSt10filesystem7__cxx114pathENS0_5permsERSt10error_code+0x0): warning: mkdir is not implemented and will always fail
/opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/bin/ld: esp-idf/mbedtls/mbedtls/library/libmbedx509.a(x509_crt.c.obj):(.literal.mbedtls_x509_crt_parse_path+0x20): warning: closedir is not implemented and will always fail
/opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/bin/ld: esp-idf/mbedtls/mbedtls/library/libmbedx509.a(x509_crt.c.obj):(.literal.mbedtls_x509_crt_parse_path+0x1c): warning: opendir is not implemented and will always fail
/opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/bin/ld: /opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/lib/esp32s3/no-rtti/libnosys.a(dir.o): in function `opendir':
/builds/idf/crosstool-NG/.build/xtensa-esp-elf/src/newlib/libgloss/libnosys/dir.c:16: multiple definition of `opendir'; esp-idf/vfs/libvfs.a(vfs.c.obj):/opt/esp/idf/components/vfs/vfs.c:733: first defined here
/opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/bin/ld: /opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/lib/esp32s3/no-rtti/libnosys.a(dir.o): in function `closedir':
/builds/idf/crosstool-NG/.build/src/newlib-git-7cdcb0b0/newlib/libc/include/dirent.h:76: multiple definition of `closedir'; esp-idf/vfs/libvfs.a(vfs.c.obj):/opt/esp/idf/components/vfs/vfs.c:805: first defined here
/opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/bin/ld: /opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/lib/esp32s3/no-rtti/libnosys.a(dir.o): in function `mkdir':
/builds/idf/crosstool-NG/.build/xtensa-esp-elf/src/newlib/libgloss/libnosys/dir.c:30: multiple definition of `mkdir'; esp-idf/vfs/libvfs.a(vfs.c.obj):/opt/esp/idf/components/vfs/vfs.c:818: first defined here
/opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/bin/ld: /opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/lib/esp32s3/no-rtti/libnosys.a(dir.o): in function `getcwd':
/builds/idf/crosstool-NG/.build/xtensa-esp-elf/src/newlib/libgloss/libnosys/dir.c:37: multiple definition of `getcwd'; esp-idf/newlib/libnewlib.a(realpath.c.obj):/opt/esp/idf/components/newlib/realpath.c:111: first defined here
/opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/bin/ld: /opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/lib/esp32s3/no-rtti/libnosys.a(dir.o): in function `chdir':
/builds/idf/crosstool-NG/.build/xtensa-esp-elf/src/newlib/libgloss/libnosys/dir.c:42: multiple definition of `chdir'; esp-idf/newlib/libnewlib.a(realpath.c.obj):/opt/esp/idf/components/newlib/realpath.c:120: first defined here
/opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/bin/ld: /opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/lib/esp32s3/no-rtti/libnosys.a(dir.o): in function `readdir':
/builds/idf/crosstool-NG/.build/src/newlib-git-7cdcb0b0/newlib/libc/include/dirent.h:56: multiple definition of `readdir'; esp-idf/vfs/libvfs.a(vfs.c.obj):/opt/esp/idf/components/vfs/vfs.c:750: first defined here
/opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/bin/ld: /opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/lib/esp32s3/no-rtti/libnosys.a(dir.o): in function `telldir':
/builds/idf/crosstool-NG/.build/src/newlib-git-7cdcb0b0/newlib/libc/include/dirent.h:73: multiple definition of `telldir'; esp-idf/vfs/libvfs.a(vfs.c.obj):/opt/esp/idf/components/vfs/vfs.c:776: first defined here
/opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/bin/ld: /opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/lib/esp32s3/no-rtti/libnosys.a(dir.o): in function `seekdir':
/builds/idf/crosstool-NG/.build/xtensa-esp-elf/src/newlib/libgloss/libnosys/dir.c:73: multiple definition of `seekdir'; esp-idf/vfs/libvfs.a(vfs.c.obj):/opt/esp/idf/components/vfs/vfs.c:789: first defined here
/opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/bin/ld: /opt/esp/tools/xtensa-esp-elf/esp-13.2.0_20240530/xtensa-esp-elf/bin/../lib/gcc/xtensa-esp-elf/13.2.0/../../../../xtensa-esp-elf/lib/esp32s3/no-rtti/libnosys.a(dir.o): in function `rewinddir':
/builds/idf/crosstool-NG/.build/xtensa-esp-elf/src/newlib/libgloss/libnosys/dir.c:101: multiple definition of `rewinddir'; esp-idf/vfs/libvfs.a(vfs.c.obj):/opt/esp/idf/components/vfs/vfs.c:800: first defined here
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

More Information.

No response

[igrr]

I think the key point here is not the usage of fatfs component, but the usage of std::filesystem. I also got the same issue with the latest toolchain when trying std::filesystem.

We didn't have any test cases which used std::filesystem, so we didn't notice this issue when doing the last toolchain upgrade. We'll try to fix this soon. In the meantime, please try this patch which can help you temporarily bypass this issue: #13736 (comment)

[AntoineSX]

Thank you ! The patch works. What's the timeframe for having a fix on the release branch ? I just need a ballpark figure.

[AntoineSX]

Any update ?

[ivmarkov]

@igrr Sorry to bother. Do you have an ETA when commit 684d3e6 will be merged into the release/v5.3 branch?

The Rust esp-idf-* crates are currently stuck on version 5.2 because of this ^^^ issue and we can't support 5.3 with these, as we have precisely the same problem during CI.

For us - btw - the problem only manifests itself with the "duplicate symbol" errors and we don't have missing symbols during linkage:

getcwd

          /home/runner/work/esp-idf-hal/esp-idf-hal/.embuild/espressif/tools/riscv32-esp-elf/esp-13.2.0_20240530/riscv32-esp-elf/bin/../lib/gcc/riscv32-esp-elf/13.2.0/../../../../riscv32-esp-elf/bin/ld: /home/runner/work/esp-idf-hal/esp-idf-hal/.embuild/espressif/tools/riscv32-esp-elf/esp-13.2.0_20240530/riscv32-esp-elf/bin/../lib/gcc/riscv32-esp-elf/13.2.0/../../../../riscv32-esp-elf/lib/rv32imc_zicsr_zifencei/ilp32/no-rtti/libnosys.a(dir.o): in function `getcwd':
          /builds/idf/crosstool-NG/.build/riscv32-esp-elf/src/newlib/libgloss/libnosys/dir.c:39: multiple definition of `getcwd'; esp-idf/newlib/libnewlib.a(realpath.c.obj):/home/runner/work/esp-idf-hal/esp-idf-hal/.embuild/espressif/esp-idf/release-v5.3/components/newlib/realpath.c:111: first defined here

chdir

          /home/runner/work/esp-idf-hal/esp-idf-hal/.embuild/espressif/tools/riscv32-esp-elf/esp-13.2.0_20240530/riscv32-esp-elf/bin/../lib/gcc/riscv32-esp-elf/13.2.0/../../../../riscv32-esp-elf/bin/ld: /home/runner/work/esp-idf-hal/esp-idf-hal/.embuild/espressif/tools/riscv32-esp-elf/esp-13.2.0_20240530/riscv32-esp-elf/bin/../lib/gcc/riscv32-esp-elf/13.2.0/../../../../riscv32-esp-elf/lib/rv32imc_zicsr_zifencei/ilp32/no-rtti/libnosys.a(dir.o): in function `chdir':
          /builds/idf/crosstool-NG/.build/riscv32-esp-elf/src/newlib/libgloss/libnosys/dir.c:42: multiple definition of `chdir'; esp-idf/newlib/libnewlib.a(realpath.c.obj):/home/runner/work/esp-idf-hal/esp-idf-hal/.embuild/espressif/esp-idf/release-v5.3/components/newlib/realpath.c:120: first defined here

But I think anyway the "duplicate symbol" error is a side effect of at least one missing symbol, because otherwise libnosys.a would've not be linked-in (right?) which leads to the "duplicate symbols"?