workaround esp-idf version detection #39

madmo · 2024-02-20T17:07:34Z

If version.txt is missing some part of the cmake toolchain tries to read refs/heads/fetchgit in the idf repo which failed.

This is not a perfect solution, but works nicely when using cargo, and fixes some of the issues in #5

jonahbron · 2024-03-06T11:04:44Z

pkgs/esp-idf/default.nix

@@ -1,5 +1,5 @@
 { rev ? "v5.1.2"
-, sha256 ? "sha256-uEf3/3NPH+E39VgQ02AbxTG7nmG5bQlhwk/WcTeAUfg="
+, sha256 ? "sha256-qjZ438LUQLzyQDKkeo4WMO0jxfkFh6dReuLq/6rftpw="


Is this change intentional? I'm trying to test using your fork but this appears to be breaking it.

Hmm, it was working when I made the change. If I rebuild now it also stopped working for me. I will update the hash, and we will see if this fixes the issue for now.

madmo · 2024-03-06T16:21:16Z

Ok, back to the drawing board. The issue is that fetchgit with leaveDotGit is not always reproducible: NixOS/nixpkgs#8567.

jonahbron · 2024-03-07T07:44:04Z

pkgs/esp-idf/default.nix

+    git init .
+    git config user.email "nixbld@localhost"
+    git config user.name "nixbld"
+    git commit --date="1970-01-01 00:00:00" --allow-empty -m "make idf happy"


This seems quite clever. Is it working on your end?

I'm getting this error now:

To add an exception for this directory, call: git config --global --add safe.directory /nix/store/mfmmywqch5m4bshhmh5lb01np9yjdlgk-esp-idf-v5.1.2```

I tried fiddling with this for a few hours but didn't crack it. You can see the things I tried here:

https://github.com/jonahbron/nixpkgs-esp-dev/commits/madmo/?author=jonahbron

Hoping you have more luck!

jonahbron · 2024-03-08T08:41:17Z

Seeing other comments around this affecting non-ESP related stuff, might have some useful resources

NixOS/nixpkgs#169193

madmo · 2024-03-22T08:37:28Z

For me it is working now. Had a issue with the installPhase script writing to the source dir, which in turn caused some hash issue.

jonahbron · 2024-03-23T02:18:17Z

I can now get into a shell with nix develop with your fork. However, I'm still having trouble pulling it into another flake for building.

{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
    nixpkgs-esp-dev.url = "github:madmo/nixpkgs-esp-dev";
  };

  outputs = { self, nixpkgs, nixpkgs-esp-dev }: let
    forAllSystems = function:
      nixpkgs.lib.genAttrs [
        "x86_64-linux"
      ] (system: function (import nixpkgs {inherit system;}));
  in {
    packages = forAllSystems (pkgs: let
      berkeley-db-1_xx = pkgs.fetchFromGitHub {
        owner = "pfalcon";
        repo = "berkeley-db-1.xx";
        rev = "35aaec4418ad78628a3b935885dd189d41ce779b";
        sha256 = "sha256-XItxmpXXPgv11LcnL7dty6uq1JctGokHCU8UGG9ic04=";
      };
      micropython-src = pkgs.fetchFromGitHub {
        owner = "micropython";
        repo = "micropython";
        rev = "v1.22.2";
        sha256 = "sha256-67CAR34VrMOzvNkukDeGRnUfoOLO66R37wsrRHjpp5E=";
      };
    in {
      esp32genericc3 = pkgs.stdenv.mkDerivation {
        name = "esp32genericc3";
        nativeBuildInputs = [
          pkgs.git
          nixpkgs-esp-dev.packages.${pkgs.system}.esp-idf-esp32c3
        ];

        src = micropython-src;
        phases = ["unpackPhase" "patchPhase" "buildPhase"];

        patchPhase = ''
          rmdir lib/berkeley-db-1.xx
          ln -s ${berkeley-db-1_xx} lib/berkeley-db-1.xx
        '';
        
        buildPhase = ''
          make V=1 -C mpy-cross
          make V=1 -C ports/esp32 BOARD=ESP32_GENERIC_C3
        '';
      };
    });
  };
}

This is incomplete but I can't even get past the buildPhase before it hits the Git error

...
make: Leaving directory '/build/source/mpy-cross'
make: Entering directory '/build/source/ports/esp32'
idf.py -D MICROPY_BOARD=ESP32_GENERIC_C3 -D MICROPY_BOARD_DIR=/build/source/ports/esp32/boards/ESP32_GENERIC_C3  -B build-ESP32_GENERIC_C3 build || (echo -e "See \033[1;31mhttps://github.com/micropython/micropython/wiki/Build-Troubleshooting\033[0>
Executing action: all (aliases: build)
Running cmake in directory /build/source/ports/esp32/build-ESP32_GENERIC_C3
Executing "cmake -G Ninja -DPYTHON_DEPS_CHECKED=1 -DPYTHON=/nix/store/41253d4jjd23jjdaiidgnb8q2qambh1a-python3-3.11.7-env/bin/python3.11 -DESP_PLATFORM=1 -DMICROPY_BOARD=ESP32_GENERIC_C3 -DMICROPY_BOARD_DIR=/build/source/ports/esp32/boards/ESP32_G>
-- Found Git: /nix/store/bd0dcc9qzpfz5xfz7fm6cwf7q64cyfhi-git-2.44.0/bin/git (found version "2.44.0")
-- git rev-parse returned 'fatal: detected dubious ownership in repository at '/nix/store/kgczq8mgnydbsj0almzx8kqs5rzjd9av-esp-idf-v5.1.2'

Any idea why the issue is persisting when run in a derivation?

madmo · 2024-03-23T06:39:18Z

@jonahbron This is related to the safe.directory setting of git. I completely disabled this security feature on my machine.

jonahbron · 2024-03-23T06:57:49Z

After hours of fiddling I got it to almost work. But now, it looks like the compiler is trying to download files over the network!

Dependencies lock doesn't exist, solving dependencies.
CMake Error at /nix/store/kgczq8mgnydbsj0almzx8kqs5rzjd9av-esp-idf-v5.1.2/tools/cmake/build.cmake:540 (message):
  ERROR: Solver failed processing dependency "espressif/mdns" from the
  manifest file "/build/source/ports/esp32/main_esp32c3/idf_component.yml".

  HTTPSConnectionPool(host='components-file.espressif.com', port=443): Max
  retries exceeded with url: /components/espressif/mdns.json (Caused by
  NameResolutionError("<urllib3.connection.HTTPSConnection object at
  0x7ffff5c2b190>: Failed to resolve 'components-file.espressif.com' ([Errno
  -3] Temporary failure in name resolution)"))

Call Stack (most recent call first):
  /nix/store/kgczq8mgnydbsj0almzx8kqs5rzjd9av-esp-idf-v5.1.2/tools/cmake/project.cmake:547 (idf_build_process)
  CMakeLists.txt:53 (project)


-- Configuring incomplete, errors occurred!
cmake failed with exit code 1, output of the command is in the /build/source/ports/esp32/build-ESP32_GENERIC_C3/log/idf_py_stderr_output_1601 and /build/source/ports/esp32/build-ESP32_GENERIC_C3/log/idf_py>
See https://github.com/micropython/micropython/wiki/Build-Troubleshooting
make: *** [Makefile:66: all] Error 1
make: Leaving directory '/build/source/ports/esp32'
CMake Error at /nix/store/kgczq8mgnydbsj0almzx8kqs5rzjd9av-esp-idf-v5.1.2/tools/cmake/build.cmake:540 (message):
  ERROR: Solver failed processing dependency "espressif/mdns" from the
  manifest file "/build/source/ports/esp32/main_esp32c3/idf_component.yml".

  HTTPSConnectionPool(host='components-file.espressif.com', port=443): Max
  retries exceeded with url: /components/espressif/mdns.json (Caused by
  NameResolutionError("<urllib3.connection.HTTPSConnection object at
  0x7ffff5c2b190>: Failed to resolve 'components-file.espressif.com' ([Errno
  -3] Temporary failure in name resolution)"))

Call Stack (most recent call first):
  /nix/store/kgczq8mgnydbsj0almzx8kqs5rzjd9av-esp-idf-v5.1.2/tools/cmake/project.cmake:547 (idf_build_process)
  CMakeLists.txt:53 (project)

This might be make building purely in a flake a no-go 😢

jonahbron · 2024-03-23T10:03:18Z

It's not my favorite, but the solution (plus a bunch of other little fixes) was to make it into a known-output derivation; adding a outputHash prop etc. so that the derivation could make the network calls to Espressif for pulling in dependency components.

Here is my flake for anyone interested in how it works.

https://git.sr.ht/~jonahbron/doorbell/tree/963d4cc33cbba10d555237a4b4e2ef8df6f24776/item/mpy-flake/flake.nix

Take specific note of the workaround I applied to the Git problem, specifically marking only the correct directory safe, and using the GIT_CONFIG_GLOBAL env var to apply that config.

I'd love to see this PR submitted, it's helping me immensely!

jonahbron · 2024-03-24T07:56:04Z

Settled onto what I think is quite a nice Nix expression for this purpose.

https://git.sr.ht/~jonahbron/doorbell/tree/83e24a1477a65b0d3902d2492baab623cfc7baed/item/firmware.nix

Hopefully this can be a useful reference so nobody has to burn as much time figuring this out as I have 😄

jonahbron · 2024-03-24T08:04:17Z

Actually, because I necessarily have to have the Git fix in my project too, this PR has no efffect. Removing it my build still works.

https://git.sr.ht/~jonahbron/doorbell/tree/701ca6b16861b7183e72719be843a34308d2cc5b/item/flake.nix

@madmo At this point the issue might only be coming down to the "dubious ownership" issue, which I've solved with this code:

https://git.sr.ht/~jonahbron/doorbell/tree/701ca6b16861b7183e72719be843a34308d2cc5b/item/firmware.nix#L133

If version.txt is missing some part of the cmake toolchain tries to read refs/heads/fetchgit in the idf repo which failed. This is not a perfect solution, but works nicely when using cargo.

jonahbron reviewed Mar 6, 2024

View reviewed changes

madmo force-pushed the master branch 2 times, most recently from aa89fb4 to b96f3c2 Compare March 6, 2024 15:32

madmo marked this pull request as draft March 6, 2024 16:19

madmo force-pushed the master branch from b96f3c2 to dc25355 Compare March 6, 2024 16:21

jonahbron reviewed Mar 7, 2024

View reviewed changes

madmo force-pushed the master branch from dc25355 to 129e53e Compare March 7, 2024 10:14

jonahbron mentioned this pull request Mar 10, 2024

Incomplete Git Repository Detection (IDFGH-12856) espressif/esp-idf#13345

Closed

3 tasks

madmo force-pushed the master branch from 129e53e to 45ca892 Compare March 22, 2024 08:35

madmo marked this pull request as ready for review March 22, 2024 08:37

madmo force-pushed the master branch from 45ca892 to 4e8936e Compare March 22, 2024 09:51

workaround esp-idf version detection

970f693

If version.txt is missing some part of the cmake toolchain tries to read refs/heads/fetchgit in the idf repo which failed. This is not a perfect solution, but works nicely when using cargo.

madmo force-pushed the master branch from 4e8936e to 970f693 Compare April 29, 2024 07:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

workaround esp-idf version detection #39

workaround esp-idf version detection #39

madmo commented Feb 20, 2024

jonahbron Mar 6, 2024

madmo Mar 6, 2024

madmo commented Mar 6, 2024

jonahbron Mar 7, 2024

jonahbron Mar 7, 2024

jonahbron Mar 7, 2024

jonahbron commented Mar 8, 2024

madmo commented Mar 22, 2024

jonahbron commented Mar 23, 2024 •

edited

Loading

madmo commented Mar 23, 2024

jonahbron commented Mar 23, 2024

jonahbron commented Mar 23, 2024 •

edited

Loading

jonahbron commented Mar 24, 2024

jonahbron commented Mar 24, 2024

workaround esp-idf version detection #39

Are you sure you want to change the base?

workaround esp-idf version detection #39

Conversation

madmo commented Feb 20, 2024

jonahbron Mar 6, 2024

Choose a reason for hiding this comment

madmo Mar 6, 2024

Choose a reason for hiding this comment

madmo commented Mar 6, 2024

jonahbron Mar 7, 2024

Choose a reason for hiding this comment

jonahbron Mar 7, 2024

Choose a reason for hiding this comment

jonahbron Mar 7, 2024

Choose a reason for hiding this comment

jonahbron commented Mar 8, 2024

madmo commented Mar 22, 2024

jonahbron commented Mar 23, 2024 • edited Loading

madmo commented Mar 23, 2024

jonahbron commented Mar 23, 2024

jonahbron commented Mar 23, 2024 • edited Loading

jonahbron commented Mar 24, 2024

jonahbron commented Mar 24, 2024

jonahbron commented Mar 23, 2024 •

edited

Loading

jonahbron commented Mar 23, 2024 •

edited

Loading