Node 20.3.0 images give error /usr/bin/env: 'node': Text file busy

[culli]

Environment

• Platform: macos 13.4 M1
• Docker Version: 24.0.2
• Node.js Version: 20.3.0
• Image Tag: node:20-alpine3.17

Expected Behavior

yarn tsc should run correctly

Current Behavior

 > [service 5/5] RUN yarn build:
#0 0.298 yarn run v1.22.19
#0 0.314 $ tsc
#0 0.325 /usr/bin/env: 'node': Text file busy
#0 0.334 error Command failed with exit code 126.

Possible Solution

downgrade to node:20-alpine3.16 (or node:20.2.0-bullseye-slim)

Steps to Reproduce

dockerfile:

FROM node:20-alpine3.17 as install
WORKDIR /service
COPY ./service ./
RUN yarn install --frozen-lockfile
RUN yarn build

yarn build creates a directory, compiles some protobuf, then execs tsc (bin from typescript dependency)

Additional Information

Things work correctly on 3.16, but give the text file busy error with 3.17 and 3.18. Also tried bullseye-slim and node:20.3.0-bullseye-slim gives the same error, but 20.2.0 works correctly.

Maybe this is an architecture mismatch or file permissions problem?

[culli]

I can provide a more complete repro node project if that is needed.

[danlordan]

+1, getting an issue on this as well, had to switch to node:18 to get builds to run, instead of node:latest

[Eeemil]

A colleague of mine running a Mac M1 has the same issue, but it works for me on PC (Ubuntu). Might be an issue for Mac M1 only.

[egl74]

Can (temporarily) be fixed by using OrbStack instead of Docker for Mac

[mfn]

Same here with node:20.3-slim running in Docker Desktop 4.20.1 on macOS 13.4 Apple Silicon.

Reverting to node:20.2-slim worked.

Only affected shell/bash scripts trying to start node (e.g. yarn start in a NextJS project), when I ran it manually it worked.

Only saw this on local dev and not pro container (but prod != ARM, so… 🤷🏼 )

[yann1731]

A colleague of mine running a Mac M1 has the same issue, but it works for me on PC (Ubuntu). Might be an issue for Mac M1 only.

Can confirm I got this issue on M1 mac and Pop!_os

[bjonnh-work]

This is breaking in docker on x86 as well as M1 for us, and we get the same error on containers that start yarn.

[book000]

This issue was also observed in the following environments:

• Windows 10 22H2 (Build 19045.2965)
• CPU: AMD Ryzen 9 5900X (x86_64)
• Docker Desktop 4.19.0 (106363)
• Image tag: node:20-alpine
• Package manager: Yarn

It is not limited to a specific Node.js project, but occurs in all projects that use node:20-alpine. I have confirmed that replacing it with node:18-alpine works.

---

However, this problem does not occur in the following environments:

• Ubuntu 22.04.2 LTS
• CPU: Intel Core i7-6700 (x86_64)
• Docker 24.0.2

[jerlendds]

This can be recreated in my project from commit b76bc9b with the frontend Dockerfile but I have since pushed this version: node:18-alpine.

Seemingly related: Docker forums
My related issue with more details can be found here: jerlendds/osintbuddy#49

[evanw]

FYI there is another case of people hitting this problem here: evanw/esbuild#3156.

[JoostK]

Sharing an additional data point here as well (from evanw/esbuild#3156 (comment)): the issue only reproduces for me using Docker Engine 24, whereas the image works fine using Docker Engine 20. Tested on M1 Max.

[GabeDuarteM]

This was also happening with me and all my coworkers that use mac for development.

The "solution" we found for now was to use OrbStack instead, it's a drop-in replacement for docker desktop, and seems to solve this problem. While not ideal, it did unblock us while we wait for the official solution.

[ahamlinman]

I started looking more closely at the libuv v1.45.0 upgrade after @JoostK mentioned it in the esbuild thread, particularly some of the changes around io_uring support. There is an environment variable (undocumented and unstable, intended for debugging purposes only) to disable libuv's use of io_uring, and with UV_USE_IO_URING=0 set in my container's environment I can no longer reproduce the issue during esbuild installs with any of the Node.js v20.3.0 image variants I've tested (M1 Max, Docker Desktop v4.20.1, Docker Engine v24.0.2).

I also have an Arch Linux ARM virtual machine on my Mac separate from Docker Desktop, and tried building and installing libuv v1.45.0 there. In that case, I couldn't reproduce the same issue during an esbuild install, even though I could confirm with strace that Node.js was dispatching io_uring operations. That makes me wonder if the kernel version is partly at play (6.3.7 on my Arch VM vs. 5.15.49-linuxkit-pr in the Docker VM). (edit: I set up a separate system with a separate build of Linux 5.15.49, and couldn't reproduce either on the system or with the Docker images. Looking at kernel versions alone may not be a useful lead.)

Given @evanw's note in the original thread about possible changes to fs.renameSync, it might be notable that rename is apparently one of the operations now backed by io_uring, per libuv/libuv#4012 (my guess is that the *Sync functions in Node.js still use libuv under the hood, but I'm not certain). If that really is where this comes from, the discussion may ultimately belong with libuv, or even with Node.js if it's somehow particular to Node's usage of libuv. (edit: This entire part is incorrect, please see below. Thank you @santigimeno for the correction, I apologize for the error.)

[harvey-appleton]

Experiencing same issue here, but downgrading to node:20-alpine3.16 seems to have fixed it. Seems the issue reappears for us in 20-alpine3.17 (which includes Node 20.3.0) - perhaps (guessing here) it's related to a line in the changelog for Node v20.3.0 which says:

[bfcb3d1d9a] - deps: upgrade to libuv 1.45.0, including significant performance improvements to file system operations on Linux (Santiago Gimeno) nodejs/node#48078

(which does backup what @ahamlinman mentioned above)

[mgattei]

Same here for a pipeline that run semantic-release. Gitlab - node:alpine

- yarn global add handpick
- handpick --target=devDependencies --manager=yarn
- yarn semantic-release

Error:

$ yarn global add handpick
yarn global v1.22.19
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Installed "[email protected]" with binaries:
      - handpick
Done in 0.96s.
$ handpick --target=devDependencies --manager=yarn
Picking DIRTY devDependencies via YARN
Done 854 packages in 71.30 seconds
$ yarn semantic-release
yarn run v1.22.19
$ /builds/project/node_modules/.bin/semantic-release
env: can't execute 'node': Text file busy
error Command failed with exit code 126.

By retrying several times, it seems to work after a while.... I solved the problem temporarily by downgrading to an older node image.

[jkuchar]

Not sure what causes this, but this is what I have found:

• problem happens on version 20.3.0 on ubuntu 22.04 (manually installed), Debian 10, 11, 12 (node images from docker hub)
• problem does NOT happen with node 20.2.0 or node 19

So I would guess this is some bug in node itself?

[santigimeno]

Given @evanw's note in the original thread about possible changes to fs.renameSync, it might be notable that rename is apparently one of the operations now backed by io_uring, per libuv/libuv#4012 (my guess is that the *Sync functions in Node.js still use libuv under the hood, but I'm not certain). If that really is where this comes from, the discussion may ultimately belong with libuv, or even with Node.js if it's somehow particular to Node's usage of libuv.

FWIW, that specific libuv commit is not yet in any release so it didn't get to Node.js. Also, the fs sync operations aren't io_uring backed.

[santigimeno]

• problem happens on version 20.3.0 on ubuntu 22.04 (manually installed),

@jkuchar can you elaborate exactly what steps to follow? I'd like to have a reproducer in that environment. Thanks!

[jkuchar]

This was my test scenario environment:

FROM ubuntu:22.04

# NodeJS & Chromium for tests
RUN curl -sL https://deb.nodesource.com/setup_20.x | bash -
RUN apt install -y nodejs

[danopia]

I'm seeing this within a CircleCI linux environment when attempting to bump node from 20.2.0-alpine3.17 to 20.3.1-alpine3.17.

This is the docker server info from CircleCI:

Server Engine Details:
  Version:          20.10.18
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.6
  Git commit:       e42327a
  Built:            2022-09-08T23:09:30.000000000+00:00
  OS/Arch:          linux/amd64
  Experimental:     false

The error happened during `docker build`, when yarn tried running `react-scripts build`:

Step 15/23 : RUN yarn build && yarn test -- --watchAll=false
 ---> Running in e344e5cf9846
yarn run v1.22.19
$ react-scripts build
env: can't execute 'node': Text file busy
error Command failed with exit code 126.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
The command '/bin/sh -c yarn build && yarn test -- --watchAll=false' returned a non-zero code: 126

[FernandoKGA]

Not sure what causes this, but this is what I have found:

* problem happens on version 20.3.0 on ubuntu 22.04 (manually installed), Debian 10, 11, 12 (node images from docker hub)

* problem does NOT happen with node 20.2.0 or node 19

So I would guess this is some bug in node itself?

I could not reproduce in my system which is a Ubuntu 22.04 that was upgraded automatically from the Ubuntu 20.04.

I notice a difference between my installed packages and my colleagues packages:

 @ docker-buildx-plugin : 0.10.5-1~ubuntu.20.04~focal -> 0.10.5-1~ubuntu.22.04~jammy
 @ docker-ce : 5:24.0.2-1~ubuntu.20.04~focal -> 5:24.0.2-1~ubuntu.22.04~jammy
 @ docker-ce-cli : 5:24.0.2-1~ubuntu.20.04~focal -> 5:24.0.2-1~ubuntu.22.04~jammy
 @ docker-ce-rootless-extras : 5:24.0.2-1~ubuntu.20.04~focal -> 5:24.0.2-1~ubuntu.22.04~jammy
 @ docker-compose-plugin : 2.18.1-1~ubuntu.20.04~focal -> 2.18.1-1~ubuntu.22.04~jammy

My Docker packages are 20.04~focal, although I have the same version as him, the OS package is different.

We have the same versions for Docker: 24.0.2, build cb74dfc and the uname -a show pretty much the same kernel and configs: 5.19.0-45-generic #46~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Jun 7 15:06:04 UTC 20 x86_64 x86_64 x86_64 GNU/Linux.

The docker compose was different, I was on v2.2.3 and him at v2.15.1, I upgraded to the latest v2.19.0, but still couldn't reproduce.

We will try to see if the environment variable that @ahamlinman mentioned (UV_USE_IO_URING=0) can allow him to build the image. Just read @ahamlinman comment again, this won't work anyway by now.

The production builds at our company didn't fail with the upgrade to node:20-alpine, but it is using another version of Docker to build the images.

[malix0]

I just add my experience. I had an old Docusaurus image built FROM node:20-alpine3.17 that worked without problem, the node version here was node 20.2.0 I had to rebuild this image than I switch it to node:20-alpine so I start getting this error because node was updated to latest 20.10.0 version and searching on internet I reached this issue. I solved the error switching back to node 20.2.0 (FROM node:20.2.0-alpine3.17)

Edit: excuse me, I just had no time to read all the thread, I read it now and maybe this post is not so useful, but I want to add that my host is Ubuntu 22.04.1 with kernel 5.19.0-32, so as wrote by @santigimeno maybe this kernel version will never receive fix that can solve this, so at the moment my solution is to use node 20.2.0

[kode54]

Experiencing this on Arch Linux (x86_64) as of updating to kernel 6.9.1. Failing to build AUR thelounge package with these Text file busy errors. nodejs 22.0.0-1 from Arch repos.

Edit: Dang, I thought this was the nodejs repo, my bad.

[MartinX3]

@kode54 Thank you for the hint.
I also updated to kernel 6.9.1 and if I run husky via pnpm it results into the 126 bad interpreter: cannot execute or modify file (busy) error.
Starting husky directly from cmd doesn't make this error.

So it must be caused by the 6.9.x kernel.

[luisbc92]

Just adding another datapoint, node fails in various different places with bad interpreter: Text file busy on Arch Linux 6.9.1 kernel.

Switching back to 6.8.x (lts kernel) fixes it.

[thanhquang1988]

Just adding another datapoint, node fails in various different places with bad interpreter: Text file busy on Arch Linux 6.9.1 kernel.

Switching back to 6.8.x (lts kernel) fixes it.

I have same issue with Arch 6.9, but after upgrade nodejs from 20.9 => 20.13 problem fixed also

[hoaithing]

I have same issue with Arch 6.9, but after upgrade nodejs from 20.9 => 20.13 problem fixed also

I have upgraded to 20.13 but the issue still be remaining.

[remisiki]

Just adding another datapoint, node fails in various different places with bad interpreter: Text file busy on Arch Linux 6.9.1 kernel.
Switching back to 6.8.x (lts kernel) fixes it.

I have same issue with Arch 6.9, but after upgrade nodejs from 20.9 => 20.13 problem fixed also

same issue here on arch 6.9.1 and node 22.2.0. npm works without errors and only yarn fails.

[FLX-0x00]

Cant get this working with arch linux 6.9.1-arch. tried nodejs-lts-hydrogen 18.20, nodejs-lts-iron 20.13 or nodejs 22.2

[is0ly]

yarn set version stable
yarn install

from yarn site - fixed this wired behavior without switching kernel version

[felixonmars]

Cant get this working with arch linux 6.9.1-arch. tried nodejs-lts-hydrogen 18.20, nodejs-lts-iron 20.13 or nodejs 22.2

Please try Arch Linux's new kernel 6.9.2-1 and see if it makes a difference.

[FLX-0x00]

Cant get this working with arch linux 6.9.1-arch. tried nodejs-lts-hydrogen 18.20, nodejs-lts-iron 20.13 or nodejs 22.2

Please try Arch Linux's new kernel 6.9.2-1 and see if it makes a difference.

Still the same error even with new kernel after reboot. Switched to the linux-lts kernel.

[MartinX3]

At my side it got fixed with the upgrade from pnpm 9.1.2 to 9.1.3.
And I also use the 6.9.2 kernel now.

I don't know if the fix is in any of these commits or it is just coincidence:
pnpm/pnpm@v9.1.2...v9.1.3

[zguig52]

Workaround to avoid downgrading the kernel:
UV_USE_IO_URING=0 your_command_using_node

Ex: UV_USE_IO_URING=0 yarn dev

[ZulluBalti]

At my side it got fixed with the upgrade from pnpm 9.1.2 to 9.1.3. And I also use the 6.9.2 kernel now.

I don't know if the fix is in any of these commits or it is just coincidence: pnpm/[email protected]

I think it's a coincident as I"m also using pnpm version 9.1.3 with kernel 6.9.2-arch1-1 and still getting this error

[ZulluBalti]

Cant get this working with arch linux 6.9.1-arch. tried nodejs-lts-hydrogen 18.20, nodejs-lts-iron 20.13 or nodejs 22.2

Please try Arch Linux's new kernel 6.9.2-1 and see if it makes a difference.

Nope, it doesn't make any difference still getting this error with 6.9.2-1

[MartinX3]

@ZulluBalti woopsi.
You're right.
After a computer restart the issue reappeared at my site as well.

[skabber]

Same error /usr/bin/env: ‘node’: Text file busy using nixos kernel 6.9.2. Backing out to 6.8.11 fixes the issue.

[mzndr]

For anyone wondering on how to downgrade your arch kernel, you can downgrade to previous installations (of any package) from chache:
pacman -U /var/cache/pacman/pkg/linux-<your-version>.pkg.tar.zst

[skabber]

Here is the issue tracked on nodejs/node nodejs/node#53051