Getting one garbage character from container.exec()

[jamesmurdza]

I've written two helper functions using Dockerode. The idea is to run a command synchronously and then return the results as a string.

It works well, except each chunk that is returned by stream.on('data') starts with a garbage character, like "X", "*", "!", etc. I can't figure out why.

This happens for multiple commands, such as "bash" and "cat".

Help is much appreciated!

My code below:

async function waitForStreamEnd(stream: NodeJS.ReadableStream): Promise<void> {
  return new Promise<void>((resolve, reject) => {
    try {
      stream.on('end', async () => {
        resolve();
      });
    } catch (err) {
      reject(err);
    }
  });
}

async function runCommandInContainer(container: Docker.Container, command: string[]): Promise<string> {
  const exec = await container.exec({
    Cmd: command,
    AttachStdout: true,
    AttachStderr: true,
  });
  const stream = await exec.start({ hijack: true, stdin: true });
  let output = "";
  stream.on('data', (data) => {
    output += data;
  });
  await waitForStreamEnd(stream);
  return output;
}

[Akimyou]

const readStream = async (
  stream: stream.Duplex
): Promise<{ stdin: string; stdout: string; stderr: string }> => {
  let stdin = '';
  let stdout = '';
  let stderr = '';

  return new Promise<{ stdin: string; stdout: string; stderr: string }>((resolve, reject) => {
    let buffer = Buffer.alloc(0); // Accumulate unprocessed data

    stream.on('data', (chunk: Buffer) => {
      buffer = Buffer.concat([buffer, chunk]);

      while (buffer.length >= 8) {
        // Parse the header
        const header = buffer.slice(0, 8);
        const streamType = header[0]; // First byte indicates the stream type (0, 1, or 2)
        const size = header.readUInt32BE(4); // Last 4 bytes indicate the payload size (big-endian)

        // Check if the full frame is available
        if (buffer.length < 8 + size) {
          break; // Wait for more data if the payload is incomplete
        }

        // Extract the payload
        const payload = buffer.slice(8, 8 + size).toString('utf-8');

        // Append the payload to the appropriate stream
        if (streamType === 0) {
          stdin += payload;
        } else if (streamType === 1) {
          stdout += payload;
        } else if (streamType === 2) {
          stderr += payload;
        }

        // Remove the processed frame from the buffer
        buffer = buffer.slice(8 + size);
      }
    });

    stream.on('end', () => {
      resolve({ stdin, stdout, stderr });
    });

    stream.on('error', (err) => {
      reject(err);
    });
  });
};

May be this function can help you.

[jamesmurdza]

Interesting, I will try this. Any explanation why this would work?

[Fhwang0926]

add more comment
i also has it issue but i guess it is docker' api problem, is not?

for example, exec with ping

i just console.log of exec result

[jamesmurdza]

@Fhwang0926 Were you able to figure out any solution?

[ewanmellor]

When you do not use the tty flag, Docker sends data with the first four bytes set to tell you whether the data is from stdout or stderr, and its length. This is the "garbage" that you're seeing at the start of the line.

See "Stream format" within https://docs.docker.com/reference/api/engine/version/v1.43/#tag/Container/operation/ContainerAttach

[Akimyou]

const readStream = async (
  stream: stream.Duplex
): Promise<{ stdin: string; stdout: string; stderr: string }> => {
  let stdin = '';
  let stdout = '';
  let stderr = '';

  return new Promise<{ stdin: string; stdout: string; stderr: string }>((resolve, reject) => {
    let buffer = Buffer.alloc(0); // Accumulate unprocessed data

    stream.on('data', (chunk: Buffer) => {
      buffer = Buffer.concat([buffer, chunk]);

      while (buffer.length >= 8) {
        // Parse the header
        const header = buffer.slice(0, 8);
        const streamType = header[0]; // First byte indicates the stream type (0, 1, or 2)
        const size = header.readUInt32BE(4); // Last 4 bytes indicate the payload size (big-endian)

        // Check if the full frame is available
        if (buffer.length < 8 + size) {
          break; // Wait for more data if the payload is incomplete
        }

        // Extract the payload
        const payload = buffer.slice(8, 8 + size).toString('utf-8');

        // Append the payload to the appropriate stream
        if (streamType === 0) {
          stdin += payload;
        } else if (streamType === 1) {
          stdout += payload;
        } else if (streamType === 2) {
          stderr += payload;
        }

        // Remove the processed frame from the buffer
        buffer = buffer.slice(8 + size);
      }
    });

    stream.on('end', () => {
      resolve({ stdin, stdout, stderr });
    });

    stream.on('error', (err) => {
      reject(err);
    });
  });
};

May be this function can help you.

Thanks for @ewanmellor , i write the new func to read data, it work now. @jamesmurdza

[YuHuanTin]

const readStream = async (
  stream: stream.Duplex
): Promise<{ stdin: string; stdout: string; stderr: string }> => {
  let stdin = '';
  let stdout = '';
  let stderr = '';

  return new Promise<{ stdin: string; stdout: string; stderr: string }>((resolve, reject) => {
    let buffer = Buffer.alloc(0); // Accumulate unprocessed data

    stream.on('data', (chunk: Buffer) => {
      buffer = Buffer.concat([buffer, chunk]);

      while (buffer.length >= 8) {
        // Parse the header
        const header = buffer.slice(0, 8);
        const streamType = header[0]; // First byte indicates the stream type (0, 1, or 2)
        const size = header.readUInt32BE(4); // Last 4 bytes indicate the payload size (big-endian)

        // Check if the full frame is available
        if (buffer.length < 8 + size) {
          break; // Wait for more data if the payload is incomplete
        }

        // Extract the payload
        const payload = buffer.slice(8, 8 + size).toString('utf-8');

        // Append the payload to the appropriate stream
        if (streamType === 0) {
          stdin += payload;
        } else if (streamType === 1) {
          stdout += payload;
        } else if (streamType === 2) {
          stderr += payload;
        }

        // Remove the processed frame from the buffer
        buffer = buffer.slice(8 + size);
      }
    });

    stream.on('end', () => {
      resolve({ stdin, stdout, stderr });
    });

    stream.on('error', (err) => {
      reject(err);
    });
  });
};

May be this function can help you.

This is so useful that I feel the library's 'modem.demuxStream' is just a toy. When I really want to get the output content instead of outputting it directly to stdout or stderr (there is no function to separate them), I can only parse it manually. Of course, I am not sure if there is a problem with my writing. As a beginner, I find it completely difficult to understand the readme.
Thank you very much for your code!

[jamesmurdza]

Wow, thanks for finally solving this!