[C++] Corrupt MCAP file if close is not called.

[timab25]

Is it possible to ensure that an mcap file is still readable if the program that was writing it failed, and didn't call close on the mcap writer?

[emersonknapp]

You should be able to use the MCAP CLI via mcap recover to fix the file. mcap doctor may also tell you more information about the issue

[timab25]

The output of those are "Invalid magic at start of file, found: []"

I am writing protobuf messages to the mcap file in C++.

[emersonknapp]

That's strange - the mcap file is written linearly and the very first bytes written should be the magic and the header. If your program exits prematurely, generally the effect you will see is a truncated file, meaning the footer/closing magic are missing, incomplete records, that type of problem.

What size is the output file? (wc -c my.mcap) - I'm wondering if it's totally empty at 0B.

[emersonknapp]

Note: the FileWriter uses std::file as its output, which has in-memory buffering up to a certain size (8192 default on my system). If your file doesn't get this large before the crash, then when the program crashes you will not see any data written out. That's my best hypothesis for what you're seeing

Here is a trivial example:

#include "mcap/mcap.hpp"

namespace mcap {

class UnbufferedFileWriter : public IWritable {
public:
  ~UnbufferedFileWriter() override
  {
    end();
  }

  Status open(std::string_view filename)
  {
    end();
    file_ = std::fopen(filename.data(), "wb");
    // NOTE: THIS IS THE ONE NEW LINE
    std::setbuf(file_, nullptr);
    if (!file_) {
      const auto msg = internal::StrCat("failed to open file \"", filename, "\" for writing");
      return Status(mcap::StatusCode::OpenFailed, msg);
    }
    return StatusCode::Success;
  }

  void handleWrite(const std::byte * data, uint64_t size) override
  {
    assert(file_);
    const size_t written = std::fwrite(data, 1, size, file_);
    (void)written;
    assert(written == size);
    size_ += size;
  }

  void end() override
  {
    if (file_) {
      std::fclose(file_);
      file_ = nullptr;
    }
    size_ = 0;
  }

  uint64_t size() const override
  {
    return size_;
  }

private:
  std::FILE * file_ = nullptr;
  uint64_t size_ = 0;
};

}
int main() {
  mcap::Status res;
  mcap::McapWriterOptions options{""};

  mcap::McapWriter writer1;
  res = writer1.open("buffered.mcap", options);

  mcap::UnbufferedFileWriter ufwriter{};
  res = ufwriter.open("unbuffered.mcap");
  mcap::McapWriter writer2;
  writer2.open(ufwriter, options);

  mcap::Metadata metadata;
  metadata.metadata = {{"foo", "bar"}};
  res = writer1.write(metadata);
  res = writer2.write(metadata);
  std::abort();
}

In the above, I've created a copy-paste of the internal FileWriter that has one additional line to disable buffering. The std::abort crashes the program uncleanly, not allowing any destructors to run.

$ g++ mcapbuffer.cpp -I mcap/cpp/mcap/include -D MCAP_IMPLEMENTATION -lzstd -llz4 -o mcapbuffer
$ ./mcapbuffer                                                                                                                                                                                                           3s 
[1]    1854247 IOT instruction (core dumped)  ./mcapbuffer
$ wc -c *.mcap
 0 buffered.mcap
69 unbuffered.mcap
69 total

So you can see the difference, that since buffered never hit the minimum write size nothing was written. max(BUFSIZ, chunk_size) is the most data you could lose on a crash, but if the total file size is below BUFSIZ then you will get a totally empty file like observed.

One conclusion here is that perhaps we should add a WriterOption to disable or change the size of output buffering.