Bug: sometimes an _id field gets removed from a record payload

[oykmnk]

Bug description

When using the MongoDB Source connector and sending a payload containing an _id field, it is sometimes removed from the record payload but not from the record key. This issue occurs when there are more than two destinations, which makes it a peculiar bug.

The Snapshot iterator has the following code:

return sdk.Util.Source.NewRecordSnapshot(
	sdkPosition, metadata,
	sdk.StructuredData{idFieldName: element[idFieldName]}, sdk.StructuredData(element),
), nil

Here, the map called "element" is used to construct a record key (sdk.StructuredData{idFieldName: element[idFieldName]}) and a payload (sdk.StructuredData(element)). The map is not modified afterward, so it's unclear why the payload sometimes doesn't contain the _id field while the record key always does.

Steps to reproduce:

1. Add data to a MongoDB collection
2. Create a pipeline with a MongoDB Source connector for the collection, and File, Postgres, and Materialize Destination connectors
3. Start the pipeline
4. Check the transmitted payload (may need to repeat the process as the bug is not stable)

Version

Conduit v0.5.2, MongoDB connector 4736288

[alarbada]

Hello @oykmnk, I know it's been a while, I've got a few questions before delving deeper into this:

1. Do you remember with what kind of schema did this bug happen?
2. Do you remember the mongodb connection parameters?
3. Do you remember when did the bug happen? Only when snapshotting, or also when the connector was on cdc mode?
4. By any chance, do you remember what pipeline config file were you using.

I'm asking because I tried to reproduce the bug with a simple pipeline from mongodb to file and I couldn't reproduce the bug. It also looks like this happened in an older conduit version, we are now on v0.12.2.