Protected names are not sanitized correctly in Data Lake Sink / Influx sink

[bossenti]

Apache StreamPipes version

dev (current development state)

Affected StreamPipes components

Connect

What happened?

When reading the following CSV file with the FileStream adapter and persisting the created the data stream in the data storage, simple string values are lost.
Only string values that are explicitly quoted are kept, but this should not be necessary for a CSV file.

ts,testId,name,assetId
1000,1,Tom,“5“
2000,2,Albert,“1“

How to reproduce?

Expected behavior

No response

Additional technical information

No response

Are you willing to submit a PR?

None

[muyangye]

Hi @bossenti, I have looked into the issue. The event is "correct" ("correct" meaning key value pairs are the same with the file) in frontend, Kafka producer, and Kafka consumer. However, once the code hits Influx Store, the runtime name of some keys are modified. The root cause is that "name" key in the file stream is a keyword of Influx reserved keywords list and Streampipes sanitizes DataLake measure for those keys conflicting with Influx reserved keywords. If you change "name" to something else not in the list everything works.

To fix this, I propose to also sanitize event instead of only sanitizing DataLake measure. Let me know what do you think🙂!

[bossenti]

Hi @muyangye,

you are right, changing the column name makes it work.

As you said, sanitation should prevent naming conflicts and therefore rename the name column. But the expected outcome is then that the values are part of the DataLakeMeasure with the sanitized measure name, which doesn't seem to work here.

Where would you place the sanitation of the event? Sanitation is a specific requirement of the InfluxDb in this case, so I'm a bit hesitant to affect other usages of the event stream as well.

[muyangye]

Just published a PR! The scope of the sanitization is limited to InfluxDb.