cdl_runner::db_client: insert error

[latifatbraimah]

The biggest bottlenecks of the loader are really:

• The fact we have to iterate through every record in order to determine if we need to delete a specific value.
• The fact we don't batch together DELETEs/INSERTs at all.
• The fact we're processing each table one at a time, without threading out.

All of these facts do slow it down considerably, and we've in the past avoided fixing these bottlenecks because we don't want people to rely on this full time working with Canvas Data. It's meant to be an example application showing users how to interact with the API, and how they could do things like "avoid historical refreshes".

We could one day down the road, fix these features, and turn the CDL into a fully supported solution, but that would have to be a product decision.

Originally posted by @securityinsanity in #6 (comment)

@securityinsanity, @UniversalsuperB made a remark on finding out that some id columns gets padded with 555555. I believe this is why I get the following error when I tried to run this command:
[root@localhost home]# RUST_LOG=info ./target/release/cdl-runner

Error: 
 INFO 2018-11-13T16:42:52Z: cdl_runner: Setting up API Client...
 INFO 2018-11-13T16:42:52Z: cdl_runner: Connecting to RocksDB Store....
 INFO 2018-11-13T16:42:52Z: cdl_runner: Skipping dump: f2861f59-f11c-4c97-8d09-c349e7f4f3f6 
 INFO 2018-11-13T16:42:52Z: cdl_runner: Skipping dump: 259dfc10-69b7-4f9f-bd89-013082e6c1ef due to only final selected
 INFO 2018-11-13T16:42:52Z: cdl_runner: Skipping dump: a8d4271f-2013-4ff0-8813-b8aa79e85e0b due to only final selected
 INFO 2018-11-13T16:42:53Z: cdl_runner: Connecting to the DB
ERROR 2018-11-13T16:50:35Z: cdl_runner::db_client: insert error
ERROR 2018-11-13T16:50:35Z: cdl_runner::db_client: MySqlError { ERROR 1264 (22003): Out of range value for column 'position' at row 1 }
ERROR 2018-11-13T16:50:35Z: cdl_runner::importer: process -> for line in finalized_string -> !is_volatile -> ins_res -> is_err
ERROR 2018-11-13T16:50:35Z: cdl_runner::importer: Error(MysqlErr, State { next_error: None, backtrace: None })
 INFO 2018-11-13T16:50:35Z: cdl_runner: Done!

My question is: Is there a fix for this?

[Mythra]

Hey @latifatbraimah ,

As mentioned in the issue you're posting this from. This is a known bottleneck, and not one we plan on fixing for the reasons outlined in this comment. I'm unsure as to why exactly this was moved to it's own issue.

However, if you have any more questions I recommend posting them on the original issue (which is still open).

Thanks,