两个taos3.0之前迁移，都是本地的，用的是datax3.0，脏数据导致数据同步失败

[yanjunwan12138]

对应错误

2023-06-08 17:45:35.245 [job-0] INFO StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00%
2023-06-08 17:45:45.248 [job-0] INFO StandAloneJobContainerCommunicator - Total 10000 records, 256450 bytes | Speed 25.04KB/s, 1000 records/s | Error 2000 records, 51270 bytes | All Task WaitWriterTime 0.006s | All Task WaitReaderTime 0.338s | Percentage 0.00%
2023-06-08 17:45:45.249 [job-0] ERROR JobContainer - 运行scheduler 模式[standalone]出错.
2023-06-08 17:45:45.249 [job-0] ERROR JobContainer - Exception when job run
com.alibaba.datax.common.exception.DataXException: Code:[Framework-14], Description:[DataX传输脏数据超过用户预期，该错误通常是由于源端数据存在较多业务脏数据导致，请仔细检查DataX汇报的脏数据日志信息, 或者您可以适当调大脏数据阈值 .]. - 脏数据条数检查不通过，限制是[0]条，但实际上捕获了[2000]条.
at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:30) ~[datax-common-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.util.ErrorRecordChecker.checkRecordLimit(ErrorRecordChecker.java:58) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.job.scheduler.AbstractScheduler.schedule(AbstractScheduler.java:89) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.job.JobContainer.schedule(JobContainer.java:535) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:119) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.Engine.start(Engine.java:93) [datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.Engine.entry(Engine.java:175) [datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.Engine.main(Engine.java:208) [datax-core-0.0.1-SNAPSHOT.jar:na]
2023-06-08 17:45:45.250 [job-0] INFO StandAloneJobContainerCommunicator - Total 10000 records, 256450 bytes | Speed 250.44KB/s, 10000 records/s | Error 2000 records, 51270 bytes | All Task WaitWriterTime 0.006s | All Task WaitReaderTime 0.338s | Percentage 0.00%
2023-06-08 17:45:45.252 [job-0] ERROR Engine -

经DataX智能分析,该任务最可能的错误原因是:
com.alibaba.datax.common.exception.DataXException: Code:[Framework-14], Description:[DataX传输脏数据超过用户预期，该错误通常是由于源端数据存在较多业务脏数据导致，请仔细检查DataX汇报的脏数据日志信息, 或者您可以适当调大脏数据阈值 .]. - 脏数据条数检查不通过，限制是[0]条，但实际上捕获了[2000]条.
at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:30)
at com.alibaba.datax.core.util.ErrorRecordChecker.checkRecordLimit(ErrorRecordChecker.java:58)
at com.alibaba.datax.core.job.scheduler.AbstractScheduler.schedule(AbstractScheduler.java:89)
at com.alibaba.datax.core.job.JobContainer.schedule(JobContainer.java:535)
at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:119)
at com.alibaba.datax.core.Engine.start(Engine.java:93)
at com.alibaba.datax.core.Engine.entry(Engine.java:175)
at com.alibaba.datax.core.Engine.main(Engine.java:208)

对应job.json

{
"job":{
"content":[
{
"reader":{
"name":"tdengine30reader",
"parameter":{
"username":"root",
"password":"taosdata",
"connection":[
{
"table":[
"weather"
],
"jdbcUrl":[
"jdbc:TAOS-RS://127.0.0.1:6041/wanyanjun?timestampFormat=TIMESTAMP"
]
}
],
"column":[
"ts",
"temperature",
"humidity",
"location",
"groupid"
]
}
},
"writer":{
"name":"tdengine30writer",
"parameter":{
"username":"root",
"password":"taosdata",
"column":[
"ts",
"temperature",
"humidity",
"location",
"groupid"
],
"connection":[
{
"table":[
"weather"
],
"jdbcUrl":"jdbc:TAOS-RS://192.168.3.212:6041/wanyanjun"
}
],
"encoding":"UTF-8",
"batchSize":1000,
"ignoreTagsUnmatched":true
}
}
}
],
"setting":{
"speed":{
"channel":5
},
"errorLimit":{
"record":0
}
}
}
}

[yu285]

我们看下