Elasticsearch Update Operation Tutorial

[shoaibjdev]

Hi,

Is there a tutorial on ElasticSearch update operation?

Pipeline Flow:
Have create a minimal pipeline to read data from ES 6.2.2 (origin) and push it to RDBMS (destination 1). Thereafter we update same ES document (destination 2) to update a field @Version from 1 to 2. So that on the next run of the pipeline this document from ES will not be picked for moving to RDBMS.

Can someone point to the right direction as we are getting below error, after following the Streamset documents at the Destination (Also if someone could point on debugging the flow it would be helpful as the error is not clear where exactly the issue is) :

2018-05-28 08:27:35,171 [user:*admin] [pipeline:ESUpdatePipeline/ESUpdatePipeline5fdde3c4-2c84-480b-a853-f2085333df2c] [runner:] [thread:ProductionPipelineRunnable-ESUpdatePipeline5fdde3c4-2c84-480b-a853-f2085333df2c-ESUpdatePipeline] INFO Pipeline - Destroying pipeline with reason=FAILURE
2018-05-28 08:27:35,177 [user:*admin] [pipeline:ESUpdatePipeline/ESUpdatePipeline5fdde3c4-2c84-480b-a853-f2085333df2c] [runner:] [thread:ProductionPipelineRunnable-ESUpdatePipeline5fdde3c4-2c84-480b-a853-f2085333df2c-ESUpdatePipeline] INFO Pipeline - Processing lifecycle stop event
2018-05-28 08:27:35,177 [user:*admin] [pipeline:ESUpdatePipeline/ESUpdatePipeline5fdde3c4-2c84-480b-a853-f2085333df2c] [runner:] [thread:ProductionPipelineRunnable-ESUpdatePipeline5fdde3c4-2c84-480b-a853-f2085333df2c-ESUpdatePipeline] INFO Pipeline - Pipeline finished destroying with final reason=FAILURE
2018-05-28 08:27:35,185 [user:*admin] [pipeline:ESUpdatePipeline/ESUpdatePipeline5fdde3c4-2c84-480b-a853-f2085333df2c] [runner:] [thread:ProductionPipelineRunnable-ESUpdatePipeline5fdde3c4-2c84-480b-a853-f2085333df2c-ESUpdatePipeline] ERROR ProductionPipelineRunnable - An exception occurred while running the pipeline, com.streamsets.pipeline.api.StageException: ELASTICSEARCH_17 - Could not index '347' records: One or more operations failed
com.streamsets.pipeline.api.StageException: ELASTICSEARCH_17 - Could not index '347' records: One or more operations failed
at com.streamsets.pipeline.stage.destination.elasticsearch.ElasticsearchTarget.write(ElasticsearchTarget.java:342)
at com.streamsets.pipeline.configurablestage.DTarget.write(DTarget.java:34)
at com.streamsets.datacollector.runner.StageRuntime.lambda$execute$2(StageRuntime.java:249)
at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:195)
at com.streamsets.datacollector.runner.StageRuntime.execute(StageRuntime.java:257)
at com.streamsets.datacollector.runner.StagePipe.process(StagePipe.java:219)
at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.processPipe(ProductionPipelineRunner.java:801)
at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.lambda$executeRunner$3(ProductionPipelineRunner.java:846)
at com.streamsets.datacollector.runner.PipeRunner.executeBatch(PipeRunner.java:136)
at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.executeRunner(ProductionPipelineRunner.java:845)
at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.runSourceLessBatch(ProductionPipelineRunner.java:823)
at com.streamsets.datacollector.execution.runner.common.ProductionPipelineRunner.processBatch(ProductionPipelineRunner.java:472)
at com.streamsets.datacollector.runner.StageRuntime$3.run(StageRuntime.java:321)
at java.security.AccessController.doPrivileged(Native Method)
at com.streamsets.datacollector.runner.StageRuntime.processBatch(StageRuntime.java:317)
at com.streamsets.datacollector.runner.StageContext.processBatch(StageContext.java:252)
at com.streamsets.pipeline.stage.origin.elasticsearch.ElasticsearchSource$ElasticsearchTask.run(ElasticsearchSource.java:231)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)