Getting same records infinitie number of times

[aisb459]

When I perform request to the specific REST API (get request). I get this back:
{"PWID": "1","LOT_NUMBER": "jz3177"}

But when I perform the request through this source connector, I get same records infinitie number of times

{"schema":{"type":"struct","fields":[{"type":"string","optional":false,"doc":"HTTP Record Value","field":"value"},{"type":"string","optional":true,"doc":"HTTP Record Key","
field":"key"},{"type":"int64","optional":true,"doc":"HTTP Record Timestamp","field":"timestamp"}],"optional":false,"name":"com.github.castorm.kafka.connect.http.Value","doc
":"Message Value"},"payload":{"value":"{\"PWID\":\"1\",\"LOT_NUMBER\":\"jz3177\"}","key":"1","timestamp":1623412068790}}
{"schema":{"type":"struct","fields":[{"type":"string","optional":false,"doc":"HTTP Record Value","field":"value"},{"type":"string","optional":true,"doc":"HTTP Record Key","
field":"key"},{"type":"int64","optional":true,"doc":"HTTP Record Timestamp","field":"timestamp"}],"optional":false,"name":"com.github.castorm.kafka.connect.http.Value","doc
":"Message Value"},"payload":{"value":"{\"PWID\":\"1\",\"LOT_NUMBER\":\"jz3177\"}","key":"1","timestamp":1623412099052}}
{"schema":{"type":"struct","fields":[{"type":"string","optional":false,"doc":"HTTP Record Value","field":"value"},{"type":"string","optional":true,"doc":"HTTP Record Key","
field":"key"},{"type":"int64","optional":true,"doc":"HTTP Record Timestamp","field":"timestamp"}],"optional":false,"name":"com.github.castorm.kafka.connect.http.Value","doc
":"Message Value"},"payload":{"value":"{\"PWID\":\"1\",\"LOT_NUMBER\":\"jz3177\"}","key":"1","timestamp":1623412129386}}

.
.
.
same thing infinite number of times

I expect it only once as I get only one record as response from my rest api

{
	"connector.class": "com.github.castorm.kafka.connect.http.HttpSourceConnector",
	"http.request.url": "request url",
	"http.timer.interval.millis": "30000",
	"kafka.topic": "http-topic3",
	"http.request.template.factory": "com.github.castorm.kafka.connect.http.request.template.NoTemplateFactory",
	"http.request.headers": "Accept:application/json",
	"tasks.max": 1,
	"max.retries": "0",
	"http.response.record.offset.pointer": "key=/PWID"
}

what am I missing in properties? I am new to kafka connect

if I apply transforms and get only this {"PWID":"1","LOT_NUMBER":"jz3177"} then is there any way to avoid duplicate records in topic in which PWID is primary key?

Please help !!

[castorm]

Hi @aisb459,

The connector is meant to scroll through the results returned by the API.

Apparently that API is not meant to be scrolled, is it?

Best regards.

[aisb459]

@castorm yes..Can we set maximum number of http requests connector can make to 1?

[castorm]

Hi @aisb459,

At the moment I'm afraid that's not possible.

You could just remove the connector if all you want is 1 record.

[aisb459]

Hi Castorm, Yes.. The api is not meant to change.. Is there a way I can set the connector to make only one http request? Also is there a way I can provide time interval between json records in a http request(didn't mean interval between http requests)..? Thanks.

…

On Sun, 13 Jun, 2021, 10:06 PM Cástor Rodríguez, ***@***.***> wrote: Hi @aisb459 <https://github.com/aisb459>, The connector is meant to scroll through the results returned by the API. Apparently that API is not meant to be scrolled, is it? Best regards. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#131 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUOHK4G2VIQEBZWXKSBYKMDTSTM7BANCNFSM46Q4WNBQ> .

[castorm]

Hi @aisb459,

I'm afraid the connector doesn't allow you to limit to a single request.

As you probably already know, you can configure the interval in between requests, not sure what you mean about interval in between records, can you elaborate on that?

Thanks,
Best regards.

[aisb459]

Hi, Suppose if my api response is the following: [ { "PWID": "1", "LOT_NUMBER": "jz3177" }, { "PWID": "2", "LOT_NUMBER": "jz3177" }, { "PWID": "3", "LOT_NUMBER": "jz3177" } ] can I give intervals between each json object which means { "PWID": "1", "LOT_NUMBER": "jz3177" } **n seconds interval** { "PWID": "2", "LOT_NUMBER": "jz3177" } **n seconds interval** { "PWID": "1", "LOT_NUMBER": "jz3177" } Thanks

…

On Wed, Jun 16, 2021 at 11:52 PM Cástor Rodríguez ***@***.***> wrote: Hi @aisb459 <https://github.com/aisb459>, I'm afraid the connector doesn't allow you to limit to a single request. As you probably already know, you can configure the interval in between requests, not sure what you mean about interval in between records, can you elaborate on that? Thanks, Best regards. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#131 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUOHK4FXHA2RGDUABZZIOLLTTDTWBANCNFSM46Q4WNBQ> .

[castorm]

Hi @aisb459,

I believe there is something that could work for you, however, you'd have to give up timestamps.

The reason it doesn't work for you as it is, is because the plugin has no way of knowing whether an element has been seen or not.

The way the plugin usually does this is by requesting results ordered by the creation/update date of the elements, and when provided that order, by looking at the timestamps of the last element seen.

Your records have no timestamp, so Kafka connect will add one for you, making a record in kafka different to the next record read in the request, even if they have the same key.

To work around this, and taking advantage of the numerical key, you could instruct the plugin to treat the key also as timestamp:

"http.response.record.offset.pointer": "key=/PWID, timestamp=/PWID"

Provided your API will always return incremental keys, that should work.

Best regards.

[aisb459]

Hi @castorm
"http.response.record.offset.pointer": "key=/PWID, timestamp=/PWID"
This really helps.
Thank you so much