Welcome to kafka-connect-http Discussions!

[castorm]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[OlyaVashchenko]

Hello there!
I'm new to Kafka Connect and I try to use your connector to query data (via get method) from the url.
Url returns me a JSON response. I can see the response in my console-consumer. For test purposes I use console.
I want to go further and test SMT (for example, change field name). Unfortunately, the transformation doesn't seem to work.
Here's my .properties file:

name=http-source
connector.class=com.github.castorm.kafka.connect.http.HttpSourceConnector
http.request.url=https://www.nbrb.by/api/exrates/currencies/28
http.timer.interval.millis=30000
kafka.topic=my-test-topic

transforms=RenameField
transforms.RenameField.type=org.apache.kafka.connect.transforms.ReplaceField$Value
transforms.RenameField.renames=Cur_Abbreviation:Cur_Abbr

[castorm]

Welcome @OlyaVashchenko,

I hope the connector fits your needs! Thank you for sharing your experience.

Any questions, ideas or contributions of any type are welcome!

Best regards.

[OlyaVashchenko]

Sorry @castorm , I edited my question :) (used the wrong key combination for new line)
The connector is great when used alone, but with SMT (Single Message Transformation) smth is wrong. I don't know whether it's me or the connector itself

[PuneethRaghav]

Hi
How to modify this connector for pagination support

[PuneethRaghav]

How to modify this connector for pagination support

Hi @castorm
How to modify this connector for pagination support(I mean if we have page number and total pages are coming in API response)

[G4rtzEY]

Hi all,

We are trying to set up an HTTPS Kafka connection and we are getting the following error:

org.apache.kafka.connect.errors.RetriableException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

This is our current connector configuration:

name: Connector_name
config:
connector.class: com.github.castorm.kafka.connect.http.HttpSourceConnector
tasks.max: 1
http.request.method: GET
http.request.url: https://example.com
http.request.headers: 'Content-Type: application/json'
http.auth.type: Basic
http.auth.user: user
http.auth.password: password
http.timer.interval.millis: 30000
http.timer.catchup.interval.millis: 1000
kafka.topic: topicname

Could you please advise us on how to set up secure connections?

Thank you in advance.