http-callout

HTTP Callout Example

HTTP Callout shows how to call external HTTP servers from inside the operators. In this example, we will read car events and send requests to an HTTP server for additional information about the maker.

The car events are generated via an HTTP-Source connector.

Dataflow Primitives

The dataflow uses the following primitives:

• map
• capabilities
  • http

Step-by-step

• Take a look at the dataflow.yaml to get an idea of what we're doing.

• Make sure to Install SDF & Start a Cluster.

1. Run the Dataflow

Use sdf command line tool to run the dataflow:

sdf run --ui

Use --ui to open the Studio.

2. Test the Dataflow

Start the HTTP Sink Connector

In a new terminal change direcotry to ./connectors, download the connector binary, and start 2 connectors:

cd ./connectors
cdk hub download infinyon/[email protected]
cdk deploy start --ipkg infinyon-http-source-0.4.3.ipkg -c car-callout-connector.yaml

To see the events, run fluvio consume cars.

For additional context, checkout connectors.

Consume form Makers

The http lookup result is sent to a makers topic. Consume from makers topic:

fluvio consume makers

Hit to exit.

Check Metrics

The SDF engine will automatically collect metrics for the dataflow. In the sdf terminal, show the number of records processed by get-car-maker:

show state lookup-car-maker/get-car-maker/metrics

Key    Window  succeeded  failed
stats  *       36         0

Run show state to see the other parameters available.

Congratulations! You've successfully built and run a dataflow!

Clean-up

Note that your connector will continue to run in the background until you shut them down:

cdk deploy shutdown --name car-callout

Exit sdf terminal and clean-up. The --force flag removes the topics:

sdf clean --force