HTTPRequest operator

HTTPRequest operator proposal

Goal

A new operator that supports all HTTP 1.1 request methods (GET, POST, PUT, DELETE, etc.) and is designed to work well with HTTP(S) REST services including Bluemix services.

Issues

• Interacting with Bluemix services is not always supported by existing operators and/or functions in com.ibm.streamsx.inet, some examples:
  • Updating secure gateway destinations requires PUTand a non-standard authorization header - https://www.ng.bluemix.net/docs/api/content/api/securegateway/swagger/index.html
  • Similar request for Twitter: https://github.com/IBMStreams/streamsx.inet/issues/210
• Some comments on the existing operators: https://github.com/IBMStreams/streamsx.inet/issues/105
• Current operators do not (all?) support certification based authentication
• Not well integrated with JSON: https://github.com/IBMStreams/streamsx.inet/issues/187

All existing issues should be looked at when designing this operator.

Use cases

Commons use cases for an single invocation of the operator.

1. Interaction with a single REST service, e.g. Bluemix secure gateway or Twitter. Authentication & connection configuration is common to all requests, multiple methods may be executed against the same REST service (same or different URLs, e.g. different resources in the same service).
2. Execution of requests against internet services (single or multiple URLs) that do not require authentication (public facing data), most likely GETs.

Functionality concepts

• Operator makes an HTTP(S) Request for each tuple arriving on its input port.
• HTTP response becomes an output tuple on an optional output port or an optional error port(?).
• Details of the specific request may be fixed or come from the input tuple (e.g. URL may be fixed but the method may be an attribute in the tuple, or both URL and method can come from the attribute)
• A substitution mechanism for the URL would be good, e.g. the value of input attribute team replaces ${team} in the URL, exact syntax TBD, may look at Apache HTTP Server URL rewriting:

 param
    fixedUrl: "http://api.example.com/scores/${team}"

• Connection configuration (authentication, retries, timeouts etc.) is driven by operator parameters and/or application configurations
• JSON support will be the default and primary focus.
• Multiple common authentication schemes will be supported (e.g. BASIC, certificate, ...).
• Lessons learnt from existing operators will be applied (e.g. extra headers, and look at existing issues requesting new functionality).

Scenarios

Scenarios that should be implemented to demonstrate the operator is ready for purpose and simple to use:

• Automatically update Bluemix Secure Gateway destination for invocation of HTTPTupleView, so that upon relocation after failure clients continue to access the HTTPTupleView web-site correctly.
• Build Streams applications that uses Bluemix services:
• Weather ?
• Twitter ?

Code

WIP pull request (https://github.com/IBMStreams/streamsx.inet/pull/241) has an initial version of the operator that can be used as starting point. it can be iterated over to produce the final operator based upon an agile style process.

• Initial starting version supports GET,PUT,POST with no output port, or error handling, and authentication only through extra-headers.
• At some point we may want to create a feature branch for this operator or just develop in master with the understanding its api may change as it is developed. I.e. master would not be released with a version of the operator until it is in an acceptable state.
• Eventually HTTPPost would be deprecated or re-implemented to be a restricted version of HTTPRequest.