Skip to content

Commit

Permalink
Merge pull request #343 from IBMStreams/develop
Browse files Browse the repository at this point in the history 
Develop
  • Loading branch information
@joergboe
joergboe authored Mar 20, 2019
2 parents a30676f + 90b69c5 commit 70c49d5
Show file tree
Hide file tree
Showing 39 changed files with 442 additions and 263 deletions.
15 changes: 15 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,21 @@
# Changes
=======

## v3.1.0:
* Update apache http client library to v 4.5.7
* HTTPRequest operator: Allows tls-client authentication
* HTTPRequest operator: Enable TLS1.2 for all connections
* HTTPRequest operator: Relaxed requirements for parameter sslTrustStorePassword
* HTTPRequest operator: server certificate does not require host match
* Description added: how to create https certificates
* Faster build
* One common spl doc for samples generated
* HTTP Test server works now also with streams java ssl engine
* Added tests: HTTPLineTest, URLEncodeDecode
* Removed old script tests
* Cloud test suite added
* Samples: Makefiles take the streams studio settings into account

## v3.0.0:
* The http rest functions and the WebSocket server functions are now moved into [streamsx.inetserver toolkit](https://github.com/IBMStreams/streamsx.inetserver/releases)
* Operator HTTPRequestAdd - new parameters extraHeaderAttribute, accessTokenAttribute and tokenTypeAttribute
Expand Down
53 changes: 44 additions & 9 deletions DEVELOPMENT.md
View file
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# Running applications that use the Internet Toolkit

To create applications that use the Internet Toolkit, you must configure either Streams Studio
or the SPL compiler to be aware of the location of the toolkit.
or the SPL compiler to be aware of the location of the toolkit.

## Before you begin

Install IBM InfoSphere Streams. Configure the product environment variables by entering the following command:
Install IBM InfoSphere Streams. Configure the product environment variables by entering the following command:
source product-installation-root-directory/product-version/bin/streamsprofile.sh

## About this task
Expand All @@ -25,15 +25,15 @@ Alternatively, you can fully qualify the operators that are provided by toolkit
where MyMain is the name of the SPL main composite.
**Note**: These command parameters override the **STREAMS_SPLPATH** environment variable.
* Add the toolkit location in InfoSphere Streams Studio.
2. Develop your application. To avoid the need to fully qualify the operators, add a use directive in your application.
2. Develop your application. To avoid the need to fully qualify the operators, add a use directive in your application.
* For example, you can add the following clause in your SPL source file:
use com.ibm.streamsx.inet::*;
use com.ibm.streamsx.inet.http::*;
You can also specify a use clause for individual operators by replacing the asterisk (\*) with the operator name. For example:
You can also specify a use clause for individual operators by replacing the asterisk (\*) with the operator name. For example:
use com.ibm.streamsx.inet::InetSource;
3. Build your application. You can use the **sc** command or Streams Studio.
4. Start the InfoSphere Streams instance.
5. Run the application. You can submit the application as a job by using the **streamtool submitjob** command or by using Streams Studio.
4. Start the InfoSphere Streams instance.
5. Run the application. You can submit the application as a job by using the **streamtool submitjob** command or by using Streams Studio.



Expand All @@ -55,7 +55,7 @@ The top-level build.xml contains the main targets:
Execute the comman ant -p to display the target information.

The release should use Java 8 for the Java compile to allow the widest use of the toolkit (with Streams 4.0.1 or later). (Note Streams 4.0.1 ships Java 8).
The build script inserts the commit hash into the toolkit version if the version number has a form like X.Y.Z.__dev__
The build script inserts the commit hash into the toolkit version if the version number has a form like X.Y.Z.__dev__
This change in the info.xml file can be removed with ant target revertversion.

# Test the toolkit
Expand All @@ -67,9 +67,9 @@ To read more about the configuration of the test servers look into file
The test run can be customized with file
[tests/FrameworkTests/tests/TestProperties.sh](tests/FrameworkTests/tests/TestProperties.sh)

To find out more about the test utility go into directory
To find out more about the test utility go into directory
`tests/FrameworkTests`
and run the commands:
and run the commands:
```
./runTest.sh -h
./runTest.sh --man
Expand All @@ -79,3 +79,38 @@ and run the commands:

To look into the test results open firefox and open the *index.html* file in the test workdir.

# Update the github.io pages

Checkout the label of the current version:
```
git checkout <vx.y.z>
```

Clean the workspace and generate the spl docs:
```
ant clean-all
ant spldoc
```

Make an archive of the docs:
```
tar -cvzf alldocs.tgz doc/ samples/doc/
```

Checkout the branch gh-pages, move the spl-doc of the prior version and overwrite with the new spl-doc files:
```
git checkout gh-pages
mkdir <prior version>
git mv doc/ <prior version>
rm -rf doc/ samples/doc/
tar -xvzf alldocs.tgz
rm alldocs.tgz
```

Checkin the changes:
```
git add doc --all
git add samples/doc/ --all
git commit -m 'SPLDOC for new version <vx.y.z>'
git push
```
157 changes: 55 additions & 102 deletions build.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,33 +20,10 @@

<target name="all"
description="Build the toolkit code, build the samples index and generate the spl documentation"
depends="setcommitversion, toolkit, samples, spldoc_work"
depends="toolkit, samples, spldoc"
/>

<target name="setcommitversion">
<!-- Extract the git commit hash -->
<exec executable="git" outputproperty="commithash">
<arg value="rev-parse"/>
<arg value="--short"/>
<arg value="HEAD"/>
</exec>
<!-- Update the info.xml -->
<replace file="${toolkit.dir}/info.xml" token="__dev__" value="commit_${commithash}" summary="yes"/>
<!-- Extract info from the toolkit's info.xml -->
<xmlproperty file="${toolkit.dir}/info.xml" prefix="tkinfo" keepRoot="no"/>
<echo message="Toolkit Version: ${tkinfo.identity.version}"/>
<echo message="Git Hash: ${commithash}"/>
</target>

<target name="revertversion" description="Revert the version change (commithash) in info.xml">
<exec executable="git">
<arg value="checkout"/>
<arg value="--"/>
<arg value="${toolkit.dir}/info.xml"/>
</exec>
</target>

<target name="clean" depends="spldocclean,samplesclean,releaseinfo-clean"
<target name="clean" depends="spldoc-clean,samples-clean,releaseinfo-clean"
description="clean up toolkit code, samples code and all spldocs">
<delete includeemptydirs="true">
<fileset dir="${basedir}" includes="release-*/"/>
Expand All @@ -59,46 +36,73 @@
<ant dir="${toolkit.test}" target="clean-all"/>
</target>

<target name="toolkit"
<target name="toolkit" depends="get_name"
description="Build any code and index the toolkit." >
<ant dir="${toolkit.dir}" target="all"/>
</target>


<target name="get_name">
<xmlproperty file="${toolkit.dir}/info.xml" prefix="tkinfo" keepRoot="no"/>
<echo message="Toolkit Name: ${tkinfo.identity.name}"/>
<echo message="Toolkit Version: ${tkinfo.identity.version}"/>
</target>

<target name="toolkitIndexOnly" description="Index the toolkit.">
<ant dir="${toolkit.dir}" target="cleanToolkitxml" />
<ant dir="${toolkit.dir}" target="indextoolkit" />
</target>

<target name="spldoc" depends="setcommitversion, spldoc_work, revertversion"
description="Create SPLDOC with git commit hash included in the version"
/>

<target name="spldoc_work" depends="samples"> <!-- can not be called independently thus has no description -->
<target name="spldoc" depends="get_name, samples"
description="Create spldoc for toolkit and all samples">
<!-- build the toolkit doc -->
<antcall target="spldoctoolkit">
<param name="tkdir" value="${toolkit.dir}"/>
<param name="tktitle" value="IBMStreams ${tkinfo.identity.name} Toolkit"/>
</antcall>
<echo message="Tookit to SPLDOC: ${toolkit.dir}"/>
<exec executable="${spl-md}" failonerror="true">
<arg value="--include-composite-operator-diagram"/>
<arg value="--author"/>
<arg value="IBMStreams Open Source Community at GitHub - https://github.com/IBMStreams/${ant.project.name}"/>
<arg value="--doc-title"/>
<arg value="IBMStreams ${tkinfo.identity.name} Toolkit"/>
<arg value="--toolkit-path"/>
<arg value="${toolkit.dir}"/>
<arg value="--warn-no-comments"/>
<arg value="--check-tags"/>
</exec>
<!-- build the sample doc for each sample-->
<subant target="spldocsample" genericantfile="${basedir}/build.xml">
<dirset dir="samples" includes="*" excludes="doc"/>
</subant>
<exec executable="bash" outputproperty="samples.list" failonerror="true">
<arg value="-c"/>
<arg value="for x in samples/*/info.xml; do echo -n ${x%/*}; echo -n : ; done"/>
</exec>
<echo message="All samples to SPLDOC: ${samples.list}"/>
<exec executable="${spl-md}" failonerror="true">
<arg value="--include-composite-operator-diagram"/>
<arg value="--author"/>
<arg value="IBMStreams Open Source Community at GitHub - https://github.com/IBMStreams/${ant.project.name}"/>
<arg value="--doc-title"/>
<arg value="IBMStreams ${tkinfo.identity.name} Samples"/>
<arg value="--toolkit-path"/>
<arg value="${samples.list}"/>
<arg value="--warn-no-comments"/>
<arg value="--check-tags"/>
<arg value="--include-source"/>
<arg value="--output-directory"/>
<arg value="samples/doc/spldoc"/>
</exec>
</target>

<target name="samples" depends="toolkit" description="Build the toolkit index for the samples">
<subant target="indextoolkit" genericantfile="${basedir}/build.xml">
<dirset dir="samples" includes="*" excludes="doc"/>
</subant>
</target>
<target name="samplesclean" description="Clean up samples index and samples code">
<target name="samples-clean" description="Clean up samples index and samples code">
<subant genericantfile="${basedir}/build.xml">
<target name="cleantoolkit"/>
<target name="cleansample"/>
<dirset dir="samples" includes="*" excludes="doc"/>
</subant>
</target>

<target name="build-all-samples" depends="toolkit" description="Compile all samples">
<target name="samples-build" depends="toolkit" description="Compile all samples">
<subant target="buildsample" genericantfile="${basedir}/build.xml">
<dirset dir="samples" includes="*" excludes="doc"/>
</subant>
Expand Down Expand Up @@ -150,70 +154,21 @@
<delete dir="${basedir}/output" quiet="true"/>
<delete dir="${basedir}/doc" quiet="true"/>
</target>
<!-- build the toolkit doc -->
<target name="spldoctoolkit">
<property name="tkdir" value="${basedir}"/>
<echo message="Tookit to SPLDOC: ${tkdir}"/>
<exec executable="${spl-md}">
<arg value="--include-composite-operator-diagram"/>
<arg value="--author"/>
<arg value="IBMStreams Open Source Community at GitHub - https://github.com/IBMStreams/${ant.project.name}"/>
<arg value="--doc-title"/>
<arg value="${tktitle}"/>
<arg value="--directory"/>
<arg value="${tkdir}"/>
<arg value="--warn-no-comments"/>
</exec>
</target>
<!-- build the sample doc for each sample -->
<target name="spldocsample">
<property name="tkdir" value="${basedir}"/>
<echo message="Sample to SPLDOC: ${tkdir}"/>
<exec executable="${spl-md}">
<arg value="--include-composite-operator-diagram"/>
<arg value="--author"/>
<arg value="IBMStreams Open Source Community at GitHub - https://github.com/IBMStreams/${ant.project.name}"/>
<arg value="--doc-title"/>
<arg value="${tktitle}"/>
<arg value="--directory"/>
<arg value="${tkdir}"/>
<arg value="--warn-no-comments"/>
<arg value="--include-source"/>
</exec>
</target>
<!-- build the common doc for all samples -->
<target name="spldoc-all-samples" depends="samples" description="build the doc for all samples">
<property name="tkdir" value="${basedir}"/>
<echo message="All samples to SPLDOC: ${tkdir}"/>
<exec executable="${spl-md}">
<arg value="--include-composite-operator-diagram"/>
<arg value="--author"/>
<arg value="IBMStreams Open Source Community at GitHub - https://github.com/IBMStreams/${ant.project.name}"/>
<arg value="--doc-title"/>
<arg value="${tktitle}"/>
<arg value="-t"/>
<arg value="samples"/>
<arg value="--warn-no-comments"/>
<arg value="--include-source"/>
<arg value="--output-directory"/>
<arg value="samples/doc/spldoc"/>
</exec>
</target>

<target name="spldocclean" description="clean the toolkit doc and the all samples doc">
<target name="spldoc-clean" description="clean the toolkit doc and the all samples doc">
<delete dir="${basedir}/doc" quiet="true"/>
<delete dir="${basedir}/samples/doc"/>
</target>

<!-- Targets to build releases -->
<target name="release" description="Make a toolkit release archive - purge workspace and build toolkit from scratch" depends="clean">
<antcall target="release-target" inheritAll="true"/>
<antcall target="release-target" inheritAll="true"/>
</target>

<target name="release-target" depends="releaseinfo">
<!-- prepare toolkit code and docs -->
<antcall target="all" inheritAll="true"/>
<antcall target="samplesclean" inheritAll="true"/> <!-- remove toolkit index from samples -->
<antcall target="samples-clean" inheritAll="true"/> <!-- remove toolkit index from samples -->
<!-- get os type string -->
<property name="osVersion" value="${os.version}"/>
<condition property="osPlat" value="el7">
Expand All @@ -231,20 +186,20 @@
<!-- create release bundle -->
<mkdir dir="${release.dir}" />
<property name="releasefilename" value="${release.dir}/streamsx.inet.toolkit-${tkinfo.identity.version}-${osPlat}-${os.arch}-${commithash.short}-${DSTAMP}-${TSTAMP}.tgz"/>
<echo message="Create release archive ${releasefilename}"/>
<tar compression="gzip" longfile="gnu"
destfile="${releasefilename}"
basedir="${basedir}"
includes="${toolkit.string}/** samples/** README.md LICENSE.md"
includes="${toolkit.string}/** samples/** doc/** README.md LICENSE.md"
excludes="**/.gitignore **/.gitkeep ${toolkit.string}/.project ${toolkit.string}/.settings/** ${toolkit.string}/.classpath ${toolkit.string}/icons/ ${toolkit.string}/impl/java/ ${toolkit.string}/impl/cpp/ ${toolkit.string}/impl/src/ ${toolkit.string}/impl/Makefile ${toolkit.string}/build.xml "
/>
<checksum file="${releasefilename}"/>
<checksum algorithm="sha1" file="${releasefilename}"/>
<!-- revert commithash -->
<antcall target="revertversion" inheritAll="true"/>
</target>

<!-- Extract the git commit hash and make release info -->
<target name="releaseinfo" depends="clean-ignored" description="Make the release information file ${release.info.file}">
<target name="releaseinfo" depends="get_name,clean-ignored" description="Make the release information file">
<echo message="create info-file ${release.info.file}"/>
<exec executable="git" outputproperty="commithash.long" failonerror="true">
<arg value="rev-parse" />
<arg value="HEAD" />
Expand All @@ -262,16 +217,14 @@
<arg value="-c" />
<arg value="cat ${release.info.file}" />
</exec>
<!-- Extract info from the toolkit's info.xml -->
<xmlproperty file="${toolkit.dir}/info.xml" prefix="tkinfo" keepRoot="no"/>
</target>

<target name="releaseinfo-clean" description="remove the release information file ${release.info.file}">
<target name="releaseinfo-clean" description="remove the release information file">
<delete file="${release.info.file}"/>
</target>

<!--- Targets to purge the workspace before a release archive is ptroduced -->
<target name="clean-ignored" depends="warn-unclean" description="Remove all git-ignored files (exclude model files) and warn if workspace has uncommited changes"/>
<!--- Targets to purge the workspace before a release archive is produced -->
<target name="clean-ignored" depends="warn-unclean" description="Remove all git-ignored files and warn if workspace has uncommited changes"/>

<target name="warn-unclean" depends="check-unclean" if="has.uncommited">
<echo>!Your workspace is not clean!</echo>
Expand Down
12 changes: 6 additions & 6 deletions com.ibm.streamsx.inet/.classpath
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@
<classpathentry kind="src" path="impl/java/src"/>
<classpathentry kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER"/>
<classpathentry kind="con" path="com.ibm.streams.java/com.ibm.streams.operator"/>
<classpathentry kind="lib" path="opt/httpcomponents-client-4.5.5/lib/commons-codec-1.10.jar"/>
<classpathentry kind="lib" path="opt/httpcomponents-client-4.5.5/lib/commons-logging-1.2.jar"/>
<classpathentry kind="lib" path="opt/httpcomponents-client-4.5.5/lib/httpclient-4.5.5.jar"/>
<classpathentry kind="lib" path="opt/httpcomponents-client-4.5.5/lib/httpclient-cache-4.5.5.jar"/>
<classpathentry kind="lib" path="opt/httpcomponents-client-4.5.5/lib/httpcore-4.4.9.jar"/>
<classpathentry kind="lib" path="opt/httpcomponents-client-4.5.5/lib/httpmime-4.5.5.jar"/>
<classpathentry kind="lib" path="opt/httpcomponents-client-4.5.7/lib/commons-codec-1.11.jar"/>
<classpathentry kind="lib" path="opt/httpcomponents-client-4.5.7/lib/commons-logging-1.2.jar"/>
<classpathentry kind="lib" path="opt/httpcomponents-client-4.5.7/lib/httpclient-4.5.7.jar"/>
<classpathentry kind="lib" path="opt/httpcomponents-client-4.5.7/lib/httpclient-cache-4.5.7.jar"/>
<classpathentry kind="lib" path="opt/httpcomponents-client-4.5.7/lib/httpcore-4.4.11.jar"/>
<classpathentry kind="lib" path="opt/httpcomponents-client-4.5.7/lib/httpmime-4.5.7.jar"/>
<classpathentry kind="lib" path="opt/signpost-1.2/lib/signpost-core-1.2.jar"/>
<classpathentry kind="lib" path="opt/signpost-1.2/lib/signpost-commonshttp4-1.2.1.1.jar"/>
<classpathentry kind="con" path="org.eclipse.jdt.junit.JUNIT_CONTAINER/4"/>
Expand Down
2 changes: 1 addition & 1 deletion com.ibm.streamsx.inet/build.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ All required libs are now available in the source repository
<path id="cp.manifest">
<fileset dir="opt">
<include name="signpost-1.2/lib/*.jar"/>
<include name="httpcomponents-client-4.5.5/lib/*.jar"/>
<include name="httpcomponents-client-4.5.7/lib/*.jar"/>
</fileset>
</path>

Expand Down
Loading

0 comments on commit 70c49d5

Please sign in to comment.