Skip to content

Commit

Permalink
Docs maintenance (#625)
Browse files Browse the repository at this point in the history
* A bunch of small changes

* Add link to guide

* more small changes

* mention Pod overrides

* use tab blocks for install instructions

* Update docs/modules/nifi/pages/usage_guide/configuration-environment-overrides.adoc

Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com>

* Update docs/modules/nifi/pages/usage_guide/configuration-environment-overrides.adoc

Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com>

* Update docs/modules/nifi/pages/usage_guide/updating.adoc

Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com>

* Update docs/modules/nifi/pages/usage_guide/zookeeper-connection.adoc

Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com>

* Update docs/modules/nifi/pages/usage_guide/zookeeper-connection.adoc

Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com>

* Update docs/modules/nifi/pages/usage_guide/security.adoc

Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com>

* Update docs/modules/nifi/pages/usage_guide/operations/cluster-operations.adoc

Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com>

* Update docs/modules/nifi/pages/usage_guide/extra-volumes.adoc

Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com>

* Update docs/modules/nifi/pages/usage_guide/configuration-environment-overrides.adoc

Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com>

* Update docs/modules/nifi/pages/usage_guide/custom_processors.adoc

Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com>

* Update docs/modules/nifi/pages/usage_guide/custom_processors.adoc

Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com>

* Update docs/modules/nifi/pages/usage_guide/configuration-environment-overrides.adoc

Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com>

* Update docs/modules/nifi/pages/usage_guide/custom_processors.adoc

Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com>

* Update docs/modules/nifi/pages/usage_guide/external_ports.adoc

Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com>

* Update installation.adoc

* Update installation.adoc

* Update docs/modules/nifi/pages/getting_started/installation.adoc

Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com>

---------

Co-authored-by: Nick <NickLarsenNZ@users.noreply.github.com>
  • Loading branch information
fhennig and NickLarsenNZ authored Jun 17, 2024
1 parent 2c056b1 commit c4c9026
Showing 17 changed files with 107 additions and 73 deletions.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
3 changes: 2 additions & 1 deletion docs/modules/nifi/pages/getting_started/first_steps.adoc
Original file line number Diff line number Diff line change
@@ -31,7 +31,8 @@ The xref:zookeeper:znodes.adoc[ZNode] makes sure that the NiFi cluster will oper

=== Apache NiFi

The NiFi cluster requires authentication. Create a set of credentials for this purpose:
The NiFi cluster requires authentication.
Create a set of credentials for this purpose:

[source,bash]
----
3 changes: 2 additions & 1 deletion docs/modules/nifi/pages/getting_started/index.adoc
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
= Getting started

This guide will get you started with Apache NiFi using the Stackable operator. It will guide you through the installation of the operator and its dependencies, setting up your first NiFi cluster.
This guide will get you started with Apache NiFi using the Stackable operator.
It will guide you through the installation of the operator and its dependencies, setting up your first NiFi cluster.

== Prerequisites

37 changes: 22 additions & 15 deletions docs/modules/nifi/pages/getting_started/installation.adoc
Original file line number Diff line number Diff line change
@@ -1,25 +1,28 @@
= Installation

On this page you will install the Stackable Operator for Apache NiFi and operators for its dependencies - ZooKeeper -
as well as the commons, secret and listener operator which are required by all Stackable Operators.
On this page you will install the Stackable operator for Apache NiFi and operators for its dependencies - ZooKeeper -
as well as the commons, secret and listener operator which are required by all Stackable operators.

== Stackable Operators
== Stackable operators

There are two ways to install Stackable Operators:
There are two ways to install Stackable operators:

. Using xref:management:stackablectl:index.adoc[stackablectl]
. Using Helm

=== stackablectl
* Using xref:management:stackablectl:index.adoc[stackablectl]
* Using Helm

[tabs]
====
stackablectl::
+
--
The `stackablectl` command line tool is the recommended way to interact with operators and dependencies. Follow the
xref:management:stackablectl:installation.adoc[installation steps] for your platform if you choose to work with
`stackablectl`.
After you have installed `stackablectl` and have a Kubernetes cluster up and running, run the following command to
install all operators necessary for NiFi:
[source,bash]
[source,shell]
----
include::example$getting_started/getting_started.sh[tag=stackablectl-install-operators]
----
@@ -32,25 +35,29 @@ include::example$getting_started/install-operator-output.txt[tag=stackablectl-in
----
TIP: Consult the xref:management:stackablectl:quickstart.adoc[] to learn more about how to use `stackablectl`.
--
=== Helm

Helm::
+
--
You can also use Helm to install the operators. Add the Stackable Helm repository:
[source,bash]
[source,shell]
----
include::example$getting_started/getting_started.sh[tag=helm-add-repo]
----
Then install the Stackable Operators:
Then install the Stackable operators:
[source,bash]
----
include::example$getting_started/getting_started.sh[tag=helm-install-operators]
----
Helm will deploy the operators in a Kubernetes Deployment and apply the CRDs for the Apache NiFi service (as well as the
CRDs for the required operators). You are now ready to deploy Apache NiFi in Kubernetes.
Helm will deploy the operators in a Kubernetes Deployment and apply the CRDs for the Apache NiFi service (as well as the CRDs for the required operators).
You are now ready to deploy Apache NiFi in Kubernetes.
--
====

== What's next

Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
= Command Line parameters
= Command line parameters

This operator accepts the following command line parameters:

4 changes: 2 additions & 2 deletions docs/modules/nifi/pages/reference/environment-variables.adoc
Original file line number Diff line number Diff line change
@@ -16,7 +16,7 @@ export PRODUCT_CONFIG=/foo/bar/properties.yaml
stackable-nifi-operator run
----

or via docker:
or via Docker:

----
docker run \
@@ -44,7 +44,7 @@ export WATCH_NAMESPACE=test
stackable-nifi-operator run
----

or via docker:
or via Docker:

[source]
----
Original file line number Diff line number Diff line change
@@ -1,22 +1,27 @@
= Configuration & Environment Overrides
= Configuration & environment overrides
:nifi-docs: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties
:java-security-docs: https://docs.oracle.com/en/java/javase/11/security/java-security-overview1.html

The cluster definition also supports overriding configuration properties and environment variables, either per role or per role group, where the more specific override (role group) has precedence over the less specific one (role).
The NiFi cluster definition supports overriding configuration properties, environment variables, and Pod attributes. The configuration overrides can be applied either per role, or per role group where the more specific override (role group) has precedence over the less specific one (role).

IMPORTANT: Do not override port numbers.
This will lead to cluster malfunction.

== Configuration Overrides
== Configuration overrides

Apache NiFi runtime configuration is stored in several files listed below. The `configOverrides` block enables you to customize parameters in these files.
The complete list of the configuration options can be found in the https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties[Apache NiFi documentation].
Apache NiFi runtime configuration is stored in several files listed below.
The `configOverrides` block enables you to customize parameters in these files.
The complete list of the configuration options can be found in the {nifi-docs}[Apache NiFi documentation].

The following files can be edited directly via the `configOverrides` mechanism:

- `bootstrap.conf`
- `nifi.properties`
- `security.properties`
* `bootstrap.conf`
* `nifi.properties`
* `security.properties`

Overrides are key, value pairs defined under one of the configuration files from the list above. They must match the names values as expected by NiFi. In the example below, a property `nifi.flow.configuration.archive.enabled` is being explicitly set to 'false', overriding the default value.
Overrides are key-value pairs defined under one of the configuration files from the list above.
They must match the property names as expected by NiFi.
In the example below, a property `nifi.flow.configuration.archive.enabled` is being explicitly set to 'false', overriding the default value.

The following snippet shows how to disable workflow file backups in the NifiCluster definition:

@@ -32,9 +37,13 @@ WARNING: Please be aware that by overriding config settings in this section you

=== The security.properties file

The `security.properties` file is used to configure JVM security properties. It is very seldom that users need to tweak any of these, but there is one use-case that stands out, and that users need to be aware of: the JVM DNS cache.
The `security.properties` file is used to configure JVM security properties.
It is very rare that you need to tweak any of these, but there is one use-case that stands out, and that users need to be aware of: the JVM DNS cache.

The JVM manages its own cache of successfully resolved host names as well as a cache of host names that cannot be resolved. Some products of the Stackable platform are very sensible to the contents of these caches and their performance is heavily affected by them. As of version 1.21.0 Apache Nifi performs poorly if the positive cache is disabled. To cache resolved host names, and thus increase the performance your Nifi cluster, you can configure the TTL of entries in the positive cache like this:
The JVM manages its own cache of successfully resolved host names as well as a cache of host names that cannot be resolved.
Some products of the Stackable platform are very sensitive to the contents of these caches and their performance is heavily affected by them.
As of version 1.21.0, Apache Nifi performs poorly if the positive cache is disabled.
To cache resolved host names, and thus increase the performance your Nifi cluster, you can configure the TTL of entries in the positive cache like this:

[source,yaml]
----
@@ -47,9 +56,9 @@ The JVM manages its own cache of successfully resolved host names as well as a c

NOTE: The operator configures DNS caching by default as shown in the example above.

For details on the JVM security see https://docs.oracle.com/en/java/javase/11/security/java-security-overview1.html
For details on the JVM security, consult the {java-security-docs}[Java security overview].

== Environment Variables
== Environment variables

Environment variables can be (over)written by adding the `envOverrides` property.

@@ -78,3 +87,9 @@ nodes:
config: {}
replicas: 1
----

== Pod overrides

Pod overrides allow you to configure any attributes that can be configured on a Pod, such as tolerations or labels on the Pod.

Read the xref:concepts:overrides.adoc#pod-overrides[Pod overrides concepts page] to learn more.
31 changes: 17 additions & 14 deletions docs/modules/nifi/pages/usage_guide/custom_processors.adoc
Original file line number Diff line number Diff line change
@@ -1,20 +1,21 @@
= Loading Custom Components
= Loading custom components
:nifi-docs-custom-components: https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#introduction

It is possible to develop https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#introduction[custom components] for Apache NiFi to extend the available functionality - most often this will probably be custom processors.
It is possible to develop {nifi-docs-custom-components}[custom components] for Apache NiFi to extend the available functionality - most often this will probably be custom processors.

For these to be available in the NiFi UI, they need to be present in the classpath at startup, so the nar bundles have to be injected into your NiFi instance.

The Stackable Data Platform supports two ways to achive this goal, both of which are described below.

Please note that building your own Docker image is the recommended solution, as giving multiple pods access to a shared PVC can often be tricky (see comment on access mode in the second section).

== Custom Docker Image
== Custom Docker image

You can extend the official Stackable NiFi image and simply copy all required nar files into the classpath of Nifi.
The benefit of this method is that there is no need for any config overrides or extra mounts, you can simply use any ouf our NiFi examples, swap the image and your components will be available.
You can extend the official Stackable NiFi image and copy all required _nar_ files into the classpath of Nifi.
The benefit of this method is that there is no need for any config overrides or extra mounts, you can use any ouf our NiFi examples, swap the image and your components will be available.
But this means you will need to have access to a registry to push your custom image to.

A simple Dockerfile would look like show in the following listing.
The basic Dockerfile below shows how to achieve this:

[source,Dockerfile]
----
@@ -32,11 +33,13 @@ spec:
custom: "docker.company.org/nifi:1.25.0-customprocessor"
----

== Using the Official Image
If you don't want to create a custom image, then you can use the extra volume mount functionality to load components and configure NiFi with to read these from the extra volumes.
Also read the xref:guides:custom-images.adoc[Using customized product images] guide for additional information.

For this to work you'll need to prepare a PVC containing your components.
Usually the best way to do this will be to mount the PVC into a temporary container and then simply `kubectl cp` the nar files into that volume.
== Using the official image
If you don't want to create a custom image or don't have access to an image registry, you can use the extra volume mount functionality to mount a volume containing your custom components and configure NiFi to read these from the mounted volumes.

For this to work you'll need to prepare a PersistentVolumeClaim (PVC) containing your components.
Usually the best way to do this will be to mount the PVC into a temporary container and then `kubectl cp` the _nar_ files into that volume.

The following listing shows an example of creating the PVC and the Pod:

@@ -74,7 +77,7 @@ spec:
name: nifi-processor
----

<1> Please note that this access mode will mean that you can only use this volume with a single NiFi pod. For a distributed scenario, an additional list value of `ReadOnlyMany` could be specified here if this is supported.
<1> Please note that this access mode will mean that you can only use this volume with a single NiFi Pod. For a distributed scenario, an additional list value of `ReadOnlyMany` could be specified here if this is supported.

The command to then copy the nar bundle into the PVC is:

@@ -83,9 +86,9 @@ The command to then copy the nar bundle into the PVC is:
kubectl cp /path/to/component.nar processorcopy:/volume/
----

Now you can mount the extra volume into your NiFi instance as described in xref:nifi:usage_guide/extra-volumes.adoc[] .
Now you can mount the extra volume into your NiFi instance as described in xref:nifi:usage_guide/extra-volumes.adoc[].

After this is done your components will be available within the NiFi pods and NiFi can be configured to load these at startup by adding an extra directory to the classpath:
After this is done your components will be available within the NiFi Pods and NiFi can be configured to load these at startup by adding an extra directory to the classpath:


[source,yaml]
@@ -120,4 +123,4 @@ spec:
----

<1> Specify your prepared PVC here.
<2> The directory name after `userdata` has to match the name of the volume, while `mycustomlibs` is a name you can freely change.
<2> The directory name after `userdata` has to match the name of the volume, while `myCustomLibs` is a name you can freely change.
13 changes: 6 additions & 7 deletions docs/modules/nifi/pages/usage_guide/external_ports.adoc
Original file line number Diff line number Diff line change
@@ -1,19 +1,18 @@
= Expose NiFi processor ports
= Exposing NiFi processor ports

== Exposing processors
Some NiFi processors allow external tools to connect to them to trigger workflows or send data.
For this to work from outside the Kubernetes cluster NiFi has been deployed to, it is necessary to create https://kubernetes.io/docs/concepts/services-networking/service/[Service] or https://kubernetes.io/docs/concepts/services-networking/ingress/[Ingress] objects to configure Kubernetes routes for these ports.
For this to work from outside the Kubernetes cluster NiFi has been deployed to, it is necessary to create Service or Ingress objects to configure Kubernetes routes for these ports.

Stackable doesn't place any restriction on the type of service that can be used here.

== Example

A simple example for this is shown below.
An example for this is shown below.
It consists of a NiFi cluster with three nodes called `simple-nifi` and has a processor that listens to a TCP port and forwards lines written into that port as flowfiles.

image:https://user-images.githubusercontent.com/1070361/212958154-941cef1d-e370-4d08-b37d-b789b242c062.png[image]
image:listening-processor-example.png[A ListenTCP processor]

The processor has been configured to listen on port 8123, so this port is available on all NiFi pods for processes that run inside the Kubernetes cluster.
The processor has been configured to listen on port 8123, so this port is available on all NiFi Pods for processes that run inside the Kubernetes cluster.

In order to make this port available to external processes as well, a LoadBalancer type service can be used for example:

@@ -39,7 +38,7 @@ spec:
Depending on the Kubernetes configuration this snippet will request and external ip address and open port 8080 on that ip address.
Any requests to this port will be forwarded to port 8123 on the NiFi pods - and thus the processor.

If LoadBalancer services are not available, a NodePort service would be an alternative:
If LoadBalancer Services or Ingress resources are not available for use, a NodePort Service would be an alternative:

[source,yaml]
----
4 changes: 2 additions & 2 deletions docs/modules/nifi/pages/usage_guide/extra-volumes.adoc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
= Adding External Files to the NiFi Servers
= Adding external files to the NiFi servers

Since Apache NiFi allows executing pretty much arbitrary workflows depending on which processors are used, it may become necessary to add external files to the pods.
Since Apache NiFi allows executing arbitrary workflows depending on which processors are used, it may become necessary to add external files to the Pods.
These could for example be client certificates used to configure a `PollHTTP` processor, a keytab to obtain a Kerberos ticket, or similar things.

In order to make these files available the operator allows specifying extra volumes that will be added to the NiFi pods.
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@

= Cluster operation

Apache NiFi installations can be configured with different cluster operations like pausing reconciliation or stopping the cluster. See xref:concepts:operations/cluster_operations.adoc[cluster operations] for more details.
Apache NiFi installations can be configured with different cluster operations such as pausing reconciliation or stopping the cluster.
See xref:concepts:operations/cluster_operations.adoc[cluster operations] for more details.
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@

= Allowed Pod disruptions

You can configure the permitted Pod disruptions for NiFi nodes as described in xref:concepts:operations/pod_disruptions.adoc[].
Original file line number Diff line number Diff line change
@@ -30,7 +30,7 @@ nodes:

In the above example, all nodes in the default group will request `12Gi` of storage the various folders.

== Resource Requests
== Resource requests

include::home:concepts:stackable_resource_requests.adoc[]

Loading

0 comments on commit c4c9026

Please sign in to comment.