Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs maintenance #625

Merged
merged 23 commits into from
Jun 17, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
bd0d592
A bunch of small changes
Jun 4, 2024
9453c18
Add link to guide
Jun 4, 2024
70c3e85
more small changes
Jun 5, 2024
c115996
mention Pod overrides
Jun 5, 2024
eaa6e81
use tab blocks for install instructions
Jun 5, 2024
aad3132
Update docs/modules/nifi/pages/usage_guide/configuration-environment-…
fhennig Jun 11, 2024
38e016d
Update docs/modules/nifi/pages/usage_guide/configuration-environment-…
fhennig Jun 11, 2024
cc6c762
Update docs/modules/nifi/pages/usage_guide/updating.adoc
fhennig Jun 11, 2024
8bb0bf8
Update docs/modules/nifi/pages/usage_guide/zookeeper-connection.adoc
fhennig Jun 11, 2024
24d3a08
Update docs/modules/nifi/pages/usage_guide/zookeeper-connection.adoc
fhennig Jun 11, 2024
c96fe47
Update docs/modules/nifi/pages/usage_guide/security.adoc
fhennig Jun 11, 2024
b6cf8a0
Update docs/modules/nifi/pages/usage_guide/operations/cluster-operati…
fhennig Jun 11, 2024
94114fc
Update docs/modules/nifi/pages/usage_guide/extra-volumes.adoc
fhennig Jun 11, 2024
fc4b3dc
Update docs/modules/nifi/pages/usage_guide/configuration-environment-…
fhennig Jun 11, 2024
05437d1
Update docs/modules/nifi/pages/usage_guide/custom_processors.adoc
fhennig Jun 11, 2024
25845fe
Update docs/modules/nifi/pages/usage_guide/custom_processors.adoc
fhennig Jun 11, 2024
8e8b71b
Update docs/modules/nifi/pages/usage_guide/configuration-environment-…
fhennig Jun 11, 2024
12e8e73
Update docs/modules/nifi/pages/usage_guide/custom_processors.adoc
fhennig Jun 11, 2024
cbad64a
Update docs/modules/nifi/pages/usage_guide/external_ports.adoc
fhennig Jun 11, 2024
921518f
Update installation.adoc
fhennig Jun 11, 2024
35da3e8
Update installation.adoc
fhennig Jun 11, 2024
afa2212
Merge branch 'main' into docs-maintenance
fhennig Jun 17, 2024
40af56b
Update docs/modules/nifi/pages/getting_started/installation.adoc
fhennig Jun 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
3 changes: 2 additions & 1 deletion docs/modules/nifi/pages/getting_started/first_steps.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,8 @@ The xref:zookeeper:znodes.adoc[ZNode] makes sure that the NiFi cluster will oper

=== Apache NiFi

The NiFi cluster requires authentication. Create a set of credentials for this purpose:
The NiFi cluster requires authentication.
Create a set of credentials for this purpose:

[source,bash]
----
Expand Down
3 changes: 2 additions & 1 deletion docs/modules/nifi/pages/getting_started/index.adoc
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
= Getting started

This guide will get you started with Apache NiFi using the Stackable operator. It will guide you through the installation of the operator and its dependencies, setting up your first NiFi cluster.
This guide will get you started with Apache NiFi using the Stackable operator.
It will guide you through the installation of the operator and its dependencies, setting up your first NiFi cluster.

== Prerequisites

Expand Down
37 changes: 22 additions & 15 deletions docs/modules/nifi/pages/getting_started/installation.adoc
Original file line number Diff line number Diff line change
@@ -1,25 +1,28 @@
= Installation

On this page you will install the Stackable Operator for Apache NiFi and operators for its dependencies - ZooKeeper -
as well as the commons, secret and listener operator which are required by all Stackable Operators.
On this page you will install the Stackable operator for Apache NiFi and operators for its dependencies - ZooKeeper -
as well as the commons, secret and listener operator which are required by all Stackable operators.

== Stackable Operators
== Stackable operators

There are two ways to install Stackable Operators:
There are two ways to install Stackable operators:

. Using xref:management:stackablectl:index.adoc[stackablectl]
. Using Helm

=== stackablectl
* Using xref:management:stackablectl:index.adoc[stackablectl]
* Using Helm

[tabs]
====
stackablectl::
+
--
The `stackablectl` command line tool is the recommended way to interact with operators and dependencies. Follow the
xref:management:stackablectl:installation.adoc[installation steps] for your platform if you choose to work with
`stackablectl`.

After you have installed `stackablectl` and have a Kubernetes cluster up and running, run the following command to
install all operators necessary for NiFi:

[source,bash]
[source,shell]
----
include::example$getting_started/getting_started.sh[tag=stackablectl-install-operators]
----
Expand All @@ -32,25 +35,29 @@ include::example$getting_started/install-operator-output.txt[tag=stackablectl-in
----

TIP: Consult the xref:management:stackablectl:quickstart.adoc[] to learn more about how to use `stackablectl`.
--

=== Helm

Helm::
+
--
You can also use Helm to install the operators. Add the Stackable Helm repository:

[source,bash]
[source,shell]
----
include::example$getting_started/getting_started.sh[tag=helm-add-repo]
----

Then install the Stackable Operators:
Then install the Stackable operators:

[source,bash]
----
include::example$getting_started/getting_started.sh[tag=helm-install-operators]
----

Helm will deploy the operators in a Kubernetes Deployment and apply the CRDs for the Apache NiFi service (as well as the
CRDs for the required operators). You are now ready to deploy Apache NiFi in Kubernetes.
Helm will deploy the operators in a Kubernetes Deployment and apply the CRDs for the Apache NiFi service (as well as the CRDs for the required operators).
You are now ready to deploy Apache NiFi in Kubernetes.
--
====

== What's next

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
= Command Line parameters
= Command line parameters

This operator accepts the following command line parameters:

Expand Down
4 changes: 2 additions & 2 deletions docs/modules/nifi/pages/reference/environment-variables.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ export PRODUCT_CONFIG=/foo/bar/properties.yaml
stackable-nifi-operator run
----

or via docker:
or via Docker:

----
docker run \
Expand Down Expand Up @@ -44,7 +44,7 @@ export WATCH_NAMESPACE=test
stackable-nifi-operator run
----

or via docker:
or via Docker:

[source]
----
Expand Down
Original file line number Diff line number Diff line change
@@ -1,22 +1,27 @@
= Configuration & Environment Overrides
= Configuration & environment overrides
:nifi-docs: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties
:java-security-docs: https://docs.oracle.com/en/java/javase/11/security/java-security-overview1.html

The cluster definition also supports overriding configuration properties and environment variables, either per role or per role group, where the more specific override (role group) has precedence over the less specific one (role).
The NiFi cluster definition supports overriding configuration properties, environment variables, and Pod attributes. The configuration overrides can be applied either per role, or per role group where the more specific override (role group) has precedence over the less specific one (role).

IMPORTANT: Do not override port numbers.
This will lead to cluster malfunction.

== Configuration Overrides
== Configuration overrides

Apache NiFi runtime configuration is stored in several files listed below. The `configOverrides` block enables you to customize parameters in these files.
The complete list of the configuration options can be found in the https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties[Apache NiFi documentation].
Apache NiFi runtime configuration is stored in several files listed below.
The `configOverrides` block enables you to customize parameters in these files.
The complete list of the configuration options can be found in the {nifi-docs}[Apache NiFi documentation].

The following files can be edited directly via the `configOverrides` mechanism:

- `bootstrap.conf`
- `nifi.properties`
- `security.properties`
* `bootstrap.conf`
* `nifi.properties`
* `security.properties`

Overrides are key, value pairs defined under one of the configuration files from the list above. They must match the names values as expected by NiFi. In the example below, a property `nifi.flow.configuration.archive.enabled` is being explicitly set to 'false', overriding the default value.
Overrides are key-value pairs defined under one of the configuration files from the list above.
They must match the property names as expected by NiFi.
In the example below, a property `nifi.flow.configuration.archive.enabled` is being explicitly set to 'false', overriding the default value.

The following snippet shows how to disable workflow file backups in the NifiCluster definition:

Expand All @@ -32,9 +37,13 @@

=== The security.properties file

The `security.properties` file is used to configure JVM security properties. It is very seldom that users need to tweak any of these, but there is one use-case that stands out, and that users need to be aware of: the JVM DNS cache.
The `security.properties` file is used to configure JVM security properties.
It is very rare that you need to tweak any of these, but there is one use-case that stands out, and that users need to be aware of: the JVM DNS cache.

The JVM manages its own cache of successfully resolved host names as well as a cache of host names that cannot be resolved. Some products of the Stackable platform are very sensible to the contents of these caches and their performance is heavily affected by them. As of version 1.21.0 Apache Nifi performs poorly if the positive cache is disabled. To cache resolved host names, and thus increase the performance your Nifi cluster, you can configure the TTL of entries in the positive cache like this:
The JVM manages its own cache of successfully resolved host names as well as a cache of host names that cannot be resolved.
Some products of the Stackable platform are very sensitive to the contents of these caches and their performance is heavily affected by them.
As of version 1.21.0, Apache Nifi performs poorly if the positive cache is disabled.

Check notice on line 45 in docs/modules/nifi/pages/usage_guide/configuration-environment-overrides.adoc

View workflow job for this annotation

GitHub Actions / LanguageTool

[LanguageTool] docs/modules/nifi/pages/usage_guide/configuration-environment-overrides.adoc#L45

Possible spelling mistake found. (EN_MULTITOKEN_SPELLING_TWO[2]) Suggestions: `Apache NiFi` Rule: https://community.languagetool.org/rule/show/EN_MULTITOKEN_SPELLING_TWO?lang=en-US&subId=2 Category: MULTITOKEN_SPELLING
Raw output
docs/modules/nifi/pages/usage_guide/configuration-environment-overrides.adoc:45:22: Possible spelling mistake found. (EN_MULTITOKEN_SPELLING_TWO[2])
 Suggestions: `Apache NiFi`
 Rule: https://community.languagetool.org/rule/show/EN_MULTITOKEN_SPELLING_TWO?lang=en-US&subId=2
 Category: MULTITOKEN_SPELLING
To cache resolved host names, and thus increase the performance your Nifi cluster, you can configure the TTL of entries in the positive cache like this:

[source,yaml]
----
Expand All @@ -47,9 +56,9 @@

NOTE: The operator configures DNS caching by default as shown in the example above.

For details on the JVM security see https://docs.oracle.com/en/java/javase/11/security/java-security-overview1.html
For details on the JVM security, consult the {java-security-docs}[Java security overview].

== Environment Variables
== Environment variables

Environment variables can be (over)written by adding the `envOverrides` property.

Expand Down Expand Up @@ -78,3 +87,9 @@
config: {}
replicas: 1
----

== Pod overrides

Pod overrides allow you to configure any attributes that can be configured on a Pod, such as tolerations or labels on the Pod.

Read the xref:concepts:overrides.adoc#pod-overrides[Pod overrides concepts page] to learn more.
31 changes: 17 additions & 14 deletions docs/modules/nifi/pages/usage_guide/custom_processors.adoc
Original file line number Diff line number Diff line change
@@ -1,20 +1,21 @@
= Loading Custom Components
= Loading custom components
:nifi-docs-custom-components: https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#introduction

It is possible to develop https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#introduction[custom components] for Apache NiFi to extend the available functionality - most often this will probably be custom processors.
It is possible to develop {nifi-docs-custom-components}[custom components] for Apache NiFi to extend the available functionality - most often this will probably be custom processors.

For these to be available in the NiFi UI, they need to be present in the classpath at startup, so the nar bundles have to be injected into your NiFi instance.

The Stackable Data Platform supports two ways to achive this goal, both of which are described below.

Please note that building your own Docker image is the recommended solution, as giving multiple pods access to a shared PVC can often be tricky (see comment on access mode in the second section).

== Custom Docker Image
== Custom Docker image

You can extend the official Stackable NiFi image and simply copy all required nar files into the classpath of Nifi.
The benefit of this method is that there is no need for any config overrides or extra mounts, you can simply use any ouf our NiFi examples, swap the image and your components will be available.
You can extend the official Stackable NiFi image and copy all required _nar_ files into the classpath of Nifi.
The benefit of this method is that there is no need for any config overrides or extra mounts, you can use any ouf our NiFi examples, swap the image and your components will be available.
But this means you will need to have access to a registry to push your custom image to.

A simple Dockerfile would look like show in the following listing.
The basic Dockerfile below shows how to achieve this:

[source,Dockerfile]
----
Expand All @@ -32,11 +33,13 @@ spec:
custom: "docker.company.org/nifi:1.25.0-customprocessor"
----

== Using the Official Image
If you don't want to create a custom image, then you can use the extra volume mount functionality to load components and configure NiFi with to read these from the extra volumes.
Also read the xref:guides:custom-images.adoc[Using customized product images] guide for additional information.

For this to work you'll need to prepare a PVC containing your components.
Usually the best way to do this will be to mount the PVC into a temporary container and then simply `kubectl cp` the nar files into that volume.
== Using the official image
If you don't want to create a custom image or don't have access to an image registry, you can use the extra volume mount functionality to mount a volume containing your custom components and configure NiFi to read these from the mounted volumes.

For this to work you'll need to prepare a PersistentVolumeClaim (PVC) containing your components.
Usually the best way to do this will be to mount the PVC into a temporary container and then `kubectl cp` the _nar_ files into that volume.

The following listing shows an example of creating the PVC and the Pod:

Expand Down Expand Up @@ -74,7 +77,7 @@ spec:
name: nifi-processor
----

<1> Please note that this access mode will mean that you can only use this volume with a single NiFi pod. For a distributed scenario, an additional list value of `ReadOnlyMany` could be specified here if this is supported.
<1> Please note that this access mode will mean that you can only use this volume with a single NiFi Pod. For a distributed scenario, an additional list value of `ReadOnlyMany` could be specified here if this is supported.

The command to then copy the nar bundle into the PVC is:

Expand All @@ -83,9 +86,9 @@ The command to then copy the nar bundle into the PVC is:
kubectl cp /path/to/component.nar processorcopy:/volume/
----

Now you can mount the extra volume into your NiFi instance as described in xref:nifi:usage_guide/extra-volumes.adoc[] .
Now you can mount the extra volume into your NiFi instance as described in xref:nifi:usage_guide/extra-volumes.adoc[].

After this is done your components will be available within the NiFi pods and NiFi can be configured to load these at startup by adding an extra directory to the classpath:
After this is done your components will be available within the NiFi Pods and NiFi can be configured to load these at startup by adding an extra directory to the classpath:


[source,yaml]
Expand Down Expand Up @@ -120,4 +123,4 @@ spec:
----

<1> Specify your prepared PVC here.
<2> The directory name after `userdata` has to match the name of the volume, while `mycustomlibs` is a name you can freely change.
<2> The directory name after `userdata` has to match the name of the volume, while `myCustomLibs` is a name you can freely change.
13 changes: 6 additions & 7 deletions docs/modules/nifi/pages/usage_guide/external_ports.adoc
Original file line number Diff line number Diff line change
@@ -1,19 +1,18 @@
= Expose NiFi processor ports
= Exposing NiFi processor ports

== Exposing processors
Some NiFi processors allow external tools to connect to them to trigger workflows or send data.
For this to work from outside the Kubernetes cluster NiFi has been deployed to, it is necessary to create https://kubernetes.io/docs/concepts/services-networking/service/[Service] or https://kubernetes.io/docs/concepts/services-networking/ingress/[Ingress] objects to configure Kubernetes routes for these ports.
For this to work from outside the Kubernetes cluster NiFi has been deployed to, it is necessary to create Service or Ingress objects to configure Kubernetes routes for these ports.

Stackable doesn't place any restriction on the type of service that can be used here.

== Example

A simple example for this is shown below.
An example for this is shown below.
It consists of a NiFi cluster with three nodes called `simple-nifi` and has a processor that listens to a TCP port and forwards lines written into that port as flowfiles.

image:https://user-images.githubusercontent.com/1070361/212958154-941cef1d-e370-4d08-b37d-b789b242c062.png[image]
image:listening-processor-example.png[A ListenTCP processor]

The processor has been configured to listen on port 8123, so this port is available on all NiFi pods for processes that run inside the Kubernetes cluster.
The processor has been configured to listen on port 8123, so this port is available on all NiFi Pods for processes that run inside the Kubernetes cluster.

In order to make this port available to external processes as well, a LoadBalancer type service can be used for example:

Expand All @@ -39,7 +38,7 @@ spec:
Depending on the Kubernetes configuration this snippet will request and external ip address and open port 8080 on that ip address.
Any requests to this port will be forwarded to port 8123 on the NiFi pods - and thus the processor.

If LoadBalancer services are not available, a NodePort service would be an alternative:
If LoadBalancer Services or Ingress resources are not available for use, a NodePort Service would be an alternative:

[source,yaml]
----
Expand Down
4 changes: 2 additions & 2 deletions docs/modules/nifi/pages/usage_guide/extra-volumes.adoc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
= Adding External Files to the NiFi Servers
= Adding external files to the NiFi servers

Since Apache NiFi allows executing pretty much arbitrary workflows depending on which processors are used, it may become necessary to add external files to the pods.
Since Apache NiFi allows executing arbitrary workflows depending on which processors are used, it may become necessary to add external files to the Pods.
These could for example be client certificates used to configure a `PollHTTP` processor, a keytab to obtain a Kerberos ticket, or similar things.

In order to make these files available the operator allows specifying extra volumes that will be added to the NiFi pods.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@

= Cluster operation

Apache NiFi installations can be configured with different cluster operations like pausing reconciliation or stopping the cluster. See xref:concepts:operations/cluster_operations.adoc[cluster operations] for more details.
Apache NiFi installations can be configured with different cluster operations such as pausing reconciliation or stopping the cluster.
See xref:concepts:operations/cluster_operations.adoc[cluster operations] for more details.
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@

= Allowed Pod disruptions

You can configure the permitted Pod disruptions for NiFi nodes as described in xref:concepts:operations/pod_disruptions.adoc[].
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ nodes:

In the above example, all nodes in the default group will request `12Gi` of storage the various folders.

== Resource Requests
== Resource requests

include::home:concepts:stackable_resource_requests.adoc[]

Expand Down
Loading
Loading