From f146952fc4a32ad3aedefa27916c12397599ee5d Mon Sep 17 00:00:00 2001 From: Dale Date: Mon, 25 Nov 2019 22:10:04 +1100 Subject: [PATCH 1/6] METRON-2088 - Support for SOLR time-based arrays --- .../metron-solr/metron-solr-common/README.md | 119 ++++++++++++++++++ .../src/main/config/schema/bro/schema.xml | 3 + .../src/main/config/schema/error/schema.xml | 1 + .../main/config/schema/metaalert/schema.xml | 3 + .../src/main/config/schema/snort/schema.xml | 3 + .../src/main/config/schema/yaf/schema.xml | 3 + .../src/main/scripts/create_configset.sh | 54 ++++++++ 7 files changed, 186 insertions(+) create mode 100755 metron-platform/metron-solr/metron-solr-common/src/main/scripts/create_configset.sh diff --git a/metron-platform/metron-solr/metron-solr-common/README.md b/metron-platform/metron-solr/metron-solr-common/README.md index 88f74375db..ba01dfacc8 100644 --- a/metron-platform/metron-solr/metron-solr-common/README.md +++ b/metron-platform/metron-solr/metron-solr-common/README.md @@ -164,3 +164,122 @@ The `create_collection.sh` script depends on schemas installed in `$METRON_HOME/ Additional schemas should be installed in that location if using the `create_collection.sh` script. Any collection can be deleted with the `delete_collection.sh` script. These scripts use the [Solr Collection API](http://lucene.apache.org/solr/guide/7_4/collections-api.html). + +## Time routed alias support +An alias is a pointer that points to a Collection. Sending a document to an alias sends it to the collection the alias points too. +The collection an alias points to can be changed with a single, low-cost operation. Time Routed Aliases (TRAs) is a SolrCloud feature +that manages an alias and a time sequential series of collections. + +A TRA automatically creates new collections and (optionally) deletes old ones as it routes documents to the correct collection +based on the timestamp of the event. This approach allows for indefinite indexing of data without degradation of performance otherwise +experienced due to the continuous growth of a single index. + +A TRA is defined with a minimum time and a defined interval period and SOLR provides a collection for each interval for a +contiguous set of datetime intervals from the start date to the maximum received document date. Collections are created to host documents based on examining the document's event-time. If a document does not currently +have a collection created for it, then starting at the minimum date SOLR will create a collection for each interval that does not have one + up until the interval period needed to store the current document. + +See SOLR documentation [\(1\)](https://lucene.apache.org/solr/guide/7_4/time-routed-aliases.html) +[\(2\)](https://lucene.apache.org/solr/guide/7_4/collections-api.html#createalias) for more information. + +### Setting up Time routed alias support + +Using SOLR's Time-based routing requires using SOLR's native Datetime types. At the moment, Metron uses the LongTrie field type +to store dates, which is not a SOLR native datetime type. At a later stage the Metron code-base will be changed to use SOLR native datetime types, +but for now a workaround procedure has been created to allow for the use of Time-based routing, but still allows Metron to use LongTrie type. +This procedure only works for new collections, and is as follows: + +1. Add the following field type definition near the end of the schema.xml document (the entry must be inside the tags) + ``` + + ``` + +1. Add the following field definition near the start of the schema.xml document (the entry must be inside the /schema> tags) + ``` + + ``` + +1. Create the configset for the collection: Assuming that the relevant collections schema.xml and solrconfig.xml are located in +$METRON_HOME/config/schema/ folder, use the following command: + ``` + $METRON_HOME/bin/create_configset + ``` + +1. Create the time-based routing alias for the collection: +Assuming the following values: + * ALIAS_NAME: Name of the new alias + * ROUTER_START: Beginning time-period datetime in ISO-8601 standard - milliseconds potion of the date must be 0, some examples are +'2018-01-14T21:00:00:00', 'NOW/SECOND', 'NOW/DAY' + * ROUTER_FIELD: The name of the field in the incoming document that contains the datetime to route on - field must be of SOLR type DateTrie or DatePoint. + For METRON this is standardised as field `datetime`. + * ROUTER_INTERVAL: SOLR Date math format. The interval of time that each collection holds. eg "+1DAY", "+6HOUR", "+1WEEK" (`+` must be URL encoded to `%2B` ) + * ROUTER_MAXFUTUREMS: Optional field containing the number of milliseconds into the future that it is considered valid to have an event time for. + Documents with event time exceeding this time period in the future are considered invalid and an error is returned. Used as a sanity check to prevent + the creation of unnecessary collections due to corrupted datetimes in events. Defaults to 10 minutes into the future. + * ROUTER_AUTODELETEAGE: Optional field in SOLR Date math format. If this field is present, any time a collection is created, + the oldest collections are assessed for deletion. Collections are deleted if the datetime interval they represent is older then + NOW - AUTODELETE_INTERVAL. eg -2WEEK, -3MONTH, -1YEAR + * CONFIGSET: Name of the colleciton configset that was created in the previous step - this is used a template for new collections. + * create-collection.*: These allow for Create collection options (e.g. numShards or numReplicas) to be specified directly in the + create alias command. + + Then the following command will create a time-routed alias. + ``` + curl http://:8983/solr/admin/collections?action=CREATEALIAS\ + &name=\ + &router.start=\ + &router.field=\ + &router.name=time\ + &router.interval=\ + &router.maxFutureMs=\ + &create-collection.collection.configName=\ + &create-collection.numShards=2 + ``` +1. Add a Metron Parser Stellar field transformation to the parser config that adds a correctly formatted datetime string to the event as it is being parsed. + 1. Set environment variables for later reference + ``` + source /etc/defaults/metron + export HDP_HOME="/usr/hdp/current" + export PARSER_NAME= + ``` + 1. Pull the most recent sensor parser config from zookeeper + ``` + ${METRON_HOME}/bin/zk_load_configs.sh -i ${METRON_HOME}/config/zookeeper -m PULL -c PARSER -n $PARSER_NAME -z $ZOOKEEPER + ``` + 1. Open the file to the relevant sensor parser at `$METRON_HOME/config/zookeeper/parsers/$PARSER_NAME.json` + + 1. Add to the sensor parser config json field the following transformation + ``` + "fieldTransformations" : [{ + input + "transformation" : "STELLAR" + ,"output" : [ "datetime" ] + ,"config" : { + "datetime" : "DATE_FORMAT("yyyy-MM-dd'T'HH:mm:ss.SSSX",timestamp)" + } + }] + ``` + 1. Push the configuration back to zookeeper + ``` + ${METRON_HOME}/bin/zk_load_configs.sh -i ${METRON_HOME}/config/zookeeper -m PUSH -c PARSER -n $PARSER_NAME -z $ZOOKEEPER + ``` + 1. Run kafka console to monitor correct operation of the field transformation + ``` + ${HDP_HOME}/kafka-broker/bin/kafka-console-consumer.sh --bootstrap-server $BROKERLIST --topic $PARSER_NAME + ``` + +1. Config Metron SOLR indexing to push documents to the newly created Collection Alias. + 1. Pull the most recent index config from zookeeper + ``` + ${METRON_HOME}/bin/zk_load_configs.sh -i ${METRON_HOME}/config/zookeeper -m PULL -c INDEXING -n $PARSER_NAME -z $ZOOKEEPER + ``` + 1. Edit the file ${METRON_HOME}/config/zookeeper/indexing/$PARSER_NAME.json + 1. Update the solr/index field to the `ALIAS_NAME` value you configured for the SOLR time-based routing alias. + 1. Push the configuration back to zookeeper + ``` + ${METRON_HOME}/bin/zk_load_configs.sh -i ${METRON_HOME}/config/zookeeper -m PUSH -c INDEXING -n $PARSER_NAME -z $ZOOKEEPER + ``` + +1. + + diff --git a/metron-platform/metron-solr/metron-solr-common/src/main/config/schema/bro/schema.xml b/metron-platform/metron-solr/metron-solr-common/src/main/config/schema/bro/schema.xml index 6be76a0e43..81de9576a1 100644 --- a/metron-platform/metron-solr/metron-solr-common/src/main/config/schema/bro/schema.xml +++ b/metron-platform/metron-solr/metron-solr-common/src/main/config/schema/bro/schema.xml @@ -36,6 +36,8 @@ + + guid @@ -697,5 +699,6 @@ + diff --git a/metron-platform/metron-solr/metron-solr-common/src/main/config/schema/error/schema.xml b/metron-platform/metron-solr/metron-solr-common/src/main/config/schema/error/schema.xml index 4aa80efc96..743753b201 100644 --- a/metron-platform/metron-solr/metron-solr-common/src/main/config/schema/error/schema.xml +++ b/metron-platform/metron-solr/metron-solr-common/src/main/config/schema/error/schema.xml @@ -55,6 +55,7 @@ + diff --git a/metron-platform/metron-solr/metron-solr-common/src/main/config/schema/metaalert/schema.xml b/metron-platform/metron-solr/metron-solr-common/src/main/config/schema/metaalert/schema.xml index 6555bf61d7..154a3d7c3d 100644 --- a/metron-platform/metron-solr/metron-solr-common/src/main/config/schema/metaalert/schema.xml +++ b/metron-platform/metron-solr/metron-solr-common/src/main/config/schema/metaalert/schema.xml @@ -26,6 +26,8 @@ + + @@ -58,5 +60,6 @@ + \ No newline at end of file diff --git a/metron-platform/metron-solr/metron-solr-common/src/main/config/schema/snort/schema.xml b/metron-platform/metron-solr/metron-solr-common/src/main/config/schema/snort/schema.xml index 3c57574a0b..dd66bd9eb8 100644 --- a/metron-platform/metron-solr/metron-solr-common/src/main/config/schema/snort/schema.xml +++ b/metron-platform/metron-solr/metron-solr-common/src/main/config/schema/snort/schema.xml @@ -21,6 +21,8 @@ + + @@ -90,5 +92,6 @@ + diff --git a/metron-platform/metron-solr/metron-solr-common/src/main/config/schema/yaf/schema.xml b/metron-platform/metron-solr/metron-solr-common/src/main/config/schema/yaf/schema.xml index 37e5f12d7b..9665e135fb 100644 --- a/metron-platform/metron-solr/metron-solr-common/src/main/config/schema/yaf/schema.xml +++ b/metron-platform/metron-solr/metron-solr-common/src/main/config/schema/yaf/schema.xml @@ -21,6 +21,8 @@ + + @@ -96,5 +98,6 @@ + diff --git a/metron-platform/metron-solr/metron-solr-common/src/main/scripts/create_configset.sh b/metron-platform/metron-solr/metron-solr-common/src/main/scripts/create_configset.sh new file mode 100755 index 0000000000..1a46e53ecb --- /dev/null +++ b/metron-platform/metron-solr/metron-solr-common/src/main/scripts/create_configset.sh @@ -0,0 +1,54 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +METRON_VERSION=${project.version} +METRON_HOME=/usr/metron/$METRON_VERSION +ZOOKEEPER=${ZOOKEEPER:-localhost:2181} +ZOOKEEPER_HOME=${ZOOKEEPER_HOME:-/usr/hdp/current/zookeeper-client} +SECURITY_ENABLED=${SECURITY_ENABLED:-false} +NEGOTIATE='' +if [ ${SECURITY_ENABLED,,} == 'true' ]; then + NEGOTIATE=' --negotiate -u : ' +fi + +# test for errors in SOLR URL +if [[ ${SOLR_NODE} =~ .*:null ]]; then + echo "Error occurred while attempting to read SOLR Cloud configuration data from Zookeeper."; + if ! [[ ${ZOOKEEPER} =~ .*/solr ]]; then + echo "Warning! Environment variable ZOOKEEPER=$ZOOKEEPER does not contain a chrooted zookeeper ensemble address - are you sure you do not mean ZOOKEEPER=$ZOOKEEPER/solr?"; + fi + exit 1; +fi + + +# test for presence of datetime field in schema collection +DQT='"' +DATETIME_SCHEMA=" Date: Mon, 25 Nov 2019 22:21:51 +1100 Subject: [PATCH 2/6] METRON-2088: Removed some brackets that was causing the markdown to HTML converter some grief --- .../metron-solr/metron-solr-common/README.md | 23 ++++++++++--------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/metron-platform/metron-solr/metron-solr-common/README.md b/metron-platform/metron-solr/metron-solr-common/README.md index ba01dfacc8..fe10b5e631 100644 --- a/metron-platform/metron-solr/metron-solr-common/README.md +++ b/metron-platform/metron-solr/metron-solr-common/README.md @@ -189,24 +189,25 @@ to store dates, which is not a SOLR native datetime type. At a later stage the but for now a workaround procedure has been created to allow for the use of Time-based routing, but still allows Metron to use LongTrie type. This procedure only works for new collections, and is as follows: -1. Add the following field type definition near the end of the schema.xml document (the entry must be inside the tags) +1. Add the following field type definition near the end of the schema.xml document (the entry must be inside the schema tags) ``` ``` -1. Add the following field definition near the start of the schema.xml document (the entry must be inside the /schema> tags) +1. Add the following field definition near the start of the schema.xml document (the entry must be inside the schema tags) ``` ``` 1. Create the configset for the collection: Assuming that the relevant collections schema.xml and solrconfig.xml are located in -$METRON_HOME/config/schema/ folder, use the following command: +`$METRON_HOME/config/schema/` folder, use the following command: ``` - $METRON_HOME/bin/create_configset + $METRON_HOME/bin/create_configset collectΩΩion name> ``` 1. Create the time-based routing alias for the collection: Assuming the following values: + * SOLR_HOST: Host SOLR is installed on * ALIAS_NAME: Name of the new alias * ROUTER_START: Beginning time-period datetime in ISO-8601 standard - milliseconds potion of the date must be 0, some examples are '2018-01-14T21:00:00:00', 'NOW/SECOND', 'NOW/DAY' @@ -225,14 +226,14 @@ Assuming the following values: Then the following command will create a time-routed alias. ``` - curl http://:8983/solr/admin/collections?action=CREATEALIAS\ - &name=\ - &router.start=\ - &router.field=\ + curl http://$SOLR_HOST:8983/solr/admin/collections?action=CREATEALIAS\ + &name=$ALIAS_NAME\ + &router.start=$ROUTER_START\ + &router.field=$ROUTER_FIELD\ &router.name=time\ - &router.interval=\ - &router.maxFutureMs=\ - &create-collection.collection.configName=\ + &router.interval=$ROUTER_INTERVAL\ + &router.maxFutureMs=$ROUTER_MAXFUTUREMS\ + &create-collection.collection.configName=$CONFIGSET\ &create-collection.numShards=2 ``` 1. Add a Metron Parser Stellar field transformation to the parser config that adds a correctly formatted datetime string to the event as it is being parsed. From b9bc03703f1a5bf7445b13ab1c5f703a1aacf2d1 Mon Sep 17 00:00:00 2001 From: Dale Date: Mon, 25 Nov 2019 22:52:52 +1100 Subject: [PATCH 3/6] METRON-2327 - Minor formatting nits to keep th markdown to html encoder producing nicer html --- .../metron-solr/metron-solr-common/README.md | 83 ++++++++++++------- 1 file changed, 51 insertions(+), 32 deletions(-) diff --git a/metron-platform/metron-solr/metron-solr-common/README.md b/metron-platform/metron-solr/metron-solr-common/README.md index fe10b5e631..924ec598bb 100644 --- a/metron-platform/metron-solr/metron-solr-common/README.md +++ b/metron-platform/metron-solr/metron-solr-common/README.md @@ -184,72 +184,87 @@ See SOLR documentation [\(1\)](https://lucene.apache.org/solr/guide/7_4/time-rou ### Setting up Time routed alias support -Using SOLR's Time-based routing requires using SOLR's native Datetime types. At the moment, Metron uses the LongTrie field type -to store dates, which is not a SOLR native datetime type. At a later stage the Metron code-base will be changed to use SOLR native datetime types, -but for now a workaround procedure has been created to allow for the use of Time-based routing, but still allows Metron to use LongTrie type. -This procedure only works for new collections, and is as follows: +Using SOLR's tme-based routing requires using SOLR's native datetime types. At the moment, Metron uses the LongTrie field type +to store dates, which is not a SOLR native datetime type. At a later stage the Metron code-base will be changed to use SOLR native datetime types +(as the LongTrie type is deprecated), but for now a workaround procedure has been created to allow for the use of time-based routing, while at the + same time allowing for Metron to continue to use the LongTrie type. This procedure only works for new collections, and is as follows: 1. Add the following field type definition near the end of the schema.xml document (the entry must be inside the schema tags) ``` ``` + 1. Add the following field definition near the start of the schema.xml document (the entry must be inside the schema tags) ``` ``` + 1. Create the configset for the collection: Assuming that the relevant collections schema.xml and solrconfig.xml are located in -`$METRON_HOME/config/schema/` folder, use the following command: +`$METRON_HOME/config/schema/$COLLECTION_NAME` folder, use the following command: ``` - $METRON_HOME/bin/create_configset collectΩΩion name> + $METRON_HOME/bin/create_configset $COLLECTION_NAME ``` + 1. Create the time-based routing alias for the collection: Assuming the following values: * SOLR_HOST: Host SOLR is installed on + * ALIAS_NAME: Name of the new alias + * ROUTER_START: Beginning time-period datetime in ISO-8601 standard - milliseconds potion of the date must be 0, some examples are '2018-01-14T21:00:00:00', 'NOW/SECOND', 'NOW/DAY' + * ROUTER_FIELD: The name of the field in the incoming document that contains the datetime to route on - field must be of SOLR type DateTrie or DatePoint. For METRON this is standardised as field `datetime`. + * ROUTER_INTERVAL: SOLR Date math format. The interval of time that each collection holds. eg "+1DAY", "+6HOUR", "+1WEEK" (`+` must be URL encoded to `%2B` ) + * ROUTER_MAXFUTUREMS: Optional field containing the number of milliseconds into the future that it is considered valid to have an event time for. - Documents with event time exceeding this time period in the future are considered invalid and an error is returned. Used as a sanity check to prevent - the creation of unnecessary collections due to corrupted datetimes in events. Defaults to 10 minutes into the future. + Documents with an event time exceeding this time period in the future are considered invalid and an error is returned. Used as a sanity check to prevent + the creation of unnecessary collections due to corrupted datetime values in events. Defaults is to ignore anything more then 10 minutes into the future. + * ROUTER_AUTODELETEAGE: Optional field in SOLR Date math format. If this field is present, any time a collection is created, the oldest collections are assessed for deletion. Collections are deleted if the datetime interval they represent is older then - NOW - AUTODELETE_INTERVAL. eg -2WEEK, -3MONTH, -1YEAR - * CONFIGSET: Name of the colleciton configset that was created in the previous step - this is used a template for new collections. - * create-collection.*: These allow for Create collection options (e.g. numShards or numReplicas) to be specified directly in the + NOW - AUTODELETE_INTERVAL. eg -2WEEK, -3MONTH, -1YEAR. (`-` is a valid URL character that does not need to URL encoded.) + + * CONFIGSET: Name of the collection configset that was created in the previous step - this is used a template for new collections. + + * CREATE-COLLECTION.*: These allow for Create collection options (e.g. numShards or numReplicas) to be specified directly in the create alias command. - Then the following command will create a time-routed alias. + Then the following command will create a time-routed alias: ``` - curl http://$SOLR_HOST:8983/solr/admin/collections?action=CREATEALIAS\ - &name=$ALIAS_NAME\ - &router.start=$ROUTER_START\ - &router.field=$ROUTER_FIELD\ - &router.name=time\ - &router.interval=$ROUTER_INTERVAL\ - &router.maxFutureMs=$ROUTER_MAXFUTUREMS\ - &create-collection.collection.configName=$CONFIGSET\ - &create-collection.numShards=2 + curl http://$SOLR_HOST:8983/solr/admin/collections?action=CREATEALIAS\ + &name=$ALIAS_NAME\ + &router.start=$ROUTER_START\ + &router.field=$ROUTER_FIELD\ + &router.name=time\ + &router.interval=$ROUTER_INTERVAL\ + &router.maxFutureMs=$ROUTER_MAXFUTUREMS\ + &create-collection.collection.configName=$CONFIGSET\ + &create-collection.numShards=2 ``` -1. Add a Metron Parser Stellar field transformation to the parser config that adds a correctly formatted datetime string to the event as it is being parsed. + + +1. Add a Metron Parser Stellar field transformation to the parser config that adds a correctly formatted datetime string to the event as it is being parsed: 1. Set environment variables for later reference ``` source /etc/defaults/metron export HDP_HOME="/usr/hdp/current" - export PARSER_NAME= + export PARSER_NAME= ``` + 1. Pull the most recent sensor parser config from zookeeper ``` ${METRON_HOME}/bin/zk_load_configs.sh -i ${METRON_HOME}/config/zookeeper -m PULL -c PARSER -n $PARSER_NAME -z $ZOOKEEPER ``` + 1. Open the file to the relevant sensor parser at `$METRON_HOME/config/zookeeper/parsers/$PARSER_NAME.json` - 1. Add to the sensor parser config json field the following transformation + 1. Add to the sensor parser config json field the following transformation: ``` "fieldTransformations" : [{ input @@ -260,27 +275,31 @@ Assuming the following values: } }] ``` + 1. Push the configuration back to zookeeper ``` ${METRON_HOME}/bin/zk_load_configs.sh -i ${METRON_HOME}/config/zookeeper -m PUSH -c PARSER -n $PARSER_NAME -z $ZOOKEEPER ``` + 1. Run kafka console to monitor correct operation of the field transformation ``` ${HDP_HOME}/kafka-broker/bin/kafka-console-consumer.sh --bootstrap-server $BROKERLIST --topic $PARSER_NAME ``` + 1. Config Metron SOLR indexing to push documents to the newly created Collection Alias. 1. Pull the most recent index config from zookeeper - ``` - ${METRON_HOME}/bin/zk_load_configs.sh -i ${METRON_HOME}/config/zookeeper -m PULL -c INDEXING -n $PARSER_NAME -z $ZOOKEEPER - ``` + ``` + ${METRON_HOME}/bin/zk_load_configs.sh -i ${METRON_HOME}/config/zookeeper -m PULL -c INDEXING -n $PARSER_NAME -z $ZOOKEEPER + ``` + 1. Edit the file ${METRON_HOME}/config/zookeeper/indexing/$PARSER_NAME.json + 1. Update the solr/index field to the `ALIAS_NAME` value you configured for the SOLR time-based routing alias. + 1. Push the configuration back to zookeeper - ``` - ${METRON_HOME}/bin/zk_load_configs.sh -i ${METRON_HOME}/config/zookeeper -m PUSH -c INDEXING -n $PARSER_NAME -z $ZOOKEEPER - ``` - -1. + ``` + ${METRON_HOME}/bin/zk_load_configs.sh -i ${METRON_HOME}/config/zookeeper -m PUSH -c INDEXING -n $PARSER_NAME -z $ZOOKEEPER + ``` From 97d5f9e3e27caa9f4eb1b7c0bd429e43f47c0d44 Mon Sep 17 00:00:00 2001 From: Dale Date: Wed, 27 Nov 2019 09:59:17 +1100 Subject: [PATCH 4/6] METRON-2327: Added convenience configset creation script to RPM spec file --- metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec | 1 + 1 file changed, 1 insertion(+) diff --git a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec index e4b99ca4dc..ea9f8b086c 100644 --- a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec +++ b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec @@ -311,6 +311,7 @@ This package installs the Metron Solr Common files %dir %{metron_home}/config %{metron_home}/bin/create_collection.sh %{metron_home}/bin/delete_collection.sh +%{metron_home}/bin/create_configset.sh %{metron_home}/bin/install_solr.sh %{metron_home}/bin/start_solr.sh %{metron_home}/bin/start_solr_topology.sh From 43eccf0cb5a7c49c399eb31f10e89dd39ce0cfbd Mon Sep 17 00:00:00 2001 From: Dale Date: Wed, 27 Nov 2019 14:49:43 +1100 Subject: [PATCH 5/6] METRON-2327 - Tweaked some tests that had hard-coded dependencies on the sample schema files --- .../metron/solr/integration/SolrSearchIntegrationTest.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/metron-platform/metron-solr/metron-solr-common/src/test/java/org/apache/metron/solr/integration/SolrSearchIntegrationTest.java b/metron-platform/metron-solr/metron-solr-common/src/test/java/org/apache/metron/solr/integration/SolrSearchIntegrationTest.java index b6d6162bf1..1c7a2fea27 100644 --- a/metron-platform/metron-solr/metron-solr-common/src/test/java/org/apache/metron/solr/integration/SolrSearchIntegrationTest.java +++ b/metron-platform/metron-solr/metron-solr-common/src/test/java/org/apache/metron/solr/integration/SolrSearchIntegrationTest.java @@ -112,7 +112,7 @@ public void returns_column_metadata_for_specified_indices() throws Exception { { Map fieldTypes = dao.getColumnMetadata(Collections.singletonList("bro")); // Don't test all fields, just test a sample of different fields - Assert.assertEquals(263, fieldTypes.size()); + Assert.assertEquals(264, fieldTypes.size()); // Fields present in both with same type Assert.assertEquals(FieldType.TEXT, fieldTypes.get("guid")); @@ -148,7 +148,7 @@ public void returns_column_metadata_for_specified_indices() throws Exception { // getColumnMetadata with only snort { Map fieldTypes = dao.getColumnMetadata(Collections.singletonList("snort")); - Assert.assertEquals(33, fieldTypes.size()); + Assert.assertEquals(34, fieldTypes.size()); // Fields present in both with same type Assert.assertEquals(FieldType.TEXT, fieldTypes.get("guid")); From dca4a8611e48b19e28c492a6c91d04334995d442 Mon Sep 17 00:00:00 2001 From: Dale Date: Mon, 9 Dec 2019 18:43:54 +1100 Subject: [PATCH 6/6] METRON-2327 - Some additional typos --- .../metron-solr/metron-solr-common/README.md | 10 +++++----- .../src/main/scripts/create_configset.sh | 7 +++---- 2 files changed, 8 insertions(+), 9 deletions(-) diff --git a/metron-platform/metron-solr/metron-solr-common/README.md b/metron-platform/metron-solr/metron-solr-common/README.md index 924ec598bb..2cc4f0e47f 100644 --- a/metron-platform/metron-solr/metron-solr-common/README.md +++ b/metron-platform/metron-solr/metron-solr-common/README.md @@ -237,7 +237,7 @@ Assuming the following values: Then the following command will create a time-routed alias: ``` - curl http://$SOLR_HOST:8983/solr/admin/collections?action=CREATEALIAS\ + curl "http://$SOLR_HOST:8983/solr/admin/collections?action=CREATEALIAS\ &name=$ALIAS_NAME\ &router.start=$ROUTER_START\ &router.field=$ROUTER_FIELD\ @@ -245,21 +245,21 @@ Assuming the following values: &router.interval=$ROUTER_INTERVAL\ &router.maxFutureMs=$ROUTER_MAXFUTUREMS\ &create-collection.collection.configName=$CONFIGSET\ - &create-collection.numShards=2 + &create-collection.numShards=2" ``` 1. Add a Metron Parser Stellar field transformation to the parser config that adds a correctly formatted datetime string to the event as it is being parsed: 1. Set environment variables for later reference ``` - source /etc/defaults/metron + source /etc/default/metron export HDP_HOME="/usr/hdp/current" export PARSER_NAME= ``` 1. Pull the most recent sensor parser config from zookeeper ``` - ${METRON_HOME}/bin/zk_load_configs.sh -i ${METRON_HOME}/config/zookeeper -m PULL -c PARSER -n $PARSER_NAME -z $ZOOKEEPER + ${METRON_HOME}/bin/zk_load_configs.sh -o ${METRON_HOME}/config/zookeeper -m PULL -c PARSER -n $PARSER_NAME -z $ZOOKEEPER ``` 1. Open the file to the relevant sensor parser at `$METRON_HOME/config/zookeeper/parsers/$PARSER_NAME.json` @@ -290,7 +290,7 @@ Assuming the following values: 1. Config Metron SOLR indexing to push documents to the newly created Collection Alias. 1. Pull the most recent index config from zookeeper ``` - ${METRON_HOME}/bin/zk_load_configs.sh -i ${METRON_HOME}/config/zookeeper -m PULL -c INDEXING -n $PARSER_NAME -z $ZOOKEEPER + ${METRON_HOME}/bin/zk_load_configs.sh -o ${METRON_HOME}/config/zookeeper -m PULL -c INDEXING -n $PARSER_NAME -z $ZOOKEEPER ``` 1. Edit the file ${METRON_HOME}/config/zookeeper/indexing/$PARSER_NAME.json diff --git a/metron-platform/metron-solr/metron-solr-common/src/main/scripts/create_configset.sh b/metron-platform/metron-solr/metron-solr-common/src/main/scripts/create_configset.sh index 1a46e53ecb..59f8602908 100755 --- a/metron-platform/metron-solr/metron-solr-common/src/main/scripts/create_configset.sh +++ b/metron-platform/metron-solr/metron-solr-common/src/main/scripts/create_configset.sh @@ -26,6 +26,9 @@ if [ ${SECURITY_ENABLED,,} == 'true' ]; then NEGOTIATE=' --negotiate -u : ' fi +# Get the first Solr node from the list of live nodes in Zookeeper +SOLR_NODE=`$ZOOKEEPER_HOME/bin/zkCli.sh -server $ZOOKEEPER ls /live_nodes | tail -n 1 | sed 's/\[\([^,]*\).*\]/\1/' | sed 's/_solr//'` + # test for errors in SOLR URL if [[ ${SOLR_NODE} =~ .*:null ]]; then echo "Error occurred while attempting to read SOLR Cloud configuration data from Zookeeper."; @@ -35,7 +38,6 @@ if [[ ${SOLR_NODE} =~ .*:null ]]; then exit 1; fi - # test for presence of datetime field in schema collection DQT='"' DATETIME_SCHEMA="