METRON-2327 - Support for SOLR time-based arrays #1572
Conversation
…TML converter some grief
…er producing nicer html
|
You should create a new JIRA not reuse 2088 please. |
metron-platform/metron-solr/metron-solr-common/src/main/scripts/create_configset.sh
Show resolved
Hide resolved
tigerquoll
changed the title
…com/apache/metron into metron-2327_solr_timebased_routing
…the sample schema files
|
Interesting enough, when I ran the integration tests on my laptop, I got the following error: |
| # test for presence of datetime field in schema collection | ||
| DQT='"' | ||
| DATETIME_SCHEMA="<field name=${DQT}datetime${DQT} type=${DQT}datetime${DQT}" | ||
| grep --quiet --fixed-strings $DATETIME_SCHEMA $METRON_HOME/config/schema/$1/schema.xml; rc=$? |
There was a problem hiding this comment.
I see this when I run the create_configset script...
[root@solrtra-5 schema]# pwd
/usr/metron/0.7.2/config/schema
[root@solrtra-5 schema]# ../../bin/create_configset.sh bro1
grep: name="datetime": No such file or directory
grep: type="datetime": No such file or directory
adding: solrconfig.xml (deflated 72%)
adding: schema.xml (deflated 88%)
{
"responseHeader":{
"status":0,
"QTime":96}}
Configset bro1 successfully uploaded
Looks like the regex pattern needs fixing?
|
|
||
| 1. Add to the sensor parser config json field the following transformation: | ||
| ``` | ||
| "fieldTransformations" : [{ |
There was a problem hiding this comment.
I'm guessing the markdown changed some of the double quotes.. this should be something like....
{
"parserClassName":"org.apache.metron.parsers.bro.BasicBroParser",
"sensorTopic":"bro",
"parserConfig": {
"fieldTransformations" : [{
"input" : [ ]
,"transformation" : "STELLAR"
,"output" : [ "datetime" ]
,"config" : {
"datetime" : "DATE_FORMAT(\"yyyy-MM-dd'T'HH:mm:ss.SSSX\",timestamp)"
}
}]
}
}
| Assuming the following values: | ||
| * SOLR_HOST: Host SOLR is installed on | ||
|
|
||
| * ALIAS_NAME: Name of the new alias |
There was a problem hiding this comment.
In my case, I created the alias name as bro1 but I notice the indices are not being written. In the solr admin UI, I see the name of the collection as bro1_2019-12-03.
Can you provide an example for "name of the new alias" ?
There was a problem hiding this comment.
In this case the alias is what Metron is configured to read and write from - "bro1". What SOLR does behind the scenes is actually route the event to be stored in the collection "bro1_2019-12-03". If the event had been collected the next day, then it may be stored in the collection "bro1_2019-12-04". Both events are still retrievable when searching for them via the alias "bro1".
| # | ||
| METRON_VERSION=${project.version} | ||
| METRON_HOME=/usr/metron/$METRON_VERSION | ||
| ZOOKEEPER=${ZOOKEEPER:-localhost:2181} |
There was a problem hiding this comment.
We need to source /etc/default/metron before setting ZOOKEEPER. This step will fail in a multinode deployment.
There was a problem hiding this comment.
And in addition, we need to do something like this before invoking the script so that the ZK variable is set properly with the /solr suffix.
The README can be updated with this info.
| # | ||
| METRON_VERSION=${project.version} | ||
| METRON_HOME=/usr/metron/$METRON_VERSION | ||
| ZOOKEEPER=${ZOOKEEPER:-localhost:2181} |
There was a problem hiding this comment.
And in addition, we need to do something like this before invoking the script so that the ZK variable is set properly with the /solr suffix.
The README can be updated with this info.
|
|
||
| Then the following command will create a time-routed alias: | ||
| ``` | ||
| curl "http://$SOLR_HOST:8983/solr/admin/collections?action=CREATEALIAS\ |
There was a problem hiding this comment.
Minor nit... can we remove the spaces at the beginning of this block of code? Copy-pasting this will result in this command to fail.
Also, in my case I had to set the numShards to 1 for the command to complete successfully. Not sure if that is something to do with the way my solr is configured.
| ``` | ||
| ${HDP_HOME}/kafka-broker/bin/kafka-console-consumer.sh --bootstrap-server $BROKERLIST --topic $PARSER_NAME | ||
| ``` | ||
|
|
There was a problem hiding this comment.
Would it be good to include the step to start the parser topology ? :)
Contributor Comments
Updated schemas and instructions on how to enable SOLR time based arrays.
Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our Development Guidelines for the complete guide to follow for contributions.
Please refer also to our Build Verification Guidelines for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
For all changes:
For documentation related changes:
Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via
site-book/target/site/index.html:Have you ensured that any documentation diagrams have been updated, along with their source files, using draw.io? See Metron Development Guidelines for instructions.