From 48c5c3b43b63f3715043ff3e83f4253796047a21 Mon Sep 17 00:00:00 2001 From: Tom Yerex Date: Sun, 19 Apr 2020 18:19:12 -0700 Subject: [PATCH 1/4] METRON-2345 "Metron on CentOS 7" documentation does not exist --- .../development/centos7/README.md | 160 ++++++++++++++++++ 1 file changed, 160 insertions(+) create mode 100644 metron-deployment/development/centos7/README.md diff --git a/metron-deployment/development/centos7/README.md b/metron-deployment/development/centos7/README.md new file mode 100644 index 0000000000..be217693a5 --- /dev/null +++ b/metron-deployment/development/centos7/README.md @@ -0,0 +1,160 @@ + +Metron on CentOS 7 +================== + +This project fully automates the provisioning and deployment of Apache Metron and all necessary prerequisites on a single, virtualized host running CentOS 7. + +Metron is composed of many components and installing all of these on a single host, especially a virtualized one, will greatly stress the resources of the host. The host will require at least 8 GB of RAM and a fair amount of patience. It is highly recommended that you shut down all unnecessary services. + +Getting Started +--------------- + +### Prerequisites + +The computer used to deploy Apache Metron will need to have the following components installed. + + - [Ansible](https://github.com/ansible/ansible) 2.6.5+ + - [Docker](https://www.docker.com/community-edition) + - [OpenJDK](https://openjdk.java.net) 1.8 + - [Vagrant](https://www.vagrantup.com) 2.0+ + - [Vagrant Hostmanager Plugin](https://github.com/devopsgroup-io/vagrant-hostmanager) + - [Virtualbox](https://virtualbox.org) 5.0+ + - [Node.js](https://nodejs.org/) 12.16.2+ + - Python 2.7 + - Maven 3.3.9 + - C++11 compliant compiler, like [GCC](https://gcc.gnu.org/projects/cxx-status.html#cxx11) + +Running the following script can help validate whether you have all the prerequisites installed and running correctly. + + ``` + metron-deployment/scripts/platform-info.sh + ``` + +#### How do I install these on MacOS? + +Any platform that supports these tools is suitable, but the following instructions cover installation on macOS. The easiest means of installing these tools on a Mac is to use the excellent [Homebrew](http://brew.sh/) project. + +1. Install Homebrew by following the instructions at [Homebrew](http://brew.sh/). + +1. Run the following command in a terminal to install all of the required tools. + + ``` + brew cask install vagrant virtualbox docker + brew cask install homebrew/cask-versions/adoptopenjdk8 + brew install maven@3.3 git + sudo pip install ansible==2.6.5 + vagrant plugin install vagrant-hostmanager + open /Applications/Docker.app + ``` + +### Deploy Metron + +1. Ensure that the Docker service is running. + + ``` + docker info + ``` + +1. Deploy Metron + + ``` + cd metron-deployment/development/centos7 + vagrant up + ``` + + Should the process fail before completing the deployment, the following command will continue the deployment process without re-instantiating the host. + + ``` + vagrant provision + ``` + + If the process should fail again, check the versions of the supporting components are correct and use the following command to clean up the process and retry. + + ``` + vagrant destroy + ``` + +### Deployment optimizations + +1. Set environment variable + ``` + export ANSIBLE_ARGS='--extra-vars "SKIP_RPMLINT=1"' + ``` + To disable running rpmlint as part of the dev deployment task - this can save a couple of minutes of time on the deployment. + Either add this variable to your profile, or use it on the command line like + + ``` + env ANSIBLE_ARGS='--extra-vars "SKIP_RPMLINT=1"' vagrant up + ``` + +### Deployment debugging + +1. To enable more verbose logging of ansible actions during the deployment, use + ``` + env ANSIBLE_ARGS=' -vvvv' vagrant up + ``` + As this can produce large amounts of logging, it is best to redirect output to a file for later analysis. + + +### Explore Metron + +Navigate to the following resources to explore your newly minted Apache Metron environment. + +* [Metron Alerts](http://node1:4201) credentials: user/password +* [Ambari](http://node1:8080) credentials: admin/admin + +Connecting to the host through SSH is as simple as running the following command. +``` +vagrant ssh +``` + +### Working with Metron + +In addition to re-running the entire provisioning play book, you may now re-run an individual Ansible tag or a collection of tags in the following ways. The following commands will re-run the `sensor-stubs` role on the Vagrant image. This will install and start the sensor stub components. + +``` +vagrant --ansible-tags="sensor-stubs" provision +``` + +Tags are listed in the playbooks. Here are some frequently used tags: ++ `hdp-install` - Install HDP ++ `hdp-deploy` - Deploy and Start HDP Services (will start all Hadoop Services) ++ `sensors` - Deploy the sensors (see [Sensors](#sensors) for more details regarding this tag) ++ `sensor-stubs` - Deploy and start the sensor stubs. + +#### Sensors + +By default, the Metron development environment uses sensor stubs to mimic the behavior of the full sensors. This is done because the full sensors take a significant amount of time and CPU to build, install, and run. + +From time to time you may want to install the full sensors for testing (see the specifics of what that means [here](../../ansible/playbooks/sensor_install.yml)). This can be done by running the following command: + +``` +vagrant --ansible-skip-tags="sensor-stubs" up +``` + +This will skip only the `sensor-stubs` tag, allowing the ansible roles with the `sensors` tag to be run. This provisions the full sensors in a 'testing mode' so that they are more active, and thus more useful for testing (more details on that [here](../../ansible/roles/sensor-test-mode/)). **However**, when vagrant completes the sensors will NOT be running. In order to start the sensors and simulate traffic through them (which will create a fair amount of load on your test system), complete the below steps: + +``` +vagrant ssh +sudo su - +service pcap-replay restart +service yaf restart +service snortd restart +service snort-producer restart +``` From 896f5dc29e4b313a046d4f157944088293587a05 Mon Sep 17 00:00:00 2001 From: Tom Yerex Date: Sat, 25 Apr 2020 16:22:30 -0700 Subject: [PATCH 2/4] Add CentOS 7 to the list of dev environments. --- metron-deployment/development/README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/metron-deployment/development/README.md b/metron-deployment/development/README.md index 448b53fe1e..8858d6c87c 100644 --- a/metron-deployment/development/README.md +++ b/metron-deployment/development/README.md @@ -20,6 +20,7 @@ limitations under the License. This directory contains environments useful for Metron developers. These environments are not intended for proof-of-concept, testing, or production use. These are extremely resource constrained and cannot support anything beyond the most basic work loads. * Metron running on CentOS 6 +* Metron running on CentOS 7 * Metron running on Ubuntu 14 * Fastcapa From 1a37edd9b1c316a44f37f7f2eb01733313741441 Mon Sep 17 00:00:00 2001 From: Tom Yerex Date: Sat, 25 Apr 2020 19:47:49 -0700 Subject: [PATCH 3/4] Added note about GeoIP update problem. --- metron-deployment/development/README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/metron-deployment/development/README.md b/metron-deployment/development/README.md index 8858d6c87c..a7bedf27b7 100644 --- a/metron-deployment/development/README.md +++ b/metron-deployment/development/README.md @@ -71,3 +71,7 @@ Now, when you go to Swagger or the UIs, you should be able to give a user and pa If you have the Mojave OS or newer, you may run into this issue when running `vagrant up`. In order to correct this you will need to grant permissions to your relevant terminal application. Navigate to `System Preferences -> Security & Privacy -> Privacy` and add your terminal application to "Full Disk Access". See [https://github.com/hashicorp/vagrant/issues/10234](https://github.com/hashicorp/vagrant/issues/10234) for more details. + +`Warning: Unable to copy remote GeoIP database to local file, attempt 2: java.net.UnknownHostException: geolite.maxmind.com` + +In order to correct this you will need to set up an alternative distribution point for Maxmind GeoIP2 Database update as a workaround until a full solution is implemented. See [https://issues.apache.org/jira/projects/METRON/issues/METRON-2340] (https://issues.apache.org/jira/projects/METRON/issues/METRON-2340) for more details. \ No newline at end of file From e64316a991766949d8e1232abdb85891977a475c Mon Sep 17 00:00:00 2001 From: Tom Yerex Date: Sat, 25 Apr 2020 19:49:27 -0700 Subject: [PATCH 4/4] Syntax fix so links appear properly. --- metron-deployment/development/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/metron-deployment/development/README.md b/metron-deployment/development/README.md index a7bedf27b7..d7d12cfc6f 100644 --- a/metron-deployment/development/README.md +++ b/metron-deployment/development/README.md @@ -74,4 +74,4 @@ Navigate to `System Preferences -> Security & Privacy -> Privacy` and add your t `Warning: Unable to copy remote GeoIP database to local file, attempt 2: java.net.UnknownHostException: geolite.maxmind.com` -In order to correct this you will need to set up an alternative distribution point for Maxmind GeoIP2 Database update as a workaround until a full solution is implemented. See [https://issues.apache.org/jira/projects/METRON/issues/METRON-2340] (https://issues.apache.org/jira/projects/METRON/issues/METRON-2340) for more details. \ No newline at end of file +In order to correct this you will need to set up an alternative distribution point for Maxmind GeoIP2 Database update as a workaround until a full solution is implemented. See [https://issues.apache.org/jira/projects/METRON/issues/METRON-2340](https://issues.apache.org/jira/projects/METRON/issues/METRON-2340) for more details. \ No newline at end of file