elastic 7.x support #1061

uberjew666 · 2019-11-19T08:37:13Z

Update JVM startup options
Add node name to log4j2.properties
Rename augeas resource to prevent dependency loop
Rename xpack related parameters to new names
Add boolean to enable/disable xpack and required features
Add self to contributors list
Resolve variable scope warnings
Update unit tests
Remove ubuntu 14.04 acceptance tests - Not supported
https://www.elastic.co/support/matrix

Fixes:

Pull request acceptance prerequisites:

Signed the CLA (if not already signed)
Rebased/up-to-date with base branch
Tests pass
Updated CHANGELOG.md with patch notes (if necessary)
Updated CONTRIBUTORS (if attribution is requested)

elasticmachine · 2019-11-19T08:37:15Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

uberjew666 · 2019-11-19T15:48:58Z

@tylerjl @fatmcgav

Evening gents, can either of you two approve?

ernstae · 2019-12-02T19:59:08Z

FWIW, I've been running this branch in my QA environment, and it has been behaving properly.

specdevops · 2019-12-05T16:08:56Z

@tylerjl Any update on this? Unsure if I should proceed to install 6 if 7 will be released soon... :)

smasa90 · 2019-12-18T01:52:56Z

Any contribution mention for this ?? #1040

jake2184 · 2020-01-02T11:20:11Z

Is there anything preventing this being merged? I am keen to use the functionality

andrewbierbaum · 2020-01-02T17:37:41Z

@tylerjl I am also eagerly waiting on official word regarding 7.x support.

trunet · 2020-01-09T15:17:35Z

this PR is giving me this error:

java.lang.IllegalArgumentException: a key must be provided to run as a server. the key should be configured using the [xpack.security.http.ssl.key] or [xpack.security.http.ssl.keystore.path] setting

I think you have to set all xpack.security.http in addition to xpack.security.transport

uberjew666 · 2020-01-09T17:00:43Z

this PR is giving me this error:

java.lang.IllegalArgumentException: a key must be provided to run as a server. the key should be configured using the [xpack.security.http.ssl.key] or [xpack.security.http.ssl.keystore.path] setting

I think you have to set all xpack.security.http in addition to xpack.security.transport

You need to set elasticsearch::xpack: true This requires ca_certificate, certificate and private_key to be configured. This will add all required xpack settings to the config file.

trunet · 2020-01-09T17:01:43Z

I did

uberjew666 · 2020-01-09T17:12:07Z

Set elasticsearch::instance::ssl to false.

trunet · 2020-01-09T20:06:26Z

but this will disable http client -> elasticsearch tls communication and I want that.

you just have to also set where you set xpack.security.transport.*:

xpack.security.http.ssl.key
xpack.security.http.ssl.certificate
xpack.security.http.ssl.certificate_authorities

Documentation:
https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-tls.html

trunet · 2020-01-10T09:12:24Z

Set elasticsearch::instance::ssl to false.

This is indeed needed, otherwise it tries to load the keystore that is not used on this PR.

I think the only thing this needs, is the xpack.security.http.ssl.* in addition to the xpack.security.transport.*

TJM · 2020-01-13T22:21:17Z

manifests/instance.pp

@@ -440,13 +444,51 @@
      }
    }

+    if $xpack {
+      if (($ca_certificate == undef) and ($certificate == undef) and ($private_key == undef)) {


Shouldn't these be or instead of and ... or != undef ?

The way this is written, if any of them are defined, it will skip by this error check, it will only catch if all three are undef.

Good spot, it would be better to use or.

This has been added also

TJM · 2020-01-13T22:22:55Z

manifests/instance.pp

@@ -290,8 +294,8 @@
        $tls_config = {
          'xpack.security.transport.ssl.enabled' => true,
          'xpack.security.http.ssl.enabled'      => true,
-          'xpack.ssl.keystore.path'              => $_keystore_path,
-          'xpack.ssl.keystore.password'          => $keystore_password,
+          'xpack.http.ssl.keystore.path'         => $_keystore_path,


Are these options backwards compatible? Otherwise there should probably be a check on the major version and a switch between these two configurations?

We had to add a check for major version on our cluster settings... this may be useful:

$es_major_version = $elasticsearch::version.split(/\./)[0] # 7.5.1 -> 7

This has been added for ssl section

manifests/package.pp

TJM · 2020-01-13T23:25:07Z

spec/defines/005_elasticsearch_instance_spec.rb

                -Djna.nosys=true.
                -Dlog4j.shutdownHookEnabled=false.
                -Dlog4j2.disable.jmx=true.
                -XX:\+AlwaysPreTouch.
                -XX:\+HeapDumpOnOutOfMemoryError.
-                -XX:\+PrintGCDateStamps.
-                -XX:\+PrintGCDetails.
-                -XX:\+PrintTenuringDistribution.
                -XX:\+UseCMSInitiatingOccupancyOnly.
                -XX:\+UseConcMarkSweepGC.


This should have an `8:' too

I realize that the test isn't the only change but the point was the -XX:+UseConcMarkSweepGC should have an 8: prefix as it was deprecated in 9.

This has been added.

Antiarchitect · 2020-01-23T13:34:19Z

Really need this feature too. Anything I could help with?

TJM · 2020-01-23T14:42:16Z

Really need this feature too. Anything I could help with?

I took the code from this pull request and added the changes I mentioned above. I also had to modify my "profile" code to form the cluster with the new parameters. It is working for us. I figured @uberjew666 would make the changes I suggested so that it could hopefully be merged, but I can re-submit if they are unable to.

Tommy

uberjew666 · 2020-01-24T14:56:51Z

@TJM Sorry, I've not had much time to work on this. Luckily, this module is required for a project I'm working on so I will try and get the amendments sorted and pushed soon.

Antiarchitect · 2020-02-02T06:18:52Z

@TJM @uberjew666 Any news on this? Can I help somehow?

TJM · 2020-02-02T17:41:02Z

@Antiarchitect I just pushed the version we are using internally: https://github.com/TJM/puppet-elasticsearch/tree/es7-fix

Antiarchitect · 2020-02-02T18:23:45Z

@TJM Thank you so much! Anyway, I and my team have some time. So I'll wait this merged into master.

fatmcgav · 2020-02-04T13:52:38Z

Hi there,

Firstly, thank you for taking the time to contribute, and apologies for the radio silence from Elastic on this PR.

I'm going to be working on this module over the next few weeks, with an ultimate aim of updating the module to support the elasticsearch 7.x and simplifying the module to make it easier to use.

I'll be reviewing all the open issues and PRs over the next few days, and will merge or provide feedback where appropriate.

So please hang in there whilst we give this module some TLC.

Thanks again.

cla-checker-service · 2020-02-05T15:07:32Z

Author of the following commits did not sign a Contributor Agreement:
, 83a5648, a4b9222

Please, read and sign the above mentioned agreement if you want to contribute to this project

- Update JVM startup options - Add node name to log4j2.properties - Rename augeas resource to prevent dependency loop - Add boolean to enable/disable xpack and required features - Add self to contributors list - Add check for required certificate variables - Add version check for xpack related settings Maintains backwards compatibility - Resolve variable scope warnings - Update unit tests - Remove ubuntu 14.04 acceptance tests - Not supported https://www.elastic.co/support/matrix

Antiarchitect · 2020-03-16T19:29:13Z

Any progress on this, guys?

crazymind1337 · 2020-03-25T05:18:06Z

@uberjew666 looks like you have conflicts within your branch.

crazymind1337 · 2020-04-10T01:11:43Z

Since there is no progress in here I have created a new PR which is rebased: #1082

fatmcgav · 2020-12-24T00:18:04Z

Elasticsearch 7 support has been added. See #1068 (comment)

Thanks

TJM suggested changes Jan 13, 2020

View reviewed changes

TJM reviewed Jan 13, 2020

View reviewed changes

fatmcgav added es-7.x feature triage labels Feb 4, 2020

uberjew666 force-pushed the master branch from a4b9222 to 5e7278a Compare February 5, 2020 15:14

uberjew666 force-pushed the master branch from 5e7278a to 847e7a2 Compare February 5, 2020 16:43

smortex mentioned this pull request Aug 18, 2020

Module update and path to supporting 7.x #1068

Open

fatmcgav closed this Dec 24, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

elastic 7.x support #1061

elastic 7.x support #1061

uberjew666 commented Nov 19, 2019 •

edited

Loading

elasticmachine commented Nov 19, 2019

uberjew666 commented Nov 19, 2019

ernstae commented Dec 2, 2019

specdevops commented Dec 5, 2019

smasa90 commented Dec 18, 2019

jake2184 commented Jan 2, 2020 •

edited

Loading

andrewbierbaum commented Jan 2, 2020

trunet commented Jan 9, 2020

uberjew666 commented Jan 9, 2020 •

edited

Loading

trunet commented Jan 9, 2020

uberjew666 commented Jan 9, 2020

trunet commented Jan 9, 2020

trunet commented Jan 10, 2020 •

edited

Loading

TJM Jan 13, 2020 •

edited

Loading

uberjew666 Jan 14, 2020

uberjew666 Feb 5, 2020

TJM Jan 13, 2020

TJM Jan 14, 2020

uberjew666 Feb 5, 2020

TJM Jan 13, 2020

TJM Jan 14, 2020

uberjew666 Feb 5, 2020

Antiarchitect commented Jan 23, 2020

TJM commented Jan 23, 2020

uberjew666 commented Jan 24, 2020

Antiarchitect commented Feb 2, 2020

TJM commented Feb 2, 2020

Antiarchitect commented Feb 2, 2020 •

edited

Loading

fatmcgav commented Feb 4, 2020

cla-checker-service bot commented Feb 5, 2020

Antiarchitect commented Mar 16, 2020

crazymind1337 commented Mar 25, 2020

crazymind1337 commented Apr 10, 2020

fatmcgav commented Dec 24, 2020

elastic 7.x support #1061

elastic 7.x support #1061

Conversation

uberjew666 commented Nov 19, 2019 • edited Loading

elasticmachine commented Nov 19, 2019

uberjew666 commented Nov 19, 2019

ernstae commented Dec 2, 2019

specdevops commented Dec 5, 2019

smasa90 commented Dec 18, 2019

jake2184 commented Jan 2, 2020 • edited Loading

andrewbierbaum commented Jan 2, 2020

trunet commented Jan 9, 2020

uberjew666 commented Jan 9, 2020 • edited Loading

trunet commented Jan 9, 2020

uberjew666 commented Jan 9, 2020

trunet commented Jan 9, 2020

trunet commented Jan 10, 2020 • edited Loading

TJM Jan 13, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Antiarchitect commented Jan 23, 2020

TJM commented Jan 23, 2020

uberjew666 commented Jan 24, 2020

Antiarchitect commented Feb 2, 2020

TJM commented Feb 2, 2020

Antiarchitect commented Feb 2, 2020 • edited Loading

fatmcgav commented Feb 4, 2020

cla-checker-service bot commented Feb 5, 2020

Antiarchitect commented Mar 16, 2020

crazymind1337 commented Mar 25, 2020

crazymind1337 commented Apr 10, 2020

fatmcgav commented Dec 24, 2020

uberjew666 commented Nov 19, 2019 •

edited

Loading

jake2184 commented Jan 2, 2020 •

edited

Loading

uberjew666 commented Jan 9, 2020 •

edited

Loading

trunet commented Jan 10, 2020 •

edited

Loading

TJM Jan 13, 2020 •

edited

Loading

Antiarchitect commented Feb 2, 2020 •

edited

Loading