Add 'opensearch' and 'opensearch_info' modules #859
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Build failed.
|
|
Build failed.
|
|
It does not come to the integration tests, because of some sanity failures.
|
|
Build failed.
|
|
Build failed.
|
|
@alinabuzachis , besides the module itself and the integration tests, are there other files I need to add? |
|
The error below seems unrelated to my changes: |
|
Build failed.
|
|
Build failed.
|
There was a problem hiding this comment.
@sebastien-rosset I left some initial review comments with some suggestions. But the PR looks really good!
|
@sebastien-rosset These are some validation errors that need to be addressed. |
|
Build failed.
|
sebastien-rosset
changed the title
|
Build failed.
|
|
Build failed.
|
|
Build failed.
|
|
Build failed.
|
|
Build failed.
|
|
Build failed.
|
|
It looks like the |
|
Build failed.
|
|
Merge Failed. This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset. |
|
Build failed.
|
|
Build failed.
|
|
Build failed.
|
tremble
previously requested changes
Jan 14, 2022
There was a problem hiding this comment.
Thanks for the time you've spent on this module, it's very much appreciated.
I've not looked too closely at the rest of the code, this review is specifically with regards the boto3/botocore requirements.
Please don't adjust the global botocore/boto3 requirements.
The current collection policy is that we support the version of boto3/botocore released about 1 year prior to our current major release (currently botocore 1.19/boto3 1.16). While folks wanting support for a new AWS service may be happy constantly updating botocore/boto3 this can be frustrating for folks who find that their previously working environment stops working after a minor/patch release of the collection.
Individual modules, and deliberately fenced off features inside other modules can use newer versions of the AWS SDKs, however, they should be explicitly testing for them. Otherwise we do get issues/complaints caused by modules suddenly no longer working because we added support for a new feature they didn't care about. Unfortunately unless you're familiar with the types of errors you'll see it's also non-trivial to diagnose the issue (and non-trivial to automatically catch too).
Manually catching exactly which version a botocore feature was introduced generally doesn't work and as such, to ensure we don't accidentally depend on features from a later release, our tests are deliberately run against the minimum 'supported' version.
Tests for modules/features that need newer versions should themselves by testing against the minimum necessary botocore versions. This can be installed by using the helper role "setup_botocore_pip" and used by defining the ansible_python_interpreter. See ec2_asg_scheduled_action for an example. This mechanism is, unfortunately, currently poorly documented and only needed by a handful of modules, which we should fix.
get domain response on creation do not set cold storage option if version is less than 7.9 add skeleton integration test fix doc issues Add domain endpoint config options Add policy for test purpose Add integration tests, fix code such that tests can pass Add integration tests, fix code such that tests can pass Add integration tests, fix code such that tests can pass Add integration tests, fix code such that tests can pass Fix doc issues, address PR comments uncomment module defaults. add tasks in check mode Add more integration tests. Simplify argument spec. Fix linter issues. Add support for check_mode. Remove dead code. Add code comments. Uncomment module defaults add pylint comments Add more integration tests with check mode format code using black to fix pep8 issues fix integration tests fix linter issues add support for advanced security options. Fix linter issue add more examples wait until VPC is created add longer timeout for cluster upgrade test. Handle wait timeout error Customize wait timeout of integration tests, based on actual execution runtime fix pylint issues, autotune configuration add more info to wait timeout status, add unit tests upgrade boto3 for test purpose. Could be reverted later if needed add encryption at rest fix KMV key id parameter specify version of botocore compatible with boto3 Create/Delete KMS key for encryption at rest, change ES instance type to support encryption at rest add doc comment provide test execution timeline improve integration tests Add wait for deletion of Cluster provide test execution timeline split common functions to module_utils Specify minimum version of botocore Co-authored-by: Mark Chappell <[email protected]> Revert changes for boto3 version Co-authored-by: Mark Chappell <[email protected]> Specify virtual boto environment Co-authored-by: Mark Chappell <[email protected]> Specify minimum version of botocore Co-authored-by: Mark Chappell <[email protected]> Revert change for default boto3 version Co-authored-by: Mark Chappell <[email protected]> Revert change for boto3 version Co-authored-by: Mark Chappell <[email protected]> Specify minimum version of botocore Co-authored-by: Mark Chappell <[email protected]> remove import which is no longer needed improve name of test tasks, add timing information fix assertion for auto-tune refactor to simplify code make auto-tune test more robust do not raise exception if trying to delete a ES cluster that does not exist handle deletion of resources in integration tests Handle case when ES cluster is attached to the Internet Handle case when ES cluster is attached to the Internet Handle case when ES cluster is attached to the Internet add code comments Handle case when ES cluster is attached to the Internet renamed ebs_options.enabled to ebs_enabled to make it consistent with the API improve code comments Add more test assertions reformat code reformat code fix syntax error in main.yml use key_arn instead of key_id format code custom_endpoint_enabled, fix assertion fix imports adding opensearch to runtime.yml, is it used for module defaults? split the integration test tasks in multiple files remove extra line Add route53 zone for custom opensearch endpoint Add timeout for HTTPs endpoint. Create Route53 CNAME record Delete Route53 zone Add blocks Increase timeout. Set custom endpoint fix doc issues move auto-tune at the end of the integration tests add unit tests. add function to compare versions. add support for upgrade through intermediate versions improve documentation don't wait for auto-tune background tasks to complete fix zone discovery issue when running outside us-east-1. fix name of es cluster, it must start with a letter fix test assertions fix code formatting fix linter issue move ensure_tags to module_utils use 'compare_policies' fix linter issue fix doc issues create single block for integration test add code comments, rename get_domain to get_domain_status add opensearch_info module improve integration tests fix deletion of VPC resources fix test assertion, add documentation fix indentation removed unused variables. Add assertions fix variable assignment add test fix camel_dict for 'Endpoints' attribute handle case when domain_status is None move test fix format issues fix doc issues add documentation move task to obtain endpoints property route53 is explicitly a global service mistakenly commented out 'always' block add 'allow_intermediate_upgrades' parameter cleanup route53 records fix validation issue with upgrade compatible versions fix doc issues fix validation issue with upgrade compatible versions add missing check mode in integration test add documentation for SAML improve integration tests update KMS policy remove trailing space reduce time to execute integration tests
sebastien-rosset
force-pushed
the
opensearch
branch
from
6d179df to
3a1e66d
Compare
ansibullbot
added
community_review
and removed
merge_commit
|
recheck |
|
Is there a chance this will be reviewed? |
ansibullbot
added
needs_revision
tremble
reviewed
Jun 13, 2022
| short_description: Creates OpenSearch or ElasticSearch domain. | ||
| description: | ||
| - Creates or modify a Amazon OpenSearch Service domain. | ||
| version_added: 3.1.0 |
There was a problem hiding this comment.
Suggested change
| version_added: 3.1.0 | |
| version_added: 4.0.0 |
tremble
reviewed
Jun 13, 2022
| short_description: obtain information about one or more OpenSearch or ElasticSearch domain. | ||
| description: | ||
| - obtain information about one Amazon OpenSearch Service domain. | ||
| version_added: 3.1.0 |
There was a problem hiding this comment.
Suggested change
| version_added: 3.1.0 | |
| version_added: 4.0.0 |
Docs Build 📝Thank you for contribution!✨ This PR has been merged and your docs changes will be incorporated when they are next published. |
ansibullbot
added
community_review
and removed
needs_revision
softwarefactory-project-zuul bot
pushed a commit
that referenced
this pull request
Jun 14, 2022
opensearch - Minor fixups SUMMARY Minor fixups from #859 Module hasn't been release yet, no need for changelog ISSUE TYPE Docs Pull Request Feature Pull Request COMPONENT NAME opensearch opensearch_info ADDITIONAL INFORMATION Reviewed-by: Alina Buzachis <None>
abikouo
pushed a commit
to abikouo/community.aws
that referenced
this pull request
Oct 24, 2023
fix incomplete enforce_count() return values SUMMARY This is the Ansible way. Since a call to enforce_count() requires that a call to find_instances() has already been made, the information required to populate the instances return key is already held by the variable existing_matches. There are zero use cases where it makes sense not to output this information; if I want exactly one such instance and an existing instance matches the filter, I definitely always want to know its details. Fixes ansible-collections#963, ansible-collections#859 ISSUE TYPE Bugfix Pull Request COMPONENT NAME ec2_instance ADDITIONAL INFORMATION See linked issues. Before: { "changed": false, "invocation": { "module_args": { "availability_zone": null, "aws_access_key": null, "aws_ca_bundle": null, "aws_config": null, "aws_secret_key": null, "count": null, "cpu_credit_specification": null, "cpu_options": null, "debug_botocore_endpoint_logs": false, "detailed_monitoring": false, "ebs_optimized": true, "ec2_url": null, "exact_count": 1, "filters": { "tag:Name": "something", "tag:env": "prod" }, "image": null, "image_id": "REDACTED", "instance_ids": [], "instance_initiated_shutdown_behavior": null, "instance_role": "REDACTED", "instance_type": "m5.2xlarge", "key_name": "REDACTED", "launch_template": null, "metadata_options": null, "name": null, "network": { "source_dest_check": true }, "placement_group": null, "profile": null, "purge_tags": false, "region": "us-west-2", "security_group": null, "security_groups": [ "default" ], "security_token": null, "state": "present", "tags": { "Name": "something", "env": "prod", }, "tenancy": null, "termination_protection": null, "tower_callback": null, "user_data": "REDACTED", "validate_certs": true, "volumes": [ { "device_name": "/dev/sda1", "ebs": { "delete_on_termination": true, "encrypted": true, "volume_size": 20, "volume_type": "gp3" } }, { "device_name": "/dev/xvdo", "ebs": { "delete_on_termination": true, "encrypted": true, "volume_size": 20, "volume_type": "gp3" } }, { "device_name": "/dev/xvdp", "ebs": { "delete_on_termination": true, "encrypted": true, "volume_size": 10, "volume_type": "gp3" } }, { "device_name": "/dev/xvdi", "ebs": { "delete_on_termination": true, "encrypted": true, "volume_size": 100, "volume_type": "gp3" } } ], "vpc_subnet_id": "REDACTED", "wait": true, "wait_timeout": 300 } }, "msg": "1 instances already running, nothing to do." } After: { "changed": false, "instance_ids": [ "REDACTED" ], "instances": [ { "ami_launch_index": 0, "architecture": "x86_64", "block_device_mappings": [ { "device_name": "/dev/sda1", "ebs": { "attach_time": "2022-08-10T19:50:24+00:00", "delete_on_termination": true, "status": "attached", "volume_id": "REDACTED" } }, { "device_name": "/dev/xvdo", "ebs": { "attach_time": "2022-08-10T19:50:24+00:00", "delete_on_termination": true, "status": "attached", "volume_id": "REDACTED" } }, { "device_name": "/dev/xvdp", "ebs": { "attach_time": "2022-08-10T19:50:24+00:00", "delete_on_termination": true, "status": "attached", "volume_id": "REDACTED" } }, { "device_name": "/dev/xvdi", "ebs": { "attach_time": "2022-08-10T19:50:24+00:00", "delete_on_termination": true, "status": "attached", "volume_id": "REDACTED" } } ], "capacity_reservation_specification": { "capacity_reservation_preference": "open" }, "client_token": "REDACTED", "cpu_options": { "core_count": 4, "threads_per_core": 2 }, "ebs_optimized": true, "ena_support": true, "enclave_options": { "enabled": false }, "hibernation_options": { "configured": false }, "hypervisor": "xen", "iam_instance_profile": { "arn": "REDACTED", "id": "REDACTED" }, "image_id": "REDACTED", "instance_id": "REDACTED", "instance_type": "m5.2xlarge", "key_name": "REDACTED", "launch_time": "2022-08-10T19:50:23+00:00", "maintenance_options": { "auto_recovery": "default" }, "metadata_options": { "http_endpoint": "enabled", "http_protocol_ipv6": "disabled", "http_put_response_hop_limit": 1, "http_tokens": "optional", "instance_metadata_tags": "disabled", "state": "applied" }, "monitoring": { "state": "disabled" }, "network_interfaces": [ { "attachment": { "attach_time": "2022-08-10T19:50:23+00:00", "attachment_id": "REDACTED", "delete_on_termination": true, "device_index": 0, "network_card_index": 0, "status": "attached" }, "description": "", "groups": [ { "group_id": "REDACTED", "group_name": "REDACTED" } ], "interface_type": "interface", "ipv6_addresses": [], "mac_address": "REDACTED", "network_interface_id": "REDACTED", "owner_id": "REDACTED", "private_dns_name": "REDACTED", "private_ip_address": "REDACTED", "private_ip_addresses": [ { "primary": true, "private_dns_name": "REDACTED", "private_ip_address": "REDACTED" } ], "source_dest_check": true, "status": "in-use", "subnet_id": "REDACTED", "vpc_id": "REDACTED" } ], "placement": { "availability_zone": "us-west-2a", "group_name": "", "tenancy": "default" }, "platform_details": "Linux/UNIX", "private_dns_name": "REDACTED", "private_dns_name_options": { "enable_resource_name_dns_a_record": false, "enable_resource_name_dns_aaaa_record": false, "hostname_type": "ip-name" }, "private_ip_address": "REDACTED", "product_codes": [], "public_dns_name": "", "root_device_name": "/dev/sda1", "root_device_type": "ebs", "security_groups": [ { "group_id": "REDACTED", "group_name": "REDACTED" } ], "source_dest_check": true, "state": { "code": 16, "name": "running" }, "state_transition_reason": "", "subnet_id": "REDACTED", "tags": { "Name": "something", "env": "prod", }, "usage_operation": "RunInstances", "usage_operation_update_time": "2022-08-10T19:50:23+00:00", "virtualization_type": "hvm", "vpc_id": "REDACTED" } ], "invocation": { "module_args": { "availability_zone": null, "aws_access_key": null, "aws_ca_bundle": null, "aws_config": null, "aws_secret_key": null, "count": null, "cpu_credit_specification": null, "cpu_options": null, "debug_botocore_endpoint_logs": false, "detailed_monitoring": false, "ebs_optimized": true, "ec2_url": null, "exact_count": 1, "filters": { "tag:Name": "something", "tag:env": "prod" }, "image": null, "image_id": "REDACTED", "instance_ids": [], "instance_initiated_shutdown_behavior": null, "instance_role": "REDACTED", "instance_type": "m5.2xlarge", "key_name": "REDACTED", "launch_template": null, "metadata_options": null, "name": null, "network": { "source_dest_check": true }, "placement_group": null, "profile": null, "purge_tags": false, "region": "us-west-2", "security_group": null, "security_groups": [ "default" ], "security_token": null, "state": "present", "tags": { "Name": "something", "env": "prod", }, "tenancy": null, "termination_protection": null, "tower_callback": null, "user_data": "REDACTED", "validate_certs": true, "volumes": [ { "device_name": "/dev/sda1", "ebs": { "delete_on_termination": true, "encrypted": true, "volume_size": 20, "volume_type": "gp3" } }, { "device_name": "/dev/xvdo", "ebs": { "delete_on_termination": true, "encrypted": true, "volume_size": 20, "volume_type": "gp3" } }, { "device_name": "/dev/xvdp", "ebs": { "delete_on_termination": true, "encrypted": true, "volume_size": 10, "volume_type": "gp3" } }, { "device_name": "/dev/xvdi", "ebs": { "delete_on_termination": true, "encrypted": true, "volume_size": 100, "volume_type": "gp3" } } ], "vpc_subnet_id": "REDACTED", "wait": true, "wait_timeout": 300 } }, "msg": "1 instances already running, nothing to do." } Reviewed-by: Alina Buzachis Reviewed-by: Mark Chappell
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
SUMMARY
opensearchmodule to create/update AWS OpenSearch/Elasticsearch domains.opensearch_infomodule to query AWS OpenSearch/Elasticsearch domains.Fixes #858
Requires mattclay/aws-terminator#187
ISSUE TYPE
COMPONENT NAME
Creates OpenSearch or ElasticSearch domain.
ADDITIONAL INFORMATION
botocorefor these modules is 1.21.38.