Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DRAFT] Resource permissions and sharing #166

Draft
wants to merge 345 commits into
base: main
Choose a base branch
from

Conversation

DarshitChanpura
Copy link
Owner

@DarshitChanpura DarshitChanpura commented Aug 30, 2024

@DarshitChanpura
Copy link
Owner Author

From latest ./gradlew run, it recognizes NoOpResourcePlugin.

[2024-08-30T12:42:17,976][INFO ][o.o.e.ExtensionsManager  ] [runTask-0] ExtensionsManager initialized
[2024-08-30T12:42:19,022][INFO ][o.o.a.r.ResourceService  ] [runTask-0] Security plugin disabled: Using NoOpResourcePlugin

Next test would be to determine whether it recognizes security plugin

@DarshitChanpura
Copy link
Owner Author

security plugin bootstrap: DarshitChanpura/security#10 (comment)

@DarshitChanpura DarshitChanpura changed the base branch from main to feature/segment-replication September 6, 2024 17:21
@DarshitChanpura DarshitChanpura changed the base branch from feature/segment-replication to main September 6, 2024 17:21
* The index where resource meta-data is stored
* @return the name of the parent index where resource meta-data is stored
*/
String getResourceIndex();
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can a plugin define multiple types of resources and resource indices?

*/
public class ShareWith implements ToXContentFragment {

private List<String> users;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can see pushback on introducing these concepts into the core. Is there a way to make this generic and let a plugin define an implementation?

Signed-off-by: Darshit Chanpura <[email protected]>
DarshitChanpura and others added 20 commits October 2, 2024 16:44
Signed-off-by: Darshit Chanpura <[email protected]>
Signed-off-by: Darshit Chanpura <[email protected]>
…on dynamic limit settings (opensearch-project#15986)

* Add changes to block calls in cat shards, indices and segments based on dynamic limit settings

Signed-off-by: Sumit Bansal <[email protected]>
…pensearch-project#16080)

* Separate Remote State and Publication enabled and configured methods

Signed-off-by: Shivansh Arora <[email protected]>
* add `Strings#isDigits` API

inspiration taken from [this SO answer][SO].

note that the stream is not parallelised to avoid the overhead of this
as the method is intended to be called primarily with shorter strings
where the time to set up would take longer than the actual check.

[SO]: https://stackoverflow.com/a/35150400

Signed-off-by: Ralph Ursprung <[email protected]>

* add `phone` & `phone-search` analyzer + tokenizer

this is largely based on [elasticsearch-phone] and internally uses
[libphonenumber].
this intentionally only ports a subset of the features: only `phone` and
`phone-search` are supported right now, `phone-email` can be added
if/when there's a clear need for it.

using `libphonenumber` is required since parsing phone numbers is a
non-trivial task (even though it might seem trivial at first glance!),
as can be seen in the list [falsehoods programmers believe about phone
numbers][falsehoods].

this allows defining the region to be used when analysing a phone
number. so far only the generic "unkown" region (`ZZ`) had been used
which worked as long as international numbers were prefixed with `+` but
did not work when using local numbers (e.g. a number stored as
`+4158...` was not matched against a number entered as `004158...` or
`058...`).

example configuration for an index:
```json
{
  "index": {
    "analysis": {
      "analyzer": {
        "phone": {
          "type": "phone"
        },
        "phone-search": {
          "type": "phone-search"
        },
        "phone-ch": {
          "type": "phone",
          "phone-region": "CH"
        },
        "phone-search-ch": {
          "type": "phone-search",
          "phone-region": "CH"
        }
      }
    }
  }
}
```
this creates four analyzers: `phone` and `phone-search` which do not
explicitly specify a region and thus fall back to `ZZ` (unknown region,
regional version of international dialing prefix (e.g. `00` instead of
`+` in most of europe) will not be recognised) and `phone-ch` and
`phone-search-ch` which will try to parse the phone number as a swiss
phone number (thus e.g. `00` as a prefix is recognised as the
international dialing prefix).

note that the analyzer is (currently) not meant to find phone numbers in
large text documents - instead it should be used on fields which contain
just the phone number (though extra text will be ignored) and it
collects the whole content of the field into a `String` in memory,
making it unsuitable for large field values.

this has been implemented in a new plugin which is however part of the
central opensearch repository as it was deemed too big an overhead to
have it in a separate repository but not important enough to bundle it
directly in `analysis-common` (see the discussion on the issue and the
PR for further details).

note that the new plugin has been added to the exclude list of the
javadoc check as this check is overzealous and also complains in many
cases where it shouldn't (e.g. on overridden methods - which it should
theoretically not do - or constructors which don't even exist). the
check first needs to be improved before this exclusion could be removed.

closes opensearch-project#11326

[elasticsearch-phone]: https://github.com/purecloudlabs/elasticsearch-phone
[libphonenumber]: https://github.com/google/libphonenumber
[falsehoods]: https://github.com/google/libphonenumber/blob/master/FALSEHOODS.md

Signed-off-by: Ralph Ursprung <[email protected]>

---------

Signed-off-by: Ralph Ursprung <[email protected]>
…pensearch-project#16194)

* Fix warnings from SLF4J on startup when repository-s3 is installed

Signed-off-by: Craig Perkins <[email protected]>

* Add to CHANGELOG

Signed-off-by: Craig Perkins <[email protected]>

* Fix precommit

Signed-off-by: Craig Perkins <[email protected]>

---------

Signed-off-by: Craig Perkins <[email protected]>
Signed-off-by: Darshit Chanpura <[email protected]>
* Update Apache Lucene to 9.12.0

Signed-off-by: Andriy Redko <[email protected]>

* change to IOContext READONCE in locations where the file is not expected to be read multiple times.

Signed-off-by: Marc Handalian <[email protected]>

* Use READ IOContext for all non Segment* files when copying node-node

Signed-off-by: Marc Handalian <[email protected]>

* Fixing more test failures

Signed-off-by: Andriy Redko <[email protected]>

* Move Composite912Codec under org.opensearch.index.codec.composite.composite912 package

Signed-off-by: Andriy Redko <[email protected]>

---------

Signed-off-by: Andriy Redko <[email protected]>
Signed-off-by: Marc Handalian <[email protected]>
Co-authored-by: Marc Handalian <[email protected]>
rajiv-kv and others added 26 commits December 16, 2024 14:08
)

* Bump com.gradle.develocity from 3.18.2 to 3.19

Bumps com.gradle.develocity from 3.18.2 to 3.19.

---
updated-dependencies:
- dependency-name: com.gradle.develocity
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
…ildSrc/src/testKit/thirdPartyAudit (opensearch-project#16858)

* Bump org.apache.logging.log4j:log4j-core

Bumps org.apache.logging.log4j:log4j-core from 2.24.2 to 2.24.3.

---
updated-dependencies:
- dependency-name: org.apache.logging.log4j:log4j-core
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
…est-attachment (opensearch-project#16854)

* Bump org.apache.xmlbeans:xmlbeans in /plugins/ingest-attachment

Bumps org.apache.xmlbeans:xmlbeans from 5.2.2 to 5.3.0.

---
updated-dependencies:
- dependency-name: org.apache.xmlbeans:xmlbeans
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Updating SHAs

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
…ture (opensearch-project#16857)

* Bump org.jline:jline in /test/fixtures/hdfs-fixture

Bumps [org.jline:jline](https://github.com/jline/jline3) from 3.27.1 to 3.28.0.
- [Release notes](https://github.com/jline/jline3/releases)
- [Changelog](https://github.com/jline/jline3/blob/master/changelog.md)
- [Commits](jline/jline3@jline-3.27.1...jline-3.28.0)

---
updated-dependencies:
- dependency-name: org.jline:jline
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
…y-azure (opensearch-project#16856)

* Bump com.azure:azure-core in /plugins/repository-azure

Bumps [com.azure:azure-core](https://github.com/Azure/azure-sdk-for-java) from 1.51.0 to 1.54.1.
- [Release notes](https://github.com/Azure/azure-sdk-for-java/releases)
- [Commits](Azure/azure-sdk-for-java@azure-core_1.51.0...azure-core_1.54.1)

---
updated-dependencies:
- dependency-name: com.azure:azure-core
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Updating SHAs

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Andriy Redko <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Andriy Redko <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
Signed-off-by: Darshit Chanpura <[email protected]>
…t#16850)

* Change Remote state read thread pool to Fixed type

Signed-off-by: Sooraj Sinha <[email protected]>
…16846) (opensearch-project#16870) (opensearch-project#16877)

* Update Apache Lucene to 9.12.1 (opensearch-project#16846)

Signed-off-by: Andriy Redko <[email protected]>
(cherry picked from commit bc4f44b)

* Added bwc version 1.3.21 (opensearch-project#16845)

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Andriy Redko <[email protected]>
Co-authored-by: opensearch-ci-bot <[email protected]>

---------

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Andriy Redko <[email protected]>
Co-authored-by: opensearch-ci-bot <[email protected]>
(cherry picked from commit 97f6d84)
Signed-off-by: Darshit Chanpura <[email protected]>
…egration tests as well as security policy

Signed-off-by: Darshit Chanpura <[email protected]>
Signed-off-by: Darshit Chanpura <[email protected]>
Signed-off-by: Darshit Chanpura <[email protected]>
Signed-off-by: Darshit Chanpura <[email protected]>
Signed-off-by: Darshit Chanpura <[email protected]>
…#16641)

* Changes to support IP
---------

Signed-off-by: bharath-techie <[email protected]>
We [reached agreement][1] to move this plugin to a separate repository. The
implementation was never completed here, and this code was never backported to
any release, so it is safe to remove.

[1]: opensearch-project#7771 (comment)

Signed-off-by: Andrew Ross <[email protected]>
Signed-off-by: Darshit Chanpura <[email protected]>
Signed-off-by: Darshit Chanpura <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.