Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DRAFT] Resource permissions and sharing #166

Draft
wants to merge 362 commits into
base: main
Choose a base branch
from

Conversation

DarshitChanpura
Copy link
Owner

@DarshitChanpura DarshitChanpura commented Aug 30, 2024

@DarshitChanpura
Copy link
Owner Author

From latest ./gradlew run, it recognizes NoOpResourcePlugin.

[2024-08-30T12:42:17,976][INFO ][o.o.e.ExtensionsManager  ] [runTask-0] ExtensionsManager initialized
[2024-08-30T12:42:19,022][INFO ][o.o.a.r.ResourceService  ] [runTask-0] Security plugin disabled: Using NoOpResourcePlugin

Next test would be to determine whether it recognizes security plugin

@DarshitChanpura
Copy link
Owner Author

security plugin bootstrap: DarshitChanpura/security#10 (comment)

@DarshitChanpura DarshitChanpura changed the base branch from main to feature/segment-replication September 6, 2024 17:21
@DarshitChanpura DarshitChanpura changed the base branch from feature/segment-replication to main September 6, 2024 17:21
* The index where resource meta-data is stored
* @return the name of the parent index where resource meta-data is stored
*/
String getResourceIndex();
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can a plugin define multiple types of resources and resource indices?

*/
public class ShareWith implements ToXContentFragment {

private List<String> users;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can see pushback on introducing these concepts into the core. Is there a way to make this generic and let a plugin define an implementation?

@DarshitChanpura DarshitChanpura force-pushed the resource-permissions branch 2 times, most recently from d8710cc to 23fcfba Compare October 2, 2024 20:41
DarshitChanpura and others added 21 commits October 4, 2024 18:14
* Update Apache Lucene to 9.12.0

Signed-off-by: Andriy Redko <[email protected]>

* change to IOContext READONCE in locations where the file is not expected to be read multiple times.

Signed-off-by: Marc Handalian <[email protected]>

* Use READ IOContext for all non Segment* files when copying node-node

Signed-off-by: Marc Handalian <[email protected]>

* Fixing more test failures

Signed-off-by: Andriy Redko <[email protected]>

* Move Composite912Codec under org.opensearch.index.codec.composite.composite912 package

Signed-off-by: Andriy Redko <[email protected]>

---------

Signed-off-by: Andriy Redko <[email protected]>
Signed-off-by: Marc Handalian <[email protected]>
Co-authored-by: Marc Handalian <[email protected]>
…4993)

* Avoid deep copy and other allocation improvements
* Refactoring based on PR Comments and added JavaDocs
* Added more comments
* Added character for Triggering Jenkins build
* Changes to cover collectZeroDocEntries method
* Updated comment based on change in method's functionality
* Added test to cover branches in collectZeroDocEntriesIfRequired
* Rebased and resolved changelog conflict

---------

Signed-off-by: expani <[email protected]>
…ject#15925)

* cancellation related

Signed-off-by: Kiran Prakash <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Kiran Prakash <[email protected]>

* add better cancellation reason

Signed-off-by: Kiran Prakash <[email protected]>

* Update DefaultTaskCancellationTests.java

Signed-off-by: Kiran Prakash <[email protected]>

* refactor

Signed-off-by: Kiran Prakash <[email protected]>

* refactor

Signed-off-by: Kiran Prakash <[email protected]>

* Update DefaultTaskCancellation.java

Signed-off-by: Kiran Prakash <[email protected]>

* Update DefaultTaskCancellation.java

Signed-off-by: Kiran Prakash <[email protected]>

* Update DefaultTaskCancellation.java

Signed-off-by: Kiran Prakash <[email protected]>

* Update DefaultTaskSelectionStrategy.java

Signed-off-by: Kiran Prakash <[email protected]>

* refactor

Signed-off-by: Kiran Prakash <[email protected]>

* refactor node level threshold

Signed-off-by: Kiran Prakash <[email protected]>

* use query group task

Signed-off-by: Kaushal Kumar <[email protected]>

* code clean up and refactorings

Signed-off-by: Kaushal Kumar <[email protected]>

* add unit tests and fix existing ones

Signed-off-by: Kaushal Kumar <[email protected]>

* uncomment the test case

Signed-off-by: Kaushal Kumar <[email protected]>

* update CHANGELOG

Signed-off-by: Kaushal Kumar <[email protected]>

* fix imports

Signed-off-by: Kaushal Kumar <[email protected]>

* add queryGroupService

Signed-off-by: Kaushal Kumar <[email protected]>

* refactor and add UTs for new constructs

Signed-off-by: Kaushal Kumar <[email protected]>

* fix javadocs

Signed-off-by: Kaushal Kumar <[email protected]>

* remove code clutter

Signed-off-by: Kaushal Kumar <[email protected]>

* change annotation version and task selection strategy

Signed-off-by: Kaushal Kumar <[email protected]>

* rename a util class

Signed-off-by: Kaushal Kumar <[email protected]>

* remove wrappers from resource type

Signed-off-by: Kaushal Kumar <[email protected]>

* apply spotless

Signed-off-by: Kaushal Kumar <[email protected]>

* address comments

Signed-off-by: Kaushal Kumar <[email protected]>

* add rename changes

Signed-off-by: Kaushal Kumar <[email protected]>

* address comments

Signed-off-by: Kaushal Kumar <[email protected]>

* initial changes

Signed-off-by: Kaushal Kumar <[email protected]>

* refactor changes and logical bug fix

Signed-off-by: Kaushal Kumar <[email protected]>

* add chanegs

Signed-off-by: Kaushal Kumar <[email protected]>

* address comments

Signed-off-by: Kaushal Kumar <[email protected]>

* temp changes

Signed-off-by: Kaushal Kumar <[email protected]>

* add UTs

Signed-off-by: Kaushal Kumar <[email protected]>

* add changelog

Signed-off-by: Kaushal Kumar <[email protected]>

* add task completion listener hook

Signed-off-by: Kaushal Kumar <[email protected]>

* add remaining pieces to make the feature functional

Signed-off-by: Kaushal Kumar <[email protected]>

* extend stats and fix bugs

Signed-off-by: Kaushal Kumar <[email protected]>

* fix bugs and add logic to make SBP work with wlm

Signed-off-by: Kaushal Kumar <[email protected]>

* address comments

Signed-off-by: Kaushal Kumar <[email protected]>

* fix bugs and SBP ITs

Signed-off-by: Kaushal Kumar <[email protected]>

* add missed applyCluster state change

Signed-off-by: Kaushal Kumar <[email protected]>

* address comments

Signed-off-by: Kaushal Kumar <[email protected]>

* decouple queryGroupService and cancellationService

Signed-off-by: Kaushal Kumar <[email protected]>

* replace StateApplier with StateListener interface

Signed-off-by: Kaushal Kumar <[email protected]>

* fix precommit errors

Signed-off-by: Kaushal Kumar <[email protected]>

---------

Signed-off-by: Kiran Prakash <[email protected]>
Signed-off-by: Kaushal Kumar <[email protected]>
Co-authored-by: Kiran Prakash <[email protected]>
* Segmented cache changes for TieredCache

Signed-off-by: Sagar Upadhyaya <[email protected]>

* Adding change log

Signed-off-by: Sagar Upadhyaya <[email protected]>

* Allow segment number to be power of two

Signed-off-by: Sagar Upadhyaya <[email protected]>

* Moving common tiered cache IT methods to a common base class

Signed-off-by: Sagar Upadhyaya <[email protected]>

* Adding disk took time IT test with multiple segment

Signed-off-by: Sagar Upadhyaya <[email protected]>

* Correcting changelog

Signed-off-by: Sagar Upadhyaya <[email protected]>

* Addressing comments

Signed-off-by: Sagar Upadhyaya <[email protected]>

* Fixing invalid segment count variable name

Signed-off-by: Sagar Upadhyaya <[email protected]>

* Introducing new settings for size for respective cache tier

Signed-off-by: Sagar Upadhyaya <[email protected]>

* Changing the default segmentCount logic

Signed-off-by: Sagar Upadhyaya <[email protected]>

* Fixing missing java doc issue

Signed-off-by: Sagar Upadhyaya <[email protected]>

---------

Signed-off-by: Sagar Upadhyaya <[email protected]>
Signed-off-by: Sagar <[email protected]>
* Refactoring builder tests

Signed-off-by: Bharathwaj G <[email protected]>

* adding date tests

Signed-off-by: Bharathwaj G <[email protected]>

---------

Signed-off-by: Bharathwaj G <[email protected]>
…#16110)

* Change successfulSearchShardIndices to Set<Index>

Signed-off-by: David Zane <[email protected]>

* Update CHANGELOG.md

Signed-off-by: Ankit Jain <[email protected]>

---------

Signed-off-by: David Zane <[email protected]>
Signed-off-by: Ankit Jain <[email protected]>
Co-authored-by: Ankit Jain <[email protected]>
…azure (opensearch-project#16217)

* Bump com.azure:azure-json in /plugins/repository-azure

Bumps [com.azure:azure-json](https://github.com/Azure/azure-sdk-for-java) from 1.1.0 to 1.3.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-java/releases)
- [Commits](Azure/azure-sdk-for-java@v1.1.0...v1.3.0)

---
updated-dependencies:
- dependency-name: com.azure:azure-json
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Updating SHAs

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
…es (opensearch-project#16237)

* Remove force to TLSv1.2 in gradle.properties

Signed-off-by: Craig Perkins <[email protected]>

* Add TLSv1.3

Signed-off-by: Craig Perkins <[email protected]>

---------

Signed-off-by: Craig Perkins <[email protected]>
…16238)

* Fix Cache breaking change

Signed-off-by: Sagar Upadhyaya <[email protected]>

* Empty commit to trigger build

Signed-off-by: Sagar Upadhyaya <[email protected]>

---------

Signed-off-by: Sagar Upadhyaya <[email protected]>
…omposite99Codec (opensearch-project#16227)

* Refactoring builder tests
* composite912 codec refactor changes

---------

Signed-off-by: Bharathwaj G <[email protected]>
…plugins/repository-gcs (opensearch-project#16216)

* Bump com.google.api-client:google-api-client in /plugins/repository-gcs

Bumps [com.google.api-client:google-api-client](https://github.com/googleapis/google-api-java-client) from 2.2.0 to 2.7.0.
- [Release notes](https://github.com/googleapis/google-api-java-client/releases)
- [Changelog](https://github.com/googleapis/google-api-java-client/blob/main/CHANGELOG.md)
- [Commits](googleapis/google-api-java-client@v2.2.0...v2.7.0)

---
updated-dependencies:
- dependency-name: com.google.api-client:google-api-client
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Updating SHAs

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: gaobinlong <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
Co-authored-by: gaobinlong <[email protected]>
…s-fixture (opensearch-project#16212)

* Bump com.squareup.okio:okio in /test/fixtures/hdfs-fixture

Bumps [com.squareup.okio:okio](https://github.com/square/okio) from 3.9.0 to 3.9.1.
- [Release notes](https://github.com/square/okio/releases)
- [Changelog](https://github.com/square/okio/blob/master/CHANGELOG.md)
- [Commits](square/okio@parent-3.9.0...3.9.1)

---
updated-dependencies:
- dependency-name: com.squareup.okio:okio
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
…opensearch-project#16213)

* Bump io.grpc:grpc-api from 1.57.2 to 1.68.0 in /plugins/discovery-gce

Bumps [io.grpc:grpc-api](https://github.com/grpc/grpc-java) from 1.57.2 to 1.68.0.
- [Release notes](https://github.com/grpc/grpc-java/releases)
- [Commits](grpc/grpc-java@v1.57.2...v1.68.0)

---
updated-dependencies:
- dependency-name: io.grpc:grpc-api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Updating SHAs

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

* Centralize grpc version management

Signed-off-by: Andriy Redko <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Andriy Redko <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andriy Redko <[email protected]>
DarshitChanpura and others added 8 commits December 18, 2024 19:17
Signed-off-by: Darshit Chanpura <[email protected]>
…#16641)

* Changes to support IP
---------

Signed-off-by: bharath-techie <[email protected]>
We [reached agreement][1] to move this plugin to a separate repository. The
implementation was never completed here, and this code was never backported to
any release, so it is safe to remove.

[1]: opensearch-project#7771 (comment)

Signed-off-by: Andrew Ross <[email protected]>
Signed-off-by: Darshit Chanpura <[email protected]>
Signed-off-by: Darshit Chanpura <[email protected]>
DarshitChanpura and others added 20 commits December 20, 2024 12:15
Signed-off-by: Darshit Chanpura <[email protected]>
…/repository-azure (opensearch-project#16895)

* Bump com.nimbusds:oauth2-oidc-sdk in /plugins/repository-azure

Bumps [com.nimbusds:oauth2-oidc-sdk](https://bitbucket.org/connect2id/oauth-2.0-sdk-with-openid-connect-extensions) from 11.19.1 to 11.20.1.
- [Changelog](https://bitbucket.org/connect2id/oauth-2.0-sdk-with-openid-connect-extensions/src/master/CHANGELOG.txt)
- [Commits](https://bitbucket.org/connect2id/oauth-2.0-sdk-with-openid-connect-extensions/branches/compare/11.20.1..11.19.1)

---
updated-dependencies:
- dependency-name: com.nimbusds:oauth2-oidc-sdk
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Updating SHAs

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
…stribution/packages (opensearch-project#16896)

* Bump com.netflix.nebula.ospackage-base in /distribution/packages

Bumps com.netflix.nebula.ospackage-base from 11.10.0 to 11.10.1.

---
updated-dependencies:
- dependency-name: com.netflix.nebula.ospackage-base
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
…xtures/hdfs-fixture (opensearch-project#16898)

* Bump ch.qos.logback:logback-classic in /test/fixtures/hdfs-fixture

Bumps [ch.qos.logback:logback-classic](https://github.com/qos-ch/logback) from 1.5.12 to 1.5.15.
- [Commits](qos-ch/logback@v_1.5.12...v_1.5.15)

---
updated-dependencies:
- dependency-name: ch.qos.logback:logback-classic
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
…t#16897)

* Bump lycheeverse/lychee-action from 2.1.0 to 2.2.0

Bumps [lycheeverse/lychee-action](https://github.com/lycheeverse/lychee-action) from 2.1.0 to 2.2.0.
- [Release notes](https://github.com/lycheeverse/lychee-action/releases)
- [Commits](lycheeverse/lychee-action@v2.1.0...v2.2.0)

---
updated-dependencies:
- dependency-name: lycheeverse/lychee-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
…earch-project#16844)

* Extract jars to sub dirs during thirdPartyAudit task.

Signed-off-by: Finn Carroll <[email protected]>

* Change regex to split on '-'/'.'. Ignore version.

Signed-off-by: Finn Carroll <[email protected]>

* Split on .jar for sub folder prefix.

Signed-off-by: Finn Carroll <[email protected]>

---------

Signed-off-by: Finn Carroll <[email protected]>
…ository-azure (opensearch-project#16918)

* Bump com.microsoft.azure:msal4j in /plugins/repository-azure

Bumps [com.microsoft.azure:msal4j](https://github.com/AzureAD/microsoft-authentication-library-for-java) from 1.17.2 to 1.18.0.
- [Release notes](https://github.com/AzureAD/microsoft-authentication-library-for-java/releases)
- [Changelog](https://github.com/AzureAD/microsoft-authentication-library-for-java/blob/dev/changelog.txt)
- [Commits](AzureAD/microsoft-authentication-library-for-java@v1.17.2...v1.18.0)

---
updated-dependencies:
- dependency-name: com.microsoft.azure:msal4j
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Updating SHAs

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
…ixtures/hdfs-fixture (opensearch-project#16919)

* Bump org.apache.commons:commons-text in /test/fixtures/hdfs-fixture

Bumps org.apache.commons:commons-text from 1.12.0 to 1.13.0.

---
updated-dependencies:
- dependency-name: org.apache.commons:commons-text
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
Signed-off-by: Darshit Chanpura <[email protected]>
Signed-off-by: Darshit Chanpura <[email protected]>
Signed-off-by: Darshit Chanpura <[email protected]>
Signed-off-by: Darshit Chanpura <[email protected]>
Introduce auxiliary transport to NetworkPlugin and add gRPC plugin.

Auxiliary transports are optional lifecycle components provided by
network plugins which run in parallel to the http server/native
transport. They are distinct from the existing NetworkPlugin
interfaces of 'getTransports' and 'getHttpTransports' as auxiliary
transports are optional. Each AuxTransport implements it's own
'aux.transport.type' and 'aux.transport.<type>.ports' setting. Since
Security.java initializes previous to Node.java during bootstrap
socket binding permissions are granted based on
'aux.transport.<type>.ports' for each enabled 'aux.transport.type',
falling back to a default if no ports are specified.

Signed-off-by: Finn Carroll <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.