Skip to content

Commit

Permalink
🎉 Ported to OpenSearch 1.1.0
Browse files Browse the repository at this point in the history
  • Loading branch information
aparo committed Nov 20, 2021
1 parent 9c47725 commit 2f93e03
Show file tree
Hide file tree
Showing 35 changed files with 629 additions and 547 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,4 @@
*.iml
\.*
!.travis.yml
/build
2 changes: 2 additions & 0 deletions NOTICE.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Medcl
Copyright 2012-2021 Medcl
62 changes: 13 additions & 49 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,48 +1,18 @@
IK Analysis for Elasticsearch
=============================
IK Analysis for OpenSearch
==========================

The IK Analysis plugin integrates Lucene IK analyzer (http://code.google.com/p/ik-analyzer/) into elasticsearch, support customized dictionary.
This is a port of [medcl's IK Analysis](https://github.com/medcl/elasticsearch-analysis-ik).

Analyzer: `ik_smart` , `ik_max_word` , Tokenizer: `ik_smart` , `ik_max_word`
The IK Analysis plugin integrates Lucene IK analyzer (http://code.google.com/p/ik-analyzer/) into opensearch, support customized dictionary.

Versions
--------

IK version | ES version
-----------|-----------
master | 7.x -> master
6.x| 6.x
5.x| 5.x
1.10.6 | 2.4.6
1.9.5 | 2.3.5
1.8.1 | 2.2.1
1.7.0 | 2.1.1
1.5.0 | 2.0.0
1.2.6 | 1.0.0
1.2.5 | 0.90.x
1.1.3 | 0.20.x
1.0.0 | 0.16.2 -> 0.19.0
Analyzer: `ik_smart` , `ik_max_word` , Tokenizer: `ik_smart` , `ik_max_word`

Install
-------

1.download or compile

* optional 1 - download pre-build package from here: https://github.com/medcl/elasticsearch-analysis-ik/releases

create plugin folder `cd your-es-root/plugins/ && mkdir ik`

unzip plugin to folder `your-es-root/plugins/ik`

* optional 2 - use elasticsearch-plugin to install ( supported from version v5.5.1 ):

```
./bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.3.0/elasticsearch-analysis-ik-6.3.0.zip
```
NOTE: replace `6.3.0` to your own elasticsearch version
2.restart elasticsearch
| OS | Command |
| ----- | ------- |
| 1.1.0 | `bin/opensearch-plugin install https://github.com/aparo/opensearch-analysis-ik/releases/download/1.1.0/opensearch-analisys-ik-1.1.0.zip` |



Expand Down Expand Up @@ -164,7 +134,7 @@ Result
### Dictionary Configuration

`IKAnalyzer.cfg.xml` can be located at `{conf}/analysis-ik/config/IKAnalyzer.cfg.xml`
or `{plugins}/elasticsearch-analysis-ik-*/config/IKAnalyzer.cfg.xml`
or `{plugins}/opensearch-analysis-ik-*/config/IKAnalyzer.cfg.xml`

```xml
<?xml version="1.0" encoding="UTF-8"?>
Expand Down Expand Up @@ -216,16 +186,16 @@ have fun.


```bash
git clone https://github.com/medcl/elasticsearch-analysis-ik
cd elasticsearch-analysis-ik
git clone https://github.com/medcl/opensearch-analysis-ik
cd opensearch-analysis-ik
git checkout tags/{version}
mvn clean
mvn compile
mvn package
```

拷贝和解压release下的文件: #{project_path}/elasticsearch-analysis-ik/target/releases/elasticsearch-analysis-ik-*.zip 到你的 elasticsearch 插件目录, 如: plugins/ik
重启elasticsearch
拷贝和解压release下的文件: #{project_path}/opensearch-analysis-ik/target/releases/opensearch-analysis-ik-*.zip 到你的 opensearch 插件目录, 如: plugins/ik
重启opensearch

3.分词测试失败
请在某个索引下调用analyze接口测试,而不是直接调用analyze接口
Expand All @@ -245,12 +215,6 @@ ik_max_word: 会将文本做最细粒度的拆分,比如会将“中华人民

ik_smart: 会做最粗粒度的拆分,比如会将“中华人民共和国国歌”拆分为“中华人民共和国,国歌”,适合 Phrase 查询。

Changes
------
*自 v5.0.0 起*

- 移除名为 `ik` 的analyzer和tokenizer,请分别使用 `ik_smart``ik_max_word`


Thanks
------
Expand Down
188 changes: 188 additions & 0 deletions build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,188 @@
import java.nio.file.Files
import org.opensearch.gradle.testclusters.StandaloneRestIntegTestTask
import org.gradle.api.tasks.Input;
import org.gradle.process.CommandLineArgumentProvider;

import java.util.LinkedHashMap;
import java.util.Map;
import java.util.function.Supplier;
import java.util.stream.Collectors;

buildscript {
dependencies {
classpath "org.opensearch.gradle:build-tools:${opensearchVersion}"
}
}

plugins {
id "co.riiid.gradle" version "0.4.2"
// the old co.riiid.gradle is not gradle 7.0 compatible
// id "com.github.humblerookie.gradle" version "0.4.4"
id "com.github.ben-manes.versions" version '0.36.0'
}

repositories {
mavenCentral()
}

group = 'org.opensearch.plugin'
version = "${opensearchVersion}"

apply plugin: 'java'
apply plugin: 'idea'
apply plugin: 'opensearch.opensearchplugin'
apply plugin: 'opensearch.testclusters'

opensearchplugin {
name = 'opensearch-analisys-ik'
description = 'IK Analysis for OpenSearch'
classname = 'org.opensearch.plugin.analysis.ik.AnalysisIkPlugin'
licenseFile = rootProject.file('LICENSE.txt')
noticeFile = rootProject.file('NOTICE.txt')
}


// In this section you declare the dependencies for your production and test code
dependencies {

compile 'org.apache.httpcomponents:httpclient:4.5.2'

runtimeOnly 'org.apache.logging.log4j:log4j-api:2.3'
// testCompile 'org.hamcrest:hamcrest-core:1.3'
// testCompile 'org.hamcrest:hamcrest-library:1.3'
// testCompile 'org.powermock:powermock-module-junit4:1.6.2'
testCompile 'org.powermock:powermock-api-mockito:1.6.2'
testCompile 'nl.jqno.equalsverifier:equalsverifier:1.7.5'
testCompile 'com.openpojo:openpojo:0.8.1'
// testImplementation("org.opensearch.test", "framework", opensearchVersion)
// // testCompile 'junit:junit:4.4'
// testImplementation("junit", "junit" , "4.12") {
// exclude(group = "org.hamcrest")
// }
}

// ignore javadoc warnings for now
tasks.withType(Javadoc) {
options.addStringOption('Xdoclint:none', '-quiet')
}

githubRelease.doFirst {
if (!System.getProperty('GITHUB_TOKEN', '')) {
throw new Exception('Missing property GITHUB_TOKEN')
}

// check if zip file is there
assert file("build/distributions/opensearch-analisys-ik-${version}.zip").exists()

// rename zip file
def currentVersion = version.replace('-SNAPSHOT', '')
def filename = "build/distributions/opensearch-analisys-ik-${version}.zip"
Files.copy(file("build/distributions/opensearch-analisys-ik-${version}.zip").toPath(), file(filename).toPath())

// configuration
github {
owner = 'aparo'
repo = 'opensearch-analisys-ik'
token = System.getProperty('GITHUB_TOKEN')
tagName = currentVersion
assets = [ filename ]
targetCommitish = 'main'
}
}

test {

//How to run Gradle test when all tests are UP-TO-DATE? add below line
dependsOn 'cleanTest'
//To enable this fail fast behavior so even a one unit test fail it keeps running
failFast = true

// Give a different location to the gradle html reports in a folder called gradlehtmlReports
reports.html.enabled = true
reports.html.setDestination(file("$projectDir/gradlehtmlReports"))
}

// // setup yaml rest tests
// testClusters {
// yamlRestTest
// }

// sourceSets {
// yamlRestTest
// }

// configurations {
// yamlRestTestImplementation.extendsFrom testImplementation
// yamlRestTestRuntimeOnly.extendsFrom testRuntimeOnly
// restTestSpecs
// }

// tasks.register('copyRestTestSpecs', Copy) {
// from zipTree(configurations.restTestSpecs.singleFile)
// into "$buildDir/restResources/restspec"
// }

// TaskProvider<Zip> bundle = project.getTasks().withType(Zip.class).named("bundlePlugin");

// // Register rest resources with source set
// sourceSets.yamlRestTest.getOutput().dir("$buildDir/restResources/restspec");

// tasks.register('yamlRestTest', StandaloneRestIntegTestTask) { testTask ->
// testTask.dependsOn(bundle, 'copyRestTestSpecs')

// def cluster = testClusters.yamlRestTest
// cluster.plugin(bundle.flatMap(AbstractArchiveTask::getArchiveFile))
// testTask.useCluster(testClusters.yamlRestTest)

// testTask.mustRunAfter(project.getTasks().named("test"))
// testTask.setTestClassesDirs(sourceSets.yamlRestTest.getOutput().getClassesDirs())
// testTask.setClasspath(sourceSets.yamlRestTest.getRuntimeClasspath())


// SystemPropertyCommandLineArgumentProvider nonInputProperties = new SystemPropertyCommandLineArgumentProvider()
// nonInputProperties.systemProperty("tests.rest.cluster", "${-> String.join(",", cluster.getAllHttpSocketURI())}")
// nonInputProperties.systemProperty("tests.cluster", "${-> String.join(",", cluster.getAllTransportPortURI())}")
// nonInputProperties.systemProperty("tests.clustername", "${-> cluster.getName()}")
// testTask.getJvmArgumentProviders().add(nonInputProperties)
// testTask.systemProperty("tests.rest.load_packaged", Boolean.FALSE.toString())
// }

// // this is a bit of a hack to make sure we run the test tests when releasing...
// check.dependsOn 'yamlRestTest'

// dependencies {
// yamlRestTestImplementation "org.opensearch.test:framework:$opensearchVersion"
// restTestSpecs "org.opensearch:rest-api-spec:$opensearchVersion"
// }

// // This will be available in 7.15 in build tools and not manually declared.
// public class SystemPropertyCommandLineArgumentProvider implements CommandLineArgumentProvider {
// private final Map<String, Object> systemProperties = new LinkedHashMap<>();

// public void systemProperty(String key, Supplier<String> value) {
// systemProperties.put(key, value);
// }

// public void systemProperty(String key, Object value) {
// systemProperties.put(key, value);
// }

// @Override
// public Iterable<String> asArguments() {
// return systemProperties.entrySet()
// .stream()
// .map(
// entry -> "-D"
// + entry.getKey()
// + "="
// + (entry.getValue() instanceof Supplier ? ((Supplier) entry.getValue()).get() : entry.getValue())
// )
// .collect(Collectors.toList());
// }

// // Track system property keys as an input so our build cache key will change if we add properties but values are still ignored
// @Input
// public Iterable<String> getPropertyNames() {
// return systemProperties.keySet();
// }
// }
1 change: 1 addition & 0 deletions gradle.properties
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
opensearchVersion = 1.1.0
Binary file added gradle/wrapper/gradle-wrapper.jar
Binary file not shown.
5 changes: 5 additions & 0 deletions gradle/wrapper/gradle-wrapper.properties
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-6.9.1-bin.zip
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
Loading

0 comments on commit 2f93e03

Please sign in to comment.