The JIRA River Plugin allows index Atlassian JIRA issues and issue comments into Elasticsearch. It's implemented as Elasticsearch river plugin and uses JIRA REST API to obtain issues from JIRA instance.
Please note that Rivers are going to be deprecated from Elasticsearch 1.5.
In order to install the plugin into Elasticsearch 1.3.x, simply run:
bin/plugin -url https://repository.jboss.org/nexus/content/groups/public-jboss/org/jboss/elasticsearch/elasticsearch-river-jira/1.7.2/elasticsearch-river-jira-1.7.2.zip -install elasticsearch-river-jira.
In order to install the plugin into Elasticsearch 1.4.x, simply run:
bin/plugin -url https://repository.jboss.org/nexus/content/groups/public-jboss/org/jboss/elasticsearch/elasticsearch-river-jira/1.8.7/elasticsearch-river-jira-1.8.7.zip -install elasticsearch-river-jira.
-----------------------------------------------------------------------
| JIRA River | Elasticsearch | JIRA | JIRA REST API | Release date |
-----------------------------------------------------------------------
| master | 1.4.0 | 5+ | 2 | |
-----------------------------------------------------------------------
| 1.8.7 | 1.4.0 | 5+ | 2 | 17.04.2018 |
-----------------------------------------------------------------------
| 1.8.6 | 1.4.0 | 5+ | 2 | 20.06.2017 |
-----------------------------------------------------------------------
| 1.8.5 | 1.4.0 | 5+ | 2 | 04.10.2015 |
-----------------------------------------------------------------------
| 1.8.4 | 1.4.0 | 5+ | 2 | 10.04.2015 |
-----------------------------------------------------------------------
| 1.8.3 | 1.4.0 | 5+ | 2 | 26.01.2015 |
-----------------------------------------------------------------------
| 1.8.2 | 1.4.0 | 5+ | 2 | 23.12.2014 |
-----------------------------------------------------------------------
| 1.8.1 | 1.4.0 | 5+ | 2 | 15.12.2014 |
-----------------------------------------------------------------------
| 1.8.0 | 1.4.0 | 5+ | 2 | 4.12.2014 |
-----------------------------------------------------------------------
| 1.7.2 | 1.3.0 | 5+ | 2 | 4.12.2014 |
-----------------------------------------------------------------------
| 1.7.1 | 1.3.0 | 5+ | 2 | 22.9.2014 |
-----------------------------------------------------------------------
| 1.7.0 | 1.3.0 | 5+ | 2 | 20.8.2014 |
-----------------------------------------------------------------------
| 1.6.0 | 1.2.0 | 5+ | 2 | 18.6.2014 |
-----------------------------------------------------------------------
| 1.5.6 | 1.0.0 | 5+ | 2 | 6.6.2014 |
-----------------------------------------------------------------------
| 1.5.5 | 1.0.0 | 5+ | 2 | 20.5.2014 |
-----------------------------------------------------------------------
| 1.4.5 | 0.90.5 | 5+ | 2 | 20.5.2014 |
-----------------------------------------------------------------------
For info about older releases, detailed changelog, planned milestones/enhancements and known bugs see github issue tracker please.
The JIRA river indexes JIRA issues and comments, and makes them searchable
by Elasticsearch. JIRA is pooled periodically to detect changed issues
(search operation with JQL query over updatedDate field) to update search
index in incremental update mode.
Periodical full update may be configured too to completely refresh search
index and remove issues deleted in JIRA from it (deletes are not catch by
incremental updates).
Creating the JIRA river can be done using:
curl -XPUT localhost:9200/_river/my_jira_river/_meta -d '
{
"type" : "jira",
"jira" : {
"urlBase" : "https://issues.jboss.org",
"username" : "jira_username",
"pwd" : "jira_user_password",
"jqlTimeZone" : "America/New York",
"timeout" : "5s",
"maxIssuesPerRequest" : 50,
"projectKeysIndexed" : "ORG,AS7",
"indexUpdatePeriod" : "5m",
"indexFullUpdatePeriod" : "1h",
"maxIndexingThreads" : 2
"jqlTempate" : "project='%s'%s%s ORDER BY updated ASC"
},
"index" : {
"index" : "my_jira_index",
"type" : "jira_issue"
},
"activity_log": {
"index" : "jira_river_activity",
"type" : "jira_river_indexupdate"
}
}
'
The example above lists all the main options controlling the creation and behavior of a JIRA river. Full list of options with description is here:
jira/urlBaseis required in order to connect to the JIRA REST API. It's only base URL, path to REST API is added automatically.jira/restApiVersionversion of JIRA REST API to be used. Default is2. You can use other values there, likelatest, but it is not assured river will work correctly in this case, as it is tested against2only. See issue #49.jira/usernameandjira/pwdare optional JIRA login credentials to access jira issues. Anonymous JIRA access is used if not provided.jira/jqlTimeZoneis optional identifier of timezone used to format time values into JQL when requesting updated issues. Timezone of Elasticsearch JVM is used if not provided. JQL uses timezone of jira user who perform JQL query (so this setting must reflex jira timezone of user provided byjira/usernameparameter), default timezone of JIRA in case of Anonymous access. Incorrect setting of this value may lead to some issue updates not reflected in search index during incremental update!!jira/timeouttime value, defines timeout for http/s REST request to the JIRA. Optional, 5s is default if not provided.jira/maxIssuesPerRequestdefines maximal number of updated issues requested from JIRA by one REST request. Optional, 50 used if not provided. The maximum allowable value is dictated by the JIRA configuration propertyjira.search.views.default.max. If you specify a value that is higher than this number, your request results will be truncated to this number anyway.jira/projectKeysIndexedcomma separated list of JIRA project keys to be indexed. Optional, list of projects is obtained from JIRA instance if omitted (so new projects are indexed automatically).jira/projectKeysExcludedcomma separated list of JIRA project keys to be excluded from indexing if list is obtained from JIRA instance (so used only if nojira/projectKeysIndexedis defined). Optional.jira/indexUpdatePeriodtime value, defines how often is search index updated from JIRA instance. Optional, default 5 minutes.jira/indexFullUpdatePeriodtime value, defines how often is search index updated from JIRA instance in full update mode. Optional, default 12 hours. You can use0to disable automatic full updates. Full update updates all issues in search index from JIRA, and removes issues deleted in JIRA from search index also. This brings more load to both JIRA and Elasticsearch servers, and may run for long time in case of JIRA instance with many issues. Incremental updates are performed between full updates as defined byindexUpdatePeriodparameter.jira/indexFullUpdateCronExpressioncontains Quartz Cron Expression defining when is full index update performed. Optional, if defined thenindexFullUpdatePeriodis not used. Available from version 1.7.2.jira/maxIndexingThreadsdefines maximal number of parallel indexing threads running for this river. Optional, default 1. This setting influences load on both JIRA and Elasticsearch servers during indexing. Threads are started per JIRA project update. If there is more threads allowed, then one is always dedicated for incremental updates only (so full updates do not block incremental updates for another projects).jira/jqlTemplateoptional parameter that defines template that is used for creating JQL to query updates for certain project and time period. For example if your usecase only needs to see issues of type BUG you may rewrite this template toissueType='Bug' AND project='%s'%s%s ORDER BY updated ASC. First%sis replaced with project key, second%swill be replaced byAND updatedDate >= "yyyy-MM-dd HH:mm"or empty string and third%swill be replaced byAND updatedDate <= "yyyy-MM-dd HH:mm"or empty string. Bare in mind that it is up to user to put quotation symbols around project key. Most cases work without quotation, but project keys that are also reserved words will give you errors. Note that elasticsearch-river-jira depends on issues being primarily ordered by updated field in ascending order. so for current version you need to have your JQL template end withORDER BY updated ASC. Available from version 1.8.4.index/indexdefines name of search index where JIRA issues are stored. Parameter is optional, name of river is used if omitted. See related notes later!index/typedefines type used when issue is stored into search index. Parameter is optional,jira_issueis used if omitted. See related notes later!index/field_river_name,index/field_project_key,index/field_issue_key,index/field_jira_urlindex/fields,index/value_filters,index/jira_field_issue_document_idcan be used to change structure of indexed issue document. See 'JIRA issue index document structure' chapter.index/comment_modedefines mode of issue comments indexing:none- no comments indexed,embedded- comments indexed as array in issue document,child- comment indexed as separate document with parent-child relation to issue document,standalone- comment indexed as separate document. Setting is optional,embeddedvalue is default if not provided.index/comment_typedefines type used when issue comment is stored into search index inchildorstandalonemode. Parameter is optional,jira_issue_commentis used if omitted. See related notes later!index/field_comments,index/comment_fieldscan be used to change structure of comment information in indexed documents. See 'JIRA issue index document structure' chapter.index/changelog_modedefines mode of issue changelog indexing:none- no changelog indexed,embedded- changelog indexed as array in issue document,child- changelog indexed as separate document with parent-child relation to issue document,standalone- changelog indexed as separate document. Setting is optional,nonevalue is default if not provided.index/changelog_typedefines type used when issue changelog is stored into search index inchildorstandalonemode. Parameter is optional,jira_issue_changeis used if omitted. See related notes later!index/field_changelogs,index/changelog_fieldscan be used to change structure of changelog information in indexed documents. See 'JIRA issue index document structure' chapter.index/preprocessorsoptional parameter. Defines chain of preprocessors applied to issue data read from JIRA before stored into index. See related notes later!activity_logpart defines where information about jira river index update activity are stored. If omitted then no activity information are stored.activity_log/indexdefines name of index where information about jira river activity are stored.activity_log/typedefines type used to store information about jira river activity. Parameter is optional,jira_river_indexupdateis used if ommited.
Time value in configuration is number representing milliseconds, but you can use these postfixes appended to the number to define units: s for seconds, m for minutes, h for hours, d for days and w for weeks. So for example value 5h means five fours, 2w means two weeks.
To get rid of some unwanted WARN log messages add next line to the logging configuration file of your Elasticsearch instance which is config/logging.yml:
org.apache.commons.httpclient: ERROR
And to get rid of extensive INFO messages from index update runs use:
org.jboss.elasticsearch.river.jira.JIRAProjectIndexer: WARN
Configured Search index is NOT explicitly created by river code. You need to create it manually BEFORE river creation.
curl -XPUT 'http://localhost:9200/my_jira_index/'
Type Mapping for issue is not explicitly created by river
code for configured document type. The river REQUIRES Automatic Timestamp Field and keyword analyzer for project_key and source fields to be able to correctly remove issues deleted in JIRA from index during full update! So you need to create issue type mapping manually BEFORE river creation, with next content at least:
curl -XPUT localhost:9200/my_jira_index/jira_issue/_mapping -d '
{
"jira_issue" : {
"_timestamp" : { "enabled" : true },
"properties" : {
"project_key" : {"type" : "string", "analyzer" : "keyword"},
"source" : {"type" : "string", "analyzer" : "keyword"}
}
}
}
'
Same apply for 'comment' and 'changelog' mapping if you use child or standalone mode!
curl -XPUT localhost:9200/my_jira_index/jira_issue_comment/_mapping -d '
{
"jira_issue_comment" : {
"_timestamp" : { "enabled" : true },
"properties" : {
"project_key" : {"type" : "string", "analyzer" : "keyword"},
"source" : {"type" : "string", "analyzer" : "keyword"}
}
}
}
'
curl -XPUT localhost:9200/my_jira_index/jira_issue_change/_mapping -d '
{
"jira_issue_change" : {
"_timestamp" : { "enabled" : true },
"properties" : {
"project_key" : {"type" : "string", "analyzer" : "keyword"},
"source" : {"type" : "string", "analyzer" : "keyword"}
}
}
}
'
You can store mappings in Elasticsearch node configuration alternatively.
See next chapter for description of JIRA issue indexed document structure to create better mappings meeting your needs.
If you use update activity logging then you can create index and mapping for it too:
curl -XPUT 'http://localhost:9200/jira_river_activity/'
curl -XPUT localhost:9200/jira_river_activity/jira_river_indexupdate/_mapping -d '
{
"jira_river_indexupdate" : {
"properties" : {
"river_name" : {"type" : "string", "analyzer" : "keyword"},
"project_key" : {"type" : "string", "analyzer" : "keyword"},
"update_type" : {"type" : "string", "analyzer" : "keyword"},
"result" : {"type" : "string", "analyzer" : "keyword"}
}
}
}
'
You can configure which fields from JIRA will be available in search index and under which names. See river_configuration_default.json file for example of river configuration, which is used to create default configuration.
JIRA River writes JSON document with following structure to the search index for issue by default. Issue key is used as document id in search index by default (you can change this over index/jira_field_issue_document_id setting which defines field in issue data which value is used as document id. Be careful for uniqueness of this value!).
----------------------------------------------------------------------------------------------------------------------------------------------------
| **index field** | **JIRA JSON field** | **indexed field value notes** | **river configuration** |
----------------------------------------------------------------------------------------------------------------------------------------------------
| source | N/A | name of JiraRiver document was indexed over | index/field_river_name |
----------------------------------------------------------------------------------------------------------------------------------------------------
| document_url | N/A | URL to show issue in JIRA GUI | index/field_jira_url |
----------------------------------------------------------------------------------------------------------------------------------------------------
| project_key | fields.project.key | Key of project in JIRA | index/field_project_key |
----------------------------------------------------------------------------------------------------------------------------------------------------
| project_name | fields.project.name | Name of project in JIRA | index/fields |
----------------------------------------------------------------------------------------------------------------------------------------------------
| issue_key | key | Issue key from jira - also used as ID of document in the index, eg. `ORG-12` | index/field_issue_key |
----------------------------------------------------------------------------------------------------------------------------------------------------
| issue_type | fields.issuetype.name | Name of issue type, eg. `Bug`, `Feature Request` etc. | index/fields |
----------------------------------------------------------------------------------------------------------------------------------------------------
| summary | fields.summary | Title of issue | index/fields |
----------------------------------------------------------------------------------------------------------------------------------------------------
| status | fields.status.name | Name of issue status, eg. `Open`, `Resolved`, `Closed` etc. | index/fields |
----------------------------------------------------------------------------------------------------------------------------------------------------
| created | fields.created | Full timestamp format eg. `2012-08-15T03:30:02.000-0400` | index/fields |
----------------------------------------------------------------------------------------------------------------------------------------------------
| updated | fields.updated | Full timestamp format eg. `2012-08-15T03:30:02.000-0400` | index/fields |
----------------------------------------------------------------------------------------------------------------------------------------------------
| resolutiondate | fields.resolutiondate | Full timestamp format eg. `2012-08-15T03:30:02.000-0400` | index/fields |
----------------------------------------------------------------------------------------------------------------------------------------------------
| description | fields.description | Main description text for issue. May contain JIRA WIKI syntax | index/fields |
----------------------------------------------------------------------------------------------------------------------------------------------------
| labels | fields.labels | Array od String values with all labels | index/fields |
----------------------------------------------------------------------------------------------------------------------------------------------------
| reporter | fields.reporter | Object with fields `username`, `email_address`, `display_name` | index/fields |
----------------------------------------------------------------------------------------------------------------------------------------------------
| assignee | fields.assignee | Object with fields `username`, `email_address`, `display_name` | index/fields |
----------------------------------------------------------------------------------------------------------------------------------------------------
| fix_versions | fields.fixVersions | Array containing Objects with `name` field | index/fields |
----------------------------------------------------------------------------------------------------------------------------------------------------
| components | field.components | Array containing Objects with `name` field | index/fields |
----------------------------------------------------------------------------------------------------------------------------------------------------
| comments | field.comment.comments| Array of comments (comment indexing in `embedded` mode is used by default) | index/field_comments |
----------------------------------------------------------------------------------------------------------------------------------------------------
| changelogs | changelog.histories | Array of changelog items (not indexed by default) | index/field_changelogs |
----------------------------------------------------------------------------------------------------------------------------------------------------
JIRA River uses following structure to store comment informations in search index by default. Comment id is used as document id in search index in child or standalone mode.
----------------------------------------------------------------------------------------------------------------------------------------------------
| **index field** | **JIRA comment JSON field** | **indexed field value notes** | **river configuration** |
----------------------------------------------------------------------------------------------------------------------------------------------------
| source | N/A | name of JiraRiver comment was indexed over, not used in `embedded` mode | index/field_river_name |
----------------------------------------------------------------------------------------------------------------------------------------------------
| project_key | N/A | Key of project in JIRA comment is for, not used in `embedded` mode | index/field_project_key |
----------------------------------------------------------------------------------------------------------------------------------------------------
| issue_key | N/A | key of issue comment is for, not used in `embedded` mode | index/field_issue_key |
----------------------------------------------------------------------------------------------------------------------------------------------------
| document_url | N/A | URL to show comment in JIRA GUI | index/field_jira_url |
----------------------------------------------------------------------------------------------------------------------------------------------------
| comment_id | id | ID of comment in JIRA | index/comment_fields |
----------------------------------------------------------------------------------------------------------------------------------------------------
| comment_body | body | Comment text. May contain JIRA WIKI syntax | index/comment_fields |
----------------------------------------------------------------------------------------------------------------------------------------------------
| comment_created | created | Full timestamp format eg. `2012-08-15T03:30:02.000-0400` | index/comment_fields |
----------------------------------------------------------------------------------------------------------------------------------------------------
| comment_updated | updated | Full timestamp format eg. `2012-08-15T03:30:02.000-0400` | index/comment_fields |
----------------------------------------------------------------------------------------------------------------------------------------------------
| comment_author | author | Object with fields `username`, `email_address`, `display_name` | index/comment_fields |
----------------------------------------------------------------------------------------------------------------------------------------------------
| comment_updater | updateAuthor | Object with fields `username`, `email_address`, `display_name` | index/comment_fields |
----------------------------------------------------------------------------------------------------------------------------------------------------
JIRA River uses following structure to store changelog informations in search index by default. Changelog item id is used as document id in search index in child or standalone mode.
----------------------------------------------------------------------------------------------------------------------------------------------------
| **index field** | **JIRA comment JSON field** | **indexed field value notes** | **river configuration** |
----------------------------------------------------------------------------------------------------------------------------------------------------
| source | N/A | name of JiraRiver item was indexed over, not used in `embedded` mode | index/field_river_name |
----------------------------------------------------------------------------------------------------------------------------------------------------
| project_key | N/A | Key of project in JIRA item is for, not used in `embedded` mode | index/field_project_key |
----------------------------------------------------------------------------------------------------------------------------------------------------
| issue_key | N/A | key of issue item is for, not used in `embedded` mode | index/field_issue_key |
----------------------------------------------------------------------------------------------------------------------------------------------------
| document_url | N/A | URL to show issue for this changelog in JIRA GUI | index/field_jira_url |
----------------------------------------------------------------------------------------------------------------------------------------------------
| change_id | id | ID of change item in JIRA | index/changelog_fields |
----------------------------------------------------------------------------------------------------------------------------------------------------
| change_items | items | Array of changed items objects (see later) | index/changelog_fields |
----------------------------------------------------------------------------------------------------------------------------------------------------
| change_created | created | Full timestamp format eg. `2012-08-15T03:30:02.000-0400` | index/changelog_fields |
----------------------------------------------------------------------------------------------------------------------------------------------------
| change_author | author | Object with fields `username`, `email_address`, `display_name` | index/changelog_fields |
----------------------------------------------------------------------------------------------------------------------------------------------------
Example of change item object from JIRA:
{
"field": "Fix Version",
"fieldtype": "jira",
"from": null,
"fromString": null,
"to": "10225",
"toString": "1.0.0.GA"
}
You can also implement and configure some preprocessors, which allows you to change/extend issue information loaded from JIRA and store these changes/extensions to the search index. This allows you for example value normalizations, or creation of some index fields with values aggregated from more issue fields.
Framework called structured-content-tools is used to implement these preprocessors. Example how to configure preprocessors is visible here. Some generic configurable preprocessor implementations are available as part of the structured-content-tools framework.
JIRA river supports next REST commands for management purposes. Note my_jira_river in examples is name of jira river you can call operation for, soi replace it with real name for your calls.
Get state info about jira river operation:
curl -XGET localhost:9200/_river/my_jira_river/_mgm_jr/state
Stop jira river indexing process. Process is stopped permanently, so even after complete elasticsearch cluster restart or river migration to another node. You need to restart it over management REST API (see next command):
curl -XPOST localhost:9200/_river/my_jira_river/_mgm_jr/stop
Restart JIRA river indexing process. Configuration of river is reloaded during restart. You can restart running indexing, or stopped indexing (see previous command):
curl -XPOST localhost:9200/_river/my_jira_river/_mgm_jr/restart
Force full index update for all jira projects:
curl -XPOST localhost:9200/_river/my_jira_river/_mgm_jr/fullupdate
Force full index update for jira project with key projectKey:
curl -XPOST localhost:9200/_river/my_jira_river/_mgm_jr/fullupdate/projectKey
Force incremental index update for all jira projects:
curl -XPOST localhost:9200/_river/my_jira_river/_mgm_jr/incrementalupdate
Force incremental index update for jira project with key projectKey:
curl -XPOST localhost:9200/_river/my_jira_river/_mgm_jr/incrementalupdate/projectKey
List names of all JIRA Rivers running in ES cluster:
curl -XGET localhost:9200/_jira_river/list
This software is licensed under the Apache 2 license, quoted below.
Copyright 2012 - 2014 Red Hat Inc. and/or its affiliates and other contributors as indicated by the @authors tag.
All rights reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may not
use this file except in compliance with the License. You may obtain a copy of
the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations under
the License.