This TYPO3 extension aims to enable the query and display of arbitrary Solr indexes.
It provides the ability to configure many aspects of the query – e.g. query fields, facets, filtering through TypoScript – and set up the display through Fluid templates. Partials for standard display features as well as a number of View Helpers that help creating those templates are included in the extension.
For developing this extension clone this repository and install the dependencies via composer with composer install.
The extensions aims are to provide:
- access to arbitrary Solr indexes in TYPO3
- minimal configuration for a basic display
- rich templating abilities for creating complex displays
For the most basic example you need to:
add the find plug-in to a TYPO3 page
include the find plug-in’s template in your site/page’s template
configure your index information in the TypoScript template for that page. At minimum you should set the address of your default Solr index and the fields to display in the result list:
plugin.tx_find.settings { connections { default { options { host = solr.local port = 8080 path = /solr/myIndex } } } standardFields { title = title snippet = publisher } }With this setup, find will give you a search form for your index’ default search field and display a result list with the document’s »title« field as the heading of the item and the »publisher« field added as a snippet beneath it. Each result can be clicked to show the detail view for that document which presents a definition list of the fields in the Solr document.
you can then add facets for text fields and numeric range selection, e.g.:
plugin.tx_find.settings { facets { 10 { id = format field = format } 20 { id = year type = Histogram field = publishDateSort } } }additional query fields for an extended search option can also be added:
plugin.tx_find.settings { queryFields { 10 { id = title type = Text extended = 1 } } }
There are many more TypoScript configuration options. Please refer to the rest of this document and the example configurations provided in the Projects folder to learn about them.
A few example projects can be found on github. While you do not have the Solr indexes they are made for, they may give you an idea what more elaborate configurations may look like and how their templates are set up. The projects are:
- gbv: basic configuration for a big library index with multiple search fields and facets but without customised display
- hans: simple configuration for a smaller library index with multiple search fields, facets and mildly customised templates
- jfk-institut-zeitungen: simple configuration with a single search field, a non-trivial facet and a custom template without detail view
- Edfu: elaborate configuration with two search fields, two facets and highly customised display
- GermaniaSacra: elaborate configuration with many search fields, facets, data export, complex queries and highly customised display
- test: configuration to demonstrate the settings of this Readme as well as the included View Helpers
These projects are structured in a common way (with potentially only a subset of the folders / files existing)
- projectname.ts: TypoScript configuration for the setup
- EXT-Private: symlink to the extension’s Resources/Private folder; this is used as a scheme to link the extension’s partials and templates into the custom configuration so they can be used in the custom configuration
- Templates/Search: typically some of these are symlinked to the originals
- Index.html and Detail.html for the display
- Index.data and Detail.data for loading data
- Suggest.data for autocomplete / suggest responses
- Partials: Contains symlinks to the folders inside the extension’s »Partials« folder. If the default partials are to be overridden, a an actual folder with partials can be placed here. You can also add your own partials / folder of partials here. The standard partials provided by the extension are in the folders
- Components: elements used to create the page
- Display: create markup for fields in the document
- Facets: create facets
- Formats: various output formats used by the data format
- Page: standard elements to add to the page
- Pager: creates the pager for result lists
- Language: localisation files or symlinks to the extension’s localisation files
- Resources: JavaScript, CSS, images used by the project’s templates and partials
All settings discussed in this section are inside the plugin.tx_find.settings array of the TypoScript configuration.
You can have multiple Solr connections. Every connection needs to use a provider and an options array.
The plugin.tx_find.settings.activeConnection determines the currently used connection. The default value is default.
The options settings array in a connection definition is used to configure access to the Solr index. It contains:
host[127.0.0.1]: hostname of the server running the indexport[8080]: port of the Solr service on the serverpath[/solr/]: path of the Index on the Solr servertimeout[5]: number of seconds before a Solr request times outscheme[http]: URI scheme of the connection
Example:
plugin.tx_find.settings { connections { default { provider = Subugoe\Find\Service\SolrServiceProvider options { host = 127.0.0.1 port = 8080 path = /solr/ timeout = 5 scheme = http } } } }
When using the eDisMax feature solr offers, add
features.eDisMax = 1
The queryFields setting configures the search form. It is a numbered array of arrays, one for each query field that can be used. The query fields have a number of parameters depending on their type:
id(required): the id for the query field; this is used in URL parameters (tx_find_find[q][myID]) and to identify the localised label for the query fieldtype(required): the type of the query field; the partial with this name inPartials/Form/Fieldsis used to create the field for input form; the default set of partials provides the Text, Range, Hidden, Select, SelectFacet and Radio options, a few of which depend on specific code in the controller to create the right queriesquery: a sprintf string with the Solr query for this field, e.g.title_search:%s; if not given the default query$id:%sis used (where$idis the value of theidfield); this lets you use more complex queries (e.g. querying several fields at once or adding{!join}to a query), it also supports multiple parameters (see the Range type);extended[0]: if true, the query field will only be visible in the extended search formnoescape[0]: if true, the extension will not escape the user input before querying the index; this allows technically inclined users to run their own Solr queries; but it opens the risk of users accidentally entering invalid queries which will cause Solr exceptions (which the standard setup catches and offers the user a link for running an escaped query)phrase[0]: if true, the string in the field will be phrase escaped – rather than term escaped – before being placed in the Solr queryhidden[0]: if true, the input field will not be displayed; however the field will be displayed if a term for it is passed in a search parameter
The default configuration sets up a general Text query for index 0 with id default, as well as a Hidden field to transport the state of the search form with for index 10001 with id extended and a Text field for raw Solr queries for index 10002 with id raw. Please be aware of these configurations so you can override or delete them as needed.
Some of the search field types have custom behaviour and specific configuration options.
The Text field can be the simplest field available. It also allows advanced behaviour by adding autocomplete or a checkbox to select an alternate query style.
queryAlternate: an array of alternative queries that can be configured for the Text type; it creates a checkbox next to the input field which toggles between the providedqueryand the firstqueryAlternateautocomplete[0]: if true, a field of Text type will be hooked up for autocompletion using Solr suggest queryautocompleteDictionary: name of the dictionary the Solr suggest query should usedefault: default values to use in the query if no value is provided by the user (yet); may be a single value string (e.g. for the default state of checkboxes) or an array (especially useful for range queries)
Examples:
plugin.tx_find.settings.queryFields {
10 {
id = name
type = Text
}
11 {
id = name2
type = Text
query = name:%1$s
queryAlternate.1 = name:%1$s~0.7
noescape = 1
}
12 {
id = name3
type = Text
autocomplete = 1
autocompleteDictionary = name
}
}
The Range field creates two text inputs for the arguments q.{fieldInfo.id}.0 and q.{fieldInfo.id}.1. This can be used with a query like from:[* TO %2$s] AND to:[%1$s TO *] if your index has from and to fields. E.g.:
plugin.tx_find.settings.queryFields.20 {
id = year
type = Range
query = from:[* TO %2$s] AND to:[%1$s TO *]
default.0 = *
default.1 = *
}
Hidden
The Hidden field creates an input element of type hidden to pass additional parameters through the form. E.g.:
plugin.tx_find.settings.queryFields.30 {
id = hidden
type = Hidden
default = surprise
}
The Select field creates a popup menu. The popup menu is set up using options, with the default selection‘s key in the key. E.g.:
plugin.tx_find.settings.queryFields.40 {
id = country
type = Select
options {
blank =
key = value
key2 = value2
}
default = key
}
The SelectFacet field creates a popup menu using the data from a facet that has been loaded. For this to work the facet needs to have been configured and its id has to be set as the facetID parameter. E.g.:
plugin.tx_find.settings {
queryFields.50 {
id = status
type = SelectFacet
facetID = status
query = status_facet:%s
}
facets.50 {
id = status
field = status_facet
hidden = 1
fetchMinimum = 0
}
}
Creates radio buttons for the array set in the options array. E.g.:
plugin.tx_find.settings {
60 {
id = version
extended = 1
type = Radio
options {
1 = Steak
2 = Chicken
3 = Pancake
}
default = 2
}
}
Two fields in the index document can be designated as the document’s title and a snippet that are used to display the result list. This enable a simple initial configuration and should have many cases covered. If you need to display more complex information in the result list, that can be achieved by replacing the Display/Result partial or – the Index template.:
plugin.tx_find.settings {
standardFields {
title = title
snippet = detail
}
}
By default all records in the index will be displayed when no search term is given: the query *:* is used for this. You can change this default query (e.g. to a query with no results):
plugin.tx_find.settings {
defaultQuery = *:*
}
Faceting can be configured in TypoScript using the facets setting. It is a numbered list of arrays. Each array can have the keys:
id(required): ID used to identify the facettype[List]: the type of facet to use (see below for the types provided by the extension)field: the Solr field to use for the facet, if not given the field given by theidwill be usedsortOrder[count]: usingindexgives alphabetically sorted facet entries, by default facet items are sorted by the number of resultsfetchMinimum[1]: the minimum number of facet entries needed to display the facet; the facet will not be shown at all if there are fewer entries than thisfetchMaximum[100]: the maximum number of facet entries to loadquery: sprintf style formatted string to use as a filter query if the facet is selected; by default the facet’s field is used with the selected termfacetQuery: array of facet query configuration arrays to use for creating specific facets; each of the arrays has the keysidto identify the facet query andquerythe Solr query to create the facet forselectedByDefault: array with keys field value and values 1 to indicate facet terms that should be selected when no facet selection is given (especially useful with theTabsfacet type.excludeOwnFilter[0]: if set to 1 the filters created by the facet itself will not be used when computing the result count for its itemshidden[0]: whether to hide the facet from display (e.g. to use the facet data in some other part of the page like aSelectFacetquery field)
To change the defaults for these fields you can use the facetsDefaults setting and set your preferred default values there.
The provided facet types are given by the partials in Partials/Facets/Facet. The partial is picked using the type configured for the facet.
This is a default facet list displaying the facet name with a result count behind it. You can use CSS to hide the result count if it is not needed.
displayDefault[6]: the number of facet items to display by default (the remaining ones are initially hidden and can be revealed by the user)autocomplete: whether to offer an autocompletion search field above the facet items (helpful for facets with many items)sortPrefixSeparator: this string is used to split the facet’s name into two parts and only display the second part; this way you can sort by the first part (e.g. with zero-padded numbers) and still have good looking facet names
Examples:
plugin.tx_find.settings.facets {
10 {
id = orden
field = orden_facet
autocomplete = 1
}
20 {
id = band
field = band_facet
sortOrder = index
displayDefault = 8
sortPrefixSeparator = ####
}
}
The Tabs style is a slight variation of the plain list. As the name suggests, it can be used to create »Tabs« above the search results to allow users to pick a view on a certain subset of the data. This can be particularly useful with predefined facet queries to define the desired subsets.
You will have to add the partial for this type of facet to your template yourself to make it appear at the top of the page. When doing so use the hidden setting to ensure it does not appear along with the other facets as well. Turning on the excludeOwnFilter setting will ensure you get correct result counts for the facet items which are not currently selected.
Example: a Tabs facet with facet queries for four specific document »typ« values, selecting the »formular« option by default:
plugin.tx_find.settings.facets {
30 {
id = typ
excludeOwnFilter = 1
type = Tabs
hidden = 1
selectedByDefault {
formular = 1
}
facetQuery {
10 {
id = formular
query = typ:formular
}
20 {
id = wort
query = typ:wort
}
30 {
id = gott
query = typ:gott
}
40 {
id = ort
query = typ:ort
}
}
}
}
This facet is made for numeric fields. It will draw a histogram to visualise the number of results per number in the index. It is a nice way to visualise a »year« facet. You typically want a high fetchMaximum setting for the histogram facet.
barWidth: the »width« of each of the bars in the histogram; if you cover a wide number range it can be worthwhile to group the bars in wider ranges; doing so requires an index that already contains rounded data (e.g. the rounded number of the decade instead of the precise year)
Example:
plugin.tx_find.settings.facets {
40 {
id = decade
field = decade
type = Histogram
sortOrder = index
fetchMaximum = 1000
barWidth = 10
}
}
This face creates a tiny Google map from a Solr field containing specifically formatted geohashes. Please look at the Partials/Facets/Facet/Map.html partial for details. (This can still be improved in many ways.)
Example:
plugin.tx_find.settings.facets {
50 {
id = map
field = geohash
type = Map
sortOrder = index
fetchMaximum = 1000
}
}
Sort behaviour can be configured using the sort setting. It is an array of arrays with the fields id and sortCriteria. The latter is a Solr sort order string, i.e. a comma-separated list of the form fieldName [a|de]sc. The id of the default sort order should be default.
If the array has several elements, a popup menu for selecting the sort order is added to the user interface.
Example:
plugin.tx_find.settings.sort {
1 {
id = default
sortCriteria = year desc,name asc
}
}
Use the paging setting to adjust navigation in the results. In this array you can set:
perPage[20]: the number of results per pagemenu[array()]: array of numbers used to create a menu from which users can pick the number of results per pagemaximumPerPage[1000]: the maximum number of results to fetch from Solr, even if the query asks for moredetailPagePaging[1]: if 1 this enables paging between detail pages for a search
Example:
plugin.tx_find.settings.paging {
perPage = 100
maximumPerPage = 1000
detailPagePaging = 1
}
The detailPagePaging option enables a feature that lets you have pretty and unique URLs for the pages corresponding to your Solr documents when RealURL is used and have paging between a result’s detail page and the detail pages before and after it.
This is achieved by POSTing information about the previous query along with the ID of the requested record sent in the URL. A drawback of this solution is that using the browser’s back button may trigger a »Do you want to submit this form again?« dialogue. Turn this option off if you don’t need paging between the detail pages.
You can also set the the number of results and the initial document in a server reply using the count and position arguments. The count will be limited by the maximumPerPage setting.
If your index contains documents that should never be shown (e.g. because they are not published yet or because you are using them for search but {!join} them to other documents for display), you can add filter queries using the additionalFilters setting. The setting is an array with the filter queries as values:
plugin.tx_find.settings.additionalFilters {
1 = published:true
}
By default the complete Solr document is loaded and all its fields can be used. If your documents are very large or you want to avoid unnecessary highlighting, it can be useful to explicitly state which fields are to be fetched. The dataFields setting lets you do this. It has four sections default, index, detail, data. The extension will start with the fields configured in default and add the fields given in the array for the action that is used.
Each of these sections is an array with keys default (again), allow and disallow. Typically you will only need the default array but the other keys can be used to force-add fields or explicitly prohibit the output of fields. (Please note that this will not be a failsafe guarantee that users cannot see those field due to the various actions or field name wildcards.) You may also use the data-fields argument in action URLs to overwrite the default configuration. For technical reasons the keys for the fields need to begin with a letter rather than just be a number (e.g. use f1 instead of 1).
Example configuration to only load minimal fields by default and load all fields for the detail and data actions:
plugin.tx_find.settings {
dataFields {
default {
default {
f0 = id
f1 = kloster
}
}
detail {
default {
f0 = *
}
}
data < plugin.tx_find.settings.dataFields.detail
}
}
The use of Solr’s result highlighting is configured by the highlight setting. Similar to the dataFields setting, it contains arrays default, index, detail and data. Each of which can contain the following fields:
fields[{f1 = *}]: an array of field names; its keys should begin with a letter for technical reasons (i.ef1instead of1fragsize[100]: the maximum length of the highlighted fragmentquery: a custom sprintf-style query templat to use for highlightinguseQueryTerms[0]: set to 1 to create highlight queries for each query term from the search formuseFacetTerms[0]: set to 1 to create highlight queries for each selected facet termalternateFields: an array with keys the field name and values the corresponding alternate field name for situations where the highlighting does not work in the field itself
Please note that particularly the final three options do not seem ideal yet and are considered experimental. Changes may be needed in the future to make these more versatile in complicated situations.
Example highlighting just a few fields by default and a wider range of fields for the detail action. As it is used with {!join} queries, the terms from queries and facets are explicitly added again for highligh queries:
plugin.tx_find.settings.highlight {
default {
fields {
f1 = kloster
}
useQueryTerms = 1
useFacetTerms = 1
}
detail {
fields {
f2 = patrozinium
f3 = ort
f4 = bistum
f5 = land
f6 = orden
f7 = bemerkung_kloster
}
}
}
When displaying field content with the Partials/Display/Field/Content partial (or its siblings who use it) the linkFieldContent argument can be give to not just display the field content but insert a link to search all documents with the same value in that field. Without further configuration this will create a raw query where the user may see the Solr query. It can be desirable to instead hide the Solr query syntax and define a (potentially hidden) query field for that Solr field and then just fill that field with the term only.
The queryFieldForDataField setting is an array with keys Solr field names and values IDs of queryFields which creates the mapping needed for creating those queries.
Example:
plugin.tx_find.settings {
queryFields {
100 {
id = city
type = Text
hidden = 1
phrase = 1
}
queryFieldForDataField {
city = city
}
}
By default the extension creates links that jump to the extension’s content (#tx_find) on the target page to maximise the visible space for search results and information that may otherwise be wasted for the page’s head. You can remove the anchor to jump to or pick another one on your page (e.g. to leave site navigation visible) with the jumpToID setting:
plugin.tx_find.settings.jumpToID = menu
Settings include the JSPaths and CSSPaths arrays which can be used to configure JavaScript and CSS files to be added to the page. The default setup configures files at positions 10 and 20 of CSSPaths and at position 10 of JSPaths. E.g.:
plugin.tx_find.settings {
CSSPaths.30 = EXT:find/Projects/test/Resources/test.css
JSPaths.20 = EXT:find/Projects/test/Resources/test.js
}
jQuery has to be included manually on your TypoScript PAGE-object of choice. Example:
page.includeJSFooterlibs.jquery = https://cdnjs.cloudflare.com/ajax/libs/jquery/3.1.1/jquery.min.js page.includeJSFooterlibs.jquery.external = 1 page.includeJSFooterlibs.jquery.integrity = sha256-hVVnYaiADRTO2PzUGmuLJr8BLUSjGIZsDYGmIJLv2b8= page.includeJSFooterlibs.jqueryUi = https://cdnjs.cloudflare.com/ajax/libs/jqueryui/1.12.1/jquery-ui.js page.includeJSFooterlibs.jqueryUi.external = 1 page.includeJSFooterlibs.jqueryUi.integrity = sha256-T0Vest3yCU7pafRw9r+settMBX6JkKN06dqBnpQ8d30=
You can set the languageRootPath pointing to your own Localisations folder. The partials included with the extension will honour that path when accessing their localisation files. As with the partials and templates it can be handy to symlink some of the default localisation files and just override or add the additional terminology you need:
plugin.tx_find.settings.languageRootPath = EXT:find/Projects/test/Language/
Localisation files used by the included partials are:
- locallang.xml: general terminology for the search interface (the default file)
- locallang-form.xml: labels and placeholders for the search form
- locallang-facets.xml: facet names and facet item names
- locallang-fields.xml: field labels
Please refer to the initial comments in those files for the conventions used to create the localisation keys based on the id s of the respective query fields, facets or fields.
You probably want to create your own templates and partials to tailor the display of your search results for your index content. To do that create a project structure along the lines of the included example projects, symlink the parts of the extension’s templates and partials you want to use and override/add your own.
The extension comes with a large number of View Helpers and partials that can be helpful in that context. Look around the Classes/ViewHelpers and Resources/Private/Partials folders to see which parts of the work have already been done for you.
Most of the View Helpers come with usage examples in the Test partial of the test project and appear on its start page (the search interface appears after those examples) once you have it set up.
The extension provides three actions:
index: the default action that performs searchesdetail: the action to display a single document (automatically triggered if theidargument is present)suggest: used for the autocomplete setup of Text query fields
Creating data exports requires the following setup:
set up a TYPO3 page type without HTML and the MIME Type you need; the extension provides type
1369315139one for JSON:tx_find_page = PAGE tx_find_page { typeNum = 1369315139 10 < tt_content.list.20.find_find config { disableAllHeaderCode = 1 additionalHeaders = Content-type:application/json;charset=utf-8 } }create a link with
f:link.actionusing thepageTypeyou need,format="data"and add the argumentdata-formatwith the name of the format you want to the querycreate a partial for your data format in
Partials/Formatswith the.datafile name extension and create the desired output there.
Example output formats, e.g. for JSON, are available in the extension. The germania-sacra project contains additional ones.
The plug-in’s query parameters are of the form tx_find_find[parameterName].
Built-in parameter names are
id: the Solr document ID of the document to show in the detail action; having the id argument will always trigger the detail actionq: for the query; this is an array with keys the queryField IDs, e.g.tx_find_find[q][default]=term; some query field types (e.g. Range) use an array as their value to cover both their input valuesfacet: for selected facets; this is an array with keys the facet IDs, and values arrays; those arrays have keys the facet term and value 1, e.g.tx_find_find[facet][typ][formular]=1page: the page of results to show (1-based)extended: whether to show the extended search formsort: the sort order to use
These arguments are not typically exposed but can be useful when providing data exports through the extension, specifically when using the data action.
start: the document index to start at in the result setcount: the number of documents to ask Solr for (limited by thepaging.maximumPerPagesetting)data-format: when using thedataformat, this file inPartials/Formatswill be used to create the output
underlyingQuery: information about the underlying query, sent in the POST body whenpaging.detailPagePagingis turned on; it may contain the keysq,facet,position,countandsort.
The extension includes a hook for RealURL autoconfiguration. It mainly handles the parameter name for the detail view by using a id/documentID path segment.
URLs for queries and faceting are not prettified and – if required – will need to be manually configured due to the query parameter names depending on the id s used for fields and facets.
- TYPO3 6.2 or higher
- PHP 5.5 or higher
- Sven-S. Porst, SUB Göttingen
- Ingo Pfennigstorf, SUB Göttingen
The extension’s Solr connectivity is provided by the Solarium PHP Solr client (github).