ESConfig: configurable Elasticsearch document types to allow splitting index #1296
Commits on Oct 27, 2024
-
add MetaCPAN::ESConfig module to centralize ES config
Centralize Elasticsearch configuration in MetaCPAN::ESConfig. Allow overridden values from the main config file. This module is not meant to have any behavior aside from holding the configuration.
-
adapt Mapping script to use ESConfig module
ESConfig knows how to find mapping data. Use it to find the mapping data as well as index configuration. The mapping data should be able to be moved into json files rather than json wrapped in a module. This can happen in the future.
-
remove analysis configuration from MetaCPAN::Model
The analysis set in MetaCPAN::Model wasn't used for anything directly, generate the index deployment statements. The index settings we actually use lives in MetaCPAN::Script::Mapping::DeployStatement, so the declarations in MetaCPAN::Model were used for nothing.
-
-
configure MetaCPAN::Model via ESConfig
Rather than searching for modules on disk, use the explicit configuration in ESConfig to configure MetaCPAN::Model.
-
check for compilation errors in document set modules
ElasticSearchX::Model ignores all errors in a ::Set package can't be loaded, and uses a generic ElasticSearchX::Model::Document::Set object. It's fine for the module to be missing, but compilation errors should be reported.
-
disable critic rule prohibiting prototypes
PPI and thus Perl::Critic don't understand signatures, so the rule ends up prohibiting signatures as well
-
add doc method to MetaCPAN::Model
Allows getting a "type" object from a document name rather than needing to specify an index and type.
-
Previously when trying to refresh indices, scripts would call $self->index->refresh. This would refresh the "currently used" index. That doesn't make any sense when splitting each type into its own index. This was also using ElasticSearchX::Model, which we want to get rid of. Instead, call ->indices->refresh via the Search::Elasticsearch object. This will refresh all indices, which is fine for our purposes. In the future, we could consider being more selective about which indices we are refreshing, but this is no worse than the old behavior.
-
find ESXM types via model rather than index
Rather than using the same index to find other types, find them via the model. This means the types don't need to be in the same index.
-
find es index/type via ESConfig rather than passing index around
Many parts of the code treated the index as the parent of all data, so it was the thing being passed around. That will no be true in the future. Instead, ESConfig can give the path (index+type) of each named document type. Convert most places passing around index to use es_doc_path.
-
-
-
-
-
add ES and ESModel Catalyst model classes
Replaces the CPAN and User model classes. Removes magic namespace creation. Just return the Search::Elasticsearch object, or the model object.
-
-
-
-
-
create distributions with upsert
Trying to count distribution documents before creating is vulnerable to concurrency and consistency issues. Instead, use an upsert to create it.
-
-
-
-
-
always use suggester for autocomplete
We have two autocomplete end points. The old one is no longer used by the front end. The new one uses the suggest API. Rewrite both end points to use the suggest SPI, just returning data in different forms.
-
-
fix script query syntax when using newer Elasticsearch
Older versions expect the key "inline", newer expect "source".
-
remove use_dis_max from query_string query
use_dis_max is the default, and isn't supported in newer versions
-
