Search Index Bug Fix

There was a bug observed sporadically with the data category search index template using Django 1.4.x. This was most-likely due to a self include of a template. This behavior was not necessary so it was removed to prevent a recursive include.

Quick Enhancements

Enhancements

• Field and Concept names (and their plural forms) have been corrected to use the __unicode__ representation of the instance rather than merely relying on the name field. This ensures something is returned when doing unicode(field) and field.get_plural_name().
• DataField supplementary fields have been added to the admin.

DCO & Lots of other stuff

In an effort to make the contributing to Avocado more transparent and safer for all, a Developer Certificate of Origin (DCO) has been added to the CONTRIBUTING.md. As a contributor, this certifies that you are legally allowed to contribute the code submitted as patches (in pull requests or otherwise).

This requires that all contributors to the repository now sign off their commits using the git commit --signoff (or -s for short). A script has been added to the Travis-CI config to check for the presence of the Signed-off-by: line. If any commit does not contain this, the build will error which will prevent core contributors from merging your code.

Features

PR #181 introduces four new optional fields on the DataField model. These are names of fields on the instance's model that are supplementary to the base field denoted byfield_name:

• label_field_name
• order_field_name
• search_field_name
• code_field_name

The model field object can be accessed through the corresponding datafield property (without the _name suffix). This feature supersedes the need for direct integration of the Lexicon and ObjectSet classes in the DataField API since the supplementary fields can now be set explicitly. Detection of these types will be removed in 2.4, however the detection in the avocado init will remain for now.

• The DataConcept.type and DataField.type fields have been added to lay the foundation for refactoring how fields and concepts behave. Read more here #88
• The DataField API now supports making use of pre-defined choices on the field instance (#157)

Enhancements

• Parent categories are now included in the search index for fields and concepts (#142)
• Categories can be auto-created during an avocado init call when the --categories flag is passed
  • Fields and concepts that are created will be associated with the category which is named after the associated model.
• The Registry.register function can now be used as a decorator when registering classes or instances (#95)
• The parent category of field and concepts are now included in the free-text search index (#142)
• Categories can now be optionally created during an avocado init call by passing the --categories flag

Bugs

• Fixed hard-coded label field name for objectset-based models
• Fixed condition where revision data is null and causes an error during creating a diff
• The published field is now correctly exposed and editable from the DataCategory admin list view (#196)
• Query sharing now use a case-insensitive lookup on email addresses (#197)
• Fixed an issue when metadata migrations would fail due to a schema dependency (#185)
  • This now includes a depends_on attribute with the latest Avocado migration

Haystack 2.0.0, Object Sets

Enhancements

• The built-in support for object sets have been replaced with the more optimal third-party django-objectset library
  • This requires 'django-objectset` to be installed for you project
• The free-text search support has been updated to use Haystack 2.0.0
  • Note, this requires a minor configuration change in the Django project settings file - see below.
• The force_distinct option for the exporter read() method now ensures rows are distinct for the whole iterable
• The exporter read() method now takes optional offset and limit arguments to control the window and number of formatted rows to return

Bugs

• Concepts without associated fields are now properly excluded in the published manager method
• The data cache API no longer caches data from methods that had arguments supplied
  • This may be supported in the future
• Fix the --no-fake option in the metadata migration command

Required Changes

In your project's configuration file, replace the following settings:

HAYSTACK_SITECONF = 'avocado.search_sites'
HAYSTACK_SEARCH_ENGINE = 'whoosh'
HAYSTACK_WHOOSH_PATH = os.path.join(os.path.dirname(__file__), 'whoosh.index')

with the dict-based setting:

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.whoosh_backend.WhooshEngine',
        'PATH': os.path.join(os.path.dirname(__file__), 'whoosh.index'),
    }
}

Data Cache Hotfix

Fixes an issue where arguments were being ignored when the data cache API was being used. The fix prevents the data cache from being used when arguments are supplied. This may be refactored in a future release to handle differentiating cache based relative to the arguments.

Public Data Queries

Features

• Support for flagging DataQuery instances as public. This can be used by clients to differentiate public vs private queries.

Enhancements

• Changed internal structure of DataView to be more robust for error handling
  • This is backwards compatible with the previous structure
• Checks if METADATA_MIGRATION_APP is defined prior to attempting a safe backup

Changes

• The avocado init command now immediately publishes fields and auto-creates concepts for each field to more quickly get up and running on new instances.
  • The previous behavior can be retained by passing the --no-publish and --no-concepts flags

Bugs

• Fixed #136 which prevented using the data cache feature

Turn Off Data Cache (until fixed)

DATA_CACHE_ENABLED is now disabled by default until #136 is fixed.

Shared Queries & Versioning

Features

• Support for saving and sharing queries** (see #119)
• Automatic versioning of DataContext, DataView, and DataQuery objects (see #115)

Enhancements

• Add accessed field on DataContext, DataView, and DataQuery for tracking purposes
• Add DataConcept.viewable boolean field for toggling whether concepts are visible as output (see #69)
• DataContext, DataView, and DataQuery are now registered with the admin

Changes

• Introduced avocado.query.validators.Validator and FieldValidator for an improved foundation for query processing
• avocado.core.loader.Registry has been simplified
• Rename OPERATOR_MAP and SIMPLE_TYPE_MAP settings to OPERATORS and SIMPLE_TYPES, respectively
• Replace DataConcept.queryview for DataConcept.querable boolean (see #68)
• Removed cached_property method for various DataField properties (see 231264)
• Replace DataField.data_modified with data_version field
• Removed archived and published fields from DataContext, DataView, and DataQuery
• Remove DataView.count since it was not being used, nor made sense

Bugs

• Add support for exclusive ranges on date, time, datetime and number fields
• Use os.makedirs in migration subcommand to ensure the full path to migration fixtures is created

Bye Bye SciPy

• Drop NumPy and SciPy dependencies for built-in k-means algorithm
  • This is to simplify the install process and remove heavyweight dependencies for use of a single algorithm
• Move export templates under avocado/templates to remove the need to add avocado.export to INSTALLED_APPS
• Add new QueryProcessor class (see 052e7d0 for details)
• Change DataField.search to not return an iterator
  • This enables downstream slice without needing to evaluate the data structure (primarily for SearchQuerySet instances)
• Add avocado.events package for simple tracking of events such as accessing a DataConcept or DataField
• Add DataQuery model which represents a joint data structure of a DataContext and DataView
  • The JSON corresponding to each structure is stored, rather than referencing instances directly. This is for simplicity.
• Add support for n-grams search and fix outstanding search tempalte (see a5c273e)
  • Note, re-indexing may take much longer than before since it is now correctly indexing the field values

Refactor

• Add convenience manager method on DataView and DataContext for get the default template
  • Improve unicode representation for templates