Custom fields (allegro#2426)

Custom fields in GUI and API

.gitignore

@@ -71,10 +71,12 @@ node_modules/
 src/ralph/static/css/
 src/ralph/static/vendor/
 src/ralph/var/
+src/ralph/admin/static/elements/elements-min.html
 
 # TODO: move to src/ralph/static
 src/ralph/admin/static/elements/elements-min.html
 
 debian/ralph-core*
 
 .idea
+!src/ralph/lib

docs/development/custom_fields.md

@@ -0,0 +1,14 @@
+# Custom fields
+
+## How to attach custom fields to your model?
+
+Mix `WithCustomFieldsMixin` class to your model definition (import it from `ralph.lib.custom_fields.models`)
+
+## Admin integration
+
+To use custom fields in Django Admin for your model, mix `CustomFieldValueAdminMaxin` class to your model admin (import it from `ralph.lib.custom_fields.admin`)
+
+## Django Rest Framework integration
+
+To use custom fields in Django Rest Framework, mix `WithCustomFieldsSerializerMixin` class to your API serializer (import it from `ralph.lib.custom_fields.api`)
+

docs/user/custom_fields.md

@@ -0,0 +1,121 @@
+# Custom fields
+
+Ralph's custom fields features:
+
+* it could be attached to any model
+* field could have limitation on value type (ex. have to be int, string, url, bool)
+* field could have limitation on possible choices (like html's select field)
+
+## Defining your own custom fields
+
+To define your own custom fields, go to `http://<YOUR-RALPH-URL>/custom_fields/customfield/` or select it in menu under `Settings > Custom fields`.
+
+Possible options:
+
+* `name` - name of your custom field
+* `attribute name` - this is slugged name of the custom field. It's used as a key in API.
+* `type` - custom field type. Possible choices here are:
+    * `string`
+    * `integer`
+    * `date`
+    * `url`
+    * `choice list`
+* `choices` - fill it if you've chosen `choices` type. This is list of possible choices for your custom field. Separate choices by `|`, ex. `abc|def|ghi`.
+* `default value` - if you fill it, this value will be used as a default for your custom field,
+
+Example:
+
+![custom-field-definition](/img/custom-field-add.png "Example of custom field definition")
+
+
+## Attaching custom fields to objects
+
+You could attach custom fields to any object type (if it was enabled by developers to do it for particular type).
+
+Ralph's custom fields works pretty much the same as any other fields here. First type (part of) the name of the custom field into the `Key` field.
+
+
+![custom-field-autocomplete](/img/custom-field-autocomplete.png "Custom fields autocompletion")
+
+Then select custom field of your choice in the autocomplete list. Notice that (for some types) value field might change it's type, for example to select list. Type or select desirable value and save the changes!
+
+![custom-field-select-value](/img/custom-field-select-value.png "Custom fields - value selection")
+
+> For any object, there could be at most one value for each custom field attached to it (in other words, you cannot have the same custom field attached to single object multiple times).
+
+## API
+
+You could change custom fields through Ralph API as simple as using it's GUI!
+
+## Reading custom fields
+
+Custom fields are attached in read-only form to any API (applicable) resource as a key-value dictionary.
+
+> The key in this dictionary is `attribute_name` defined on custom field. As pointed above, it's slugged name of the custom field.
+
+Example:
+```
+{
+    ...
+    "custom_fields": {
+        "monitoring": "zabbix",
+        "docker_version": "1.11"
+    },
+    ...
+}
+```
+
+## Filtering
+
+You could easily filter objects by value of custom field of your choice. Preprend `attribute_name` by `customfield__` in URL of list of objects to select only matching to custom field of your choice, for example: `http://<YOUR-RALPH-URL>/api/data-center-assets/?customfield__docker_version=1.10`.
+
+
+## Changing custom fields
+
+To preview custom fields in REST-friendly way, go to `http://<YOUR-RALPH-URL>/api/<YOUR-RESOURCE-URL>/customfields/`, for example `http://<YOUR-RALPH-URL>/api/assetmodels/1234/customfields/`. Here you have custom fields attached to this particular object (in this case to model with id `1234`).
+
+Example:
+```
+{
+    "count": 2,
+    "next": null,
+    "previous": null,
+    "results": [
+        {
+            "id": 1,
+            "custom_field": {
+                "name": "docker version",
+                "attribute_name": "docker_version",
+                "type": "string",
+                "default_value": "1.10"
+            },
+            "value": "1.11",
+            "url": "http://<YOUR-RALPH-URL>/api/assetmodels/1234/customfields/1/"
+        },
+        {
+            "id": 29,
+            "custom_field": {
+                "name": "monitoring",
+                "attribute_name": "monitoring",
+                "type": "choice list",
+                "default_value": "zabbix"
+            },
+            "value": "zabbix",
+            "url": "http://<YOUR-RALPH-URL>/api/assetmodels/1234/customfields/29/"
+        }
+    ]
+}
+```
+
+
+You could attach here new custom field value for this object (make POST request on custom fields list) or update any existing custom field value (make PUT or PATCH request on selected custom field value, ex. `http://<YOUR-RALPH-URL>/api/assetmodels/1234/customfields/29/`). For example you could make POST to `http://<YOUR-RALPH-URL>/api/assetmodels/1234/customfields/` request with following data to attach new custom field to Asset Model with ID `1234`:
+```
+{
+    "value": "http://ralph.allegrogroup.com/manual.pdf",
+    "custom_field": "manual_url"
+}
+```
+
+> You could use custom field ID or attribute name to point it in API.
+
+> Notice that every action here will happen in context of particular object - every custom field will be attached to resource pointed by current url (ex. `/assetmodels/1234`).

gulpfile.js

@@ -11,10 +11,11 @@ var gulp = require('gulp'),
     sourcemaps = require('gulp-sourcemaps');
 
 var config = {
-
   bowerDir: './bower_components/',
+    bowerDir: './bower_components/',
+    elementsRoot: 'src/ralph/admin/static/elements/',
     srcRoot: 'src/ralph/static/src/',
     staticRoot: 'src/ralph/static/',
-
   vendorRoot: 'src/ralph/static/vendor/'
+    vendorRoot: 'src/ralph/static/vendor/'
 }
 
 var sass_config = {
@@ -93,7 +94,7 @@ gulp.task('clean:elements', function () {
   ]);
 });
 gulp.task('vulcanize', function () {
-    return gulp.src('src/ralph/admin/static/elements/elements.html')
+    return gulp.src(config.elementsRoot + 'elements.html')
         .pipe(vulcanize({
             abspath: '',
             excludes: [],
@@ -102,7 +103,7 @@ gulp.task('vulcanize', function () {
             inlineCss: true,
             inlineScripts: true
         }))
-        .pipe(rename('src/ralph/admin/static/elements/elements-min.html'))
+        .pipe(rename(config.elementsRoot + 'elements-min.html'))
         .pipe(gulp.dest('.'));
 });
 gulp.task('polymer-dev', function() {
@@ -116,7 +117,7 @@ gulp.task('polymer-dev', function() {
 gulp.task('watch', function() {
     // run "gulp dev" before
     gulp.watch(config.srcRoot + 'scss/**/*.scss', ['scss']);
-    gulp.watch('src/ralph/admin/static/elements/elements.html', ['vulcanize']);
+    gulp.watch([config.elementsRoot + '*.html', '!' + config.elementsRoot + '*-min.html'], ['vulcanize']);
 });
 
 gulp.task('dev', function(callback) {

mkdocs.yml

@@ -10,12 +10,14 @@ pages:
       - Quickstart: user/quickstart.md
       - Advanced guide: user/guide.md
       - Domains management: user/domains.md
+      - Custom fields: user/custom_fields.md
       - Dashboards: user/dashboards.md
       - API: user/api.md
   - Developer guide:
       - Before you start: CONTRIBUTING.md
       - Architecture: development/overview.md
       - RalphAdmin: development/admin.md
+      - Custom fields: development/custom_fields.md
       - Transitions: development/transitions.md
       - Addons : development/addons.md
       - Packaging: development/packaging.md

requirements/base.txt

@@ -11,6 +11,7 @@ django-taggit==0.17.1
 django-taggit-serializer==0.1.5
 djangorestframework==3.2.2
 djangorestframework_xml==1.2.0
+drf-nested-routers==0.11.1
 mysqlclient==1.3.6
 python-dateutil==2.4.2
 pytz==2015.4

requirements/dev.txt

@@ -4,4 +4,4 @@ Werkzeug
 pudb
 ipython
 ipdb
-isort==4.2.2
+isort==4.2.5

src/ralph/admin/mixins.py

@@ -9,6 +9,7 @@
 from django.contrib import admin
 from django.contrib.admin.templatetags.admin_static import static
 from django.contrib.auth import get_permission_codename
+from django.contrib.contenttypes.admin import GenericTabularInline
 from django.core.urlresolvers import reverse
 from django.db import models
 from django.http import HttpResponseRedirect
@@ -369,6 +370,14 @@ class RalphStackedInline(
     pass
 
 
+class RalphGenericTabularInline(
+    RalphInlineMixin,
+    RalphAutocompleteMixin,
+    GenericTabularInline
+):
+    pass
+
+
 class RalphBaseTemplateView(TemplateView):
     def get_context_data(self, **kwargs):
         context = super().get_context_data(**kwargs)

src/ralph/admin/sitetrees.py

@@ -265,6 +265,7 @@ def section(section_name, app, model):
                 section(_('Regions'), 'accounts', 'Region'),
                 section(_('Transitions'), 'transitions', 'TransitionModel'),
                 section(_('Report template'), 'reports', 'Report'),
+                section(_('Custom fields'), 'custom_fields', 'CustomField'),
             ]
         )
     ])

src/ralph/admin/templates/admin/edit_inline/tabular.html

@@ -92,3 +92,7 @@ <h2>{{ inline_admin_formset.opts.verbose_name_plural|capfirst }}</h2>
     </div>
   </div>
 </div>
+
+{% block extra_scripts %}
+
+{% endblock %}

src/ralph/api/routers.py

@@ -6,10 +6,11 @@
 from rest_framework.response import Response
 from rest_framework.reverse import reverse
 
+from ralph.lib.custom_fields.api.routers import NestedCustomFieldsRouterMixin
 from ralph.lib.permissions.api import RalphPermission
 
 
-class RalphRouter(routers.DefaultRouter):
+class RalphRouter(NestedCustomFieldsRouterMixin, routers.DefaultRouter):
     """
     Acts like DefaultRouter + checks if user has permissions to see viewset.
     Viewsets for which user doesn't have permissions are hidden in root view.

src/ralph/api/serializers.py

@@ -5,7 +5,7 @@
 
 import reversion
 from django.core.exceptions import ValidationError as DjangoValidationError
-from django.core.exceptions import ObjectDoesNotExist
+from django.core.exceptions import NON_FIELD_ERRORS, ObjectDoesNotExist
 from django.db import transaction
 from django.db.models import Q
 from django.db.models.fields import exceptions
@@ -235,15 +235,24 @@ def _validate_model_clean(self, attrs):
         try:
             instance.clean()
         except DjangoValidationError as e:
-            # convert Django ValidationError to rest framework
-            # ValidationError to display errors per field
-            # (the standard behaviour of DRF is to dump all Django
-            # ValidationErrors into "non_field_errors" result field)
-            raise RestFrameworkValidationError(detail=dict([
-                (key, value if isinstance(value, list) else [value])
-                for key, value in e.message_dict.items()
+            raise self._django_validation_error_to_drf_validation_error(e)
+        self._extra_instance_validation(instance)
+
+    def _django_validation_error_to_drf_validation_error(self, exc):
+        # convert Django ValidationError to rest framework
+        # ValidationError to display errors per field
+        # (the standard behaviour of DRF is to dump all Django
+        # ValidationErrors into "non_field_errors" result field)
+        if hasattr(exc, 'error_dict'):
+            return RestFrameworkValidationError(detail=dict(list(exc)))
+        else:
+            return RestFrameworkValidationError(detail=dict([
+                (NON_FIELD_ERRORS, [value]) for value in exc
             ]))
 
+    def _extra_instance_validation(self, instance):
+        pass
+
     def validate(self, attrs):
         attrs = super().validate(attrs)
         if self._validate_using_model_clean:

src/ralph/api/viewsets.py

@@ -14,6 +14,7 @@
 )
 from ralph.api.serializers import RalphAPISaveSerializer, ReversedChoiceField
 from ralph.api.utils import QuerysetRelatedMixin
+from ralph.lib.custom_fields.api import CustomFieldsFilterBackend
 from ralph.lib.permissions.api import (
     PermissionsForObjectFilter,
     RalphPermission
@@ -61,7 +62,7 @@ class RalphAPIViewSetMixin(QuerysetRelatedMixin, AdminSearchFieldsMixin):
         PermissionsForObjectFilter, filters.OrderingFilter,
         ExtendedFiltersBackend, LookupFilterBackend,
         PolymorphicDescendantsFilterBackend, TagsFilterBackend,
-        ImportedIdFilterBackend,
+        ImportedIdFilterBackend, CustomFieldsFilterBackend
     ]
     permission_classes = [RalphPermission]
     save_serializer_class = None

src/ralph/assets/admin.py

@@ -20,6 +20,7 @@
 )
 from ralph.assets.models.components import ComponentModel, GenericComponent
 from ralph.data_importer import resources
+from ralph.lib.custom_fields.admin import CustomFieldValueAdminMixin
 from ralph.lib.table import Table
 
 
@@ -121,7 +122,7 @@ class ProfitCenterAdmin(RalphAdmin):
 
 
 @register(AssetModel)
-class AssetModelAdmin(RalphAdmin):
+class AssetModelAdmin(CustomFieldValueAdminMixin, RalphAdmin):
 
     resource_class = resources.AssetModelResource
     list_select_related = ['manufacturer', 'category']

src/ralph/assets/api/serializers.py

@@ -25,6 +25,7 @@
     Service,
     ServiceEnvironment
 )
+from ralph.lib.custom_fields.api import WithCustomFieldsSerializerMixin
 from ralph.licences.api_simple import SimpleBaseObjectLicenceSerializer
 
 
@@ -167,7 +168,7 @@ class Meta:
         model = Category
 
 
-class AssetModelSerializer(RalphAPISerializer):
+class AssetModelSerializer(WithCustomFieldsSerializerMixin, RalphAPISerializer):
 
     category = CategorySerializer()
 
@@ -195,7 +196,10 @@ class Meta:
         model = AssetHolder
 
 
-class BaseObjectSerializer(RalphAPISerializer):
+class BaseObjectSerializer(
+    WithCustomFieldsSerializerMixin,
+    RalphAPISerializer
+):
     """
     Base class for other serializers inheriting from `BaseObject`.
     """

src/ralph/assets/models/assets.py

@@ -18,6 +18,7 @@
     ModelVisualizationLayout,
     ObjectModelType
 )
+from ralph.lib.custom_fields.models import WithCustomFieldsMixin
 from ralph.lib.mixins.fields import NullableCharField
 from ralph.lib.mixins.models import (
     AdminAbsoluteUrlMixin,
@@ -120,6 +121,7 @@ class AssetModel(
     PermByFieldMixin,
     NamedMixin.NonUnique,
     TimeStampMixin,
+    WithCustomFieldsMixin,
     models.Model
 ):
     # TODO: should type be determined based on category?