Skip to content

Data Model Design

Jump to bottom Edit New page
Ilya Sher edited this page Nov 3, 2018 · 18 revisions

NGS Data Model Design - WIP

This page describes data model that NGS will be using to manipulate resources. Resource is any kind of external object. Examples:

  • EC2 instance
  • Terraform logical resource
  • File
    • On local machine
    • In S3 bucket
  • A line for specific machine in ssh "known hosts" file

Requirements

  • Unified resource manipulation API. The work has been started in the AWS library, see Resource API methods below.
  • Easy way to correlate and operate on resources from different systems. This is especially important for the UI. For example, an AWS Instance can have information about it coming from different sources:
    • EC2 (describe-instances API call)
    • EC2 (instance load balancer health information)
    • CloudTrail (created time, created by user, modified time, modified by user fields)
    • Consul (and service checks)
    • Monitoring system (instance services status)
    • Graphing system
    • Terraform/CloudFormation (logical resource name)
    • Output (probably processed) of an SSH command executed against the server
  • Generating textual representation of resources
    • There should be a way to generate textual representation of both
      • An existing resource
      • Steps taken to create a resource (these will include more semantic information as opposed to existing resource)
    • Textual representation examples:
      • NGS code
      • AWS CLI code
      • Terraform/CloudFormation definition
    • Handle specific vs abstract. Examples:
      • is it specifically ami-xxxxxx or is it latest Debian 9.X AMI for the region?
      • is it subnet-xxxxxx and subnet-yyyyyy or these are subnets tagged with role=front?
      • is it operation on instance i-xxxxxx and i-yyyyyy or on instances with specific tags or AMIs?
  • Synchronous APIs adapters for asynchronous APIs: support polling commands after update
    • Polling commands should be able to report following situations:
      • Should continue waiting
      • Permanent error
      • Success (expected value is set)
  • Support for "wait" commands, such as SSH availability after EC2 instance creation
  • Change reporting (did converge actually modified the resource; if yes, which properties and how)
  • Modular design. It should be easy to add:
    • New Schema pieces
    • New modules/connectors to work with new APIs
  • Manipulating schema
    • Schema should be easily imported/exported
    • API/CLI to view/edit the schema (the data model)

Design

  • Each resource type will have schema definition
  • Schema will contain
    • CLI (and other?) commands used to create and delete particular resource type
    • Fields names and types
    • Fields values format. This will allow to for example to understand
      • That i-xxxxxx means EC2 instance type, unique id. Command line tools could be greatly improved.
      • That digits with dots are IPs, which can be on instances, load balancers, in DNS records, etc.
      • s3://xxxx/yyyy is a unique id of an object in S3 bucket.
      • arn:... formats
    • Fields properties (such as: read-only, auto-generated, field is unique id)
      • These will be NGS types? Will allow something like if f is UniqueId { ... }.
    • CLI (and other?) commands used to retrieve and update any particular field
    • Relations between resources (between types as well as between )
      • Directed arrows with types. Examples:
        • EC2 instance -> is in VPC -> VPC
        • Subnet -> is in VPC -> VPC
        • EC2 instance -> has disk -> EBS volume
        • ELB -> has instance -> EC2 instance
        • AWS Lambda -> uses artifact -> S3 Object
      • For each relation type:
        • Required? For example EC2 VPC instance is required to have "is in VPC" arrow
        • API methods to establish and tear down relation ???
  • Resource API methods
    • create - creates a resource with given properties
    • delete - deletes given resource
    • converge - create or update a resource to converge to specified properties.
      • have a way to specify properties that will not be converged to but only used during creation

This approach is somewhat similar to configuration management and orchestration systems. Important differences are:

  • I'm viewing it mostly from scripting and UI angle where small steps are the focus, not where all the information is passed to the system and then there is phase in which all the resources are converged to the desired state. Somewhat similar is Puppet RAL.
  • Another difference from "classical" configuration management systems is that most of the resources are expected to be manipulated via APIs. While the former focuses on "providers" which do the work, the proposed model focuses on the data and it's semantic meaning.
  • Easy extensibility as all information gathering and manipulation operations are done via running external commands (or maybe calling APIs).

Schema (examples based)

Object types schema

  • Type name - restype:aws/ec2/instance ???
  • Display name - such as AWS EC2 Instance
  • CLI name - i (because of i-xxxxxx) ???

Resource types' relations (example 1):

  • Relation name - reltype:aws/vpc/in-vpc
  • Display name - is in VPC
  • Reverse relation name - reltype:aws/vpc/vpc-contains
  • CLI name - in-vpc ???
  • Left type - restype:aws/ec2/instance
  • Right type - restype:aws/vpc/vpc
  • Commands
    • Connect - n/a

Resource types' relations (example 2):

  • Relation name - reltype:aws/vpc/has-eip
  • Display name - 'has Elastic IP'
  • Reverse relation name - reltype:aws/vpc/eip-is-on
  • CLI name - has-eip ???
  • Left type - restype:aws/ec2/instance
  • Right type - restype:aws/vpc/eip
  • Commands
    • Connect - ["aws", "ec2", "associate-address", "--allocation-id", "$right.AllocationId", "--instance-id", "$left.InstanceId"]

Remarks

  • AWS boto has JSON files which describe different services and their API.
    • Included
      • Data structures to pass when calling API
      • Data structures returned from API calls
    • Not included
      • The model - what are the fields and types of the objects.
        • The data structures which describe API call results include pagination, idempotency tokens, etc. Then you need to guess where the resources are in the data structure.
        • Not all semantic data is present. For example, InstanceId is just a String. In practice it's (1) auto-generated (2) unique id and (3) has special string format.
      • Filters description. That's also the reason that AWS CLI does not have filters completion.

TODO

  • "wait" commands
    • schema
    • methods
  • Design command line tools for manipulating resources, similar in spirit to na but more generic.

Open Issues / Maybe TODO

  • Bring ideas to make development of na easier.
  • converge is not allowed to delete and the create a replacing resource in order to "modify" read-only properties such as AWS Instance AMI. Maybe this should be allowed under the condition that it will not surprise the caller, maybe allow_delete=true or delete_policy=... or something similar.
  • Servers and containers
  • Ability to create DAG of described resources and parallelize similar to Terraform?
    • Doesn't fit the simple top-to-bottom execution model.
    • Doesn't fit "it's just a library for simpler more powerful coding" model.
    • Great performance boost.
    • Need to think about general DAG facility in NGS, maybe then it will fit well.
  • Ability to list and handle together a resources of different types.
  • Should these also be resources? Looks too much like a configuration management system then:
    • A user
    • A service
  • Which standards and tools can help?
    • RAML / Swagger would be of any use?
  • Maybe puppet RAL can be used by NGS is yet another API for manipulation of resources? Will there be enough schema information?
  • Completion support
  • If a file is say Nginx configuration file, when the content is set, should it be validated (by running Nginx syntax check nginx -t for example) ?
  • Sometimes change status (whether a resource of a particular property of a resource was updated) is important, sometimes it is not. Maybe different API calls or commands can be used.
  • It would be nice to handle creation/deletion and per-property changes specifically (meaning having different events triggered) as opposed to classical systems where any kind of change triggers update of another resource.
  • Tools to manage the data model?
    • It would be nice to create big chunks of AWS model from the boto JSON files for example
    • Central exchange?
    • Imagine running an AWS CLI command + wizard to generate additional piece of schema
  • Collaboration?
    • Locking support (Terraform style)?
    • Central logging?
  • Transport support for remote resources
    • A file on remote machine can be creating using SSH transport for example
  • How to track which properties are OK to change and which will be overridden by Terraform/CloudFormation? This situation needs at least to cause a warning.
    • A property which is in lifecycle > ignore_changes should be OK
    • Would this require complete understanding of Terraform data model (all providers)? That's quite a task.
    • When used interactively, it should be suggested to modify the definition (Terraform/CloudFormation) when trying to modify an instance of a resource.
    • A good starting point with relatively small amount of work is just to notify that a resource at hand has Terraform/CloudFormation definition.
  • Support import/export of data
    • Examples
      • Input from hosts.csv list of machines to be operated on by ip or InstanceId column.
      • Output to hosts.json when including data (columns) from EC2 and Nagios.
  • Schema for schema?

NGS official website is at https://ngs-lang.org/

Add a custom sidebar
Clone this wiki locally