Add entities GraphQL query

[wKich]

Motivation

At this point the Backstage GraphQL plugin doesn't allow to query entities with various filters, which was significant stumbling block for usage, because it requires to do additional work like queries entities with REST API, getting entities refs, encode them to Node ids and then query GraphQL API.

Approach

As we have nicely typed GraphQL schema it would be great to have similar type support for filter argument of entities query. To achieve that we need to understand how Catalog Entity fields are mapped to GraphQL types. So the first step is to walkthrough any Entity interface implementations' fields and store fieldName => @field(at: "path") information. Each field might be a primitive type such as String or Int or it might be an object type/interface. If the fieldName's type is composite we have to go deeper and store info about all nested fields.

With that information we can start generating filter input type. Actually three filter types. Because filter argument of entities query has 3 parts:

• order - to specify in which way sort entities
• search - a full text search across whole entity record or just a few specific fields
• match - strict equality for sets of fields to specific values

As a result we have to generate a few input types that allow us to query entities with a filter like this:

{ entities(filter: {
  order: [
    { fieldA: ASC }
    { fieldB: DESC }
    { fieldC: [{ fieldD: ASC }] }
    { fieldE: { order: ASC } }
    {
      fieldE: {
        fields: [{ fieldF: DESC }, { fieldG: ASC }]
      }
    }
  ]
  search: {
    term: "substring"
    fields: {
      fieldA: true
      fieldB: true
      fieldC: { fieldD: true }
      fieldE: {
        include: true
        fields: { fieldF: true, fieldG: true }
      }
    }
  }
  match: [
    { fieldA: ["value1", "value2"], fieldB: ["value3"] }
    { fieldC: { fieldD: ["value4"] } }
    {
      fieldE: {
        values: ["value5", "value6"],
        fields: { fieldF: ["value7"], fieldG: ["value8"] }
      }
    }
  ]
}) {
# ...
}}

Let's walkthrough each type.

order

it's an array with EntityOrderField items. Each item represents a compilation of various Entity and its implementations fields with OrderDirection enum type.

input EntityOrderField {
  annotations: [EntityRawOrderField!]
  apiVersion: OrderDirection
  definition: OrderDirection
  description: OrderDirection
  kind: OrderDirection
  labels: [EntityRawOrderField!]
  lifecycle: OrderDirection
  links: [EntityOrderField_Links!]
  name: OrderDirection
  namespace: OrderDirection
  parameters: [EntityRawOrderField!]
  presence: OrderDirection
  profile: [EntityOrderField_Profile!]
  relations: [EntityOrderField_Relations!]
  steps: [EntityOrderField_Steps!]
  tags: OrderDirection
  target: OrderDirection
  targets: OrderDirection
  title: OrderDirection
  type: OrderDirection
}

To sort properly you must describe a sorting priority of fields, with EntityOrderField items. Each item must include only one field with OrderDirection value.

# ✅ Correct
order: [
  { kind: ASC }
  { type: DESC }
]

# ❌ Incorrect
order: [
  { kind: ASC, type: DESC }
]

You may notice that some fields have different types instead of OrderDirection, like profile: [EntityOrderField_Profile!] or steps: [EntityOrderField_Steps!]. This is because most top level fields are primitives, unlike profile and steps and some others, they are objects which have their own nested fields.

order: [
  { profile: [
    { email: ASC }
    { displayName: DECS }
  ]}
]

# The same
order: [
  { profile: [{ email: ASC }] }
  { profile: [{ displayName: DESC }] }
]

There is one special case that I'd like to describe. In the nature how fields are combined it is possible that an order field might refer to a primitive and object for different type, like this:

interface Entity {
  # ...
}

type Component @implements(interface: "Entity") {
  location: String @field(at: "spec.location")
}

type User @implements(interface: "Entity") {
  location: AddressLocation @field(at: "spec.address")
}

type AddressLocation {
  country: String!
 # ...etc
}

So as the result EntityOrderField will look like:

input EntityOrderField {
  location: EntityOrderField_Location
  # ...
}

input EntityOrderField_Location {
  order: OrderDirection
  fields: [EntityOrderField__Location!]
}

input EntityOrderField__Location {
  country: OrderDirection
  # ...
}

The location field instead of an array is EntityOrderField_Location type, which has two fields order and fields. The order is used for sorting Component.location and fields field for sorting User.location.* fields

# ✅ Correct
order: [
  location: { order: ASC }
  location: {
    fields: [{ country: DESC }]
  }
]

# ❌ Incorrect
order: [
  location: {
    order: ASC
    fields: [{ country: DESC }]
  }
]

You may notice annotations/labels/parameters have strange type [EntityRawOrderField!] this is because those fields are loose typed and usually value is just simple dictionary, but sometimes you might want to filter entities by such fields if you know by which key you would like to do it.

order: [
  {
    annotations: [{
      field: "backstage.io/source-location"
      order: ASC
    }]
  }
]

search

The search field has some similar structure as order, except that entity's fields priority doesn't matter, so it's possible to use plain object. EntityTextFilter input type has two fields

• term - a substring value which is used for searching across all entity fields or specific ones
• fields - is EntityTextFilterFieldsset type of fields to limit search scope

Each field of EntityTextFilterFields is Boolean type except some of them which refers to fields with non-primitive values

input EntityTextFilter {
  term: String!
  fields: EntityTextFilterFields
}

input EntityTextFilterFields {
  annotations: [String!]
  apiVersion: Boolean
  definition: Boolean
  description: Boolean
  kind: Boolean
  labels: [String!]
  lifecycle: Boolean
  links: EntityTextFilterFields_Links
  name: Boolean
  namespace: Boolean
  parameters: [String!]
  presence: Boolean
  profile: EntityTextFilterFields_Profile
  relations: EntityTextFilterFields_Relations
  steps: EntityTextFilterFields_Steps
  tags: Boolean
  target: Boolean
  targets: Boolean
  title: Boolean
  type: Boolean
}

As with order the search filter query pretty straight forward and has less chance to make a mistake and will look like

search: {
  term: "Lorem ipsum"
  fields: {
    # Primitive fields
    kind: true
    type: true
    # Object type field
    profile: {
      displayName: true
      email: true
    }
    # Complex field that might be primitive or object
    location: {
      include: true
      fields: {
        country: true
      }
    }
  }
}

Note: Keep in mind using false as search field value doesn't have any effect and similar to just don't mention that field

You may notice that fields.profile and fields.location have similar structure properties and might ask yourself how is it possible to distinguish if include field is a real entity field or it refers to location primitive field for some entity types. Pretty simple. At first we walkthrough each @field directive we look at field's type and store that information, and when we parse entities query we are looking at stored type information

match

The match filter field is a little bit complicated. It is an array of EntityFilterExpression items. Between each EntityFilterExpression item effectively an OR is applied. And between fields of EntityFilterExpression item effectively an AND is applied

input EntityFilterExpression {
  annotations: [EntityRawFilterField!]
  apiVersion: [JSON!]
  definition: [JSON!]
  description: [JSON!]
  kind: [JSON!]
  labels: [EntityRawFilterField!]
  lifecycle: [JSON!]
  links: EntityFilterExpression_Links
  name: [JSON!]
  namespace: [JSON!]
  parameters: [EntityRawFilterField!]
  presence: [JSON!]
  profile: EntityFilterExpression_Profile
  relations: EntityFilterExpression_Relations
  steps: EntityFilterExpression_Steps
  tags: [JSON!]
  target: [JSON!]
  targets: [JSON!]
  title: [JSON!]
  type: [JSON!]
}

match filter has some similarities to /entities/by-query REST Catalog API. So it shouldn't be a problem for migration

match: [
  { kind: ["API", "Component"] }
  { name: ["a"], namespace: ["b"] }
] 

Is the same to Catalog API

filter: [
  { kind: ['API', 'Component'] },
  { 'metadata.name': 'a', 'metadata.namespace': 'b' }
]

If we look to the same example as we mentioned in order and search section with different field types it will look like:

match: [
  # Primitive fields
  { kind: ["Group"], type: ["Organisation"] }
  {
    kind: ["User"],
    # Object type field
    profile: {
      displayName: ["John Dow"]
    }
    # Complex field that might be primitive or object
    location: {
      fields: { country: ["USA"] }
    }
  }
  { kind: ["Component"], location: { values: ["bitbucket"] }
]

Bear in mind that combining values and fields matching filters in one filter expression leads to empty result

match: [{
  location: {
    values: ["bitbucket"],
    fields: { country: ["USA"] }
  }
}]

Simply because it's impossible that any entity field might be primitive and object types at the same

Using by-query Catalog API through GraphQL

It's possible to migrate from Catalog API smoothly without rewriting all client's requests to the new format. You can use rawFilter argument of entities query. Although rawFilter slightly different to catalog.queryEntities() method it's pretty similar.

catalog.queryEntities({
  orderFields: [
    { field: 'kind', order: 'asc' },
    { field: 'spec.type', order: 'desc' }
  ],
  fullTextFilter: {
    term: 'Lorem ipsum',
    fields: ['metadata.description', 'spec.definition']
  },
  filter: [
    { kind: ['API', 'Component'] },
    { 'metadata.name': 'a', 'metadata.namespace': 'b' }
  ]
})

And here is how this query will look with rawFilter:

{
  entities(
    rawFilter: {
      orderFields: [
        { field: "kind", order: ASC }
        { field: "spec.type", order: DESC }
      ]
      fullTextFilter: {
        term: "Lorem ipsum",
        fields: ["metadata.description", "spec.definition"]
      }
      filter: [
        { fields: [{ key: "kind", values: ["API", "Component"] }] }
        { fields: [
          { key: "metadata.name", values: ["a"] }
          { key: "metadata.namespace", values: ["b"] }
        ]}
      ]
    }
  ) {
    # ...
  }
}

The only difference is filter field to handle similar strictness to EntityFilterQuery TypeScript type

Transforming GraphQL query to Catalog query

With /entities/by-query isn't possible easily start paginating from the last page, there is no offset either. Catalog REST API only allows to specify filter and limit returning items and only after that you are able to go back and forth with cursors. So to implement usual for GraphQL first/after and last/before arguments I had to construct and encode Catalog REST API cursor manually. Catalog API cursor is represented as Base64 encoded JSON object

interface CatalogCursor {
  firstSortFieldValues?: [string, string];
  orderFieldValues?: [string, string];
  totalItems?: number;
  isPrevious: boolean;
  orderFields?: Array<{ field: string; order: 'asc' | 'desc' }>;
  fullTextSearch?: { term: string; fields?: string[] };
  filter?: { anyOf: Array<{ allOf: { key: string; values: string[] }[] }> }
}

• firstSortFieldValues - contains two values, first is a value of the field which is described in orderFields or entity's uid if there is no order fields, second is always entity's uid. The entity's values for this field is the first entity that found with particular sets of filters at specific order
• orderFieldValues - has similar two values, except that is used the last entity of a page.
• totalItems - pretty straight forward, number of entities that are matched with particular filter
• isPrevious - a flag is used to going backwards and set to true if last/before arguments are used
• orderFields/fullTextSearch/filter - set of filters and sorting parameters

We expect that if one of after/before arguments is passed it must be a valid Catalog API cursor, then we decode it and change isPrevious flag depends on which argument after or before is used.

If there is no after/before arguments we map fields in each filter part order/search/match to Catalog fields structure according to field mappings described in @field directives. And create a new cursor object, luckily to us firstSortFieldValues and orderFieldValues are not mandatory and we will receive the first page of entities for our filter

Then we encode our new cursor and call catalog.queryEntities() method

Possible Drawbacks or Risks

• There is potential risk of exponentially increasing Catalog query request size. It might happen when user has different GraphQL types where same field(-s) are mapped from different fields of Catalog Entity.

type A {
  foo: String @field(at: "spec.fooA")
}

type B {
  foo: String @field(at: "spec.fooB")
}

As you can see here we have two types with the same foo field, which we use to generate input filter with only one foo field. But because both foo fields are produced from different location, at entities query resolver we don't know which type user is asking for, so we have to include both spec.fooA and spec.fooB fields in Catalog Entities query.

Simple GraphQL query

{
  entities(
    match: {
      foo: ["bar"]
    }
  ) {
    # ...
  }
}

Becomes

{
  "anyOf": [{
    "allOf": [{
      "anyOf": [
        { "key": "fooA", "values": ["bar"] },
        { "key": "fooB", "values": ["bar"] }
    ]
   }]
}

I presume it might be a way to simplify such queries, because Backstage passes them directly to knex API.

• Right now, there is no way to query fields with @relation directive. So queries like { kind: ["Component"], owner: { kind: ["Group"], members: { name: ["John Dow"] } } are impossible. But I think it would be awesome to have them.

TODOs and Open Questions

• Entity.labels and Entity.annotations have custom resolvers, so we should transform filter values for these fields back to their source, or as more possible solution, just use JSONObject for filter
• We should support transforming GraphQL fields with JSONObject type to correct filter

[changeset-bot]

🦋 Changeset detected

Latest commit: c0e10f9

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages

Name	Type
@frontside/backstage-plugin-graphql-backend-module-catalog	Minor
@frontside/backstage-plugin-graphql-backend-node	Patch
@frontside/backstage-plugin-graphql-backend	Patch
backend	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[frontsidejack]

The following preview packages were published:

Name	Version	Install Command
@frontside/backstage-plugin-graphql-backend-node	0.1.6-dl-entities-query.0	yarn add @frontside/backstage-plugin-graphql-backend-node@dl-entities-query
@frontside/backstage-plugin-graphql-backend	0.1.8-dl-entities-query.0	yarn add @frontside/backstage-plugin-graphql-backend@dl-entities-query
@frontside/backstage-plugin-graphql-backend-module-catalog	0.2.7-dl-entities-query.0	yarn add @frontside/backstage-plugin-graphql-backend-module-catalog@dl-entities-query

Generated by @thefrontside/actions

[taras]

👍 Great work, this looks really good.

[webark]

Ohhhh BUDDY!! right in time for us at the start of Q2!! 😍🤩🤠🫵👏