06 - Service Catalog and CMDB: Service Management and Governance

A catalog answers three questions

A service catalog exists to answer, in a few seconds and without asking around, three questions: whose is this service, what does it depend on, who is affected if it stops. The third is the one that decides the duration of an incident: if working out whom to warn takes half an hour of phone calls, the inventory was not there. This article shows how to keep the three answers in a file, how to check them with a script and where the catalog ends and the CMDB begins.

For the data model (Component, API, Resource, System, Group, User and how to write a catalog-info.yaml) I refer you to article 4, which validates them with the official Backstage packages. Here I work on a simplified projection, to concentrate on the questions.

What a catalog entry must contain

Not everything that can be described is worth keeping up to date. The fields that are needed when something goes wrong are few:

• Owner: a single team, reachable.
• Lifecycle: in production, experimental, deprecated.
• Criticality (here called tier): how much an interruption costs, to order the interventions.
• Dependencies: on the services and resources needed to work.
• How to reach the team: the channel and the on-call arrangement.

Backstage has the owner, the lifecycle and the dependencies out of the box (spec.owner, spec.lifecycle, spec.dependsOn). Criticality, recovery objectives and the on-call channel are not part of the model: they are written as annotations with a prefix of your choosing, and a plugin or a script reads them. A clarification the documentation makes and that is forgotten: spec.owner is for display purposes and must not be used to assign permissions.

Among the values put in a catalog entry are the recovery objectives. The NIST glossary defines the Recovery Time Objective as the overall length of time a system's components can be in the recovery phase before negatively impacting the organization's mission, and the Recovery Point Objective as the point in time to which data must be recovered after an outage. The first is measured in minutes of downtime, the second in minutes of lost data, and they are two different numbers decided per service, not per company.

A catalog in a file, and a script that queries it

The following file describes three teams and six components, between services and resources, in compact form. The field names correspond to those of Backstage: owner, lifecycle, dipende (that is dependsOn). The data is invented and built so that it contains a few typical defects.

# Simplified projection of the catalog: the fields correspond to spec.owner, spec.lifecycle,
# spec.dependsOn of Backstage. Data invented for the example.
gruppi:
  team-checkout:   {membri: [anna, marco]}
  team-inventario: {membri: [luca]}
  team-notifiche:  {membri: []}

componenti:
  checkout-service:   {owner: team-checkout,   tier: 1, lifecycle: production, dipende: [checkout-db, inventario-service, notifiche-service]}
  inventario-service: {owner: team-inventario, tier: 2, lifecycle: production, dipende: [inventario-db, legacy-export]}
  notifiche-service:  {owner: team-notifiche,  tier: 3, lifecycle: production, dipende: []}
  legacy-export:      {owner: team-inventario, tier: 3, lifecycle: deprecated, dipende: [inventario-db]}
  checkout-db:        {owner: team-checkout,   tier: 1, lifecycle: production, dipende: []}
  inventario-db:      {owner: team-inventario, tier: 2, lifecycle: production, dipende: []}

There are three defects. team-notifiche has an empty list of members, so notifiche-service has an owner on paper and nobody to answer. legacy-export is deprecated but inventario-service still depends on it. And checkout-service, the most critical service (tier 1), depends on a component with no team. The script checks this, looks for cycles and computes the impact of a failure.

import argparse
import sys
from collections import deque

import yaml


def carica(percorso):
    with open(percorso, encoding="utf-8") as f:
        return yaml.safe_load(f)


def controlli(cat):
    comp, gruppi = cat["componenti"], cat["gruppi"]
    problemi = []
    for nome, c in sorted(comp.items()):
        g = gruppi.get(c["owner"])
        if g is None:
            problemi.append(f"{nome}: the owner {c['owner']} does not exist")
        elif not g["membri"]:
            problemi.append(f"{nome}: the owner {c['owner']} has no members")
        for d in c["dipende"]:
            if d not in comp:
                problemi.append(f"{nome}: depends on {d}, which is not in the catalog")
            elif comp[d]["lifecycle"] == "deprecated":
                problemi.append(f"{nome}: depends on {d}, which is deprecated")
    return problemi


def cicli(comp):
    """Returns a dependency cycle, if one exists, as a list of names."""
    stato, percorso = {}, []

    def visita(n):
        stato[n] = "in progress"
        percorso.append(n)
        for d in comp[n]["dipende"]:
            if d not in comp:
                continue
            if stato.get(d) == "in progress":
                return percorso[percorso.index(d):] + [d]
            if d not in stato and (ciclo := visita(d)):
                return ciclo
        percorso.pop()
        stato[n] = "done"
        return None

    for n in sorted(comp):
        if n not in stato and (ciclo := visita(n)):
            return ciclo
    return None


def impatto(comp, guasto):
    """Who depends, directly or not, on the failed component: (name, distance), sorted by tier."""
    inversi = {n: [] for n in comp}
    for n, c in comp.items():
        for d in c["dipende"]:
            if d in inversi:
                inversi[d].append(n)
    visti, coda = {guasto: 0}, deque([guasto])
    while coda:
        n = coda.popleft()
        for chi in sorted(inversi[n]):
            if chi not in visti:
                visti[chi] = visti[n] + 1
                coda.append(chi)
    del visti[guasto]
    return sorted(visti.items(), key=lambda kv: (comp[kv[0]]["tier"], kv[1], kv[0]))


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("catalogo")
    ap.add_argument("--guasto")
    ap.add_argument("--osservati")
    a = ap.parse_args()
    cat = carica(a.catalogo)
    comp = cat["componenti"]

    print("Catalog checks:")
    problemi = controlli(cat)
    for p in problemi:
        print(" ", p)
    ciclo = cicli(comp)
    if ciclo:
        print("  dependency cycle:", " -> ".join(ciclo))
    if not problemi and not ciclo:
        print("  no problems")

    if a.guasto:
        print(f"If {a.guasto} stops, these are affected (name, distance, tier):")
        for nome, dist in impatto(comp, a.guasto):
            print(f"  {nome}  distance {dist}  tier {comp[nome]['tier']}")

    if a.osservati:
        with open(a.osservati, encoding="utf-8") as f:
            visti = {r.strip() for r in f if r.strip()}
        print("Running but not in the catalog:", ", ".join(sorted(visti - set(comp))) or "none")
        print("In the catalog but not running:", ", ".join(sorted(set(comp) - visti)) or "none")

    sys.exit(1 if problemi or ciclo else 0)


main()

# Output (python3 catalogo.py catalogo.yaml --guasto inventario-db --osservati osservati.txt):
# Catalog checks:
#   inventario-service: depends on legacy-export, which is deprecated
#   notifiche-service: the owner team-notifiche has no members
# If inventario-db stops, these are affected (name, distance, tier):
#   checkout-service  distance 2  tier 1
#   inventario-service  distance 1  tier 2
#   legacy-export  distance 1  tier 3
# Running but not in the catalog: test-cart
# In the catalog but not running: legacy-export, notifiche-service

The impact analysis is a breadth-first traversal of the reversed graph: it starts from the failed component and climbs up the dependencies, noting at what distance every other component appears. The final ordering by tier and then by distance puts first what costs most, not what is closest. In the example, if inventario-db stops, three components are affected, and the most critical, checkout-service, appears at distance 2: no line in the file links it to the database, and the impact emerges only from the graph.

The script exits with code 1 when it finds a problem, and this is what makes it usable in a pipeline on the catalog repository. Backstage's engine generates the inverse relations by itself (dependsOn produces dependencyOf, ownedBy produces ownerOf, as described in the relations page), so in the portal the question «who depends on me» already has an answer for the first level; for the following levels and for ordering by criticality the calculation above is needed. The second test is the same catalog with one more dependency, of notifiche-service on checkout-service:

python3 catalogo.py catalogo-ciclo.yaml

# Output (exit 1):
# Catalog checks:
#   inventario-service: depends on legacy-export, which is deprecated
#   notifiche-service: the owner team-notifiche has no members
#   dependency cycle: checkout-service -> notifiche-service -> checkout-service

A dependency cycle is not always an error (two services may call each other), but it is always a fact to know about: in a restart after a general outage, two services waiting for each other do not start. The script reports it and leaves the decision to whoever manages the catalog.

A mandatory dependency and an optional one weigh differently in an incident: Backstage's catalog has a single relation, dependsOn, and does not tell them apart. If the distinction matters to you, it must be modeled separately (for example with an annotation on the individual dependency) and maintained with the same discipline as the rest, otherwise it produces false precision.

CMDB: what it adds and why it does not coincide with the catalog

The term CMDB (Configuration Management Database) comes from IT service management, and I did not read it at the source: ITIL publications are paid. What can be verified is NIST's definition of configuration item (SP 800-128): an aggregation of system components that is designated for configuration management and treated as a single entity in that process. A CMDB is, in common usage, the repository of these items with their relationships and their history.

The resources created with infrastructure as code (article 5) can also enter the catalog as Resource, and the team tag used by that article's policy is the same data the catalog calls owner. The difference from the catalog is one of origin, not of form. The catalog is declared: the owners write it, in the repository, and it contains what they decided to say (owner, criticality, dependencies). The CMDB is observed: it is populated by a process that looks at what really runs. The declared catalog is incomplete because someone forgets to add a service; the observed inventory is complete but mute, because it knows a workload exists and not whose it is. The value lies in the comparison between the two.

The same script does it with a list of observed names (here a five-line file, which in reality would come from a query of the cluster). In the output above the last two lines are the result: test-cart runs but is not in the catalog, that is it is a service nobody registered, while legacy-export and notifiche-service are in the catalog and do not appear to be running, that is entries that may be stale. Each of the two lists is a question to put to a person, not an automatic deletion.

Populating the catalog without writing it by hand

In Backstage the documented way to fill the catalog is the GitHub provider, which searches all of an organization's repositories for catalog-info.yaml files and registers them at intervals. It must be added to the backend (the @backstage/plugin-catalog-backend-module-github module) and requires a GitHub integration with permission to read contents. The configuration below follows the documentation's schema; I did not run it, because it needs a Backstage installation, and it is a schema to adapt.

catalog:
  providers:
    github:
      example:
        organization: example-org
        catalogPath: /catalog-info.yaml
        filters:
          branch: main
          repository: '.*'
        schedule:
          frequency: { minutes: 30 }
          timeout: { minutes: 3 }

The discovery here is of the description files written by the owners, not of the running services: it is a declared catalog, with the only novelty that repositories do not need to be registered by hand. The other two entry points I read in the documentation do not inventory what runs: the S3 provider reads catalog files from a bucket, and the Kubernetes plugin is described as a way of collecting and showing information from the clusters, not as an entity provider. Populating the catalog from what runs (the CMDB side) therefore requires a community plugin or a process of your own, like the comparison above; I did not verify that the plugin catalog contains no suitable one. To decide whether it is worth it, start from the question: how many times, in the last year, have you discovered an unregistered service?

Governance: few rules, checked by a machine

A catalog's governance rests on checks that nobody has to remember to make. Four are enough to start:

• Real owner: the group exists and has at least one member.
• Resolved references: every dependency points to a component that exists.
• No dependencies on deprecated components, or a declared migration date.
• Catalog and reality aligned: no unregistered workload and no entry without execution, reviewed at a fixed cadence.

The first three are checked at every change request to the catalog and block the merge, as the script does with the exit code. The fourth requires the comparison with what is observed and it is not worth blocking on: it produces questions for people, and should be read weekly. The reason it pays to do it with a script and not by word of mouth is that a catalog without checks loses the trust of those who use it in a few months, and a catalog nobody believes is worse than no catalog, because it makes people waste time looking for confirmation elsewhere.

A rule of scale: the catalog must be the only copy of the ownership data. Alerting, release and documentation tools read from there instead of keeping their own lists, otherwise the two copies diverge, and it is the same defect discussed in the manual catalog case in the Backstage article. The metrics to tell whether the catalog is helping are in article 7.

What you can do now

You know which three questions a catalog must answer and which fields are needed to do it. You can compute the impact of a failure on the dependency graph, ordering it by criticality, and recognize owners without people, broken references, dependencies on deprecated components and cycles. You can tell the declared catalog from the observed inventory and compare them. The next article moves on to measuring the platform itself: the DORA metrics and the SPACE framework.