01 - Introduction to Platform Engineering: What is an IDP and Why It Matters

What an internal platform is and which problem it solves

Platform Engineering is the discipline that builds and runs an Internal Developer Platform (IDP): a set of capabilities offered in self-service to product teams, so that nobody has to rebuild the build pipeline, the deployment, secrets management and monitoring on their own. The DORA 2024 report describes it as a socio-technical discipline: it concerns both the interactions between teams and the automation, self-service and repeatability of processes.

The problem comes from autonomy. The CNCF document on platforms explains that giving teams the freedom to choose tools and services has shifted more and more support responsibility onto them, and therefore time and mental energy taken away from the work the organization expects of them. The platform is the organizational answer: a few dedicated experts serve many teams, and whatever is common is done only once.

This article opens a series of fourteen. Here you will find the definitions, the numbers from the research (including those that are inconvenient for people who sell platforms) and two small scripts to measure, on your own data, whether the problem really exists where you work.

When every team builds its own stack

The most visible symptom is fragmentation. Before discussing platforms it pays to count it: how many different tools do the same job inside the company? The script below reads an inventory in YAML, with one row per team and one column per category, and prints how many distinct tools appear in each category and how many stack combinations exist. The four teams are invented for the example: the numbers that come out do not describe any real company.

# Inventory of an example organization: four teams, five choices each.
# The data is invented for the example, not measured on a real company.
teams:
  checkout:   {ci: jenkins,        cd: argocd, monitoring: datadog,    infra: terraform,      secrets: aws-secrets-manager}
  search:     {ci: github-actions, cd: flux,   monitoring: prometheus, infra: cloudformation, secrets: vault}
  catalog:    {ci: github-actions, cd: argocd, monitoring: prometheus, infra: terraform,      secrets: vault}
  notifiche:  {ci: gitlab-ci,      cd: argocd, monitoring: prometheus, infra: pulumi,         secrets: vault}

The inventory is deliberately trivial: each team put down its own name, and next to it the five choices it made. Nobody got anything wrong, and that is the point: each choice, taken on its own, is defensible.

import sys
from collections import Counter
import yaml

with open(sys.argv[1], encoding="utf-8") as f:
    teams = yaml.safe_load(f)["teams"]

categorie = ["ci", "cd", "monitoring", "infra", "secrets"]
for cat in categorie:
    scelte = Counter(t[cat] for t in teams.values())
    # stable order: most used first, ties broken alphabetically
    ordinate = sorted(scelte.items(), key=lambda kv: (-kv[1], kv[0]))
    dettaglio = ", ".join(f"{nome} x{n}" for nome, n in ordinate)
    print(f"{cat:<11} {len(scelte)} distinct tools  ({dettaglio})")

stack = {tuple(t[c] for c in categorie) for t in teams.values()}
print(f"different overall stacks: {len(stack)} out of {len(teams)} teams")
print(f"distinct skills needed in the organization: "
      f"{len({t[c] for t in teams.values() for c in categorie})}")

# Output (python3 sprawl.py stack_per_team.yaml):
# ci          3 distinct tools  (github-actions x2, gitlab-ci x1, jenkins x1)
# cd          2 distinct tools  (argocd x3, flux x1)
# monitoring  2 distinct tools  (prometheus x3, datadog x1)
# infra       3 distinct tools  (terraform x2, cloudformation x1, pulumi x1)
# secrets     2 distinct tools  (vault x3, aws-secrets-manager x1)
# different overall stacks: 4 out of 4 teams
# distinct skills needed in the organization: 12

The script uses a Counter per category and sorts the entries first by frequency and then alphabetically, so the output is identical on every run (I launched it twice and compared the results). The figure that matters is the last one: four teams, four different stacks, twelve tools to know how to use. Whoever changes team has to relearn everything; whoever handles a night-time incident does not know how to read the other teams' monitoring. Where only one tool appears per category, like secrets in the case of three teams out of four, a de facto platform is already emerging, even if nobody has called it that.

What a platform must have, according to the CNCF

The CNCF Platforms White Paper, written by the TAG App Delivery group, lists a few characteristics that set a successful platform apart from a pile of tools:

• Platform as a product: it is designed and evolved according to the requirements of its users, giving priority to the use cases common to several teams over those of a single one.
• User experience: capabilities are offered through consistent interfaces, and in more than one form if needed (portal, command line, API, IDE).
• Documentation and onboarding: examples and tools to start a new project, such as a starter template with its documentation, which the document calls a golden path.
• Self-service: ask for a database and receive its address and credentials without opening a ticket.
• Reduced cognitive load: teams must not have to run the database server, but they must be able to configure and observe it.
• Optional and composable: if the platform does not offer a graph database, the team must be able to run one on its own.
• Secure by default: compliance and validation according to the organization's rules.

The list is not a priority scale, but three items carry the weight of the choice. «Product» explains why a platform built without listening to its users fails. «Self-service» is what allows a few experts to serve many teams. «Optional and composable» is the item that projects neglect most often, and that you will meet again in the rest of the series as a design requirement: a mandatory, rigid platform becomes an obstacle.

Platform Engineering and DevOps

It is not a replacement. The DORA 2024 report observes that moving a capability into a shared system («shifting down») can seem at odds with the «you build it, you run it» principle, but reads it as a way of extending those practices to the whole organization: once a capability is in the platform, teams get it by adopting it. A concrete example is running unit tests: if the platform runs them and returns the results, the team writes better tests without building or maintaining the environment that runs them.

The practical difference lies in who takes care of what. With DevOps alone, each team looks after its whole stack, as in the inventory above. With a platform, a dedicated team looks after the shared capabilities and the product teams remain responsible for their own services in production.

Team Topologies: who builds the platform and for whom

The book Team Topologies by Matthew Skelton and Manuel Pais describes four team types: the stream-aligned team, which owns a slice of the domain from start to finish; the enabling team, which helps the others overcome obstacles and detects missing capabilities; the complicated subsystem team, where a high level of mathematical or technical expertise is needed; and the platform team, which offers an internal product to speed up the stream-aligned teams. This is what the project's website says, on its key concepts page, which also claims that most internal platforms become bulky and slow teams down, and proposes as a remedy the «Thinnest Viable Platform»: only the capability that is needed, with no other dependencies to manage.

To avoid that, the starting question is always the same: what do the teams ask for, and does the platform offer it? The next script compares the platform's catalog with the teams' requests and prints what remains uncovered. Here too the data is made up for the example.

# What the platform offers today and what the teams ask for (example data).
platform:
  offers:
    - ci-cd-pipeline
    - kubernetes-namespace
    - monitoring-dashboard
    - secrets-management
teams:
  checkout:
    consumes: [ci-cd-pipeline, kubernetes-namespace, monitoring-dashboard, secrets-management]
  search:
    consumes: [ci-cd-pipeline, elasticsearch-cluster, monitoring-dashboard]
  catalog:
    consumes: [ci-cd-pipeline, kubernetes-namespace, postgres-database]
  notifiche:
    consumes: [ci-cd-pipeline, message-queue, postgres-database]

In the file, offers is the current catalog and consumes is what each team uses or would like to use. The interesting case is that of the teams asking for a database or a queue that the platform does not offer.

import sys
from collections import Counter
import yaml

with open(sys.argv[1], encoding="utf-8") as f:
    dati = yaml.safe_load(f)

offerte = set(dati["platform"]["offers"])
richieste = Counter(
    voce for team in dati["teams"].values() for voce in team["consumes"]
)

print("Covered by the platform:")
for voce in sorted(offerte):
    print(f"  {voce}: used by {richieste.get(voce, 0)} team(s)")

print("Requests without an answer:")
mancanti = sorted(
    ((voce, n) for voce, n in richieste.items() if voce not in offerte),
    key=lambda vn: (-vn[1], vn[0]),
)
for voce, n in mancanti:
    print(f"  {voce}: requested by {n} team(s)")

totale = sum(richieste.values())
coperte = sum(n for voce, n in richieste.items() if voce in offerte)
print(f"Requests covered: {coperte} out of {totale}")

# Output (python3 domanda_offerta.py domanda_offerta.yaml):
# Covered by the platform:
#   ci-cd-pipeline: used by 4 team(s)
#   kubernetes-namespace: used by 2 team(s)
#   monitoring-dashboard: used by 2 team(s)
#   secrets-management: used by 1 team(s)
# Requests without an answer:
#   postgres-database: requested by 2 team(s)
#   elasticsearch-cluster: requested by 1 team(s)
#   message-queue: requested by 1 team(s)
# Requests covered: 9 out of 13

The result is a priority list that nobody had to invent: the PostgreSQL database is requested by two teams out of four and is not in the catalog, so it is the next candidate; the search cluster is requested by a single team, and it is better to let that team run it (the CNCF's «optional and composable» rule). The final 9 out of 13 is not a score to optimize, but a way of noticing that the catalog has left about a third of the demand uncovered. A platform team that does not measure the distance between what it offers and what the teams use ends up building what it enjoys building.

What the data says: the DORA 2024 report

The most solid source on the effect of platforms is DORA's (Google Cloud) Accelerate State of DevOps 2024 report, which devotes a chapter to the topic. These are survey results, so they are correlations and not proof of cause, and the definition of platform was deliberately left broad: 89% of respondents say they use one. Whoever uses an IDP turns out to be:

• more productive as an individual, by +8% compared with those who do not use one, and more effective as a team, by +10%;
• part of an organization with better overall performance, by +6%;
• 5% more productive when able to complete tasks without the intervention of an enabling team (developer independence);
• subject to a delivery throughput lower by 8% and a change stability lower by 14%.

The last item is the one that sales presentations leave out, and it is in the report: platforms add hand-offs between systems, and for those who are required to use them exclusively for the whole application lifecycle throughput drops by 6%. DORA offers three hypotheses for the instability (the platform makes it possible to fix things quickly, so more risk is taken; it does not filter defective changes enough; or it is the already unstable organizations that build platforms) and states that it does not know which one holds. For the first few years productivity follows a J-curve: an initial gain, a dip, then recovery.

For whoever designs a platform, the lesson is twofold. The gain depends on self-service and independence, not on the mere existence of a platform. And stability must be measured from the start, with the metrics you will find in the article Platform Observability: DORA and SPACE metrics: without them, you will not see the 14% drop.

Where to start: the CNCF maturity model

The CNCF also publishes a Platform Engineering Maturity Model with four levels (Provisional, Operationalized, Scalable, Optimizing) and five aspects to assess independently: investment, adoption, interfaces, operations and measurement. The document warns that there is no need to classify an organization as «level 2» or «level 4», because the same team can be at different levels on different aspects, and that each extra level requires more funding and more of people's time, so reaching the top is not a goal in itself.

An example from the model, on adoption: at the lowest level teams use shared tools sporadically and without a common line; at the «Scalable» level it is the teams that choose the platform because it reduces their load, not because a mandate imposes it on them. It is a useful distinction when you have to decide whether to impose a path or make it convenient, and you will come back to it in the article on golden paths.

A note of my own, not from the literature: a startup with a dozen developers rarely needs a platform team. It needs a written path for creating a service, a template that works and someone to keep it up to date. The rest comes when the inventory from the script above stops being an exercise.

How to read the series

The articles are designed to be read in sequence, from the why to the how and finally to the measurement. Knowing what comes next helps you decide where to stop:

1. This article: definitions, data and the case for and against.
2. Architecture of a modern IDP: planes, components and integrations.
3. Golden Paths: standardized flows and templates.
4. Backstage: portal, catalog and templates.
5. Infrastructure as Code: Terraform, Pulumi and policy as code.
6. Service Catalog and CMDB: ownership and dependencies.
7. DORA and SPACE metrics: how to measure the platform.
8. Multi-cloud and developer experience: portability and listening to users.
9. AI in the platform and security and compliance.
10. Case study, the production postmortems and on-prem AI versus cloud.

If you are short of time, read in this order: this article, the golden paths and the metrics. Backstage and Infrastructure as Code are tools: they come after you have decided what the platform offers, not before, and the articles on security and observability rest on those choices. On the blog you will also find the series on DevSecOps and on Observability, to which I point whenever the topics touch.

What you can do now

You can define an internal platform and tell it apart from a pile of tools, using the seven CNCF characteristics. You can measure an organization's fragmentation with an inventory and the distance between catalog and demand with the second script, both reproducible with the commands shown above. You can cite the DORA 2024 results in full, including the drop in throughput and stability, and you know that the gain depends on self-service and independence. The next step is the architecture: which parts a platform is made of and where the boundary runs between what decides and what executes.