11 - Runtime Security - Monitoring and Anomaly Detection

Runtime Security: Real-Time Monitoring

Runtime security is the last line of defense in the DevSecOps paradigm: protecting applications while they're running in production. SAST, DAST, and SCA find vulnerabilities before deployment, but runtime security detects and responds to active threats, anomalous behavior, and ongoing attacks.

In this article, we will explore Falco for syscall-based threat detection, eBPF for low-overhead monitoring, anomaly detection techniques, and integration with SIEM systems for complete visibility.

Falco: Runtime Threat Detection

Falco is the CNCF reference project for runtime security. It monitors Linux kernel syscalls in real-time and generates alerts when it detects behavior that violates defined security rules. Falco can detect shell spawns in containers, sensitive file access, suspicious network connections, and privilege escalation.

Kubernetes Installation with Helm

# Add Falco Helm repository
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

# Install Falco with eBPF driver
helm install falco falcosecurity/falco \
  --namespace falco --create-namespace \
  --set driver.kind=ebpf \
  --set falcosidekick.enabled=true \
  --set falcosidekick.webui.enabled=true

# Verify installation
kubectl get pods -n falco
kubectl logs -n falco -l app.kubernetes.io/name=falco
      

Custom Falco Rules

Falco includes hundreds of built-in rules, but creating custom rules to adapt to your environment is essential:

# falco-custom-rules.yaml
customRules:
  custom-rules.yaml: |-
    # Detect shell in production containers
    - rule: Shell in Production Container
      desc: Detect shell execution in production namespace
      condition: >
        spawned_process and container and
        proc.name in (bash, sh, zsh, dash) and
        k8s.ns.name = "production"
      output: >
        Shell spawned in production container
        (user=%user.name command=%proc.cmdline
        container=%container.name namespace=%k8s.ns.name
        pod=%k8s.pod.name image=%container.image.repository)
      priority: CRITICAL
      tags: [container, shell, production]

    # Detect access to credential files
    - rule: Read Sensitive Credential Files
      desc: Detect access to credential files
      condition: >
        open_read and container and
        (fd.name startswith /etc/shadow or
         fd.name startswith /root/.ssh or
         fd.name startswith /run/secrets)
      output: >
        Sensitive file read in container
        (file=%fd.name user=%user.name
        container=%container.name command=%proc.cmdline)
      priority: WARNING
      tags: [container, filesystem, credentials]

    # Detect unexpected outbound connections
    - rule: Unexpected Outbound Connection
      desc: Detect outbound connections from containers that should not have network
      condition: >
        outbound and container and
        k8s.pod.label.network-restricted = "true"
      output: >
        Unexpected outbound connection from restricted pod
        (connection=%fd.name pod=%k8s.pod.name
        namespace=%k8s.ns.name image=%container.image.repository)
      priority: ERROR
      tags: [container, network]
      

eBPF: High-Performance Kernel Monitoring

eBPF (extended Berkeley Packet Filter) is a Linux kernel technology that allows running sandboxed programs in the kernel without modifying it. eBPF is the underlying technology that Falco, Tetragon, and other tools use for low-overhead syscall monitoring.

eBPF overhead is typically less than 1-2% CPU, making continuous monitoring feasible even in high-performance environments.

Tetragon: eBPF Security Observability

Tetragon by Isovalent (now Cisco) is an eBPF-based tool that goes beyond simple monitoring: it can actively block suspicious behavior at the kernel level, such as unauthorized process execution or unexpected network connections.

# tetragon-policy.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: block-privilege-escalation
spec:
  kprobes:
    - call: "__x64_sys_setuid"
      syscall: true
      args:
        - index: 0
          type: int
      selectors:
        - matchArgs:
            - index: 0
              operator: Equal
              values:
                - "0"  # UID 0 = root
          matchNamespaces:
            - namespace: production
              operator: In
          matchActions:
            - action: Sigkill  # Terminate the process
      

Anomaly Detection

Anomaly detection uses behavioral baselines to identify deviations from normal behavior. Instead of looking for known attack patterns, it detects any unusual behavior that could indicate a compromise.

• Network baseline: monitor typical network traffic and detect unexpected connections
• Process baseline: record normal processes and alert for unknown processes
• File access patterns: monitor file access and detect anomalous reads
• Resource usage: CPU/memory spikes can indicate cryptomining or DoS

Automated Incident Response

Response speed is critical during a security incident. Automated incident response allows reacting in seconds instead of hours:

# Falcosidekick: automatic responses to alerts
# values.yaml for Helm
falcosidekick:
  config:
    # Slack notification for WARNING+ alerts
    slack:
      webhookurl: "https://hooks.slack.com/services/xxx"
      minimumpriority: "warning"

    # Execute Kubernetes action for CRITICAL alerts
    kubernetesCR:
      enabled: true
      minimumpriority: "critical"

    # Pagerduty for on-call
    pagerduty:
      routingkey: "xxx"
      minimumpriority: "critical"

  # Automatic actions
  customActions:
    # Isolate compromised pod (remove network policy label)
    - name: isolate-pod
      priority: critical
      action: kubernetes
      parameters:
        namespace: "production"
        action: "label"
        value: "quarantine=true"
      

SIEM Integration

Runtime security alerts must flow into a centralized system (SIEM) for correlation, analysis, and long-term retention. The most common SIEMs in the DevSecOps ecosystem are:

Runtime Security Best Practices

1. Deploy Falco on every cluster: runtime monitoring must cover 100% of workloads
2. Custom rules for context: adapt Falco rules to your application environment
3. Automate incident response: automatic isolation of compromised pods
4. Behavioral baseline: establish a baseline before activating anomaly alerts
5. Log retention: keep security logs for at least 90 days (ideally 1 year)
6. Documented runbooks: procedure for each alert type with investigation steps

Conclusions

Runtime security closes the DevSecOps protection circle. While pre-deploy tools (SAST, DAST, SCA, IaC scanning) prevent vulnerabilities, runtime security detects and responds to active threats in production. With Falco, eBPF, and automated incident response, organizations can react in real-time.

In the next article, we'll explore Compliance Framework, analyzing how to map DevSecOps controls to GDPR, SOC2, and ISO27001 requirements and automate conformity with compliance-as-code.

Related deep dive: 10 - Infrastructure as Code Scanning - IaC Security e Terraform Policy.

Related deep dive: 12 - Compliance Framework - GDPR, SOC2, ISO27001 Mapping e Automazione.