Runtime Security: Real-Time Monitoring
Runtime security is the last line of defense in the DevSecOps paradigm: protecting applications while they're running in production. SAST, DAST, and SCA find vulnerabilities before deployment, but runtime security detects and responds to active threats, anomalous behavior, and ongoing attacks.
In this article, we will explore Falco for syscall-based threat detection, eBPF for low-overhead monitoring, anomaly detection techniques, and integration with SIEM systems for complete visibility.
What You'll Learn
- How runtime threat detection works
- Falco: rules, configuration, and Kubernetes deployment
- eBPF: high-performance kernel monitoring
- Anomaly detection and behavioral analysis
- Automated incident response
- SIEM integration (Splunk, ELK, Datadog)
Falco: Runtime Threat Detection
Falco is the CNCF reference project for runtime security. It monitors Linux kernel syscalls in real-time and generates alerts when it detects behavior that violates defined security rules. Falco can detect shell spawns in containers, sensitive file access, suspicious network connections, and privilege escalation.
Kubernetes Installation with Helm
# Add Falco Helm repository
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
# Install Falco with eBPF driver
helm install falco falcosecurity/falco \
--namespace falco --create-namespace \
--set driver.kind=ebpf \
--set falcosidekick.enabled=true \
--set falcosidekick.webui.enabled=true
# Verify installation
kubectl get pods -n falco
kubectl logs -n falco -l app.kubernetes.io/name=falco
Custom Falco Rules
Falco includes hundreds of built-in rules, but creating custom rules to adapt to your environment is essential:
# falco-custom-rules.yaml
customRules:
custom-rules.yaml: |-
# Detect shell in production containers
- rule: Shell in Production Container
desc: Detect shell execution in production namespace
condition: >
spawned_process and container and
proc.name in (bash, sh, zsh, dash) and
k8s.ns.name = "production"
output: >
Shell spawned in production container
(user=%user.name command=%proc.cmdline
container=%container.name namespace=%k8s.ns.name
pod=%k8s.pod.name image=%container.image.repository)
priority: CRITICAL
tags: [container, shell, production]
# Detect access to credential files
- rule: Read Sensitive Credential Files
desc: Detect access to credential files
condition: >
open_read and container and
(fd.name startswith /etc/shadow or
fd.name startswith /root/.ssh or
fd.name startswith /run/secrets)
output: >
Sensitive file read in container
(file=%fd.name user=%user.name
container=%container.name command=%proc.cmdline)
priority: WARNING
tags: [container, filesystem, credentials]
# Detect unexpected outbound connections
- rule: Unexpected Outbound Connection
desc: Detect outbound connections from containers that should not have network
condition: >
outbound and container and
k8s.pod.label.network-restricted = "true"
output: >
Unexpected outbound connection from restricted pod
(connection=%fd.name pod=%k8s.pod.name
namespace=%k8s.ns.name image=%container.image.repository)
priority: ERROR
tags: [container, network]
eBPF: High-Performance Kernel Monitoring
eBPF (extended Berkeley Packet Filter) is a Linux kernel technology that allows running sandboxed programs in the kernel without modifying it. eBPF is the underlying technology that Falco, Tetragon, and other tools use for low-overhead syscall monitoring.
eBPF overhead is typically less than 1-2% CPU, making continuous monitoring feasible even in high-performance environments.
Tetragon: eBPF Security Observability
Tetragon by Isovalent (now Cisco) is an eBPF-based tool that goes beyond simple monitoring: it can actively block suspicious behavior at the kernel level, such as unauthorized process execution or unexpected network connections.
# tetragon-policy.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
name: block-privilege-escalation
spec:
kprobes:
- call: "__x64_sys_setuid"
syscall: true
args:
- index: 0
type: int
selectors:
- matchArgs:
- index: 0
operator: Equal
values:
- "0" # UID 0 = root
matchNamespaces:
- namespace: production
operator: In
matchActions:
- action: Sigkill # Terminate the process
Anomaly Detection
Anomaly detection uses behavioral baselines to identify deviations from normal behavior. Instead of looking for known attack patterns, it detects any unusual behavior that could indicate a compromise.
- Network baseline: monitor typical network traffic and detect unexpected connections
- Process baseline: record normal processes and alert for unknown processes
- File access patterns: monitor file access and detect anomalous reads
- Resource usage: CPU/memory spikes can indicate cryptomining or DoS
Common Indicators of Compromise (IoC)
| Indicator | Possible Attack | Tool |
|---|---|---|
| Shell spawn in container | RCE, backdoor | Falco |
| Connection to known malicious IP | C2 communication | Network monitoring |
| Sudden CPU spike | Cryptomining | Metrics/Prometheus |
| Reading /etc/shadow | Credential theft | Falco, Tetragon |
| curl/wget process in container | Data exfiltration | Falco |
Automated Incident Response
Response speed is critical during a security incident. Automated incident response allows reacting in seconds instead of hours:
# Falcosidekick: automatic responses to alerts
# values.yaml for Helm
falcosidekick:
config:
# Slack notification for WARNING+ alerts
slack:
webhookurl: "https://hooks.slack.com/services/xxx"
minimumpriority: "warning"
# Execute Kubernetes action for CRITICAL alerts
kubernetesCR:
enabled: true
minimumpriority: "critical"
# Pagerduty for on-call
pagerduty:
routingkey: "xxx"
minimumpriority: "critical"
# Automatic actions
customActions:
# Isolate compromised pod (remove network policy label)
- name: isolate-pod
priority: critical
action: kubernetes
parameters:
namespace: "production"
action: "label"
value: "quarantine=true"
SIEM Integration
Runtime security alerts must flow into a centralized system (SIEM) for correlation, analysis, and long-term retention. The most common SIEMs in the DevSecOps ecosystem are:
SIEM Integration
- ELK Stack (Elasticsearch, Logstash, Kibana): open source, flexible, great for log analysis
- Splunk: enterprise, powerful queries, good for compliance
- Datadog Security Monitoring: SaaS, native integration with cloud and containers
- Grafana + Loki: lightweight, great for Kubernetes environments
Runtime Security Best Practices
- Deploy Falco on every cluster: runtime monitoring must cover 100% of workloads
- Custom rules for context: adapt Falco rules to your application environment
- Automate incident response: automatic isolation of compromised pods
- Behavioral baseline: establish a baseline before activating anomaly alerts
- Log retention: keep security logs for at least 90 days (ideally 1 year)
- Documented runbooks: procedure for each alert type with investigation steps
Conclusions
Runtime security closes the DevSecOps protection circle. While pre-deploy tools (SAST, DAST, SCA, IaC scanning) prevent vulnerabilities, runtime security detects and responds to active threats in production. With Falco, eBPF, and automated incident response, organizations can react in real-time.
In the next article, we'll explore Compliance Framework, analyzing how to map DevSecOps controls to GDPR, SOC2, and ISO27001 requirements and automate conformity with compliance-as-code.