In the world of cloud-native security, there is a dangerous misconception that a clean image scan equals a secure production environment. I’ve seen countless teams rely solely on best open source container security scanners to catch vulnerabilities during the CI/CD phase, only to be blindsided by a zero-day exploit or a compromised credential that allows an attacker to execute a shell inside a running pod. This is exactly why use Falco for runtime security: it provides the visibility you need after the code is deployed.
The Challenge: The “Blind Spot” of Static Analysis
Static Analysis (SAST) and Image Scanning are essential, but they are snapshots in time. They tell you if a library is outdated or if a secret was accidentally baked into an image. However, they cannot tell you if a running container suddenly starts modifying /etc/shadow or opens an unexpected outbound connection to a known malicious IP in Eastern Europe.
When I first started managing Kubernetes clusters, I realized that once a pod was up, it was essentially a black box. If an attacker gained access via a remote code execution (RCE) vulnerability, they could move laterally through the cluster, and my logs would show almost nothing until the database was already dumped. We needed a way to monitor system calls in real-time without killing performance.
Solution Overview: What is Falco?
Falco is a CNCF graduated project that acts as a security camera for your cloud-native environment. Unlike traditional security tools that look at logs (which can be tampered with), Falco taps directly into the Linux kernel. It monitors system calls (syscalls) and matches them against a set of rules to detect suspicious activity.
The magic happens via eBPF (Extended Berkeley Packet Filter). By using eBPF, Falco can intercept events—like file opens, network connections, and process executions—with minimal overhead. As I’ve noted in my sysdig runtime security review, this deep kernel integration is what separates true runtime security from simple log aggregation.
Techniques: How Falco Detects Threats
Falco operates on a rule-based engine. A rule consists of a condition, a description, and a priority. For example, if a shell is spawned inside a container, that’s usually a red flag.
Example: Detecting a Shell in a Pod
Here is a simplified version of a Falco rule that triggers when a shell is opened in a container. This is one of the primary reasons why security engineers choose Falco over generic monitoring.
rule Spawn Shell in Pod
description: A shell was spawned in a pod, which is uncommon in production
condition: proc.name = sh and container.id != host
output: "Shell spawned in container (user=%user.name container_id=%container.id proc=%proc.name)"
priority: WARNING
In my experience, the most powerful part of Falco is the ability to create custom macros. You can define what “normal” looks like for your specific application and alert on anything that deviates from that baseline.
Implementation: Getting Falco Running
The fastest way to deploy Falco is via Helm. I recommend starting with the Falco Helm chart to get the daemonset running across all your nodes.
# Add the Falco Helm repo
helm repo add falcosecurity https://falcosecurity.github.io/charts
# Install Falco
helm install falco falcosecurity/falco
Once installed, Falco will begin monitoring your kernel. However, the raw output usually goes to stdout or syslog. To make this actionable, you’ll want to configure Falcosidekick. This companion tool allows you to push alerts to Slack, Discord, or an Elasticsearch cluster, transforming raw syscall data into a notification that actually wakes you up when there’s a breach.
As shown in the image below, the flow from kernel event to Slack notification is where the real value lies for an SRE team.
Case Study: Stopping a Reverse Shell Attack
I once worked on a project where a legacy Java app had an unpatched Log4j-style vulnerability. An attacker managed to execute a reverse shell, which allowed them to probe the internal network. Because we had Falco deployed, we received a CRITICAL alert within seconds: "Unexpected outbound network connection to unauthorized IP".
Because we had the exact container ID and process name from the Falco alert, we were able to kill the pod and isolate the node immediately. Without runtime security, we likely wouldn’t have known until the attacker escalated privileges to the node level.
Pitfalls and Trade-offs
- Alert Fatigue: If you use the default ruleset without tuning, you will get flooded with notifications. You must spend time marking “known good” behavior as exceptions.
- Kernel Compatibility: Since Falco relies on kernel modules or eBPF, OS updates can occasionally break the driver. I always recommend testing Falco in a staging environment before upgrading your Node OS.
- Resource Overhead: While eBPF is efficient, monitoring every single syscall on a high-traffic database node can lead to a noticeable CPU spike.
Conclusion
So, why use Falco for runtime security? Because static scanning is only half the battle. Falco provides the “eyes” inside your cluster, ensuring that even if an attacker bypasses your perimeter, their movements are logged and alerted in real-time.
If you are serious about hardening your infrastructure, start by deploying Falco in audit mode, tuning your rules, and integrating alerts into your existing incident response workflow.