For a long time, I treated container security as a ‘final step’—something I’d run right before a major release. I quickly learned the hard way that this approach creates a massive bottleneck. Finding a critical CVE (Common Vulnerabilities and Exposures) in your base image an hour before deployment is a nightmare.
To avoid this, you need a systemic approach. Implementing the best practices for container security scanning isn’t just about picking a tool; it’s about where, when, and how you integrate those scans into your workflow. In my experience, the goal is to shift security ‘left’ so that vulnerabilities are caught while the code is still fresh in the developer’s mind.
1. Scan Your Base Images Early and Often
The most common mistake I see is trusting ‘official’ images blindly. Even the official Node or Python images can carry hundreds of vulnerabilities if you use the full version. I always recommend starting with alpine or distroless images to minimize the attack surface.
Don’t just scan the final image; scan the base image you’re inheriting from. If the foundation is rotten, the house will fall. When choosing your tool, it’s helpful to see a breakdown of Trivy vs Grype vs Snyk for container scanning to see which handles base image layers most efficiently.
2. Implement Multi-Stage Builds
I’ve seen images bloated with build-time dependencies like GCC, Git, and SSH keys that are never used at runtime. These aren’t just a waste of space; they are security risks. Use multi-stage builds to separate the build environment from the runtime environment.
# Example of a secure multi-stage build
FROM golang:1.21-alpine AS builder
WORKDIR /app
COPY . .
RUN go build -o main .
FROM alpine:latest
WORKDIR /root/
COPY --from=builder /app/main .
CMD ["./main"]
3. Integrate Scanning into the CI/CD Pipeline
Security scanning should be an automated gate, not a manual chore. If a scan detects a ‘Critical’ or ‘High’ vulnerability, the pipeline should fail immediately. I’ve found that this forces a culture of security accountability within the team.
If you’re unsure how to set this up, check out my guide on how to automate security testing in CI/CD pipeline for concrete implementation steps.
4. Use a Private Trusted Registry
Avoid pulling images directly from public hubs in production. I recommend mirroring required public images to a private registry (like AWS ECR, Google Artifact Registry, or Harbor). This allows you to scan the image once, sign it, and know exactly what is being deployed.
5. Avoid Running as Root
By default, Docker containers run as root. If an attacker escapes the container, they have root access to the host. I always add a non-privileged user to my Dockerfiles. It’s a simple change that significantly hardens your container.
# Create a system user to run the app
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser6. Scan for Secrets and Hardcoded Credentials
Container scanning isn’t just about CVEs in packages. I’ve caught several instances where developers accidentally left an .env file or a cloud provider key inside an image layer. Use tools like truffleHog or the secret-scanning capabilities of Snyk to ensure no secrets are baked into your layers.
7. Implement a Vulnerability Management Policy
Not every vulnerability is exploitable in your specific context. If you fail every build for a ‘Medium’ vulnerability that only affects a part of the OS you don’t use, your developers will start ignoring the security reports. Establish a clear policy: Criticals must be fixed in 24 hours, Highs in 7 days, and Mediums in 30 days.
8. Scan at Rest (Continuous Scanning
A ‘clean’ image today might have a newly discovered CVE tomorrow. I’ve learned that scanning only at the time of build is insufficient. You need to scan images while they are sitting in your registry. Most modern registries now offer this feature natively.
9. Use Image Signing and Attestation
How do you know the image running in production is the same one that passed the security scan in CI? Use tools like Cosign (from the Sigstore project) to sign your images. Your Kubernetes cluster can then be configured to only pull images that have a valid security signature.
10. Limit Capabilities with Seccomp and AppArmor
Even a scanned image can be dangerous if the container has too many privileges. I always use the principle of least privilege. Limit the syscalls the container can make using Seccomp profiles to prevent an attacker from interacting with the host kernel.
Common Mistakes to Avoid
- Ignoring ‘Fixed In’ versions: Scanning is useless if you don’t update. Always check the ‘Fixed In’ version provided by the scanner.
- Over-reliance on a single tool: No scanner is perfect. I often use a combination of a fast scanner for CI (like Grype) and a deep scanner for registries (like Snyk).
- Scanning the ‘Latest’ tag: Never scan or deploy
:latest. Use immutable tags (like Git commit SHAs) to ensure you are scanning the exact code being deployed.
Measuring Your Success
To know if your best practices for container security scanning are working, track these three metrics:
- Mean Time to Remediate (MTTR): How long does it take from CVE discovery to a patched image in production?
- Vulnerability Density: Is the number of High/Critical CVEs per image decreasing over time?
- Build Failure Rate: Are security gates catching issues before they hit staging?
Ready to harden your infrastructure? Start by auditing your current base images and integrating a scanner into your next sprint.