How to Configure Vector for Log Transformation: A Practical Guide

If you’ve ever looked at your logging dashboard and seen a wall of unstructured text that’s impossible to query, you know the pain of ‘dirty data.’ I’ve spent years wrestling with log formats, and the biggest lesson I’ve learned is that transforming logs at the edge is the only way to maintain sanity. That’s where Vector comes in.

Learning how to configure Vector for log transformation is a game-changer for anyone trying to optimize their observability pipeline. Instead of letting your backend (like Loki or Elasticsearch) do the heavy lifting, Vector allows you to parse, scrub, and enrich logs before they ever leave your infrastructure. This not only makes your queries faster but can significantly lower costs—something I often discuss when explaining why is my CloudWatch bill so high.

Prerequisites

Vector installed on your machine or cluster (v0.20+ recommended).
A source of logs (e.g., a local file, syslog, or Docker logs).
Basic familiarity with YAML configuration.
A destination for your logs (e.g., Console, S3, or Loki).

Step-by-Step Vector Configuration

1. Define Your Source

Before you can transform data, you need to bring it in. In my typical setup, I use the file source to tail application logs. Here is a basic source configuration:

[sources.my_app_logs]
  type = "file"
  include = ["/var/log/app/*.log"]

2. Implementing the Remap Transform

The heart of log transformation in Vector is the remap transform. This uses VRL (Vector Remap Language), a powerful domain-specific language designed specifically for telemetry. This is the core of how to configure Vector for log transformation efficiently.

Suppose we have a log line like: 2026-04-25 16:27:02 [INFO] User 123 logged in from 1.2.3.4. We want to turn this into a structured JSON object.

[transforms.parse_and_clean]
  type = "remap"
  inputs = ["my_app_logs"]
  source = "\n"
  # Parse the log line using a regex
  . = parse_regex!(.message, r'^(?P<timestamp>\S+ \S+) \[(?P<level>\S+)\] (?P<body>.*)$')
  
  # Convert timestamp to a proper DateTime object
  .timestamp = parse_timestamp!(.timestamp, format: "%Y-%m-%d %H:%M:%S")
  
  # Add a custom field for the environment
  .env = "production"
  
  # Remove the original message to save space
  del(.message)

Visual representation of log transformation from raw text to structured JSON using Vector VRL

As shown in the image below, the transformation process takes a raw string and maps it into a structured object, which is essential for effective querying in tools like Grafana.

3. Adding Conditional Logic and Scrubbing

In a production environment, you often need to hide sensitive data (PII) or drop useless logs. I’ve found that filtering out “debug” logs in production can reduce ingest volume by up to 30%.

[transforms.filter_debug]
  type = "filter"
  inputs = ["parse_and_clean"]
  condition = '.level != "DEBUG"'

For PII scrubbing, you can add another remap step to mask IP addresses using VRL’s string manipulation functions. This ensures that you aren’t storing plain-text sensitive data in your cluster, which is a critical security step when choosing between Grafana Loki vs ELK stack for logs.

4. Routing to the Sink

Finally, connect your transformed data to a sink. For testing, the console sink is perfect.

[sinks.stdout]
  type = "console"
  inputs = ["filter_debug"]
  encoding.codec = "json"

Pro Tips for Vector Transformations

Order Matters: Always put your filter transforms as early as possible in the pipeline to avoid wasting CPU cycles on logs you intend to drop.
Use the VRL Playground: Don’t guess your regex. Use the official Vector VRL playground to test your logic against real log samples before deploying.
Avoid Over-Parsing: Transforming every single field can lead to high CPU usage. Only extract the fields you actually intend to alert on or visualize.

Troubleshooting Common Issues

Issue: Logs are disappearing after a transform.
In my experience, this usually happens because a VRL function like parse_regex! failed and the ! operator caused the event to be dropped. If you aren’t sure why logs are missing, remove the ! and check for .error fields in your output.

Issue: High CPU usage on the Vector agent.
Check if you are using complex regex on very large log lines. Try to use split() or strip() for simple delimiters before jumping to heavy regex.

What’s Next?

Now that you know how to configure Vector for log transformation, I recommend looking into Metric Aggregation. Instead of sending every log line to your backend, you can use Vector to count the number of 500 errors per minute and send that as a single metric to Prometheus, further reducing your costs.