Teams can now filter, redact, and reshape log streams using SQL queries before export, reducing downstream data volume and storage costs.
Key points
- Logpush now supports Transformers powered by Apache DataFusion and Arrow.
- The feature allows filtering out health checks and redacting sensitive fields.
- This update was made generally available during Cloudflare's Birthday Week.
Cloudflare has expanded its Logpush service with SQL-based Transformers. This feature lets teams modify log data before it reaches external destinations.
What happened
Cloudflare announced that Logpush now supports Transformers, which are SQL queries applied to log datasets before export. According to the company, this feature became generally available during their recent Birthday Week. The update is part of eight major changes to Cloudflare Observability, aiming to make features accessible on all self-serve plans.
Transformers allow users to filter records, redact sensitive information, and reshape data structures. Teams can define which fields to keep, rename, or remove using standard SQL syntax. The system applies these changes directly within the export pipeline, ensuring destinations receive data in the required format. This eliminates the need for separate processing services that teams previously had to deploy and maintain.
The underlying technology relies on Apache DataFusion and Apache Arrow. Cloudflare states that this engine handles high-throughput requirements without introducing additional failure scenarios. The system processes batches of logs, applying the SQL query to each batch before sending the resulting NDJSON records to the customer's chosen destination.
Why it matters
Observability teams often export logs to security information and event management systems or analytics platforms. Before this update, teams frequently had to build and manage separate services to clean or normalize that data. This added operational overhead and cost, as teams paid to export and store records they might discard immediately upon arrival.
By moving processing into the Logpush workflow, teams can reduce downstream data volume. For example, a team can filter out internal health-check traffic or add static provider fields to normalize records from multiple CDNs. This pre-export filtering helps lower costs associated with data ingestion and storage at the destination.
Cloudflare notes that its data pipeline ingests more than one billion events per second. The Logpush delivery service handles over 110 million records per second. Integrating Transformers into this high-volume path requires predictable resource use and reliable delivery, which the company says it has achieved through its distributed architecture.
The fix
- Define SQL queries to filter unnecessary records like health checks.
- Use Transformers to redact sensitive fields such as IP addresses.
- Attach the Transformer to a compatible NDJSON Logpush job.
Background: IP addresses
An IP address is a unique numerical label assigned to every device connected to a computer network using the Internet Protocol. It functions like a postal address, ensuring data packets reach the correct destination. Devices use these addresses to communicate, and routers direct traffic based on them.
Read the full guide: IP Addresses Explained: How Network Identity Works
What to do and how to stay safe: Cloudflare
- Review current log export costs and identify high-volume datasets.
- Test SQL filters in Transformer Studio to validate output schemas.
- Monitor downstream storage usage after enabling pre-export filtering.
- Ensure sensitive data is redacted before logs leave the network.
Step-by-step guide: What is password policies and how they protect systems
General security guidance from the Crypt Weekly newsroom. It is not confirmed advice from the organisations named in this story.
Frequently asked questions
How do I create a Transformer for my logs?
You can create and preview Transformers in the Cloudflare dashboard using Transformer Studio or via the API.
Does this feature change the log format?
The transformed results are exported in NDJSON format, matching the input serialization.
Is this available on all plans?
Yes, Logpush and Transformers are available on all self-serve plans according to Cloudflare.
