Cribl puts your IT and Security data at the center of your data management strategy and provides a one-stop shop for analyzing, collecting, processing, and routing it all at any scale. Try the Cribl suite of products and start building your data engine today!
Learn more ›Evolving demands placed on IT and Security teams are driving a new architecture for how observability data is captured, curated, and queried. This new architecture provides flexibility and control while managing the costs of increasing data volumes.
Read white paper ›Cribl Stream is a vendor-agnostic observability pipeline that gives you the flexibility to collect, reduce, enrich, normalize, and route data from any source to any destination within your existing data infrastructure.
Learn more ›Cribl Edge provides an intelligent, highly scalable edge-based data collection system for logs, metrics, and application data.
Learn more ›Cribl Search turns the traditional search process on its head, allowing users to search data in place without having to collect/store first.
Learn more ›Cribl Lake is a turnkey data lake solution that takes just minutes to get up and running — no data expertise needed. Leverage open formats, unified security with rich access controls, and central access to all IT and security data.
Learn more ›The Cribl.Cloud platform gets you up and running fast without the hassle of running infrastructure.
Learn more ›Cribl.Cloud Solution Brief
The fastest and easiest way to realize the value of an observability ecosystem.
Read Solution Brief ›Cribl Copilot gets your deployments up and running in minutes, not weeks or months.
Learn more ›AppScope gives operators the visibility they need into application behavior, metrics and events with no configuration and no agent required.
Learn more ›Explore Cribl’s Solutions by Use Cases:
Explore Cribl’s Solutions by Integrations:
Explore Cribl’s Solutions by Industry:
September 25 | 10am PT / 1pm ET
Hold my beer: lessons from one team’s data pipeline journey
Register ›Try Your Own Cribl Sandbox
Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Get inspired by how our customers are innovating IT, security and observability. They inspire us daily!
Read Customer Stories ›Sally Beauty Holdings
Sally Beauty Swaps LogStash and Syslog-ng with Cribl.Cloud for a Resilient Security and Observability Pipeline
Read Case Study ›Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Transform data management with Cribl, the Data Engine for IT and Security
Learn More ›Cribl Corporate Overview
Cribl makes open observability a reality, giving you the freedom and flexibility to make choices instead of compromises.
Get the Guide ›Stay up to date on all things Cribl and observability.
Visit the Newsroom ›Cribl’s leadership team has built and launched category-defining products for some of the most innovative companies in the technology sector, and is supported by the world’s most elite investors.
Meet our Leaders ›Join the Cribl herd! The smartest, funniest, most passionate goats you’ll ever meet.
Learn More ›Whether you’re just getting started or scaling up, the Cribl for Startups program gives you the tools and resources your company needs to be successful at every stage.
Learn More ›Want to learn more about Cribl from our sales experts? Send us your contact information and we’ll be in touch.
Talk to an Expert ›Cribl Stream offers different ways to optimize data, such as:
In this blog, I will focus on the Aggregation use case using the Aggregations function and how you can practically use the Aggregations function to format the output in different ways.
I will start with the simplest way of using Aggregations and gradually move into more complex data transformations to showcase how you can format the Aggregations output on Cribl Stream and have a minimal impact on your existing SIEM rules and searches, if any, while benefiting from the data optimization results.
Cribl is built on a shared-nothing architecture, meaning each worker process will operate separately. This will limit the expected reduction percentages, especially in large Cribl environments with many worker processes, as events arriving on separate Worker Processes will not be aggregated.
For optimal reduction benefits with the Aggregations function, it is highly recommended to use a caching tier, such as Redis, to share the state across the different processes.
This blog doesn’t cover Redis, but a good start with Redis and Cribl is the Redis Knowledge Pack on Cribl’s dispensary.
I’ve used the Palo Alto Networks Traffic data in this example. However, there are many other great candidates for Aggregations such as Network Firewalls, WAFs, Web Proxies, IIS and Cloud Flow Logs.
A sample of the PAN Traffic data in its CSV format looks like this:
We will review three scenarios based on the same sample data, each outputting a different format. The scenarios are detailed enough for you to choose from them based on your needs and required output.
We first need to parse the data and have Cribl Steam recognize the different fields so we can use them in the subsequent functions. For this, we will use the Parser function and leverage the embedded Parser Library.
Now that we have parsed the data, we can see the different fields on the output.
The “Aggregates” are the function(s) we will run on the data. They will allow us to maintain the accuracy of the numbers while sending out a single event out of many.
They also allow us to pass the number of Aggregated logs into the SIEM destination. This is very powerful for SIEM use cases that rely on the number of events such as “Brute Force Attacks.”
We will then pick the following five fields to group aggregates by. Depending on your data, you may need to add more fields to achieve an accurate grouping.
The output will look as follows:
You may have noticed that we’ve lost the original _raw
field after running the Aggregations. Depending on your use case, you may need to maintain the _raw
field and pass it to your SIEM destination.
To do so, we will update the Aggregations function with two entries:
_raw
field of the first aggregated log into a temporary message (notice the use of the __
to make as an internal Cribl field._raw
of the aggregated output.
Our aggregated output format now looks almost similar to the original log format.
You can then run a separate Eval function to clean some of the fields that you don’t need.
Now that we have seen how we maintained the _raw
field in the aggregated output, we can optimize our data further. For instance, the _raw
field still maintains the values of the first event added to the Aggregations. More specifically, the values of “bytes
”, “bytes_received
” , and “byte_sent
” are the original values of the first aggregated event which are logically not equal to the “totalbytes
”, “totalbytes_received
” , and “totalbytes_sent
”. You can simply update your SIEM rules, reports, and dashboards to use these new fields.
But in many cases, you may want to limit the changes needed on your SIEM. In this case, you can use the power of Cribl Stream functions to update the _raw field with these aggregated numbers and save time on SIEM rule changes.
To do so, we will parse the _raw
of the aggregated events, run an Eval function to replace the “bytes
” with the “totalbytes
” and serialize the data.
We can finally run an Eval to clean the output and maintain only the _raw
, _time
, and the “number of aggregated events
.”
The Aggregations function allows you to optimize different types of security data before ingesting it into your SIEM. The flexibility of the Aggregations function, along with other Cribl Stream functions, helps you format the data output in different ways to satisfy your requirements. Consider using a caching tier, like Redis, to get optimal reduction results in production environments. While we focused on the SIEM and security data, these techniques can be applied to different datasets and Cribl destinations.
Cribl, the Data Engine for IT and Security, empowers organizations to transform their data strategy. Customers use Cribl’s suite of products to collect, process, route, and analyze all IT and security data, delivering the flexibility, choice, and control required to adapt to their ever-changing needs.
We offer free training, certifications, and a generous free usage plan across our products. Our community Slack features Cribl engineers, partners, and customers who can answer your questions as you get started. We also offer a hands-on Sandbox for those interested in how companies globally leverage our products for their data challenges.
Experience a full version of Cribl Stream and Cribl Edge in the cloud with pre-made sources and destinations.
Classic choice. Sadly, our website is designed for all modern supported browsers like Edge, Chrome, Firefox, and Safari
Got one of those handy?