Cribl puts your IT and Security data at the center of your data management strategy and provides a one-stop shop for analyzing, collecting, processing, and routing it all at any scale. Try the Cribl suite of products and start building your data engine today!
Learn more ›Evolving demands placed on IT and Security teams are driving a new architecture for how observability data is captured, curated, and queried. This new architecture provides flexibility and control while managing the costs of increasing data volumes.
Read white paper ›Cribl Stream is a vendor-agnostic observability pipeline that gives you the flexibility to collect, reduce, enrich, normalize, and route data from any source to any destination within your existing data infrastructure.
Learn more ›Cribl Edge provides an intelligent, highly scalable edge-based data collection system for logs, metrics, and application data.
Learn more ›Cribl Search turns the traditional search process on its head, allowing users to search data in place without having to collect/store first.
Learn more ›Cribl Lake is a turnkey data lake solution that takes just minutes to get up and running — no data expertise needed. Leverage open formats, unified security with rich access controls, and central access to all IT and security data.
Learn more ›The Cribl.Cloud platform gets you up and running fast without the hassle of running infrastructure.
Learn more ›Cribl.Cloud Solution Brief
The fastest and easiest way to realize the value of an observability ecosystem.
Read Solution Brief ›Cribl Copilot gets your deployments up and running in minutes, not weeks or months.
Learn more ›AppScope gives operators the visibility they need into application behavior, metrics and events with no configuration and no agent required.
Learn more ›Explore Cribl’s Solutions by Use Cases:
Explore Cribl’s Solutions by Integrations:
Explore Cribl’s Solutions by Industry:
September 25 | 10am PT / 1pm ET
Hold my beer: lessons from one team’s data pipeline journey
Register ›Try Your Own Cribl Sandbox
Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Get inspired by how our customers are innovating IT, security and observability. They inspire us daily!
Read Customer Stories ›Sally Beauty Holdings
Sally Beauty Swaps LogStash and Syslog-ng with Cribl.Cloud for a Resilient Security and Observability Pipeline
Read Case Study ›Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Transform data management with Cribl, the Data Engine for IT and Security
Learn More ›Cribl Corporate Overview
Cribl makes open observability a reality, giving you the freedom and flexibility to make choices instead of compromises.
Get the Guide ›Stay up to date on all things Cribl and observability.
Visit the Newsroom ›Cribl’s leadership team has built and launched category-defining products for some of the most innovative companies in the technology sector, and is supported by the world’s most elite investors.
Meet our Leaders ›Join the Cribl herd! The smartest, funniest, most passionate goats you’ll ever meet.
Learn More ›Whether you’re just getting started or scaling up, the Cribl for Startups program gives you the tools and resources your company needs to be successful at every stage.
Learn More ›Want to learn more about Cribl from our sales experts? Send us your contact information and we’ll be in touch.
Talk to an Expert ›In the last few years, many organizations I worked with have significantly increased their cloud footprint. I’ve also seen a large percentage of newly launched companies go with cloud services almost exclusively, limiting their on-premises infrastructure to what cannot be done in the cloud — things like WiFi access points in offices or point of sale (POS) hardware for physical stores.
As a result, security teams at these organizations need better visibility into what’s actually going on in their cloud environments. Cloud providers help with visibility somewhat, but that is just not enough for security teams. After all, that’s what the cloud providers are saying: it’s a shared security responsibility model, and security teams need to do their part.
You cannot just not monitor and analyze your security data sources in the cloud, so you already have some sort of solution in place. But are you currently monitoring and analyzing all of those data sources? For example, if you use AWS, you likely started from looking at CloudTrail logs, AWS Config, maybe S3 access logs. But when you go through the list of AWS log sources, you inevitably get to VPC Flow Logs that can easily be several times larger than all your other AWS logs combined. And you ask yourself: “Can I afford sending that much data to my security tool (Splunk or Elasticsearch, SIEM, UBA, etc.)?” The answer can often be no, especially if your tools are licensed based on data volume or you require extra infrastructure to run the tool on.
If you’re not looking at what’s happening on your network in the cloud, it’s easy to miss anomalous network traffic patterns that can alert you to potentially malicious activities like data exfiltration, lateral movement attempts, communication with a command-and-control (C2) hosts, and so on.
Another challenge I consistently see is the lack of context. Even if you start analyzing VPC Flow Logs, how easy is it to figure out what each IP address means in your environment? In a cloud environment where many things are ephemeral, the issue with context becomes even bigger. You cannot just look up which container or instance had a specific IP address. That container or instance might have existed for a brief moment last Friday. If the VPC Flow Logs are not enriched with host information at the time the logs are collected, figuring out the meaning of those IPs can be very difficult later. But if you start enriching already voluminous flow logs, the volume will go up even more.
So what can you do? In this post, I’ll share a practical approach using Stream to simultaneously add context to logs while reducing their volume. We’ll use AWS VPC Flow Logs as an example here, but the logic can be applied to other cloud flow logs and security data sources.
The steps to configure Cribl Stream and VPC Flow Logs are:
Let’s dive a bit deeper into some of the Pack functionality and the Function Groups, starting from data reduction. You can begin by focusing on passing events containing flows to/from public IPs only; this will drop all East-West traffic events (communication between private IPs as defined in RFC1918). Another way to reduce data volume is to aggregate similar events. VPC Flow records are already aggregated, and you can configure aggregation intervals in AWS. You can further aggregate the events over longer windows and remove some information like port numbers for rejected traffic. Remember, you can still have all of the original events sent to S3 and, using Stream Replay, bring in the original events to your security analytics tool if you need more granularity to, for example, investigate all the accepted and rejected communication attempts from an external IP.
This is a good time to mention that it would be a great idea to have an in-memory database like Redis connected to Stream. The in-memory part is important since we are streaming large volumes of data through Stream. Also, going to disk to query a database will easily introduce latency that is unacceptable for streaming. Plus, when we aggregate events, an in-memory database can be accessed by multiple Worker Nodes, and the aggregation will be done across a Worker Group that will result in a better data reduction. AWS provides a managed Redis service as ElastiCache for Redis. When using the Pack with Redis, remember to specify where your Redis is by adding the Redis URL in the Redis Functions of the Pack.
You can decide to have JSON formatter flow events going to your security analytics tool, but if your tool can work with CSV events you can also use the CSV format to lower the data volume.
Now to the enrichment part. You can add Geo IP location data to your flow events (see this blog for an overview). You can also add asset information to your flow events. For example, if there is an IP address like 10.0.123.123 in your log, what does it tell you? Not much on its own, typically. But what if you can see in the same event that the IP address 10.0.123.123 is a critical EC2 instance with the hostname of customer_accounts_db? That would be more meaningful for alerting, investigations, threat hunting, etc.
There are two ways to enrich the events: via a lookup and using Redis. The lookup assets.csv is included with the Pack. You can upload your asset information with the columns matching the ones in the lookup. The other option is to use Redis. Using Redis gives you more flexibility in how you create, read, update and delete asset data, and you don’t need to re-upload your CSV lookup files to Stream Leader Node which, in turn, needs to distribute the lookups to the Worker Nodes.
VPC Flow Logs are now reduced and enriched. The screenshot below shows some example data reduction and enrichment results: 92% event count reduction and 89% data volume reduction in a small lab. Your data reduction mileage will vary depending on your environment, but it can be significant enough for you to start sending VPC Flow Logs to your security analytics tool without blowing up your license or infrastructure costs. Also, take a look at the before and after tables for the data context that makes your security team more efficient, and eliminates guesswork when it comes to IP addresses’ location a year ago, or mapping IPs to hosts at a later time when those hosts no longer exist.
As you can see in the example below, after significantly trimming down the volume, you still have good visibility into what’s happening on your network in the cloud — who talks to whom, over what protocol, who transferred a significant amount of data, what was let through, and what communication was blocked. You are sacrificing some granularity but, again, you can always get the original VPC Flow Log data using Stream Replay.
The example I’ve walked through here shows that you can reduce data volume and enrich events at the same time. Even better, you may not need to get yet another security tool just to analyze VPC Flow Logs because those are too voluminous to put into the security analytics tool that you already use.
If you don’t run Cribl Stream yet, and would like to learn more about it and experiment with sample data, Stream Sandboxes are free. They’re the best way to quickly see what’s possible and how easy it is to achieve great data manipulation results.
The fastest way to get started with Cribl Stream is to sign-up at Cribl.Cloud. You can process up to 1 TB of throughput per day at no cost. Sign-up and start using Stream within a few minutes.
Experience a full version of Cribl Stream and Cribl Edge in the cloud with pre-made sources and destinations.
Classic choice. Sadly, our website is designed for all modern supported browsers like Edge, Chrome, Firefox, and Safari
Got one of those handy?