Cribl puts your IT and Security data at the center of your data management strategy and provides a one-stop shop for analyzing, collecting, processing, and routing it all at any scale. Try the Cribl suite of products and start building your data engine today!
Learn more ›Evolving demands placed on IT and Security teams are driving a new architecture for how observability data is captured, curated, and queried. This new architecture provides flexibility and control while managing the costs of increasing data volumes.
Read white paper ›Cribl Stream is a vendor-agnostic observability pipeline that gives you the flexibility to collect, reduce, enrich, normalize, and route data from any source to any destination within your existing data infrastructure.
Learn more ›Cribl Edge provides an intelligent, highly scalable edge-based data collection system for logs, metrics, and application data.
Learn more ›Cribl Search turns the traditional search process on its head, allowing users to search data in place without having to collect/store first.
Learn more ›Cribl Lake is a turnkey data lake solution that takes just minutes to get up and running — no data expertise needed. Leverage open formats, unified security with rich access controls, and central access to all IT and security data.
Learn more ›The Cribl.Cloud platform gets you up and running fast without the hassle of running infrastructure.
Learn more ›Cribl.Cloud Solution Brief
The fastest and easiest way to realize the value of an observability ecosystem.
Read Solution Brief ›Cribl Copilot gets your deployments up and running in minutes, not weeks or months.
Learn more ›AppScope gives operators the visibility they need into application behavior, metrics and events with no configuration and no agent required.
Learn more ›Explore Cribl’s Solutions by Use Cases:
Explore Cribl’s Solutions by Integrations:
Explore Cribl’s Solutions by Industry:
September 25 | 10am PT / 1pm ET
Hold my beer: lessons from one team’s data pipeline journey
Register ›Try Your Own Cribl Sandbox
Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Get inspired by how our customers are innovating IT, security and observability. They inspire us daily!
Read Customer Stories ›Sally Beauty Holdings
Sally Beauty Swaps LogStash and Syslog-ng with Cribl.Cloud for a Resilient Security and Observability Pipeline
Read Case Study ›Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Transform data management with Cribl, the Data Engine for IT and Security
Learn More ›Cribl Corporate Overview
Cribl makes open observability a reality, giving you the freedom and flexibility to make choices instead of compromises.
Get the Guide ›Stay up to date on all things Cribl and observability.
Visit the Newsroom ›Cribl’s leadership team has built and launched category-defining products for some of the most innovative companies in the technology sector, and is supported by the world’s most elite investors.
Meet our Leaders ›Join the Cribl herd! The smartest, funniest, most passionate goats you’ll ever meet.
Learn More ›Whether you’re just getting started or scaling up, the Cribl for Startups program gives you the tools and resources your company needs to be successful at every stage.
Learn More ›Want to learn more about Cribl from our sales experts? Send us your contact information and we’ll be in touch.
Talk to an Expert ›In the era of big data, data lakes have emerged as a popular way to store and process massive amounts of data. Fortunately, with Cribl Search and Cribl Stream, you can create a Data Loop to optimize the use of your data lake by saving Search results as part of an investigation. Our four-part video series explains how to set up Cribl Search and Cribl Stream to establish a Data Loop using the Amazon S3 Data Lake destination in Cribl Stream and the Cribl Stream in_cribl_http
source. This Data Loop serves as a temporary storage location for Cribl Search results obtained from an initial investigation. For example, if an IOC triggers an alert in your SIEM (which stores only 30 days of data), you can expand your investigation on the IP addresses in that alert by querying 12 months of data stored in cost-effective Amazon S3 storage via the Data Loop. If you’d like to follow along, create a Cribl.Cloud account for free to get started!
In the very first video of the series, we delve into the concept of a data loop and why it is beneficial to use Cribl Search and Cribl Stream to optimize the use of a data lake. The video gives a concise overview of Cribl Search and Cribl Stream, and how they work in tandem to create a data loop. We then provide step-by-step instructions on how to configure the Cribl Stream “Amazon S3 Data Lake” Destination to transfer data from Stream to an S3 bucket that has been optimized specifically for Cribl Search’s access. Finally, we demonstrate sending sample data to the S3 bucket and present a before-and-after view of the bucket to showcase the impact of the test data.
In the second video of our series, we delve into the nuts and bolts of configuring Cribl Search to access the data that we’ve stored in the S3 bucket. The video guides you step-by-step through the process of configuring the Search S3 dataset provider by using the Stream Data Lake destination as a model for the authentication information. From there, we proceed to walk through the process of creating a Dataset to access the Provider that we’ve just established. To wrap things up, we demonstrate how to search through the test data that we’ve previously stored in the S3 bucket.
The third video of our series focuses on utilizing Cribl Stream to manage data. The presenter takes us through the process of configuring the Cribl Stream in_cribl_http
source in tandem with the Cribl Search send
operator to collect data. We are able to witness live data results being sent from Search to Stream.
Afterward, we demonstrate creating a Route in Stream to direct the incoming data from Search (via the in_cribl_http) Source to the Data Lake by using the Amazon S3 Data Lake Destination. This step employs a passthru
pipeline to ensure that the data is not altered in transit.
Finally, we add a layer of enrichment by appending a tag to the outbound data and modifying the Data Lake Destination to adjust the S3 bucket hierarchy to optimize Search access later.
The final section of our video series showcases how to put the data loop to use with a real-world dataset. We utilize the public domain “Boss of the SOC v3” [BOTSv3] dataset, which is readily available on GitHub. First, we employ Cribl Search to sift through and explore the BOTSv3 data that is stored in an S3 bucket to locate some specific data.
Following that, we create a fictitious scenario in which we identify a particular set of data as suspicious (in this instance, we picked mysql
data from a set of wire data analysis). We pinpointed the two IP addresses that were involved in the conversation and then expanded our investigation to all events that contained at least one of those two IP addresses. We saved this dataset to the Data Loop, thereby making it accessible in Search.
Finally, we illustrate how to utilize the Data Lake source in Search to search through the subset of data identified in the fictitious scenario in an optimized manner. We also demonstrate the advantage of utilizing the Data Lake source over the original dataset in terms of speed, which amounts to a 5x increase!
To sum it up, Cribl Search and Cribl Stream can be employed to establish a data loop that optimizes the use of an Amazon S3 bucket for intermediate storage. By gathering data from Search and looping it back to an S3 data lake destination, users can effortlessly store and process large quantities of data. Users can also filter and transform the data before it is transmitted to the data lake, allowing for complete control over the data that is stored.
Cribl Search and Cribl Stream work in unison to provide a powerful tool that enhances an existing SIEM or analytics tool by expanding and deepening forensic or audit investigations. This tool is highly valuable for businesses and organizations that rely heavily on data to make informed decisions. If you’d like to try it on your own, you can create a free Cribl.Cloud account today!
Experience a full version of Cribl Stream and Cribl Edge in the cloud with pre-made sources and destinations.
Classic choice. Sadly, our website is designed for all modern supported browsers like Edge, Chrome, Firefox, and Safari
Got one of those handy?