Cribl puts your IT and Security data at the center of your data management strategy and provides a one-stop shop for analyzing, collecting, processing, and routing it all at any scale. Try the Cribl suite of products and start building your data engine today!
Learn more ›Evolving demands placed on IT and Security teams are driving a new architecture for how observability data is captured, curated, and queried. This new architecture provides flexibility and control while managing the costs of increasing data volumes.
Read white paper ›Cribl Stream is a vendor-agnostic observability pipeline that gives you the flexibility to collect, reduce, enrich, normalize, and route data from any source to any destination within your existing data infrastructure.
Learn more ›Cribl Edge provides an intelligent, highly scalable edge-based data collection system for logs, metrics, and application data.
Learn more ›Cribl Search turns the traditional search process on its head, allowing users to search data in place without having to collect/store first.
Learn more ›Cribl Lake is a turnkey data lake solution that takes just minutes to get up and running — no data expertise needed. Leverage open formats, unified security with rich access controls, and central access to all IT and security data.
Learn more ›The Cribl.Cloud platform gets you up and running fast without the hassle of running infrastructure.
Learn more ›Cribl.Cloud Solution Brief
The fastest and easiest way to realize the value of an observability ecosystem.
Read Solution Brief ›Cribl Copilot gets your deployments up and running in minutes, not weeks or months.
Learn more ›AppScope gives operators the visibility they need into application behavior, metrics and events with no configuration and no agent required.
Learn more ›Explore Cribl’s Solutions by Use Cases:
Explore Cribl’s Solutions by Integrations:
Explore Cribl’s Solutions by Industry:
September 25 | 10am PT / 1pm ET
Hold my beer: lessons from one team’s data pipeline journey
Register ›Try Your Own Cribl Sandbox
Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Get inspired by how our customers are innovating IT, security and observability. They inspire us daily!
Read Customer Stories ›Sally Beauty Holdings
Sally Beauty Swaps LogStash and Syslog-ng with Cribl.Cloud for a Resilient Security and Observability Pipeline
Read Case Study ›Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Transform data management with Cribl, the Data Engine for IT and Security
Learn More ›Cribl Corporate Overview
Cribl makes open observability a reality, giving you the freedom and flexibility to make choices instead of compromises.
Get the Guide ›Stay up to date on all things Cribl and observability.
Visit the Newsroom ›Cribl’s leadership team has built and launched category-defining products for some of the most innovative companies in the technology sector, and is supported by the world’s most elite investors.
Meet our Leaders ›Join the Cribl herd! The smartest, funniest, most passionate goats you’ll ever meet.
Learn More ›Whether you’re just getting started or scaling up, the Cribl for Startups program gives you the tools and resources your company needs to be successful at every stage.
Learn More ›Want to learn more about Cribl from our sales experts? Send us your contact information and we’ll be in touch.
Talk to an Expert ›Imagine sending logs to cost-effective storage, converting them into efficient metrics, and forwarding only essential data for analysis. This change can slash ingest and long-term storage expenses by an order of magnitude! Enter Cribl Search—an ingenious solution that skillfully navigates storage, transforms logs into actionable metrics, and seamlessly channels vital data to your analysis systems. The result? Over 99.94% reduction in volume, enhanced efficiency, and substantial cost savings. In the example below, I am sending across 132.54MB per hour of logs to an S3 bucket. Using Cribl Search I run a scheduled search every hour to generate 70.96KB in metrics to a Prometheus endpoint, in this case, to Grafana. Feel free to choose your desired metrics, data lake, and achieve the same results.
That’s not just a space-saving trick; it’s a jaw-dropping reduction of nearly 99.94%! In practical terms, it means that the logs, which once occupied a substantial chunk of storage, now occupy only a fraction of a fraction. This kind of transformation is the power of converting logs into efficient metrics, making data management not just efficient but downright magical.
Sending approximately 132.54 megabytes of Apache web logs per hour to an S3 bucket, consistently over a month, equates to roughly 94.29GB of uncompressed data or a lean 11.79GB when compressed with gzip, which Cribl Stream utilizes as its default compression method.
192.168.0.1 - - [07/Jul/2023:12:00:01 +0000] "GET /search?query=laptop&session_id=7f1c5d8b-2a32-4b85-9df7-094a8e1ef19c&user_id=jsmith@gmail.com HTTP/1.1" 200 2048 "https://techboss.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" 156
192.168.0.1 - - [07/Jul/2023:12:01:02 +0000] "GET /product?product_id=XYZ789&product_name=laptop&session_id=7f1c5d8b-2a32-4b85-9df7-094a8e1ef19c&user_id=jsmith@gmail.com HTTP/1.1" 200 4096 "https://techboss.com/search?query=laptop" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" 241
192.168.0.1 - - [07/Jul/2023:12:02:03 +0000] "GET /reviews?product_id=XYZ789&product_name=laptop&session_id=7f1c5d8b-2a32-4b85-9df7-094a8e1ef19c&user_id=jsmith@gmail.com HTTP/1.1" 200 3072 "https://techboss.com/product?product_id=XYZ789" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" 189
192.168.0.1 - - [07/Jul/2023:12:03:04 +0000] "POST /cart/add?product_id=XYZ789&product_name=laptop&price=1979.97&quantity=1&session_id=7f1c5d8b-2a32-4b85-9df7-094a8e1ef19c&user_id=jsmith@gmail.com HTTP/1.1" 302 512 "https://techboss.com/product?product_id=XYZ789" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" 178
192.168.0.1 - - [07/Jul/2023:12:04:05 +0000] "GET /cart?session_id=7f1c5d8b-2a32-4b85-9df7-094a8e1ef19c&user_id=jsmith@gmail.com&product_id=XYZ789&product_name=laptop&price=1979.97 HTTP/1.1" 200 3072 "https://techboss.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" 214
192.168.0.1 - - [07/Jul/2023:12:05:06 +0000] "POST /checkout/start?session_id=7f1c5d8b-2a32-4b85-9df7-094a8e1ef19c&user_id=jsmith@gmail.com&product_id=XYZ789&product_name=laptop&price=1979.97 HTTP/1.1" 200 1024 "https://techboss.com/reviews?product_id=XYZ789" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" 210
192.168.0.1 - - [07/Jul/2023:12:06:07 +0000] "POST /checkout/address?session_id=7f1c5d8b-2a32-4b85-9df7-094a8e1ef19c&product_id=XYZ789&product_name=laptop&price=1979.97&user_id=jsmith@gmail.com&address=123 Main Street&city=New York&state=NY&postal_code=10001&country=USA&phone=(636) 123-4567 HTTP/1.1" 200 512 "https://techboss.com/checkout/start" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" 189
192.168.0.1 - - [07/Jul/2023:12:07:09 +0000] "POST /checkout/payment?session_id=7f1c5d8b-2a32-4b85-9df7-094a8e1ef19c&product_id=XYZ789&product_name=laptop&price=1979.97&user_id=jsmith@gmail.com HTTP/1.1" 200 512 "https://techboss.com/checkout/address" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" 221
192.168.0.1 - - [07/Jul/2023:12:08:09 +0000] "POST /checkout/confirm?session_id=7f1c5d8b-2a32-4b85-9df7-094a8e1ef19c&product_id=XYZ789&product_name=laptop&price=1979.97&user_id=jsmith@gmail.com HTTP/1.1" 302 256 "https://techboss.com/checkout/payment" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" 195
192.168.0.1 - - [07/Jul/2023:12:09:10 +0000] "GET /checkout/success?session_id=7f1c5d8b-2a32-4b85-9df7-094a8e1ef19c&product_id=XYZ789&product_name=laptop&price=1979.97&user_id=jsmith@gmail.com HTTP/1.1" 200 1024 "https://techboss.com/checkout/confirm" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" 234
192.168.0.1 - - [07/Jul/2023:12:10:11 +0000] "POST /email/send?session_id=7f1c5d8b-2a32-4b85-9df7-094a8e1ef19c&product_id=XYZ789&product_name=laptop&price=1979.97&user_id=jsmith@gmail.com&email=jsmith@gmail.com&subject=Purchase+Confirmation HTTP/1.1" 200 128 "https://techboss.com/checkout/success" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
While storing this compressed data in an S3 bucket for just a month, the cost is a mere $0.27 – an impressively frugal choice. Over the course of a year, this translates to an annual expenditure of approximately $3.24, highlighting the remarkable cost efficiency. S3, in addition to delivering these substantial savings, ensures top-notch features such as high availability, data durability, and Cribl Replay and Cribl Search natively support it. In contrast, many premium search vendors command prices well beyond $600.00 per year for similar data volumes, making them over 185 times costlier. Now, picture the potential financial impact if we were dealing with terabytes or even petabytes of data!
Sending logs to cost-effective object storage is a strategic move that can yield substantial benefits for your data management strategy. By utilizing object storage, you not only save on storage costs but also gain the flexibility to schedule Cribl searches and generate a wide array of metrics across any desired timeframe. This newfound agility in data processing allows you to optimize your data pipeline for cost reduction, ensuring compliance with audit requirements, and enhancing operational intelligence. In essence, it’s a win-win situation that empowers your organization to make data-driven decisions efficiently and economically. Let’s take a look at an example with Cribl Search that converts logs to metrics.
In the following Cribl Search query, we analyze the dataset of Apache server logs within a one-hour time window, summarize the most requested endpoints by site, and then present the top ten endpoints based on their frequency of access.
dataset="Apache_Logs_to_Metrics_Dataset" earliest=-65m@m latest=-5m@m | summarize endpoints=count() by request_uri_path,site | top 10 by endpoints
If you want these metrics sent to your preferred datalake just add a | send. Notice the summarized report of how many metric events were sent back through Cribl Stream and onto the destination of choice.
dataset="Apache_Logs_to_Metrics_Dataset" earliest=-65m@m latest=-5m@m | summarize endpoints=count() by request_uri_path,site | extend _destination="endpoints", _time=now() | send
If you append tee=true at the end of the send command you will see the exact metrics that will be sent such as:
dataset="Apache_Logs_to_Metrics_Dataset" earliest=-65m@m latest=-5m@m | summarize endpoints=count() by request_uri_path,site | extend _destination="endpoints", _time=now() | send tee=true
Take the bloated logs in, use Cribl Routes to send to S3, and Cribl Search to send the metrics you need to your preferred destination. In this example, I am sending to a Prometheus endpoint in Grafana Cloud.
A quick search in their interface to display the top 10 endpoints by site would look similar to this query:
topk(10, sum(endpoints) by (request_uri_path, site))
In summary, this query identifies the top 10 combinations of request_uri_path and site based on the sum of the endpoints metric. It helps you find which paths and sites have the highest traffic based on the endpoints count.
The transition from logs to metrics is a game-changer in data management. This remarkable transformation can reduce data volume by nearly 99.94%, resulting in substantial cost savings and operational efficiency. Storing logs in object storage comes with the added benefits of high availability, data durability, and easy access, thanks to features like Replay in Cribl Stream and Cribl Search. With Cribl Search, running tailored searches and generating metrics is a breeze, making data optimization more accessible than ever before. Embrace this transformative shift from logs to metrics with Cribl Search, and unlock new possibilities in data management and cost efficiency.
Ready to get started? Head over to Cribl.Cloud to sign up for a free account to gain instant access to all of our products!
Cribl, the Data Engine for IT and Security, empowers organizations to transform their data strategy. Customers use Cribl’s suite of products to collect, process, route, and analyze all IT and security data, delivering the flexibility, choice, and control required to adapt to their ever-changing needs.
We offer free training, certifications, and a free tier across our products. Our community Slack features Cribl engineers, partners, and customers who can answer your questions as you get started and continue to build and evolve. We also offer a variety of hands-on Sandboxes for those interested in how companies globally leverage our products for their data challenges.
Experience a full version of Cribl Stream and Cribl Edge in the cloud with pre-made sources and destinations.
Classic choice. Sadly, our website is designed for all modern supported browsers like Edge, Chrome, Firefox, and Safari
Got one of those handy?