Cribl puts your IT and Security data at the center of your data management strategy and provides a one-stop shop for analyzing, collecting, processing, and routing it all at any scale. Try the Cribl suite of products and start building your data engine today!
Learn more ›Evolving demands placed on IT and Security teams are driving a new architecture for how observability data is captured, curated, and queried. This new architecture provides flexibility and control while managing the costs of increasing data volumes.
Read white paper ›Cribl Stream is a vendor-agnostic observability pipeline that gives you the flexibility to collect, reduce, enrich, normalize, and route data from any source to any destination within your existing data infrastructure.
Learn more ›Cribl Edge provides an intelligent, highly scalable edge-based data collection system for logs, metrics, and application data.
Learn more ›Cribl Search turns the traditional search process on its head, allowing users to search data in place without having to collect/store first.
Learn more ›Cribl Lake is a turnkey data lake solution that takes just minutes to get up and running — no data expertise needed. Leverage open formats, unified security with rich access controls, and central access to all IT and security data.
Learn more ›The Cribl.Cloud platform gets you up and running fast without the hassle of running infrastructure.
Learn more ›Cribl.Cloud Solution Brief
The fastest and easiest way to realize the value of an observability ecosystem.
Read Solution Brief ›Cribl Copilot gets your deployments up and running in minutes, not weeks or months.
Learn more ›AppScope gives operators the visibility they need into application behavior, metrics and events with no configuration and no agent required.
Learn more ›Explore Cribl’s Solutions by Use Cases:
Explore Cribl’s Solutions by Integrations:
Explore Cribl’s Solutions by Industry:
September 25 | 10am PT / 1pm ET
Hold my beer: lessons from one team’s data pipeline journey
Register ›Try Your Own Cribl Sandbox
Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Get inspired by how our customers are innovating IT, security and observability. They inspire us daily!
Read Customer Stories ›Sally Beauty Holdings
Sally Beauty Swaps LogStash and Syslog-ng with Cribl.Cloud for a Resilient Security and Observability Pipeline
Read Case Study ›Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Transform data management with Cribl, the Data Engine for IT and Security
Learn More ›Cribl Corporate Overview
Cribl makes open observability a reality, giving you the freedom and flexibility to make choices instead of compromises.
Get the Guide ›Stay up to date on all things Cribl and observability.
Visit the Newsroom ›Cribl’s leadership team has built and launched category-defining products for some of the most innovative companies in the technology sector, and is supported by the world’s most elite investors.
Meet our Leaders ›Join the Cribl herd! The smartest, funniest, most passionate goats you’ll ever meet.
Learn More ›Whether you’re just getting started or scaling up, the Cribl for Startups program gives you the tools and resources your company needs to be successful at every stage.
Learn More ›Want to learn more about Cribl from our sales experts? Send us your contact information and we’ll be in touch.
Talk to an Expert ›Ledion Bitincka is Cribl's CTO and co-founder. Ledion has over a decade of engineering ex... Read Moreperience developing next-generation technologies and leading the launch of enterprise products. He was the Advanced Development Architect at Splunk where he introduced Search-Time Schema and led the design of Hunk and SmartStore. Read Less
Yesterday, we officially announced Cribl Edge, a next-generation observability agent. You can find more about its features here. In this post, I am going to walk you through the journey of incepting and building this new product.
Our most important core value at Cribl is “Customers First, Always.” and that involves actively listening and being on the lookout for any pains our customers might be experiencing. We have been consistently hearing from them that collecting logs, metrics, and traces from their infrastructure is a painful task. There are usually several agents, each coming from a different vendor, with different configuration, managing, and monitoring capabilities. But where exactly do the pain points lie?
Identifying the real pain point is a critical step in building successful products. For any product to become successful it needs user adoption. Real pain is a key trigger to action, which leads to adopting new strategies, a new product, or service. We decided to explore this further by breaking the data collection problem down to its first principles and asking lots and lots of questions, like these:
Discover, collect, transform, and route relevant observability data.
Data is collected by different agents configured and managed by different teams. The agents tend to specialize in logs or metrics collection and usually have an ecosystem affinity, e.g. Elastic Beats agents work better in the Elasticsearch ecosystem, the Cloudwatch Logs Agent only works in AWS, etc.
If we had not heard our customers’ pain we’d likely accept the state of the world and stop here. We had to dig deeper. The next few questions highlight how we came to identify the dimensions in which current solutions are broken.
The very first question asked when setting up an agent is: Which data should it collect? Today’s agents rely on the operator to specify exactly what directories to monitor and what file patterns to look for. Well, how do the operators know? In smaller organizations the operators of the observability infrastructure are the application owners so they inherently know, but in larger organizations this involves working with the application teams. As organizations deploy new applications and retire old ones, this process becomes more time consuming and error prone resulting in missing data and observability gaps. There’s goat to be a better way …
The workflow for making configuration changes varies for different agents, but generally, it follows this pattern:
If something goes wrong, we simply repeat the above steps until we get it right. There’s gotta be a better way …
Observability data collection is effectively a very large distributed system problem. Many agents today expose some metrics about their individual operation, however, the task of fleet monitoring and management is left as an exercise to the operator. There’s goat to be a better way …
Answering these questions helped us understand the fundamental pain-points of data collection that our customers were experiencing. We set out to build an observability agent which helps operators identify relevant data that needs collecting and provide them with a centralized way to configure, monitor, and manage their entire fleet. And in true engineering fashion, while building Edge we came across a new set of powerful capabilities, which we didn’t originally intend, so read on …
When addressing the log monitoring configuration problem we started by asking: Is there a way we can auto-discover the log files? The trivial solution would be to periodically scan the entire filesystem – it would work, but be quite heavy on the filesystem! Another solution would be to use AppScope, to scope all applications and have them send the data to Edge. This would solve the filesystem scanning problem but would introduce a config problem as you’d need to use AppScope on all of your applications. We went back to using the first principles questioning: what is a log file? A file where a process is writing operational messages. This answer drove us to explore a novel path: Look at the files that a process has opened for writing, and auto-tail the ones that look like log files. This method of discovering log files would be resilient to application modifications or even unplanned applications running on the endpoint. We were shocked that (to our collective knowledge) none of the existing agents address log discovery this way … such is the power of solving from first principles.
To address the pain of configuration by trial-and-error, we built Cribl Edge with a Web UI that allows users to discover, explore, process, and route data to 40+ destinations. We knew we wanted to let users see what metrics were discovered and give them a preview of what they were. We knew we wanted to let users see what log files were discovered and allow them to peek into them and assess if they were interesting candidates to be collected. We knew we wanted to give users the ability to do that on any Cribl Edge node directly from the central management layer (Leader). What we didn’t know is how powerful the combination of those features would be in providing a highly effective troubleshooting experience, all without moving a single byte from the endpoint!
The feedback we’ve received from our customers trying Cribl Edge falls into two broad buckets – “holy sh*t” and “this product gets me”. If your day is consumed by collecting observability data, I highly recommend taking Cribl Edge for a spin, you’ll love it! And if you are building new products or services, make sure to spend lots of time understanding your customers’ problems by breaking it down to the first principles and solutions from there.
Experience a full version of Cribl Stream and Cribl Edge in the cloud with pre-made sources and destinations.
Classic choice. Sadly, our website is designed for all modern supported browsers like Edge, Chrome, Firefox, and Safari
Got one of those handy?