Cribl puts your IT and Security data at the center of your data management strategy and provides a one-stop shop for analyzing, collecting, processing, and routing it all at any scale. Try the Cribl suite of products and start building your data engine today!
Learn more ›Evolving demands placed on IT and Security teams are driving a new architecture for how observability data is captured, curated, and queried. This new architecture provides flexibility and control while managing the costs of increasing data volumes.
Read white paper ›Cribl Stream is a vendor-agnostic observability pipeline that gives you the flexibility to collect, reduce, enrich, normalize, and route data from any source to any destination within your existing data infrastructure.
Learn more ›Cribl Edge provides an intelligent, highly scalable edge-based data collection system for logs, metrics, and application data.
Learn more ›Cribl Search turns the traditional search process on its head, allowing users to search data in place without having to collect/store first.
Learn more ›Cribl Lake is a turnkey data lake solution that takes just minutes to get up and running — no data expertise needed. Leverage open formats, unified security with rich access controls, and central access to all IT and security data.
Learn more ›The Cribl.Cloud platform gets you up and running fast without the hassle of running infrastructure.
Learn more ›Cribl.Cloud Solution Brief
The fastest and easiest way to realize the value of an observability ecosystem.
Read Solution Brief ›Cribl Copilot gets your deployments up and running in minutes, not weeks or months.
Learn more ›AppScope gives operators the visibility they need into application behavior, metrics and events with no configuration and no agent required.
Learn more ›Explore Cribl’s Solutions by Use Cases:
Explore Cribl’s Solutions by Integrations:
Explore Cribl’s Solutions by Industry:
September 25 | 10am PT / 1pm ET
Hold my beer: lessons from one team’s data pipeline journey
Register ›Try Your Own Cribl Sandbox
Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Get inspired by how our customers are innovating IT, security and observability. They inspire us daily!
Read Customer Stories ›Sally Beauty Holdings
Sally Beauty Swaps LogStash and Syslog-ng with Cribl.Cloud for a Resilient Security and Observability Pipeline
Read Case Study ›Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Transform data management with Cribl, the Data Engine for IT and Security
Learn More ›Cribl Corporate Overview
Cribl makes open observability a reality, giving you the freedom and flexibility to make choices instead of compromises.
Get the Guide ›Stay up to date on all things Cribl and observability.
Visit the Newsroom ›Cribl’s leadership team has built and launched category-defining products for some of the most innovative companies in the technology sector, and is supported by the world’s most elite investors.
Meet our Leaders ›Join the Cribl herd! The smartest, funniest, most passionate goats you’ll ever meet.
Learn More ›Whether you’re just getting started or scaling up, the Cribl for Startups program gives you the tools and resources your company needs to be successful at every stage.
Learn More ›Want to learn more about Cribl from our sales experts? Send us your contact information and we’ll be in touch.
Talk to an Expert ›One of the things I love about Cribl Stream is that it is an island of simplicity in the sea of log complexity. It’s easy to run a single instance of Cribl Stream and manage a fair amount of data without too much effort. But most reasonably sized implementations will eventually need to go distributed, either for scalability or due to geographic constraints (or a variety of other reasons). This can introduce a lot of complexity, but Cribl Stream uses worker groups to keep that in check. Let’s talk a little bit about the architecture of a distributed Cribl Stream environment.
Here, one instance is deemed the “master”, and becomes the control plane for the environment. It is responsible for pushing configuration to all of the workers, consolidating their logs and metadata, and acting as the overall UI for the environment.
The workers are the data plane. While they get their configuration from the master, and pass metadata back to it, each worker is wholly responsible for data that comes to it – there is no handoff/rerouting among them.
If you have a single-instance environment that you only need to scale, you can easily have a single configuration for all workers, and just put a single load balancer in front of them all. In fact, this is what happens with our Free, One, or Standard license – a single worker group is created. But if you have any other constraints (for example, handling data from different regions with different compliance requirements, or across clouds, etc.), you are going to need different configurations for different groups of workers. In Cribl Stream, that is done with “Worker Groups” – a feature available with a Cribl Stream Enterprise license.
A worker group, simply, is just a set of worker nodes that share the same configuration. This means that data sent to any of the nodes will be processed the same way. How you carve up your workloads into worker groups is really specific to your needs/workloads/constraints. Here are a few examples:
Let’s say an organization has on-premises workloads that use one tech stack, plus cloud workloads that use a completely different tech stack. In this case, a likely worker group configuration would be a cloud worker group configured to support the tech stack in the cloud, and a data-center worker group configured to support the tech stack in the data center. Additionally, putting the cloud worker group directly in that cloud can help with networking costs, since Cribl Stream can reduce the data leaving the cloud network.
In this scenario, an organization has multiple data centers in different geographic locations, and needs to keep the data from all of them “local” (whether for bandwidth management or compliance reasons). Each location/data center can have its own worker group with unique configurations for that location.
This organization has wildly variable workloads, and wants to “carve out” worker capacity to ensure that one workload doesn’t consume all of the resources. A dedicated worker group for primary sources, and a separate one for the “chatty” source systems, will protect each from the other.
What if an organization has a workload that must have the lowest delivery time possible, but also has workloads that have no delivery SLA? By separating these into multiple worker groups, you can segregate the workloads. And since you can apply things like autoscaling per worker group, you can flex-scale the “lowest-latency” workload while allowing the lower-priority workloads to divvy up available resources.
Again, these are just examples – how you use worker groups is completely up to you and your requirements. In our corporate Cribl Stream environment, we split the worker groups up into “streaming” sources versus “collection” sources (sources that are collected using our data collection feature). Why? Primarily because we want to be able to scale each group according to its workload, without impacting the other group’s workload.
We run scheduled collectors in the collection group, and applying autoscaling there can be really advantageous. At the same time, the profile of the streaming data is somewhat constant, so it’s generally a more consistent workload. For us, that division of work makes sense.
Worker Groups are a simple concept – just a group of servers that are all configured the same way. The abstraction simplifies the configuration of a complex distributed system. Depending on where your workloads are, how many data centers you have, and your specific use case(s), how you separate your workloads into worker groups can vary. One of the benefits of Cribl Stream is the flexibility to adapt to different use cases and requirements. No matter which worker group configuration you choose, Cribl Stream can support you and your business goals.
Ready to get started today? Cribl Stream gives you a dedicated Cribl Stream environment that you can start using immediately. We take care of the data infrastructure management and scaling, so you don’t need to, making this the easiest and fastest way to realize the value of an Observability Pipeline.
Experience a full version of Cribl Stream and Cribl Edge in the cloud with pre-made sources and destinations.
Classic choice. Sadly, our website is designed for all modern supported browsers like Edge, Chrome, Firefox, and Safari
Got one of those handy?