Cribl puts your IT and Security data at the center of your data management strategy and provides a one-stop shop for analyzing, collecting, processing, and routing it all at any scale. Try the Cribl suite of products and start building your data engine today!
Learn more ›Evolving demands placed on IT and Security teams are driving a new architecture for how observability data is captured, curated, and queried. This new architecture provides flexibility and control while managing the costs of increasing data volumes.
Read white paper ›Cribl Stream is a vendor-agnostic observability pipeline that gives you the flexibility to collect, reduce, enrich, normalize, and route data from any source to any destination within your existing data infrastructure.
Learn more ›Cribl Edge provides an intelligent, highly scalable edge-based data collection system for logs, metrics, and application data.
Learn more ›Cribl Search turns the traditional search process on its head, allowing users to search data in place without having to collect/store first.
Learn more ›Cribl Lake is a turnkey data lake solution that takes just minutes to get up and running — no data expertise needed. Leverage open formats, unified security with rich access controls, and central access to all IT and security data.
Learn more ›The Cribl.Cloud platform gets you up and running fast without the hassle of running infrastructure.
Learn more ›Cribl.Cloud Solution Brief
The fastest and easiest way to realize the value of an observability ecosystem.
Read Solution Brief ›Cribl Copilot gets your deployments up and running in minutes, not weeks or months.
Learn more ›AppScope gives operators the visibility they need into application behavior, metrics and events with no configuration and no agent required.
Learn more ›Explore Cribl’s Solutions by Use Cases:
Explore Cribl’s Solutions by Integrations:
Explore Cribl’s Solutions by Industry:
September 25 | 10am PT / 1pm ET
Hold my beer: lessons from one team’s data pipeline journey
Register ›Try Your Own Cribl Sandbox
Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Get inspired by how our customers are innovating IT, security and observability. They inspire us daily!
Read Customer Stories ›Sally Beauty Holdings
Sally Beauty Swaps LogStash and Syslog-ng with Cribl.Cloud for a Resilient Security and Observability Pipeline
Read Case Study ›Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Transform data management with Cribl, the Data Engine for IT and Security
Learn More ›Cribl Corporate Overview
Cribl makes open observability a reality, giving you the freedom and flexibility to make choices instead of compromises.
Get the Guide ›Stay up to date on all things Cribl and observability.
Visit the Newsroom ›Cribl’s leadership team has built and launched category-defining products for some of the most innovative companies in the technology sector, and is supported by the world’s most elite investors.
Meet our Leaders ›Join the Cribl herd! The smartest, funniest, most passionate goats you’ll ever meet.
Learn More ›Whether you’re just getting started or scaling up, the Cribl for Startups program gives you the tools and resources your company needs to be successful at every stage.
Learn More ›Want to learn more about Cribl from our sales experts? Send us your contact information and we’ll be in touch.
Talk to an Expert ›February 23, 2020
Working with data in Cribl LogStream is best done when live events flow through the system. In this post we’ll walk through how to generate sample data for the purposes of iterating through routes, pipelines, functions configurations as well as troubleshooting general issues.
In version 2.1 we introduced a new purpose-built feature called Datagens. Datagens replay template files a given rate (Events Per Second) and appear and behave exactly as regulars Sources i.e. their events can be routed, processed and outputted to a destination just like all other events. Several datagen template files ship with the product out of the box and others can be created from samples files or live captures.
To create a datagen, navigate to Sources > Datagens and click Add New. Select a Data Generator File from the dropdown – e.g., apache_common.log
and set it at 5 EPS/worker process. Hit Save. Repeat for another, e.g., syslog.log
and set it at 10EPS/worker process.
Learn about the features of Cribl LogStream in our interactive sandbox!
Next, go to Monitoring > Sources, select/search datagen
and notice that both datagens are now delivering data at approximately the configured EPS rate (in my case I have two worker processes).
Yup, that’s it – from 0 to a 30EPS sample in less than 5min!
Of course! The Datagen engine can easily generate and replay templates from custom events. Go to Preview > Paste a Sample and add a sample, say, something like these AWS VPC Flow logs:
2 123456789010 eni-abc123de 172.31.16.139 172.31.16.21 20641 22 6 20 4249 1418530010 1418530070 ACCEPT OK 2 123456789010 eni-abc123de 172.31.9.69 172.31.9.12 49761 3389 6 20 4249 1418530010 1418530070 REJECT OK 2 123456789010 eni-1a2b3c4d - - - - - - - 1431280876 1431280934 - NODATA 2 123456789010 eni-4b118871 - - - - - - - 1431280876 1431280934 - SKIPDATA 2 123456789010 eni-1235b8ca 203.0.113.12 172.31.16.139 0 0 1 4 336 1432917027 1432917142 ACCEPT OK 2 123456789010 eni-1235b8ca 172.31.16.139 203.0.113.12 0 0 1 4 336 1432917094 1432917142 REJECT OK 2 123456789010 eni-f41c42bf 2001:db8:1234:a100:8d6e:3477:df66:f105 2001:db8:1234:a102:3304:8879:34cf:4071 34892 22 6 54 8855 1477913708 1477913820 ACCEPT OK
From the Event Breaker dropdown select AWS VPC Flow to ensure that the pasted text gets broken properly (notice the Event Breaker on newlines) and each event gets timestamped correctly (highlighted purple). Next, click on Create A Datagen File.
Note: if your sample is different and you can’t find a proper Event Breaker you can create one by expanding Custom Breaker Settings section.
On the next screen, titled Create Datagen File, enter a file name, e.g., vpc-flow-datagen.log
and ensure that the timestamp template format is correct: ${timestamp: %s}
. ${timestamp: <format>}
is a literal template string that the datagen engine uses to insert current time – in each newly generated event – in the given format. In this case %s
is the desired strftime format for the timestamp (i.e. epoch). Click Save as Datagen File
To start using the newly created datagen file, go back to Sources > Datagens and define one as we did above.
—
If you’re trying to deploy LogStream on your environment or have any questions please join our community Slack– we’d love to help you out. If you’re looking to join a fast-paced, innovative team drop us a line at hello@cribl.io– we’re hiring!
Rick Salsa Sep 19, 2024
Josh Biggley Sep 17, 2024
Classic choice. Sadly, our website is designed for all modern supported browsers like Edge, Chrome, Firefox, and Safari
Got one of those handy?