Cribl puts your IT and Security data at the center of your data management strategy and provides a one-stop shop for analyzing, collecting, processing, and routing it all at any scale. Try the Cribl suite of products and start building your data engine today!
Learn more ›Evolving demands placed on IT and Security teams are driving a new architecture for how observability data is captured, curated, and queried. This new architecture provides flexibility and control while managing the costs of increasing data volumes.
Read white paper ›Cribl Stream is a vendor-agnostic observability pipeline that gives you the flexibility to collect, reduce, enrich, normalize, and route data from any source to any destination within your existing data infrastructure.
Learn more ›Cribl Edge provides an intelligent, highly scalable edge-based data collection system for logs, metrics, and application data.
Learn more ›Cribl Search turns the traditional search process on its head, allowing users to search data in place without having to collect/store first.
Learn more ›Cribl Lake is a turnkey data lake solution that takes just minutes to get up and running — no data expertise needed. Leverage open formats, unified security with rich access controls, and central access to all IT and security data.
Learn more ›The Cribl.Cloud platform gets you up and running fast without the hassle of running infrastructure.
Learn more ›Cribl.Cloud Solution Brief
The fastest and easiest way to realize the value of an observability ecosystem.
Read Solution Brief ›Cribl Copilot gets your deployments up and running in minutes, not weeks or months.
Learn more ›AppScope gives operators the visibility they need into application behavior, metrics and events with no configuration and no agent required.
Learn more ›Explore Cribl’s Solutions by Use Cases:
Explore Cribl’s Solutions by Integrations:
Explore Cribl’s Solutions by Industry:
September 25 | 10am PT / 1pm ET
Hold my beer: lessons from one team’s data pipeline journey
Register ›Try Your Own Cribl Sandbox
Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Get inspired by how our customers are innovating IT, security and observability. They inspire us daily!
Read Customer Stories ›Sally Beauty Holdings
Sally Beauty Swaps LogStash and Syslog-ng with Cribl.Cloud for a Resilient Security and Observability Pipeline
Read Case Study ›Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Transform data management with Cribl, the Data Engine for IT and Security
Learn More ›Cribl Corporate Overview
Cribl makes open observability a reality, giving you the freedom and flexibility to make choices instead of compromises.
Get the Guide ›Stay up to date on all things Cribl and observability.
Visit the Newsroom ›Cribl’s leadership team has built and launched category-defining products for some of the most innovative companies in the technology sector, and is supported by the world’s most elite investors.
Meet our Leaders ›Join the Cribl herd! The smartest, funniest, most passionate goats you’ll ever meet.
Learn More ›Whether you’re just getting started or scaling up, the Cribl for Startups program gives you the tools and resources your company needs to be successful at every stage.
Learn More ›Want to learn more about Cribl from our sales experts? Send us your contact information and we’ll be in touch.
Talk to an Expert ›Cribl Stream often gets compared to more general purpose streams processing engines or other open source log shippers. “How is this different from LogStash or Fluentd” is a frequent question from our prospects. In the past few months, we’ve been laying out our vision of an Observability Pipeline, and how one can achieve these benefits, even building your own. We’ve seen people build pipelines on top of log shippers like LogStash or Fluentd, but it is usually a long and expensive journey.
In this post we’ll compare the performance of Crib Stream vs LogStash and Fluentd for one of the simplest and common use cases our customers run into – adjust the timestamp of events received from a syslog server.
We’re interested in how efficiently can these systems perform. The chart below shows the CPU normalized throughput, (K events/sec) that each system was able to sustain. In the Parse test case Cribl Stream outperforms LogStash by a factor of 8.75x, in the parse and forward by about 6.5x and and in full test case by about 6.5x. In the Parse test case Cribl Stream outperforms Fluentd by about 26%, in the parse and forward by a factor of 4.3x and and in full test case by about 6.7x. If you’re using Apache NiFi we highly recommend checking out our earlier post showing a 55x outperformance.
New to Cribl LogStream? Take our sandbox for a drive
Three separate test cases were performed
Testing was performed on AWS c5.2xlarge (8CPU, 16GB RAM) instance running on AWS Linux (ami-0e8c04af2729ff1bb).
$ uname -a Linux ip-172-31-14-129.us-west-2.compute.internal 4.14.165-131.185.amzn2.x86_64 #1 SMP Wed Jan 15 14:19:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ ./cribl/bin/cribl version Version: 2.1.0-d58cfc78 Installation type: standalone # LogStash 7.6.0 requires java 1.8 $ sudo yum install -y java-1.8.0 $ wget https://artifacts.elastic.co/downloads/logstash/logstash-oss-7.6.0.tar.gz $ tar -xf logstash-oss-7.6.0.tar.gz $ cd logstash-7.6.0 $ bin/logstash-plugin install logstash-output-syslog # install fluentd # curl -L https://toolbelt.treasuredata.com/sh/install-amazon2-td-agent3.sh | sh # test data $ zcat syslog.log.gz | wc 15875552 269725643 1944436637
Each system was ran with default out of the box configuration settings. Cribl Stream, LogStash and Fluentd were configured to listen on a local port and updated the configurations to perform each of the test cases. When sending data out, each system was configured to send data to another localhost listener that simply drops the data. Two concurrent senders were used to send the test dataset. We measure and report the CPU consumption of all target system processes before and after sending of the data completes. We then use the CPU time to compute the thruput instead of wall time as we’re interested in resource consumption normalized throughput.
normalized_thruput = sent_events / cpu_time
We’ve used the following scripts to measure the CPU consumption. Note, these measurements underestimate the CPU consumption by a small amount as the sender can complete before the destination system has fully processed the last buffered chunks of data.
top -bn1 | grep cribl zcat syslog.log.gz | nc localhost 6666 & zcat syslog.log.gz | nc localhost 6666 & wait top -bn1 | grep cribl
asyncFuncTimeout: 1000 functions: - id: eval filter: "true" conf: add: - name: _time value: _time + 3600
input { syslog { port => 6666 } } filter { ruby { code => 'event.set("@timestamp", LogStash::Timestamp.new(Time.at(event.get("@timestamp").to_i + 3600)))' } } output { syslog { facility => "%{facility_label}" severity => "%{severity_label}" appname => "%{program}" sourcehost => "%{logsource}" procid => "%{pid}" codec => plain { format => "%{message}" } rfc => rfc3164 port => 5555 host => localhost protocol => tcp } }
<system> workers 2 </system> <source> @type syslog port 6666 bind 0.0.0.0 tag system <transport tcp> </transport> <parse> message_format rfc3164 </parse> </source> #update time <filter **> @type record_transformer enable_ruby renew_time_key new_time remove_keys new_time <record> new_time ${Time.at(time).to_f + 3600} </record> </filter> #send data out using papertrail plugin <match **> @type papertrail papertrail_host 127.0.0.1 papertrail_port 5555 </match>
This performance comparison only looks at one simple use case, pointing out that there’s a potential 85% savings in log shipping infrastructure costs. In the coming weeks we’ll post performance comparison results against a few other tools we’ve come across in the market. If we find we are slower in basic use cases, we will happily highlight it. We are by definition experts on our system and not others, if we’ve missed a “GoFaster = true” setting or if you have any other questions or suggestions please reach out!
One more thing, we’re hiring! If the problems above excite you, drop us a line at hello@cribl.io or better yet talk to us live by joining our Cribl Community.
The fastest way to get started with Cribl LogStream is to sign-up at Cribl.Cloud. You can process up to 1 TB of throughput per day at no cost. Sign-up and start using LogStream within a few minutes.
Experience a full version of Cribl Stream and Cribl Edge in the cloud with pre-made sources and destinations.
Classic choice. Sadly, our website is designed for all modern supported browsers like Edge, Chrome, Firefox, and Safari
Got one of those handy?