Cribl puts your IT and Security data at the center of your data management strategy and provides a one-stop shop for analyzing, collecting, processing, and routing it all at any scale. Try the Cribl suite of products and start building your data engine today!
Learn more ›Evolving demands placed on IT and Security teams are driving a new architecture for how observability data is captured, curated, and queried. This new architecture provides flexibility and control while managing the costs of increasing data volumes.
Read white paper ›Cribl Stream is a vendor-agnostic observability pipeline that gives you the flexibility to collect, reduce, enrich, normalize, and route data from any source to any destination within your existing data infrastructure.
Learn more ›Cribl Edge provides an intelligent, highly scalable edge-based data collection system for logs, metrics, and application data.
Learn more ›Cribl Search turns the traditional search process on its head, allowing users to search data in place without having to collect/store first.
Learn more ›Cribl Lake is a turnkey data lake solution that takes just minutes to get up and running — no data expertise needed. Leverage open formats, unified security with rich access controls, and central access to all IT and security data.
Learn more ›The Cribl.Cloud platform gets you up and running fast without the hassle of running infrastructure.
Learn more ›Cribl.Cloud Solution Brief
The fastest and easiest way to realize the value of an observability ecosystem.
Read Solution Brief ›Cribl Copilot gets your deployments up and running in minutes, not weeks or months.
Learn more ›AppScope gives operators the visibility they need into application behavior, metrics and events with no configuration and no agent required.
Learn more ›Explore Cribl’s Solutions by Use Cases:
Explore Cribl’s Solutions by Integrations:
Explore Cribl’s Solutions by Industry:
September 25 | 10am PT / 1pm ET
Hold my beer: lessons from one team’s data pipeline journey
Register ›Try Your Own Cribl Sandbox
Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Get inspired by how our customers are innovating IT, security and observability. They inspire us daily!
Read Customer Stories ›Sally Beauty Holdings
Sally Beauty Swaps LogStash and Syslog-ng with Cribl.Cloud for a Resilient Security and Observability Pipeline
Read Case Study ›Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Transform data management with Cribl, the Data Engine for IT and Security
Learn More ›Cribl Corporate Overview
Cribl makes open observability a reality, giving you the freedom and flexibility to make choices instead of compromises.
Get the Guide ›Stay up to date on all things Cribl and observability.
Visit the Newsroom ›Cribl’s leadership team has built and launched category-defining products for some of the most innovative companies in the technology sector, and is supported by the world’s most elite investors.
Meet our Leaders ›Join the Cribl herd! The smartest, funniest, most passionate goats you’ll ever meet.
Learn More ›Whether you’re just getting started or scaling up, the Cribl for Startups program gives you the tools and resources your company needs to be successful at every stage.
Learn More ›Want to learn more about Cribl from our sales experts? Send us your contact information and we’ll be in touch.
Talk to an Expert ›In this post, we’ll walk through our journey of launching Cribl Stream Cloud on AWS Graviton instances. In order to put our journey into perspective, it is worth spending a few moments to describe the product and its resource requirements.
Cribl Stream is our first product to be launched as a cloud service. Stream is a streams-processing engine specifically designed for observability data (logs, metrics, and traces). Stream allows you to implement an observability pipeline, helping you parse, restructure, and enrich telemetry data in flight – ensuring that you get the right data where you want, in the formats you need. It natively supports receiving data from, and sending it to, over 50 sources and destinations, including Amazon S3, Amazon Kinesis Data Firehose, Syslog, Elastic Cloud, Splunk, New Relic, etc.
From day one, we had two key architecture design requirements for Stream: a) resiliency and b) resource efficiency. Customers depend on data successfully passing through Stream to gain operational visibility into their systems and applications, making resiliency an obvious requirement. According to the IDC, observability data is one of the fastest-growing (25+% CAGR) data sources in the enterprise, and the systems used to collect, process, and store that type of data can have a tremendous infrastructure footprint. A resource-efficient solution allows customers to minimize their infrastructure costs.
As a streams-processing engine, Stream is primarily CPU- and secondarily network-bound. The vast majority of the CPU cycles are spent deserializing received data, processing it, and ultimately serializing it out to one or more destinations.
When designing our Stream Cloud offering, our resource-efficiency key requirement needed to be updated. In a cloud environment, resource efficiency is not sufficient. A solution also needs to be cost-efficient, including: choosing the right architecture, proper sizing, and designing for elasticity.
With cost in mind, we started researching and profiling Stream on different AWS instance types, including Intel (c5, m5zn), AMD (c5a) and AWS Graviton2 (c6g) instances. Our findings indicated that under high load, on-demand AWS Graviton2 (c6g) instances would provide 45%, 24%, and 217% better price performance when compared to c5, c5a, and m5zn instances, respectively. (Note: m5zn instances were included in the test primarily because of their high clock frequency.)
The savings above, when combined with the 15–20% savings on completely idle instances, were great incentives for us to ensure that Stream ran natively and optimally on ARM64. To that end, we had to make two changes:
Stream is built on and shipped with the NodeJS runtime. In order for it to run on AWS Graviton2 instances, we had to make sure to compile NodeJS for ARM64, and update our packaging scripts. That took us around 1 man-week, with most of that time spent troubleshooting build pipelines that failed after running for 1+hr…the joys of release engineering 🙂
A key difference between AWS Graviton2 instances and their x64 counterparts is that on AWS Graviton2 a vCPU is a physical core, while on x64 a vCPU is mapped to a hyperthread. When deciding the scale-up factor, Stream detects the number of vCPUs and, by default, spawns max(N-2,1) worker processes – with the heuristic here being to reserve some resources for the OS.
However, on instances with lower numbers of vCPUs, this heuristic would leave more resources unutilized on AWS Graviton2 instances. Changing this heuristic was the only change we had to make to the Stream codebase to improve AWS Graviton2 support.
Having launched and operated our service for a few months, we now have instance utilization data to make a better decision on how to better optimize costs. Our analysis showed a workload profile that would benefit from burstable performance instances. These instances provide a lower price point for a dedicated baseline performance with the ability to burst, at a premium. To understand the performance characteristics of these instances we ran the “All Cores” test from above on t4g.medium and t3/a.medium, in unlimited mode. We found that t4g.medium instances have the same performance as their c6g.large, indicating that they’re powered by equivalent/same underlying hardware. However, the performance of the t3/a.medium instances is not comparable to that of the c5/a.large instances – likely attributed to the difference in CPU families used. The results are summarized in the chart below.
The above analysis drove our decision to switch to using burstable performance instances, a decision which reduced our infrastructure spend by ~20% (over the compute optimized instances).
Most recently, Cribl has announced that we’ve achieved the AWS Graviton Ready designation, part of the AWS Service Ready Program. This designation recognizes that Cribl has demonstrated a successful integration with AWS Graviton and adherence to AWS best practices, along with demonstrated customer success. We estimate that our decision to launch Stream Cloud on AWS Graviton2 will reduce our overall infrastructure costs by ~10%, which we’ll pass on to our customers.
The fastest way to get started with Cribl Stream is to sign-up at Cribl.Cloud. You can process up to 1 TB of throughput per day at no cost. Sign-up and start using Stream within a few minutes.
Experience a full version of Cribl Stream and Cribl Edge in the cloud with pre-made sources and destinations.
Classic choice. Sadly, our website is designed for all modern supported browsers like Edge, Chrome, Firefox, and Safari
Got one of those handy?