Cribl puts your IT and Security data at the center of your data management strategy and provides a one-stop shop for analyzing, collecting, processing, and routing it all at any scale. Try the Cribl suite of products and start building your data engine today!
Learn more ›Evolving demands placed on IT and Security teams are driving a new architecture for how observability data is captured, curated, and queried. This new architecture provides flexibility and control while managing the costs of increasing data volumes.
Read white paper ›Cribl Stream is a vendor-agnostic observability pipeline that gives you the flexibility to collect, reduce, enrich, normalize, and route data from any source to any destination within your existing data infrastructure.
Learn more ›Cribl Edge provides an intelligent, highly scalable edge-based data collection system for logs, metrics, and application data.
Learn more ›Cribl Search turns the traditional search process on its head, allowing users to search data in place without having to collect/store first.
Learn more ›Cribl Lake is a turnkey data lake solution that takes just minutes to get up and running — no data expertise needed. Leverage open formats, unified security with rich access controls, and central access to all IT and security data.
Learn more ›The Cribl.Cloud platform gets you up and running fast without the hassle of running infrastructure.
Learn more ›Cribl.Cloud Solution Brief
The fastest and easiest way to realize the value of an observability ecosystem.
Read Solution Brief ›Cribl Copilot gets your deployments up and running in minutes, not weeks or months.
Learn more ›AppScope gives operators the visibility they need into application behavior, metrics and events with no configuration and no agent required.
Learn more ›Explore Cribl’s Solutions by Use Cases:
Explore Cribl’s Solutions by Integrations:
Explore Cribl’s Solutions by Industry:
September 25 | 10am PT / 1pm ET
Hold my beer: lessons from one team’s data pipeline journey
Register ›Try Your Own Cribl Sandbox
Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Get inspired by how our customers are innovating IT, security and observability. They inspire us daily!
Read Customer Stories ›Sally Beauty Holdings
Sally Beauty Swaps LogStash and Syslog-ng with Cribl.Cloud for a Resilient Security and Observability Pipeline
Read Case Study ›Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Transform data management with Cribl, the Data Engine for IT and Security
Learn More ›Cribl Corporate Overview
Cribl makes open observability a reality, giving you the freedom and flexibility to make choices instead of compromises.
Get the Guide ›Stay up to date on all things Cribl and observability.
Visit the Newsroom ›Cribl’s leadership team has built and launched category-defining products for some of the most innovative companies in the technology sector, and is supported by the world’s most elite investors.
Meet our Leaders ›Join the Cribl herd! The smartest, funniest, most passionate goats you’ll ever meet.
Learn More ›Whether you’re just getting started or scaling up, the Cribl for Startups program gives you the tools and resources your company needs to be successful at every stage.
Learn More ›Want to learn more about Cribl from our sales experts? Send us your contact information and we’ll be in touch.
Talk to an Expert ›At this point, you already know how powerful syslog is (and if you don’t, check out “Introduction to Syslog”). But here’s the thing: Scaling your systems to consume high volume syslog is like fighting zombies. Weird unexpected behavior and no easy solutions. Before you fight zombies, though, you have to understand them.
So, here are the challenges for scaling syslog one by one:
Network devices send data using UDP and then forget about it, unlike a TCP connection, which can check if the destination and port are present. A TCP connection will also accept data before sending it. This matters because you cannot send UDP syslog over long distances as every network hop makes it more likely you will drop data. Locate your syslog servers as close to your UDP data sources as possible to make it more likely you are capturing all of your data. It’s critical to recognize that this is simply the nature of the beast, and you must comprehend it.
There are two well defined RFCs that describe what syslog should look like and it could be transmitted to your logging tools. Yet much of what vendors call syslog is not RFC compliant, which can cause parsing and event management pain. So, what do you do about that? First, don’t make assumptions. Second, verify your data as you onboard it. Be sure to check your timestamp and create the right source and event breakers to consume your data. Back to rule one: never assume what someone says is syslog is actually syslog and not random data delivered over UDP.
Anyone ingesting syslog has to plan for spikes and keep the worst-case scenario in mind because your typical volume baseline will not always stay consistent. When you need your observability platform to work the most is when data volumes are through the roof and everything is going badly. When your firewall is broken and vomiting data or you have a breach and the security team is knee-deep in alligators.
Plan for the spike and not your daily average. Work with your business leaders to determine your level of robustness and at what point you think your systems will start to fail. This is a business decision, so offer data to support your options and let your leaders determine what level of investment they are willing to support. Be sure to keep your documentation to deal with the inevitable finger-pointing if something goes wrong.
Securing syslog is an old issue and involves many tradeoffs. UDP packets cannot be encrypted; therefore, they can’t be secured with SSL, which can be a challenging topic to discuss with security professionals. You can secure TCP syslog but then you run into scaling issues with load balancers and the high overhead that can impact certain devices. I recommend putting your syslog servers as close as possible to the syslog sources and then using your syslog server to handle encryption, which is a reasonable compensating control.
This is the number one issue in a high-volume environment: being able to consume the data, get it into your logging systems, and not drop the data is an enormous challenge. Without thinking through the scaling challenges, you might create a massive security gap. Keep all of this in mind as you fight the good fight:
Have a discussion with your network and storage engineers about implementing a standardized data format. It is critical to collaborate with your teams and document the standards that you will use as an organization to hold these teams responsible. Every class of devices should have a master logging config that everyone can agree to. Be sure to audit formats as often as possible to catch one-offs and have a process to respond. This is a good task for your overnight teams and contributes to good data governance.
There’s a lot of discussion over the best technique to load balance syslog. Essentially, you need to know if your environment can support it. Pro tip: Don’t implement an IP multicast load balancer if your network and virtualization teams lack the necessary skills. Always implement what you can support and aspire to do more when your teams have the right skill sets. Load balancing syslog is a challenge for many network teams so be prepared to implement and then make adjustments till you get it right.
Assume you have a network with many syslog servers, which may be placed far away from the source as the network grows. It’s vital to collect data as close to the source as feasible when scaling syslog. By placing a syslog server near the source, you can send UDP to it and have it send the data to Splunk or Elastic over TCP. As a result, you get the best mix of UDP delivery, proximity to the source, and TCP delivery. If you use Cribl Stream as your syslog pipeline, it will support queuing, which means that if your destination goes down, Cribl Stream will queue it, send it to the file system, and then retransmit it when your destination comes back up.
A considerable portion of syslog data is not syslog but random data from your network devices. You have to have alternatives for cleaning, repairing, improving, and minimizing your data with Cribl Stream. You can also validate your data and use Stream’s visual validation feature to make sure your data parses as expected.
As an observability engineer, you need to work with syslog data in a scalable way as this ensures that you have quality data outputs and your operations and security teams get the right data to help them do their jobs. Using a tool like Cribl Stream is a step in the right direction to solving your problems with syslog data. Check out part 3 of our Syslog series to see how Cribl Stream helps enterprises manage syslog scaling.
The fastest way to get started with scaling syslog is with Cribl Stream. Try the Free Cloud Sandboxes to get started.
Experience a full version of Cribl Stream and Cribl Edge in the cloud with pre-made sources and destinations.
Classic choice. Sadly, our website is designed for all modern supported browsers like Edge, Chrome, Firefox, and Safari
Got one of those handy?