Cribl puts your IT and Security data at the center of your data management strategy and provides a one-stop shop for analyzing, collecting, processing, and routing it all at any scale. Try the Cribl suite of products and start building your data engine today!
Learn more ›Evolving demands placed on IT and Security teams are driving a new architecture for how observability data is captured, curated, and queried. This new architecture provides flexibility and control while managing the costs of increasing data volumes.
Read white paper ›Cribl Stream is a vendor-agnostic observability pipeline that gives you the flexibility to collect, reduce, enrich, normalize, and route data from any source to any destination within your existing data infrastructure.
Learn more ›Cribl Edge provides an intelligent, highly scalable edge-based data collection system for logs, metrics, and application data.
Learn more ›Cribl Search turns the traditional search process on its head, allowing users to search data in place without having to collect/store first.
Learn more ›Cribl Lake is a turnkey data lake solution that takes just minutes to get up and running — no data expertise needed. Leverage open formats, unified security with rich access controls, and central access to all IT and security data.
Learn more ›The Cribl.Cloud platform gets you up and running fast without the hassle of running infrastructure.
Learn more ›Cribl.Cloud Solution Brief
The fastest and easiest way to realize the value of an observability ecosystem.
Read Solution Brief ›Cribl Copilot gets your deployments up and running in minutes, not weeks or months.
Learn more ›AppScope gives operators the visibility they need into application behavior, metrics and events with no configuration and no agent required.
Learn more ›Explore Cribl’s Solutions by Use Cases:
Explore Cribl’s Solutions by Integrations:
Explore Cribl’s Solutions by Industry:
September 25 | 10am PT / 1pm ET
Hold my beer: lessons from one team’s data pipeline journey
Register ›Try Your Own Cribl Sandbox
Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Get inspired by how our customers are innovating IT, security and observability. They inspire us daily!
Read Customer Stories ›Sally Beauty Holdings
Sally Beauty Swaps LogStash and Syslog-ng with Cribl.Cloud for a Resilient Security and Observability Pipeline
Read Case Study ›Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Transform data management with Cribl, the Data Engine for IT and Security
Learn More ›Cribl Corporate Overview
Cribl makes open observability a reality, giving you the freedom and flexibility to make choices instead of compromises.
Get the Guide ›Stay up to date on all things Cribl and observability.
Visit the Newsroom ›Cribl’s leadership team has built and launched category-defining products for some of the most innovative companies in the technology sector, and is supported by the world’s most elite investors.
Meet our Leaders ›Join the Cribl herd! The smartest, funniest, most passionate goats you’ll ever meet.
Learn More ›Whether you’re just getting started or scaling up, the Cribl for Startups program gives you the tools and resources your company needs to be successful at every stage.
Learn More ›Want to learn more about Cribl from our sales experts? Send us your contact information and we’ll be in touch.
Talk to an Expert ›September 2, 2021
One of the most important aspects of security in Cribl Stream deployments is protecting the communication channel between the workers and the leader instance. Cribl has made deploying Stream a very easy process, but ease of deployment does not negate the complexity of TLS.
With my background in Information Security, I’ve worked with public key infrastructure for a long time, and want to demystify how to use Mutual TLS authentication with Cribl Stream.
A Stream Leader has two ports to secure: the user interface where you, as a human, interact with the configuration; and the distributed management port, which interacts with the workers. By default, the ports are 9000 and 4200, respectively. This post focuses on the latter.
There are two main steps to completing a TLS setup with mutual authentication. The first: enabling TLS on the Leader for the distributed management port. The second: enabling certificate authentication on the workers.
First, let’s tackle enabling TLS for the worker/leader channel. This is done on the Leader instance under Distributed Settings. Upload your private key and certificate information to global System Settings > Security > Certificates. Then, at System Settings > Distributed Settings > TLS Settings, select Enabled by sliding the toggle switch to Yes. From there, use the resulting controls to select the certificate you uploaded earlier.
On the workers, you will need to configure $CRIBL_HOME/local/_system/instance.yml
to ensure TLS is enabled. Here’s a snippet from the complete file (see cribl.yml reference file on the Docs site for more information):
distributed:
tls:
disabled: false
Note: if you are setting up a worker for the first time, and use the bootstrapping command, this setting in the configuration will automatically be changed to the appropriate value.
If you deploy your workers in a container, the CRIBL_DIST_MASTER_URL
can be configured with the TLS protocol in the URL to enable encrypted TLS communications with the Leader. This will override the default setting in instance.yml
. Here’s the format, from our Distributed Deployment documentation:
CRIBL_DIST_LEADER_URL=tls://<authToken>@leader:4200
If you just want to enable TLS encryption with one-way authentication across this channel, you can stop here.
Now that the certificate is configured on the leader for encrypted TLS communications, we can implement the client certificate exchange to enable mutual authentication. This will allow Stream to permit only explicitly authorized clients holding valid certificates to connect to the Leader.
When a client certificate is presented to a Leader, two things happen:
First, the leader validates the client certificate presented to Stream. This setting is optional (enabling with the Validate Client Certs toggle), but highly recommended. The certificate is checked against the trust store to see if it has been signed by a valid certificate authority (CA). The list of certificates in the “CA Certificates Path” box is checked first (if populated), then against the list of built-in system certificates.
Second, the Leader checks if the common name (CN) matches the Regular Expression in the configuration. The Stream default is to accept any value in the common name field. You can customize this as needed. Validation is performed against the value after the CN= string in the common name, so if your common name is CN=logstream.worker
, you would enter logstream\.worker
in the common name text box (including the backslash because the value entered is a regular expression).
On the leader, your config should now look something like this:
You’ll notice that the minimum and maximum TLS versions are also configured in the screenshot above. Additionally, the Common Name regex contains an additional check for \d+
which allows for checking for the format: logstream1.worker
, logstream2.worker
, logstream3.worker
, etc.
To bootstrap a worker, only the shared authentication token is required. Certificates should be viewed as two-factor authentication. They’re meant to be out-of-band. Placing the certificates in the config bundle defeats the purpose of two-factor authentication.
Another item to consider when configuring TLS mutual authentication on workers: Place your certificates into a separate directory outside of $CRIBL_HOME
. If you place the certificates inside $CRIBL_HOME
, they’ll be removed when the next config bundle is deployed from the Leader.
To set up mutual authentication, add a few extra items need to the worker’s instance.yml
, as shown here:
distributed:
tls:
disabled: false
privKeyPath: /path/to/certs/worker.key
certPath: /path/to/certs/worker.pem
caPath: /path/to/certs/root.pem
requestCert: true
rejectUnauthorized: true # false if ignoring untrusted certs
Or if using environment variables:
With the changes made, restart the worker. You should see the worker reconnect to the Leader successfully.
If the worker doesn’t connect, check cribl.log
on both the worker and leader for more context about the problem. You should see errors related to dist leader communications
.
If you’re using a self-signed certificate, or a certificate signed by an internal certificate authority, on your Leader, see the section below on considerations for self-signed certificate authorities. You may need to add an extra setting to adjust your environment settings.
This section provides some basic steps you can follow to build your own self-signed certificate authority. We don’t recommend running this in high security environments and should be used for testing purposes only.
openssl genrsa -out root.key 2048
openssl req -x509 -new -nodes -key root.key -sha256 -days 3165 -out root.pem -subj '/CN=LogStream CA'
openssl genrsa -out leader.key 2048
openssl req -new -key leader.key -out leader.csr -subj '/CN=leader.cribl'
openssl genrsa -out worker.key 2048
openssl req -new -key worker.key -out worker.csr -subj '/CN=worker.cribl'
openssl x509 -req -in leader.csr -CA root.pem -CAkey root.key -CAcreateserial -out leader.pem -days 365 -sha256
openssl x509 -req -in worker.csr -CA root.pem -CAkey root.key -CAcreateserial -out worker.pem -days 365 -sha256
You should now have nine files: leader.csr
, leader.key
, leader.pem
, worker.csr
, worker.key
, worker.pem
, root.key
, root.pem
, root.srl
.
The leader and worker key and pem files should be used on the respective systems. The root.pem
file is the root CA and should be used in the CA config and also for the NODE_EXTRA_CA_CERTS
environment variable on the workers.
If the Leader uses a certificate that’s not trusted by a trusted system root certificate (especially if you used the section above on building your own self-signed certificate authority), you may need to add a PEM certificate to a file and add that path to the NODE_EXTRA_CA_CERTS
environment variable. Consult our docs’ Securing topic for more information.
In closing, we’ve walked through setting up Stream with Workers using certificate-based authentication to connect to the Leader node. Now all communications are encrypted and require a valid certificate to be presented. I hope this helps you secure your Cribl Stream environment! Join us on the Cribl Slack Community to discuss or for any assistance.
The fastest way to get started with Cribl Stream is to sign-up at Cribl.Cloud. You can process up to 1 TB of throughput per day at no cost. Sign-up and start using Stream within a few minutes.
Rick Salsa Sep 19, 2024
Josh Biggley Sep 17, 2024
Classic choice. Sadly, our website is designed for all modern supported browsers like Edge, Chrome, Firefox, and Safari
Got one of those handy?