Cribl puts your IT and Security data at the center of your data management strategy and provides a one-stop shop for analyzing, collecting, processing, and routing it all at any scale. Try the Cribl suite of products and start building your data engine today!
Learn more ›Evolving demands placed on IT and Security teams are driving a new architecture for how observability data is captured, curated, and queried. This new architecture provides flexibility and control while managing the costs of increasing data volumes.
Read white paper ›Cribl Stream is a vendor-agnostic observability pipeline that gives you the flexibility to collect, reduce, enrich, normalize, and route data from any source to any destination within your existing data infrastructure.
Learn more ›Cribl Edge provides an intelligent, highly scalable edge-based data collection system for logs, metrics, and application data.
Learn more ›Cribl Search turns the traditional search process on its head, allowing users to search data in place without having to collect/store first.
Learn more ›Cribl Lake is a turnkey data lake solution that takes just minutes to get up and running — no data expertise needed. Leverage open formats, unified security with rich access controls, and central access to all IT and security data.
Learn more ›The Cribl.Cloud platform gets you up and running fast without the hassle of running infrastructure.
Learn more ›Cribl.Cloud Solution Brief
The fastest and easiest way to realize the value of an observability ecosystem.
Read Solution Brief ›Cribl Copilot gets your deployments up and running in minutes, not weeks or months.
Learn more ›AppScope gives operators the visibility they need into application behavior, metrics and events with no configuration and no agent required.
Learn more ›Explore Cribl’s Solutions by Use Cases:
Explore Cribl’s Solutions by Integrations:
Explore Cribl’s Solutions by Industry:
September 25 | 10am PT / 1pm ET
Hold my beer: lessons from one team’s data pipeline journey
Register ›Try Your Own Cribl Sandbox
Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Get inspired by how our customers are innovating IT, security and observability. They inspire us daily!
Read Customer Stories ›Sally Beauty Holdings
Sally Beauty Swaps LogStash and Syslog-ng with Cribl.Cloud for a Resilient Security and Observability Pipeline
Read Case Study ›Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Transform data management with Cribl, the Data Engine for IT and Security
Learn More ›Cribl Corporate Overview
Cribl makes open observability a reality, giving you the freedom and flexibility to make choices instead of compromises.
Get the Guide ›Stay up to date on all things Cribl and observability.
Visit the Newsroom ›Cribl’s leadership team has built and launched category-defining products for some of the most innovative companies in the technology sector, and is supported by the world’s most elite investors.
Meet our Leaders ›Join the Cribl herd! The smartest, funniest, most passionate goats you’ll ever meet.
Learn More ›Whether you’re just getting started or scaling up, the Cribl for Startups program gives you the tools and resources your company needs to be successful at every stage.
Learn More ›Want to learn more about Cribl from our sales experts? Send us your contact information and we’ll be in touch.
Talk to an Expert ›All Cloud providers such as AWS, Azure, Google Cloud Platform, and Oracle Cloud offer object storage solutions to economically store large volumes of data and retrieve it on demand. It’s far cheaper to store one petabyte of data in object storage than in block storage. As AWS S3 has become the standard, many on-premise storage appliance vendors have incorporated S3 APIs to store and retrieve data. Oracle wisely continued that trend to OCI (Oracle Cloud Infrastructure). OCI’s Object Storage offers an S3-compatibility API and is relatively easy to set up as described in their documentation.
In this blog, I’ll cover how to enable Cribl Stream to read and write from/to OCI’s Object Storage. Steps include:
It’s worth noting that OCI has additional logical groupings that include a ‘namespace’ and a ‘compartment’. The user id, group, & permission belong to a ‘compartment’. The Object Storage bucket has both a ‘namespace’ and a ‘compartment’ mapped to it. A default ‘compartment’ is created when your root account is created. Per an OCI blog on compartments, they provide an extra level of security isolation and access control. The following image from the blog nicely depicts the purpose of compartments:
And per OCI documentation, Object Storage ‘namespaces’ serve as the top level container for all buckets and objects, and is similarly created when the root user account is created. We will encounter those concepts in various configuration steps.
Steps 1-3 of the below walkthrough are optional for a root user account, as the permissions are inherited. However, as a best practice, creating a new user provides additional levels of security.
1A: Create a new user for S3 API Access: Once you’re logged into the OCI console, click on the 3 lines icon in the top left corner. Click on ‘Identity & Security’, and finally ‘Users’ per the screenshot below:
1B: Select to create a new user
From the ‘user’ screen, create a new user. In the below screenshot, a ‘cribl_oci’ user has been created
1C: Create an IAM user: From the next screen, select IAM user. Fill in the banks and optionally, create tags (i.e. purpose: S3 API access)
From within the Identity page, access the groups page. Create a new group, and add the recently created user to that group per the screenshots below:
While still in the ‘Identities’ page, select policies, and create a new policy:
OCI offers a simple UI to add one policy at a time to grant access.
Copy and paste the 3 lines below, change the bucket name to the bucket in question, and apply to your policy in advanced mode.
Allow group Cribl_ObjectStorage_Access_Group to read buckets in tenancy
Allow group Cribl_ObjectStorage_Access_Group to manage objects in tenancy where all {target.bucket.name='CriblBucket', any {request.permission='OBJECT_CREATE', request.permission='OBJECT_INSPECT'}}
Allow group Cribl_ObjectStorage_Access_Group to read objects in tenancy where target.bucket.name='CriblBucket'
The first line allows listing all buckets in the user’s tenancy. (read buckets
)
The second line allows the user to create and inspect objects for the bucket in question.
The third line allows the user to read objects within the bucket. (read objects
)
Access the specific user settings one more time, and select ‘customer secret key’ per the screenshots below. Create a new secret key, being sure to copy it before closing the window.
The access key for that secret key can be retrieved by hovering over the secret key list per the screenshot below:
*Keep the secret key and access key handy for configuration within Cribl. The secret key will only be visible once.
Per OCI documentation, the URL format is as follows: mynamespace.compat.objectstorage.us-phoenix-1.oraclecloud.com
The below screenshot of the Object Store bucket exposes the namespace and region:
In this case, the URL would be: axx6hbqzm5va.compat.objectstorage.us-phoenix-1.oraclecloud.com
Access Cribl Stream and populate the destination per the screenshots and instructions below:
You might need to wait a few minutes for the Cribl timeouts and file size settings to be reached before seeing any files. Within the OCI console, select the bucket in question, and expand the bucket hierarchy to see what was written:
Creating an S3 collector is very similar to the destination we recently created. The same S3-compatibility URL is referenced, as well as manual authentication method with secret and access keys. Also, similar to the Destination, the ‘Region field needs to remain empty.
Creating an S3 collector is very similar to the destination we recently created. The same S3-compatibility URL is references, as well as manual authentication method with secret and access keys. Also, similar to the Destination, the ‘Region field needs to remain empty.
Testing replay: Follow the sequence of screenshots below to ‘run’ a replay from the S3 collector entry in question, optionally specify a time filter, and validating results are retrieved. In the below screenshot, the files populated to the bucket in a prior step are retrieved.
After a commit and deploy, you are now ready to replay data stored in Oracle Cloud Infrastructure (OCI) buckets.
Situation: When testing sending data to an OCI bucket destination, you get the following error:
Error: 400-Bad Request
Details: “Output OCI_CriblBucket does not exist!”
Resolution: Ensure the worker is up, is communicating with the leader, and has had the configuration deployed successfully. If you just deployed, wait a few more seconds to minutes for the worker to fully come up. Try a different worker in the group if necessary.
Situation: error: BadRequest message: null
Resolution: Ensure that you left the region field empty within the Stream collector or destination configuration.
Situation: No data returned when running a replay.
Resolution:
Cribl Stream is free to try via download or in the cloud. Sign up for a free account today and see how easy it is to integrate Oracle Cloud Infrastructure Object Storage with Cribl LogStream.
The fastest way to get started with Cribl Stream is to sign-up at Cribl.Cloud. You can process up to 1 TB of throughput per day at no cost. Sign-up and start using Stream within a few minutes.
Experience a full version of Cribl Stream and Cribl Edge in the cloud with pre-made sources and destinations.
Classic choice. Sadly, our website is designed for all modern supported browsers like Edge, Chrome, Firefox, and Safari
Got one of those handy?