The High Cost of Over-Collection
In this video I talk about how observability tools rank as the 3rd-highest cost for engineering teams after their people and cloud infrastructure. That’s insane! What's even more insane? You hardly ever use the collected data. Maybe if you're running four 9s, you may need that data 1 in 10,000 times, but you're collecting it all the time. You're collecting a lot more data than you might need because a particular incident may only require something related to a particular service.
Shoreline.io's Efficient Approach
At Shoreline.io, we do operations at the edge by collecting only the minimal necessary information on a daily basis for running alarm checks. When the alarm fires, you can then collect the relevant data without transmitting it. You can collect information from the last hour - or make it as wide as you want -because it's not polluting the network. And then you can run real-time queries against that data. That information doesn't overwhelm you with a blizzard of dashboards that you need to look at during a live event. Instead, you get a runbook that gives you the information you need to diagnose and repair a specific incident connected to a specific alarm raised by your observability tool.
Case Study from Fortune 10 Company
I’m currently working with a Fortune 10 company, and they said, "We only want to collect 100 metrics most of the time. But when things happen in production, we might want to collect 10,000 to understand what's going on." Now, collecting 10,000 metrics across such a large estate is challenging and expensive – even for them. But it's much easier to modify the setting when an incident occurs so that you’re collecting data just on the resources where you need it. This strategy effectively controls observability costs while maintaining high operational rigor.
Conclusion
Adopting a targeted data collection strategy not only reduces observability costs but also enhances operational efficiency. By focusing on what's truly necessary, engineering teams can maintain high performance without being overwhelmed by data or costs.
Additional Resources
For more insights into efficient observability practices and incident automation, take a look at what our customers have to say.