Architecting Log Management for Privacy and Scale without the Headache

As companies grow, they inevitably hit a wall: observability data explodes while privacy requirements become stricter. For years, engineers have faced a painful tradeoff—either ship petabytes of sensitive data to a central cloud (incurring egress costs and compliance risks) or manage a complex self-hosted stack that is painful to scale. In this talk, we’ll break down why this struggle exists and how to fundamentally rethink our approach to high-volume data. We’ll introduce CloudPrem (based on OSS Quickwit), a Datadog BYOC solution that keeps data where it is generated while maintaining high performance. We’ll also discuss why raw speed isn’t enough. We treat User Experience (UX) as a first-class citizen to build not just a powerful system, but the simplest one for developers, SREs and agents to use. You’ll walk away with a blueprint for handling petabytes of logs—keeping them private, cost-efficient, and instantly accessible.