Architectural considerations for high-volume streaming data on GCP

Streaming data plays a pivotal role in applications and industries where real-time information powers monitoring and analysis in high-stakes industries like finance, healthcare, manufacturing, e-commerce, and other data-driven businesses. Cloud platforms, especially Google Cloud Platform (GCP), excel at processing and analyzing vast, fast-growing data. GCP's scalability, cost-effectiveness, managed services, global…

Continue ReadingArchitectural considerations for high-volume streaming data on GCP

How we Built a 19 PiB Logging Platform with ClickHouse and Saved Millions

Table of Contents # Introduction Choosing ClickHouse The scale of the problem ClickHouse Cloud infrastructure Building on top of ClickHouse Cloud Understanding your users Some early design decisions Choosing OpenTelemetry (OTel) Architectural overview Ingest processing Schema Enhancing Grafana Cross-region querying Performance and cost Looking forward Conclusion Introduction # We believe…

Continue ReadingHow we Built a 19 PiB Logging Platform with ClickHouse and Saved Millions