Datadog vs OpenTelemetry with ClickHouse and Grafana: The Ingestion Volume and Host Scale Where Observability SaaS Costs 5x More Than Self-Hosting
The Observability Bill Shock: Why Growth-Stage B2B SaaS Hits the Datadog Wall
A healthy SaaS business can grow its customer base and still discover that monitoring costs are rising faster than revenue. Each new host adds infrastructure and tracing charges, while application teams may also increase log volume, event count, and metric cardinality without changing a single vendor contract.
The alternative is not free software on free servers. OpenTelemetry Collector, ClickHouse, and Grafana replace managed convenience with an operating system that somebody must deploy, secure, upgrade, and repair when production is already on fire.
These are illustrative planning scenarios, not measured benchmarks or vendor quotes; validate current prices, workload, retention, and engineering effort before making a commitment.
Baseline Architectural Assumptions & Workload Sizing
Use AWS us-east-1, Linux, 730 hours per month, gp3 storage and a 30-day target for searchable telemetry. The illustrative Datadog reference uses annual-billing list rates: Infrastructure Pro $15 per host, APM $31 per host, log ingestion $0.10 per GB, indexed logs $1.70 per million events for 15 days, and additional indexed custom metrics $5 per 100 per month. Pro includes 100 custom metrics per infrastructure host, pooled across the account.
The self-hosted reference allocates EKS and EC2 compute, gp3, S3, networking and operations to the observability platform. Infrastructure budgets are deliberately simplified allocations, not independently validated bills of materials; 0.15, 0.20 and 0.25 FTE at $150,000 annually add $1,875, $2,500 and $3,125 per month. A 15-day Datadog index is not equivalent to a 30-day self-hosted index; the comparable 30-day Datadog calculation appears below.
Quick Decision Matrix
| Architectural & Cost Metric | Managed Datadog Enterprise | Self-Hosted OTel + ClickHouse + Grafana |
|---|---|---|
| Pricing basis | Pro infrastructure + standard APM; annual list rates | Allocated AWS infrastructure + engineering labor |
| Log ingestion and indexing | Usage-based ingestion plus indexed-event retention | Capacity, storage, transfer and operations |
| Retention in headline scenarios | 15 days searchable; 30-day price calculated separately | 30 days targeted; sizing must be tested |
| Early growth / 15 hosts | $1,150 per month illustrative | $2,155 per month illustrative |
| Mid-market / 80 hosts | $6,850 per month illustrative | $3,750 per month illustrative |
| High volume / 350 hosts | $38,400 per month illustrative | $7,975 per month illustrative |
| Latency and resilience | Managed service; confirm contract and region | Depends on cluster design, queues and on-call capacity |
The Datadog column label describes a managed enterprise use case, not the Datadog Infrastructure Enterprise SKU. That SKU has a different list price, so substitute the contracted product mix before using the matrix for budgeting.
Granular Scenario Modeling: TCO & Latency Inflection Points
Start with a transparent equation: Datadog monthly spend equals host count times $46, plus ingested gigabytes times $0.10, plus indexed millions of log events times the applicable retention price, plus any custom metrics beyond the pooled allowance. One terabyte below means 1,000 GB; all hosts are assumed to run both infrastructure monitoring and APM.
Log volume does not reveal the number of indexed events. To reproduce the requested headline totals, the examples assume approximately 261.8 million, 1,717.6 million, and 11,941.2 million indexed events respectively; these counts are illustrative back-solves, not measured traffic. All stated custom-metric counts fall below the pooled Pro allowance, so their incremental indexed metric charge is zero.
Scenario A: Early Growth — 15 Hosts, 150 GB of Logs, 500 Custom Metrics
The managed estimate is $690 for hosts, $15 for ingestion and approximately $445 to index 261.8 million events for 15 days: about $1,150 per month. Because the 500 custom metrics are below the pooled 1,500-metric allowance, adding another $25 here would double-count an included entitlement.
The self-hosted planning allocation is $280 for infrastructure and $1,875 for 0.15 FTE, or $2,155 per month. One illustrative split is $73 for an EKS control plane, $120 for shared EC2 capacity, $20 for gp3, $10 for S3 and $57 for network and miscellaneous services; the latter four allocations require an actual AWS estimate.
At this size, the operating burden can dominate the invoice, so managed tooling may be the better starting point. This budget should not be interpreted as proof that a resilient multi-zone ClickHouse cluster fits inside $280: a small single-zone or shared-capacity deployment carries a different availability risk.
Scenario B: Mid-Market — 80 Hosts, 2.5 TB of Logs, 4,000 Custom Metrics
The managed example is $3,680 for hosts, $250 for ingestion and about $2,920 for 1,717.6 million indexed events at 15 days, totaling roughly $6,850. The 4,000 metrics remain inside the pooled 8,000-metric allowance, subject to the details of the actual contract and metric billing model.
The self-hosted allocation is $1,250 for AWS plus $2,500 for 0.20 FTE, totaling $3,750. Its planning split is $73 for EKS, $650 for EC2, $210 for gp3, $37 for S3 and $280 for networking and other services; these amounts are placeholders rather than measured resource consumption.
Under these invented event counts and budgets the crossover falls somewhere between the early and mid-market scenarios. They do not establish a universal 50-to-80-host threshold: a different indexing fraction, event size, reservation, support requirement or usage pattern can move the break-even point substantially.
Scenario C: High-Volume Production — 350 Hosts, 20 TB of Logs, 25,000 Custom Metrics
The managed estimate combines $16,100 in host charges, $2,000 in ingestion and approximately $20,300 to index 11,941.2 million events for 15 days, or about $38,400. The listed metric count is within the 35,000-metric pooled Pro allowance; trace ingestion and retained spans above their APM allowances are excluded.
Self-hosting allocates $4,850 for infrastructure and $3,125 for 0.25 FTE, yielding $7,975. A budgeting split is $73 EKS, $2,600 EC2, $1,100 gp3, $250 S3 and $827 for networking and other costs, before validating instance types, replication, sustained ingestion, backup, or availability targets.
That gives a managed-to-self-hosted spending ratio of approximately 4.8 to one for the 15-day comparison. It is not evidence that self-hosting universally saves five times the money, and it is not an equal-retention comparison. Infrastructure costs are shared allocations rather than a production sizing guarantee.
Normalize Retention Before Calling the Winner
Datadog's published annual rate for 30-day indexed logs is $2.50 per million, not $1.70. Keeping the same assumed event counts, hosts and ingestion produces approximately $1,359, $8,224 and $47,953 per month across A, B and C; these are about 0.63, 2.19 and 6.01 times the respective self-hosted planning budgets.
Even this adjustment is not a complete apples-to-apples service comparison. Confirm retention and indexing for traces and metrics, included data volumes, replicas, compression, alert delivery, backup, query concurrency, support and data transfer. Datadog Pro and APM are not a substitute for a quote for every Enterprise feature.
Latency Is a Measured Boundary, Not a Price Label
Write separate objectives for telemetry ingestion delay, recent dashboard refresh and high-cardinality investigation queries. Measure p50, p95 and p99 during realistic bursts, then repeat while a node is unavailable and background merges or backups are running.
ClickHouse can perform well on compressed columnar telemetry, but expensive joins, too many tiny inserts or poorly chosen ordering keys can produce long tails. A managed service can also exhibit ingestion delay or query contention; neither architecture has a defensible numerical latency advantage without a representative benchmark.
Build an Audit-Friendly AWS Bill of Materials
Turn each placeholder into a resource inventory before approval: instance family and size, instance count, utilization, hourly rate, and whether capacity is shared with other workloads. Multiply reserved capacity by the monthly hours and include the EKS control plane separately; if an existing cluster is shared, state whether the platform receives an allocated share or the full charge.
For gp3, specify provisioned GiB, included and additional IOPS, throughput and the number of copies. Add S3 storage, API requests, retrieval, snapshots and inter-zone transfer, then use current regional prices. A single monthly infrastructure total cannot establish that the design has enough disk or bandwidth during peaks.
Assign a financial owner to each uncertainty. For example, compare the cost of keeping two replicated copies against the recovery time of a single copy restored from backup. If a separate engineering team owns the cluster, allocate its share of support, security reviews and incident response rather than pretending that those hours disappear.
Test the Break-Even Equation Against Your Own Event Shape
Suppose a service doubles log bytes but preserves its indexed event count by emitting larger structured records. Ingestion charges rise while event-indexing charges stay flat; if the same byte volume consists of shorter records, the indexed-event cost can rise instead. A volume-only calculator therefore needs an assumed average event size and index fraction.
Consider metric allowances in the same way. With 80 Pro hosts and 9,000 qualifying indexed custom metric series, an illustrative 8,000-series pooled allowance leaves 1,000 additional series; at $5 per 100, the incremental indexed-metric charge would be $50 per month. Confirm whether ingested metrics or another billing model add charges under your actual agreement.
For decision making, run low, base and high cases rather than stretching one point estimate into a universal threshold. Include a managed case that drops low-value logs before indexing and a self-hosted case with higher staffing and replicated storage. If the difference disappears under small changes, reliability and team focus may matter more than the advertised savings.
Benchmark Query and Failure Behavior Before Committing
Replay representative queries at realistic concurrency, including customer-facing incident investigations and routine dashboards. Record the hardware, ingest rate, event size, retention, compression, cache state and background merge activity alongside latency; otherwise a reported number cannot be reproduced or compared.
Then kill a collector, block an exporter and take a ClickHouse node offline in a controlled test. Measure backlog drain time, dropped telemetry, alert delay and the effect of restoring service. A migration is ready only when the team understands both the normal operating cost and the failure modes it is choosing to own.
Migration & Architectural Checkpoints
Collect, Buffer, and Sample Traces Deliberately
Deploy OpenTelemetry Collectors near application workloads, then route trace data to a gateway tier with explicit memory limits, batching, retries and persistent queues where supported. Model collector failures and downstream throttling, not just normal throughput, and monitor queue occupancy, dropped records, exporter errors and end-to-end lag.
Tail-based sampling must see the relevant spans for a trace at the same decision-making collector; use trace-aware routing and allow enough time for late spans. Retain errors and slow requests with explicit policies, while recognizing that sampling can distort counts unless metrics are generated before sampling or corrected downstream.
Design ClickHouse Tables for Retention and Recovery
Choose partitioning and sorting keys around the actual time windows, tenant boundaries and frequent filters. Use ClickHouse TTL rules to expire data and, where supported by the chosen storage policy, move older parts to an S3-backed tier; test retrieval performance and object-store request charges before promising interactive cold queries.
Thirty-day retention is not a backup strategy. Protect backups in a separate failure domain, document recovery point and recovery time objectives, and regularly restore a representative dataset. Account for replication and merge headroom rather than assuming compressed payload size equals provisioned disk.
Constrain Cardinality Before It Reaches Storage
Identifiers such as request IDs, user IDs and raw URLs can turn one metric into millions of series. Normalize paths, remove unnecessary labels, set per-service limits and review histogram bucket choices; keep diagnostic identifiers in appropriately protected logs or traces instead of metric dimensions.
Track the number of distinct label combinations and alert on its rate of change. Vendor usage dashboards and self-hosted storage both suffer from unbounded dimensions, even though the bill arrives through different line items.
Plan the Metrics Path and Dashboard Translation
Prometheus remote_write is a transport protocol, not a complete long-term metrics backend. Route it to a compatible receiver such as a Prometheus-compatible time-series store, then connect Grafana to that backend; do not assume ClickHouse accepts remote_write natively without a verified receiver or adapter.
Export existing dashboards and inventory their queries, template variables, alert rules and data-source-specific functions. Rebuild or translate Datadog queries for PromQL, ClickHouse SQL or another supported source, then compare panels against the original during a dual-running period.
Cut Over in Bounded Steps
Begin with a small service and a representative telemetry sample, sending a copy to the new stack while preserving the existing alerts. Verify counts, timestamps, trace relationships and access controls, then widen traffic only after the team can diagnose a broken pipeline.
Set a time-boxed dual-run budget and a rollback trigger for missing alerts, unacceptable lag or unexpected infrastructure growth. Record ownership for operating-system patches, Collector upgrades, ClickHouse schema migrations, S3 lifecycle rules and the overnight incident queue.
Practical Engineering FAQ
How do we reduce telemetry loss during a backend outage?
Use bounded queues, retries with backoff and persistent storage where the selected Collector components support it. Test a prolonged outage and confirm what happens when the queue fills; no buffering design guarantees zero loss across every failure mode.
Can this architecture meet regulated-data requirements?
It can be designed for encryption in transit and at rest, role-based access, audit trails, residency controls and documented deletion, but deployment alone does not establish compliance. Map each obligation to the actual services, data flows and contracts, and involve security and legal reviewers before moving sensitive telemetry.
Will existing Grafana dashboards work unchanged?
Grafana dashboards already using the same data source and query semantics may transfer with modest edits. Datadog-specific queries, alerts, tags and functions generally require translation and testing; a visually similar panel does not prove identical calculations.
Is 0.25 FTE enough for production at 350 hosts?
It is an illustrative steady-state assumption, not a staffing recommendation or service-level guarantee. An organization with strict uptime, multiple regions or limited Kubernetes experience may need much more coverage, and implementation and migration work can add substantial one-time costs.
What should we compare before signing or migrating?
Use a full month of host counts, ingested bytes, indexed events, span usage and billable metric series, and obtain a written vendor quote at the required retention. Compare it with a measured self-hosted bill of materials, labor, support, network charges and a tested recovery plan.
Contextual Related Deep-Dive
The same capacity-versus-consumption tradeoff appears in compute architecture: AWS Lambda vs ECS Fargate: The Exact RPS Inflection Point Where Serverless Becomes 4x More Expensive Than Containers.
Prices, architecture assumptions and workload characteristics change; run the numbers against your own requirements and current vendor and AWS pricing before making a production decision.
Sources: Datadog pricing list, Datadog billing definitions, and Amazon EKS pricing.