OpenTelemetry

The zero-cache service embeds the JavaScript OTLP Exporter and can send logs, traces, and metrics to any standard otel collector.

To enable otel, set the following environment variables then run zero-cache as normal:

OTEL_EXPORTER_OTLP_ENDPOINT="<your otel endpoint>"
OTEL_EXPORTER_OTLP_HEADERS="<auth headers from your otel collector>"
OTEL_RESOURCE_ATTRIBUTES="<resource attributes from your otel collector>"
OTEL_NODE_RESOURCE_DETECTORS="env,host,os"

Grafana Cloud Walkthrough

Here are instructions to setup Grafana Cloud, but the setup for other otel collectors should be similar.

  1. Sign up for Grafana Cloud (Free Tier)
  2. Click Connections > Add Connection in the left sidebar add-connection
  3. Search for "OpenTelemetry" and select it
  4. Click "Quickstart" quickstart
  5. Select "JavaScript" javascript
  6. Create a new token
  7. Copy the environment variables into your .env file or similar copy-env
  8. Start zero-cache
  9. Look for logs under "Drilldown" > "Logs" in left sidebar

Distributed Tracing

You can enable end-to-end trace correlation from your frontend through zero-cache to your API server. This allows you to see the full request flow in your tracing UI.

To enable this, provide a getTraceparent callback when creating your Zero client:

import {ZeroProvider} from '@rocicorp/zero/react'
import {propagation, context} from '@opentelemetry/api'
 
function getTraceparent() {
  const carrier: Record<string, string> = {}
  propagation.inject(context.active(), carrier)
  return carrier.traceparent
}
 
return (
  <ZeroProvider
    /* ... other options ... */
    getTraceparent={getTraceparent}
  >
    <App />
  </ZeroProvider>
)

This callback is called before sending WebSocket messages that trigger API server calls (push, changeDesiredQueries, initConnection). The returned W3C traceparent header is forwarded through zero-cache to your API server, where it can be used to continue the trace.

Metrics Reference

zero.server

MetricTypeUnitDescription
uptimeGaugesCumulative uptime, starting from when requests are served
api.requestsCounterCalls to user mutate and query APIs, including cleanup and auth-validation operations
api.request_durationHistogramsEnd-to-end user API request duration, including retries
api.attemptsCounterHTTP fetch attempts made while calling user API endpoints
api.attempt_durationHistogramsDuration of each API HTTP attempt, excluding retry delays
api.in_flightUpDownCounterAPI requests currently in flight
startup_durationHistogramsTime from starting zero-cache until it is ready
worker_startup_durationHistogramsTime from starting a worker until it is ready

zero.replica

MetricTypeUnitDescription
db_sizeGaugebytesSize of the replica's main db file (excludes WAL)
wal_sizeGaugebytesSize of the replica's WAL file
wal2_sizeGaugebytesSize of the replica's WAL2 file (only if using wal2 mode)
backup_lagGaugemsTime since last litestream backup. Expected to sawtooth from 0 to ZERO_LITESTREAM_INCREMENTAL_BACKUP_INTERVAL_MINUTES
purge_blockedCounterNumber of change-log purges blocked because the actual backup state could not be verified or is stale
litestream.restore.runsCounterLitestream restore runs
litestream.restore.attemptsCounterLitestream restore subprocess attempts
litestream.restore.db_bytesCounterbytesSQLite database bytes restored by successful Litestream restores
litestream.restore.durationHistogramsWall-clock duration of Litestream restore runs
litestream.restore.wait_durationHistogramsTime spent waiting for replication-manager snapshot status before restoring
litestream.restore.process_durationHistogramsWall-clock duration of Litestream restore subprocesses
litestream.restore.validation_durationHistogramsTime spent validating restored replica databases
litestream.backup.process_runsCounterLitestream backup process exits
litestream.backup.process_durationHistogramsRuntime of Litestream backup subprocesses before exit
litestream.backup.list_durationHistogramsTime to list the Litestream backup destination
litestream.backup.verification_durationHistogramsTime to verify backup state in the destination
litestream.snapshot.reservation_durationHistogramsTime snapshot reservations are held while view-syncers restore and subscribe

zero.replication

MetricTypeUnitDescription
upstream_lagGaugemsLatency from sending a replication report to receiving it in the stream
replica_lagGaugemsLatency from receiving a replication report to it reaching the replica
total_lagGaugemsEnd-to-end replication latency. Grows as an estimate if the next report hasn't arrived
last_total_lagGaugemsEnd-to-end latency of the most recently received report. Unlike total_lag, does not grow if reports stop arriving
eventsCounterNumber of replication events processed
transactionsCounterCount of replicated transactions
changesCounterCount of replicated changes, including DML and DDL statements
slot_healthGauge1One-hot status for the active logical replication slot: ok, unreserved, lost, missing, or unknown
slot_retained_wal_bytesGaugebytesWAL bytes retained by the active logical replication slot
slot_safe_wal_bytesGaugebytesRemaining WAL capacity before the active logical replication slot is lost; omitted when Postgres reports no value
initial_sync_runsCounterNumber of initial-sync runs
initial_sync_durationHistogramsWall-clock duration of an initial-sync run
initial_sync_copy_durationHistogramsWall-clock duration of the COPY phase for a successful initial-sync run
initial_sync_copy_other_durationHistogramsInitial-sync duration excluding SQLite flush and index time for a successful run
initial_sync_flush_durationHistogramsTotal SQLite flush time for a successful initial-sync run
initial_sync_index_durationHistogramsSQLite index creation time for a successful initial-sync run
initial_sync_rowsCounterRows copied during successful initial-sync runs
initial_sync_copy_streamCounterbytesPostgreSQL COPY stream bytes processed during initial sync, including in-progress and failed runs
initial_sync_completed_copy_streamCounterbytesPostgreSQL COPY stream bytes processed during successful initial-sync runs
initial_sync_copy_chunksCounterPostgreSQL COPY stream chunks processed during initial sync
shadow-sync-runsCounterNumber of shadow initial-sync runs, labeled by result
shadow-sync-durationHistogramsWall-clock duration of a shadow initial-sync run, labeled by result
flow_control.active_subscribersGaugeActive change-stream subscribers receiving live changes
flow_control.queued_subscribersGaugeChange-stream subscribers waiting for the current transaction to finish before activation
flow_control.pending_messagesGaugeDownstream change-stream messages not yet acknowledged by subscribers
flow_control.backlog_messagesGaugeLive change-stream messages buffered while subscribers catch up
flow_control.backlog_bytesGaugebytesLive change-stream bytes buffered while subscribers catch up
flow_control.max_backlog_bytesGaugebytesMaximum live change-stream bytes buffered by a single subscriber
flow_control.waitsCounterCompleted flow-control checkpoints
flow_control.wait_durationHistogramsTime replication waits at flow-control checkpoints

zero.sync

MetricTypeUnitDescription
max-protocol-versionGaugeHighest sync protocol version seen from connecting clients
active-clientsUpDownCounterNumber of currently connected sync clients
active-client-groupsGaugeNumber of active ViewSyncerService instances in a syncer worker
queriesGaugeActive IVM pipelines across all client groups in a syncer worker
rowsGaugeCVR-tracked rows across all client groups in a syncer worker
serving_lagGaugemsLongest time locally ready replica changes have remained unserved across active ViewSyncer client groups
serving_lag_statsGaugemsDistribution of serving lag across active ViewSyncer client groups
serving_lagging_client_groupsGaugeActive ViewSyncer client groups with locally ready replica changes not yet served to clients
view_syncer_lagHistogramsTime from replica changes becoming ready to ViewSyncer output, sampled once per minute per active client group
view_syncer_hydrationHistogramsTime from a ViewSyncer query sync requiring hydration until output, per client group
lock-wait-timeHistogramsTime spent waiting to acquire the ViewSyncerService lock per operation
pipeline-resetsCounterCount of pipeline resets, labeled by reason
hydrationCounterNumber of query hydrations
hydration-timeHistogramsTime to hydrate a query
advance-timeHistogramsTime to advance all queries for a client group after applying a transaction
poke.timeHistogramsTime per poke transaction (excludes canceled/noop pokes)
poke.transactionsCounterCount of poke transactions
poke.rowsCounterCount of poked rows
cvr.load_attemptsCounterCVR load attempts
cvr.load_durationHistogramsTime to load a CVR
cvr.flush_attemptsCounterCVR flush attempts
cvr.flush-timeHistogramsTime to flush a CVR transaction
cvr.rows-flushedCounterNumber of changed rows flushed to a CVR
websocket.open_connectionsUpDownCounterOpen client WebSocket connections
websocket.connection_attemptsCounterClient WebSocket connection attempts
websocket.connection_successesCounterClient WebSocket connections successfully initialized
websocket.connection_failuresCounterClient WebSocket connection attempts that failed before initialization
websocket.errorsCounterClient WebSocket error events
ivm.advance-timeHistogramsTime to advance IVM queries in response to a single change
ivm.conflict-rows-deletedCounterRows deleted because they conflicted with an added row
query.transformationsCounterNumber of query transformations performed
query.transformation-timeHistogramsTime to transform custom queries via API server
query.transformation-hash-changesCounterTimes a query transformation hash changed
query.transformation-no-opsCounterTimes a query transformation was a no-op
query.row-set-signature-driftsCounterUnchanged query rehydrations whose row-set signature differs from the CVR, forcing a config-version bump

zero.mutation

MetricTypeUnitDescription
crudCounterNumber of CRUD mutations processed
customCounterNumber of custom mutations processed
pushesCounterNumber of pushes processed