Private Tracking Deployment Docs

Architecture Overview

SensorFlow keeps ingestion, cleaning, raw storage, and BI dashboards on your own servers.

• Compatible with Sensors-style event names, user properties, and batch uploads.
• Ingestion, ClickHouse, and Superset can start on one host and split later.
• Raw details stay in your database for audit, replay, and secondary processing.

Sensors SDK Style Ingestion

Keep the client tracking model mostly unchanged and point uploads to the private ingestion service.

• Preserve event, distinct_id, user_id, time, and properties fields.
• Support app, web, and server batch events.
• Use dual-write validation before switching the primary path.

POST /collect/batch
Content-Type: application/json

{"events":[{"event":"ViewProduct","properties":{"sku":"A001"}}]}

Deployment Sizing

Start small and scale by event volume and query pressure.

• Validation: 4C 8G, 100-200GB SSD for ingestion, MySQL, Redis, ClickHouse, and Superset.
• Production start: 8C 16G, 500GB-1TB SSD, split ingestion from ClickHouse/Superset.
• Higher volume: 16C 32G+, multi-disk SSD or ClickHouse cluster, separate Superset.

ingestion: 2C 4G
clickhouse: 4C 8G, SSD
superset: 2C 4G
mysql/redis: shared or managed

ClickHouse Detail Warehouse

ClickHouse is suited for high-throughput event writes and wide detail queries.

• Partition by date and order by project, event name, user id, and time.
• Keep raw properties for future metrics and recomputation.
• Use retention tiers to control storage cost.

PARTITION BY toYYYYMMDD(event_time)
ORDER BY (project_id, event_name, user_id, event_time)

Superset Dashboards

Superset connects directly to ClickHouse for self-service analytics.

• Good for trends, funnels, retention, paths, and operation reports.
• SQL Lab and datasets reduce custom report development.
• Permissions can be split by team, project, or dataset.

Operations Notes

Private deployments should focus on observability, backups, and capacity planning.

• Record bad events and schema validation failures at ingestion.
• Alert on disk usage, write latency, and query latency in ClickHouse.
• Back up MySQL, ClickHouse, and Superset metadata regularly.