Complete Data Flow

SensorFlow keeps collection, ingestion, storage, and analytics on your own servers.

• Official Sensors Data SDKs collect events from Android, iOS, Web, mini programs, and servers.
• SensorFlow validates and processes incoming events.
• ClickHouse stores event details and Apache Superset provides SQL, charts, and dashboards.

Official Sensors Data SDKs -> SensorFlow -> ClickHouse -> Apache Superset

System Requirements

Start with one validation host and scale with event volume and query load.

• Minimum: 4 CPU cores, 8 GB RAM, and 100 GB SSD.
• Production start: 8 CPU cores, 16 GB RAM, and 500 GB SSD.
• Install Docker, Docker Compose, and Go 1.19 or newer.

1. Get the Open-source Service

Clone the SensorFlow repository and enter its root directory.

git clone https://github.com/data-analyze-bi/sensors.git
cd sensors

2. Start the Data Infrastructure

Docker Compose starts MySQL, Redis, ClickHouse, and Apache Superset.

• Ingestion uses port 8081, ClickHouse uses 8123/9000, and Superset uses 8088.
• Change the default Superset password and set a unique SUPERSET_SECRET_KEY in production.

cd deploy/docker
docker compose up -d --build
docker compose ps
cd ../..

3. Download and Install the License

Generate the license in the console, download the license and configuration, then place them in the repository's fixed directory.

• Store the license as binaries/sensors-payload-decoder.
• Store its configuration in binaries/.
• Never commit downloaded license files to Git.

mkdir -p binaries
mv ~/Downloads/sensors-payload-decoder-* binaries/sensors-payload-decoder
mv ~/Downloads/*.verify.json binaries/
chmod +x binaries/sensors-payload-decoder

4. Configure and Start Ingestion

Confirm the license path in conf/app.conf, install dependencies, and start the service.

# conf/app.conf
decoder_binary_path = ./binaries/sensors-payload-decoder

go mod download
go run main.go

5. Connect Official Sensors Data SDKs

Use the official SDK for each client platform and point its upload URL to SensorFlow.

• ⚠️ Third-party compatibility notice: standard upload compatibility is not an official Sensors Data certification; test SDK versions, encryption plugins, visual tracking, and auto-track features before production use.
• Use Android, iOS, Web, mini program, or React Native on clients; use the official Java, Go, Python, Node.js, or PHP SDK for servers.
• Validate event names, login IDs, property types, ClickHouse storage, and Superset metrics before switching production traffic.

https://your-domain.example/sensors/send/?token=YOUR_TOKEN

6. Verify ClickHouse

After sending a test event, confirm that it is stored in ClickHouse.

SELECT event, count() AS event_count
FROM sensors.event
GROUP BY event
ORDER BY event_count DESC
LIMIT 20;

7. Analyze in Superset

Open Superset and use its ClickHouse connection for SQL Lab, datasets, charts, and dashboards.

• Build DAU, registrations, logins, and event trends.
• Combine funnels, retention, paths, and cohorts.
• Control access by team, project, or dataset.

Updates and Operations

Stop ingestion before replacing a license, restore execute permission, and restart the service.

• Back up MySQL, ClickHouse, and Superset metadata.
• Monitor ingestion failures, disk usage, write latency, and query latency.
• Renew before expiry; purchased terms accumulate after the current end date.