Complete Data Flow

SensorFlow keeps collection, ingestion, storage, and analytics on your own servers.

• Official Sensors Data SDKs collect events from Android, iOS, Web, mini programs, and servers.
• SensorFlow validates and processes incoming events.
• ClickHouse stores event details and Apache Superset provides SQL, charts, and dashboards.

Official Sensors Data SDKs -> SensorFlow -> ClickHouse -> Apache Superset

System Requirements

Start with one validation host and scale with event volume and query load.

• Minimum: 4 CPU cores, 8 GB RAM, and 100 GB SSD.
• Production start: 8 CPU cores, 16 GB RAM, and 500 GB SSD.
• Install Docker, Docker Compose, and Go 1.19 or newer.

1. Get the Open-source Service

Clone the SensorFlow repository and enter its root directory.

git clone https://github.com/data-analyze-bi/sensors.git
cd sensors

2. Start the Data Infrastructure

Docker Compose starts MySQL, Redis, ClickHouse, and Apache Superset.

• Ingestion uses port 8081, ClickHouse uses 8123/9000, and Superset uses 8088.
• Change the default Superset password and set a unique SUPERSET_SECRET_KEY in production.

cd deploy/docker
docker compose up -d --build
docker compose ps
cd ../..

3. Download and Install the License

Generate the license in the console, download the license and configuration, then place them in the repository's fixed directory.

• Store the license as binaries/sensors-payload-decoder.
• Store its configuration in binaries/.
• Never commit downloaded license files to Git.

mkdir -p binaries
mv ~/Downloads/sensors-payload-decoder-* binaries/sensors-payload-decoder
mv ~/Downloads/*.verify.json binaries/
chmod +x binaries/sensors-payload-decoder

4. Configure and Start Ingestion

Confirm the license path in conf/app.conf, install dependencies, and start the service.

# conf/app.conf
decoder_binary_path = ./binaries/sensors-payload-decoder

go mod download
go run main.go

5. Connect Official Sensors Data SDKs

Use the official SDK for each client platform and point its upload URL to SensorFlow.

• ⚠️ Third-party compatibility notice: standard upload compatibility is not an official Sensors Data certification; test SDK versions, encryption plugins, visual tracking, and auto-track features before production use.
• Use Android, iOS, Web, mini program, or React Native on clients; use the official Java, Go, Python, Node.js, or PHP SDK for servers.
• Validate event names, login IDs, property types, ClickHouse storage, and Superset metrics before switching production traffic.

https://your-domain.example/sensors/send/?token=YOUR_TOKEN

Official SDK Downloads

These links point to repositories under the official sensorsdata GitHub organization. SensorFlow does not bundle or redistribute these SDKs. Review the LICENSE and official terms for the version you select.

Web / JavaScript SDK

Collect page views, clicks, custom events, and profiles in websites and web applications.

• Install through npm, yarn, or an officially supported browser distribution.
• Point server_url to SensorFlow and configure SPA page tracking as documented by the SDK.
• Disable debug logging and verify Beacon, CORS, and domain policies before production.

npm install sa-sdk-javascript

import sensors from 'sa-sdk-javascript';
sensors.init({
  server_url: 'https://your-domain.example/sensors/send/?token=YOUR_TOKEN',
  is_track_single_page: true, use_client_time: true, send_type: 'beacon'
});
sensors.quick('autoTrack');
sensors.login('USER_ID');
sensors.track('ButtonClick', { button_name: 'Submit' });

iOS SDK

Install from official instructions with CocoaPods or Swift Package Manager and use the SensorFlow ingestion URL.

• SensorFlow is independent from Sensors Data Cloud; do not configure a Sensors Data Cloud project Scheme for this endpoint.
• Test auto-track, login IDs, custom properties, and lifecycle events.
• Disable SDK debug logs before release.

let options = SAConfigOptions(serverURL: "https://your-domain.example/sensors/send/?token=YOUR_TOKEN", launchOptions: launchOptions)
options.enableAutoTrack = .default
SensorsAnalyticsSDK.start(configOptions: options)
SensorsAnalyticsSDK.sharedInstance()?.login("USER_ID")
SensorsAnalyticsSDK.sharedInstance()?.track("ButtonClick", withProperties: ["button_name": "Submit"])

Android SDK

Add the official Gradle dependency and initialize the SDK in Application with the SensorFlow URL.

• No Sensors Data Cloud project Scheme is required for the SensorFlow endpoint.
• Configure auto-track scope and consent timing for your privacy obligations.
• Disable logs and verify obfuscation, network permission, and HTTPS before release.

SAConfigOptions options = new SAConfigOptions("https://your-domain.example/sensors/send/?token=YOUR_TOKEN");
options.setAutoTrackEventType(SensorsDataAPI.AutoTrackEventType.APP_START | SensorsDataAPI.AutoTrackEventType.APP_END | SensorsDataAPI.AutoTrackEventType.APP_CLICK);
SensorsDataAPI.startWithConfigOptions(this, options);
SensorsDataAPI.sharedInstance().login("USER_ID");
SensorsDataAPI.sharedInstance().track("ButtonClick", properties);

WeChat Mini Program SDK

Obtain the SDK from an official source, initialize it in app.js, and allow the SensorFlow HTTPS domain.

• Add the ingestion host to the request domain allowlist.
• Select auto-track events according to product and privacy requirements.
• Verify requests on a real device and production-like domain.

sensors.init({
  server_url: 'https://your-domain.example/sensors/send/?token=YOUR_TOKEN',
  autotrack: { appLaunch: true, appShow: true, appHide: true, pageShow: true, pageShare: true }
});
sensors.login('USER_ID');
sensors.track('ButtonClick', { button_name: 'Submit' });

React Native SDK

Install the official npm package and complete native initialization for both iOS and Android.

• Keep the JavaScript package compatible with both native SDK versions.
• Point serverUrl to SensorFlow and test screen views and custom events.
• Run regression tests before React Native or native SDK upgrades.

RNSensorsAnalyticsModule.init({
  serverUrl: 'https://your-domain.example/sensors/send/?token=YOUR_TOKEN',
  autoTrackAppViewScreen: true
});
RNSensorsAnalyticsModule.login('USER_ID');
RNSensorsAnalyticsModule.track('ButtonClick', { button_name: 'Submit' });

Server SDKs

Use the official Java, Go, Python, Node.js, or PHP SDK and track trusted events after business transactions succeed.

• Reuse an application-level SDK instance.
• Keep client and server login IDs consistent.
• Flush or close before shutdown and preserve stable property types.

from sensorsanalytics import SensorsAnalytics
sa = SensorsAnalytics(server_url='https://your-domain.example/sensors/send/?token=YOUR_TOKEN')
sa.track(distinct_id='USER_ID', event_name='ProductView', properties={'product_name': 'Example', 'price': 5999})
sa.flush()
sa.close()

Validate SDK Collection

Send an integration_test event and verify each layer from the client request through ClickHouse and Superset.

• Confirm a request reaches /sensors/send/ in client network logs.
• Check ingestion logs for license, format, or property errors.
• Query integration_test in ClickHouse and repeat the check in Superset SQL Lab.
• Validate identity linking, event time, and property types.

SELECT event, distinct_id, time, properties
FROM sensors.event
WHERE event = 'integration_test'
ORDER BY time DESC
LIMIT 10;

6. Verify ClickHouse

After sending a test event, confirm that it is stored in ClickHouse.

SELECT event, count() AS event_count
FROM sensors.event
GROUP BY event
ORDER BY event_count DESC
LIMIT 20;

7. Analyze in Superset

Open Superset and use its ClickHouse connection for SQL Lab, datasets, charts, and dashboards.

• Build DAU, registrations, logins, and event trends.
• Combine funnels, retention, paths, and cohorts.
• Control access by team, project, or dataset.

Updates and Operations

Stop ingestion before replacing a license, restore execute permission, and restart the service.

• Back up MySQL, ClickHouse, and Superset metadata.
• Monitor ingestion failures, disk usage, write latency, and query latency.
• Renew before expiry; purchased terms accumulate after the current end date.