Overview
Data Pipelines enable seamless integration with external services and cloud platforms. Stream your IoT data in real-time to analytics tools, databases, data lakes, and third-party applications without writing integration code.
Supported Destinations
Cloud Services
AWS IoT, Azure IoT Hub, Google Cloud IoT, IBM Watson IoT
Databases
PostgreSQL, MongoDB, InfluxDB, TimescaleDB, MySQL
Analytics
Elasticsearch, Grafana, Tableau, Power BI, Google Analytics
Custom Webhooks
Any REST API endpoint with custom headers and authentication
Creating a Data Pipeline
Access Pipelines Section
Navigate to Data Pipelines from the main menu.
Create New Pipeline
Click the add button and provide a descriptive name for your pipeline.
Select Data Source
Choose which devices or device groups will feed data into this pipeline.
Configure Destination
Select destination type and provide connection credentials (API keys, connection strings, endpoints).
Map Data Fields
Define how device data fields map to destination schema. Apply transformations if needed.
Set Delivery Options
Configure batching, frequency, retry policies, and error handling.
Test and Activate
Send test data to verify configuration. Activate the pipeline to begin streaming.
Pipeline Configuration Options
| Option | Description | Use Case |
|---|---|---|
| Batch Size | Number of records per transmission | Optimize for destination API limits |
| Frequency | How often data is sent | Real-time vs. periodic batching |
| Data Filtering | Send only matching conditions | Reduce data volume and costs |
| Transformation | Modify data structure or values | Format conversion, unit changes |
| Retry Policy | Automatic retry on failures | Handle temporary connectivity issues |
| Error Handling | What to do when delivery fails | Dead letter queue, alerts, skip |
Example Pipelines
IoT Data to AWS S3 Data Lake
Source: All temperature sensors
Destination: AWS S3 bucket
Configuration:
- Batch 1000 records or every 5 minutes
- Partition by device ID and date
- Convert to Parquet format for analytics
- Enable server-side encryption
Real-time Analytics to Elasticsearch
Source: All devices
Destination: Elasticsearch cluster
Configuration:
- Real-time streaming (no batching)
- Index by device type and timestamp
- Include metadata for filtering
- Retry up to 3 times on failure
Sensor Data to Custom REST API
Source: Environmental monitoring devices
Destination: Custom webhook endpoint
Configuration:
- POST JSON payload on every reading
- Include Bearer token authentication
- Custom headers for API versioning
- Alert on consecutive failures
Data Transformation
Apply transformations to match destination requirements:
Supported Transformations
- Field Mapping: Rename fields to match destination schema
- Unit Conversion: Convert temperature, pressure, distance units
- Data Type Casting: String to number, timestamp formatting
- Aggregation: Calculate averages, min/max, sums over time windows
- Enrichment: Add device metadata, location, custom tags
- Filtering: Include/exclude specific fields or records
- Custom Scripts: JavaScript transformation for complex logic
Monitoring and Troubleshooting
Pipeline Metrics
Monitor pipeline health and performance:
- Records processed (success/failure counts)
- Throughput (records per second/minute)
- Delivery latency (time from source to destination)
- Error rate and failure reasons
- Queue depth (pending records)
- Cost estimation (for cloud destinations)
Common Issues and Solutions
| Issue | Cause | Solution |
|---|---|---|
| High failure rate | Invalid credentials | Verify API keys and permissions |
| Slow delivery | Small batch size | Increase batch size or frequency |
| Duplicate data | Retry on transient errors | Implement idempotency in destination |
| Missing fields | Incorrect mapping | Review field mapping configuration |
Security Best Practices
- Use secure credential storage (never hardcode secrets)
- Enable TLS/SSL for all connections
- Rotate API keys and credentials regularly
- Apply principle of least privilege (minimal permissions)
- Monitor for unauthorized access attempts
- Encrypt sensitive data before transmission
- Use VPN or private connectivity for database destinations