Azure Streaming Analytics enables you to leverage streaming ingestion mode when writing to Azure Data Explorer, delivering latencies as low as less than ten seconds.
The Azure Data Explorer output connector uses streaming ingestion by default. For your job to use streaming ingestion, you must enable streaming ingestion in your Azure Data Explorer cluster under Configurations and define your target database or table to use the streaming ingestion policy. For more information, visit Configure streaming ingestion on your Azure Data Explorer cluster.
Data Ingestion:
For stream data, Azure Data Explorer supports streaming ingestion and queued ingestion.
- Streaming ingestion: This method ensures near-real-time latency for small sets of data per table. Data is ingested in micro batches from a streaming source, initially placed in the row store, and then transferred to column store extents. For more information, see Configure streaming ingestion.
- Queued ingestion: This method is optimized for high ingestion throughput. Data is batched based on ingestion properties, with small batches then merged and optimized for fast query results. By default, the maximum queued values are 5 minutes, 1000 items, or a total size of 1 GB. The data size limit for a queued ingestion command is 6 GB. For more information, see Ingestion batching policy.
Consider using streaming ingestion for the following scenarios:
- Latency of less than a few seconds is required.
- To optimize operational processing of many tables where the stream of data into each table is relatively small (a few records per second), but the overall data ingestion volume is high (thousands of records per second).
If the stream of data into each table is high (over 4 GB per hour), consider using queued ingestion.
