https://spark.apache.org/docs/2.4.3/structured-streaming-programming-guide.html
The Internals of Spark Structured Streaming (Apache Spark 2.4.4)
已读
- 整体介绍
- watermark工作原理
- watermark工作时，需要设置withwatermark与groupby使用相同time field
未读
一、简介
专访朱诗雄：Apache Spark 中的全新流式引擎 Structured Streaming
看了这篇博客，你还敢说不会Structured Streaming？
Note_Spark_Day13：Structured Streaming
Spark教程：Spark Structured Streaming入门编程指南
Spark从入门到精通（09）：结构化流(Structured Streaming)（上）
Spark Structured Streaming高级特性
二、架构
三、watermark
四、state
五、streaming+kafka
官网：Spark Streaming + Kafka Integration Guide (Kafka broker version 0.10.0 or higher)
Tutorial: Use Apache Spark Structured Streaming with Apache Kafka on HDInsight
Processing Data in Apache Kafka with Structured Streaming in Apache Spark 2.2
How to write spark streaming DF to Kafka topic
Apache Kafka transactional writer with foreach sink, is it possible?
Spark : Best way to Broadcast KafkaProducer to Spark streaming
How to write a Dataset to Kafka topic?
六、文件
Spark2.0入门：Structured Streaming操作文件流
Spark2.0入门：Structured Streaming操作文件流
七、config
八、stage
InMemoryTableScanExec Leaf Physical Operator
九、源码
Spark Structrued Streaming源码分析–(三)Aggreation聚合状态存储与更新
十、监控
StreamingQueryListener — Intercepting Life Cycle Events of Streaming Queries
如何使用 Spark 3.0 中新加的 Structured Streaming UI 来进行异常分析
二十、未分类
MicroBatchExecution — Stream Execution Engine of Micro-Batch Stream Processing
spark运行没有启动job
Spark Structured Streaming app has no jobs and no stages
问题：
- spark broadcast kafkaproducer
- dataframe write to kafka each
- structured streaming 流式读取文件 多层
- structured streaming finished without job