Copyright | (c) 2013-2023 Brendan Hay |
---|---|
License | Mozilla Public License, v. 2.0. |
Maintainer | Brendan Hay |
Stability | auto-generated |
Portability | non-portable (GHC extensions) |
Safe Haskell | Safe-Inferred |
Language | Haskell2010 |
Synopsis
- data KafkaStreamingSourceOptions = KafkaStreamingSourceOptions' {
- assign :: Maybe Text
- bootstrapServers :: Maybe Text
- classification :: Maybe Text
- connectionName :: Maybe Text
- delimiter :: Maybe Text
- endingOffsets :: Maybe Text
- maxOffsetsPerTrigger :: Maybe Natural
- minPartitions :: Maybe Natural
- numRetries :: Maybe Natural
- pollTimeoutMs :: Maybe Natural
- retryIntervalMs :: Maybe Natural
- securityProtocol :: Maybe Text
- startingOffsets :: Maybe Text
- subscribePattern :: Maybe Text
- topicName :: Maybe Text
- newKafkaStreamingSourceOptions :: KafkaStreamingSourceOptions
- kafkaStreamingSourceOptions_assign :: Lens' KafkaStreamingSourceOptions (Maybe Text)
- kafkaStreamingSourceOptions_bootstrapServers :: Lens' KafkaStreamingSourceOptions (Maybe Text)
- kafkaStreamingSourceOptions_classification :: Lens' KafkaStreamingSourceOptions (Maybe Text)
- kafkaStreamingSourceOptions_connectionName :: Lens' KafkaStreamingSourceOptions (Maybe Text)
- kafkaStreamingSourceOptions_delimiter :: Lens' KafkaStreamingSourceOptions (Maybe Text)
- kafkaStreamingSourceOptions_endingOffsets :: Lens' KafkaStreamingSourceOptions (Maybe Text)
- kafkaStreamingSourceOptions_maxOffsetsPerTrigger :: Lens' KafkaStreamingSourceOptions (Maybe Natural)
- kafkaStreamingSourceOptions_minPartitions :: Lens' KafkaStreamingSourceOptions (Maybe Natural)
- kafkaStreamingSourceOptions_numRetries :: Lens' KafkaStreamingSourceOptions (Maybe Natural)
- kafkaStreamingSourceOptions_pollTimeoutMs :: Lens' KafkaStreamingSourceOptions (Maybe Natural)
- kafkaStreamingSourceOptions_retryIntervalMs :: Lens' KafkaStreamingSourceOptions (Maybe Natural)
- kafkaStreamingSourceOptions_securityProtocol :: Lens' KafkaStreamingSourceOptions (Maybe Text)
- kafkaStreamingSourceOptions_startingOffsets :: Lens' KafkaStreamingSourceOptions (Maybe Text)
- kafkaStreamingSourceOptions_subscribePattern :: Lens' KafkaStreamingSourceOptions (Maybe Text)
- kafkaStreamingSourceOptions_topicName :: Lens' KafkaStreamingSourceOptions (Maybe Text)
Documentation
data KafkaStreamingSourceOptions Source #
Additional options for streaming.
See: newKafkaStreamingSourceOptions
smart constructor.
KafkaStreamingSourceOptions' | |
|
Instances
newKafkaStreamingSourceOptions :: KafkaStreamingSourceOptions Source #
Create a value of KafkaStreamingSourceOptions
with all optional fields omitted.
Use generic-lens or optics to modify other optional fields.
The following record fields are available, with the corresponding lenses provided for backwards compatibility:
$sel:assign:KafkaStreamingSourceOptions'
, kafkaStreamingSourceOptions_assign
- The specific TopicPartitions
to consume. You must specify at least one
of "topicName"
, "assign"
or "subscribePattern"
.
$sel:bootstrapServers:KafkaStreamingSourceOptions'
, kafkaStreamingSourceOptions_bootstrapServers
- A list of bootstrap server URLs, for example, as
b-1.vpc-test-2.o4q88o.c6.kafka.us-east-1.amazonaws.com:9094
. This
option must be specified in the API call or defined in the table
metadata in the Data Catalog.
$sel:classification:KafkaStreamingSourceOptions'
, kafkaStreamingSourceOptions_classification
- An optional classification.
$sel:connectionName:KafkaStreamingSourceOptions'
, kafkaStreamingSourceOptions_connectionName
- The name of the connection.
$sel:delimiter:KafkaStreamingSourceOptions'
, kafkaStreamingSourceOptions_delimiter
- Specifies the delimiter character.
$sel:endingOffsets:KafkaStreamingSourceOptions'
, kafkaStreamingSourceOptions_endingOffsets
- The end point when a batch query is ended. Possible values are either
"latest"
or a JSON string that specifies an ending offset for each
TopicPartition
.
$sel:maxOffsetsPerTrigger:KafkaStreamingSourceOptions'
, kafkaStreamingSourceOptions_maxOffsetsPerTrigger
- The rate limit on the maximum number of offsets that are processed per
trigger interval. The specified total number of offsets is
proportionally split across topicPartitions
of different volumes. The
default value is null, which means that the consumer reads all offsets
until the known latest offset.
$sel:minPartitions:KafkaStreamingSourceOptions'
, kafkaStreamingSourceOptions_minPartitions
- The desired minimum number of partitions to read from Kafka. The default
value is null, which means that the number of spark partitions is equal
to the number of Kafka partitions.
$sel:numRetries:KafkaStreamingSourceOptions'
, kafkaStreamingSourceOptions_numRetries
- The number of times to retry before failing to fetch Kafka offsets. The
default value is 3
.
$sel:pollTimeoutMs:KafkaStreamingSourceOptions'
, kafkaStreamingSourceOptions_pollTimeoutMs
- The timeout in milliseconds to poll data from Kafka in Spark job
executors. The default value is 512
.
$sel:retryIntervalMs:KafkaStreamingSourceOptions'
, kafkaStreamingSourceOptions_retryIntervalMs
- The time in milliseconds to wait before retrying to fetch Kafka offsets.
The default value is 10
.
$sel:securityProtocol:KafkaStreamingSourceOptions'
, kafkaStreamingSourceOptions_securityProtocol
- The protocol used to communicate with brokers. The possible values are
"SSL"
or "PLAINTEXT"
.
$sel:startingOffsets:KafkaStreamingSourceOptions'
, kafkaStreamingSourceOptions_startingOffsets
- The starting position in the Kafka topic to read data from. The possible
values are "earliest"
or "latest"
. The default value is
"latest"
.
$sel:subscribePattern:KafkaStreamingSourceOptions'
, kafkaStreamingSourceOptions_subscribePattern
- A Java regex string that identifies the topic list to subscribe to. You
must specify at least one of "topicName"
, "assign"
or
"subscribePattern"
.
$sel:topicName:KafkaStreamingSourceOptions'
, kafkaStreamingSourceOptions_topicName
- The topic name as specified in Apache Kafka. You must specify at least
one of "topicName"
, "assign"
or "subscribePattern"
.
kafkaStreamingSourceOptions_assign :: Lens' KafkaStreamingSourceOptions (Maybe Text) Source #
The specific TopicPartitions
to consume. You must specify at least one
of "topicName"
, "assign"
or "subscribePattern"
.
kafkaStreamingSourceOptions_bootstrapServers :: Lens' KafkaStreamingSourceOptions (Maybe Text) Source #
A list of bootstrap server URLs, for example, as
b-1.vpc-test-2.o4q88o.c6.kafka.us-east-1.amazonaws.com:9094
. This
option must be specified in the API call or defined in the table
metadata in the Data Catalog.
kafkaStreamingSourceOptions_classification :: Lens' KafkaStreamingSourceOptions (Maybe Text) Source #
An optional classification.
kafkaStreamingSourceOptions_connectionName :: Lens' KafkaStreamingSourceOptions (Maybe Text) Source #
The name of the connection.
kafkaStreamingSourceOptions_delimiter :: Lens' KafkaStreamingSourceOptions (Maybe Text) Source #
Specifies the delimiter character.
kafkaStreamingSourceOptions_endingOffsets :: Lens' KafkaStreamingSourceOptions (Maybe Text) Source #
The end point when a batch query is ended. Possible values are either
"latest"
or a JSON string that specifies an ending offset for each
TopicPartition
.
kafkaStreamingSourceOptions_maxOffsetsPerTrigger :: Lens' KafkaStreamingSourceOptions (Maybe Natural) Source #
The rate limit on the maximum number of offsets that are processed per
trigger interval. The specified total number of offsets is
proportionally split across topicPartitions
of different volumes. The
default value is null, which means that the consumer reads all offsets
until the known latest offset.
kafkaStreamingSourceOptions_minPartitions :: Lens' KafkaStreamingSourceOptions (Maybe Natural) Source #
The desired minimum number of partitions to read from Kafka. The default value is null, which means that the number of spark partitions is equal to the number of Kafka partitions.
kafkaStreamingSourceOptions_numRetries :: Lens' KafkaStreamingSourceOptions (Maybe Natural) Source #
The number of times to retry before failing to fetch Kafka offsets. The
default value is 3
.
kafkaStreamingSourceOptions_pollTimeoutMs :: Lens' KafkaStreamingSourceOptions (Maybe Natural) Source #
The timeout in milliseconds to poll data from Kafka in Spark job
executors. The default value is 512
.
kafkaStreamingSourceOptions_retryIntervalMs :: Lens' KafkaStreamingSourceOptions (Maybe Natural) Source #
The time in milliseconds to wait before retrying to fetch Kafka offsets.
The default value is 10
.
kafkaStreamingSourceOptions_securityProtocol :: Lens' KafkaStreamingSourceOptions (Maybe Text) Source #
The protocol used to communicate with brokers. The possible values are
"SSL"
or "PLAINTEXT"
.
kafkaStreamingSourceOptions_startingOffsets :: Lens' KafkaStreamingSourceOptions (Maybe Text) Source #
The starting position in the Kafka topic to read data from. The possible
values are "earliest"
or "latest"
. The default value is
"latest"
.
kafkaStreamingSourceOptions_subscribePattern :: Lens' KafkaStreamingSourceOptions (Maybe Text) Source #
A Java regex string that identifies the topic list to subscribe to. You
must specify at least one of "topicName"
, "assign"
or
"subscribePattern"
.
kafkaStreamingSourceOptions_topicName :: Lens' KafkaStreamingSourceOptions (Maybe Text) Source #
The topic name as specified in Apache Kafka. You must specify at least
one of "topicName"
, "assign"
or "subscribePattern"
.