Copyright | (c) 2013-2023 Brendan Hay |
---|---|
License | Mozilla Public License, v. 2.0. |
Maintainer | Brendan Hay |
Stability | auto-generated |
Portability | non-portable (GHC extensions) |
Safe Haskell | Safe-Inferred |
Language | Haskell2010 |
Synopsis
- data S3CsvSource = S3CsvSource' {
- additionalOptions :: Maybe S3DirectSourceAdditionalOptions
- compressionType :: Maybe CompressionType
- escaper :: Maybe Text
- exclusions :: Maybe [Text]
- groupFiles :: Maybe Text
- groupSize :: Maybe Text
- maxBand :: Maybe Natural
- maxFilesInBand :: Maybe Natural
- multiline :: Maybe Bool
- optimizePerformance :: Maybe Bool
- outputSchemas :: Maybe [GlueSchema]
- recurse :: Maybe Bool
- skipFirst :: Maybe Bool
- withHeader :: Maybe Bool
- writeHeader :: Maybe Bool
- name :: Text
- paths :: [Text]
- separator :: Separator
- quoteChar :: QuoteChar
- newS3CsvSource :: Text -> Separator -> QuoteChar -> S3CsvSource
- s3CsvSource_additionalOptions :: Lens' S3CsvSource (Maybe S3DirectSourceAdditionalOptions)
- s3CsvSource_compressionType :: Lens' S3CsvSource (Maybe CompressionType)
- s3CsvSource_escaper :: Lens' S3CsvSource (Maybe Text)
- s3CsvSource_exclusions :: Lens' S3CsvSource (Maybe [Text])
- s3CsvSource_groupFiles :: Lens' S3CsvSource (Maybe Text)
- s3CsvSource_groupSize :: Lens' S3CsvSource (Maybe Text)
- s3CsvSource_maxBand :: Lens' S3CsvSource (Maybe Natural)
- s3CsvSource_maxFilesInBand :: Lens' S3CsvSource (Maybe Natural)
- s3CsvSource_multiline :: Lens' S3CsvSource (Maybe Bool)
- s3CsvSource_optimizePerformance :: Lens' S3CsvSource (Maybe Bool)
- s3CsvSource_outputSchemas :: Lens' S3CsvSource (Maybe [GlueSchema])
- s3CsvSource_recurse :: Lens' S3CsvSource (Maybe Bool)
- s3CsvSource_skipFirst :: Lens' S3CsvSource (Maybe Bool)
- s3CsvSource_withHeader :: Lens' S3CsvSource (Maybe Bool)
- s3CsvSource_writeHeader :: Lens' S3CsvSource (Maybe Bool)
- s3CsvSource_name :: Lens' S3CsvSource Text
- s3CsvSource_paths :: Lens' S3CsvSource [Text]
- s3CsvSource_separator :: Lens' S3CsvSource Separator
- s3CsvSource_quoteChar :: Lens' S3CsvSource QuoteChar
Documentation
data S3CsvSource Source #
Specifies a command-separated value (CSV) data store stored in Amazon S3.
See: newS3CsvSource
smart constructor.
S3CsvSource' | |
|
Instances
:: Text | |
-> Separator | |
-> QuoteChar | |
-> S3CsvSource |
Create a value of S3CsvSource
with all optional fields omitted.
Use generic-lens or optics to modify other optional fields.
The following record fields are available, with the corresponding lenses provided for backwards compatibility:
$sel:additionalOptions:S3CsvSource'
, s3CsvSource_additionalOptions
- Specifies additional connection options.
$sel:compressionType:S3CsvSource'
, s3CsvSource_compressionType
- Specifies how the data is compressed. This is generally not necessary if
the data has a standard file extension. Possible values are "gzip"
and "bzip"
).
$sel:escaper:S3CsvSource'
, s3CsvSource_escaper
- Specifies a character to use for escaping. This option is used only when
reading CSV files. The default value is none
. If enabled, the
character which immediately follows is used as-is, except for a small
set of well-known escapes (\n
, \r
, \t
, and \0
).
$sel:exclusions:S3CsvSource'
, s3CsvSource_exclusions
- A string containing a JSON list of Unix-style glob patterns to exclude.
For example, "[\"**.pdf\"]" excludes all PDF files.
$sel:groupFiles:S3CsvSource'
, s3CsvSource_groupFiles
- Grouping files is turned on by default when the input contains more than
50,000 files. To turn on grouping with fewer than 50,000 files, set this
parameter to "inPartition". To disable grouping when there are more
than 50,000 files, set this parameter to "none"
.
$sel:groupSize:S3CsvSource'
, s3CsvSource_groupSize
- The target group size in bytes. The default is computed based on the
input data size and the size of your cluster. When there are fewer than
50,000 input files, "groupFiles"
must be set to "inPartition"
for this to take effect.
$sel:maxBand:S3CsvSource'
, s3CsvSource_maxBand
- This option controls the duration in milliseconds after which the s3
listing is likely to be consistent. Files with modification timestamps
falling within the last maxBand milliseconds are tracked specially when
using JobBookmarks to account for Amazon S3 eventual consistency. Most
users don't need to set this option. The default is 900000
milliseconds, or 15 minutes.
$sel:maxFilesInBand:S3CsvSource'
, s3CsvSource_maxFilesInBand
- This option specifies the maximum number of files to save from the last
maxBand seconds. If this number is exceeded, extra files are skipped and
only processed in the next job run.
$sel:multiline:S3CsvSource'
, s3CsvSource_multiline
- A Boolean value that specifies whether a single record can span multiple
lines. This can occur when a field contains a quoted new-line character.
You must set this option to True if any record spans multiple lines. The
default value is False
, which allows for more aggressive
file-splitting during parsing.
$sel:optimizePerformance:S3CsvSource'
, s3CsvSource_optimizePerformance
- A Boolean value that specifies whether to use the advanced SIMD CSV
reader along with Apache Arrow based columnar memory formats. Only
available in Glue version 3.0.
$sel:outputSchemas:S3CsvSource'
, s3CsvSource_outputSchemas
- Specifies the data schema for the S3 CSV source.
$sel:recurse:S3CsvSource'
, s3CsvSource_recurse
- If set to true, recursively reads files in all subdirectories under the
specified paths.
$sel:skipFirst:S3CsvSource'
, s3CsvSource_skipFirst
- A Boolean value that specifies whether to skip the first data line. The
default value is False
.
$sel:withHeader:S3CsvSource'
, s3CsvSource_withHeader
- A Boolean value that specifies whether to treat the first line as a
header. The default value is False
.
$sel:writeHeader:S3CsvSource'
, s3CsvSource_writeHeader
- A Boolean value that specifies whether to write the header to output.
The default value is True
.
$sel:name:S3CsvSource'
, s3CsvSource_name
- The name of the data store.
$sel:paths:S3CsvSource'
, s3CsvSource_paths
- A list of the Amazon S3 paths to read from.
$sel:separator:S3CsvSource'
, s3CsvSource_separator
- Specifies the delimiter character. The default is a comma: ",", but
any other character can be specified.
$sel:quoteChar:S3CsvSource'
, s3CsvSource_quoteChar
- Specifies the character to use for quoting. The default is a double
quote: '"'
. Set this to -1
to turn off quoting entirely.
s3CsvSource_additionalOptions :: Lens' S3CsvSource (Maybe S3DirectSourceAdditionalOptions) Source #
Specifies additional connection options.
s3CsvSource_compressionType :: Lens' S3CsvSource (Maybe CompressionType) Source #
Specifies how the data is compressed. This is generally not necessary if
the data has a standard file extension. Possible values are "gzip"
and "bzip"
).
s3CsvSource_escaper :: Lens' S3CsvSource (Maybe Text) Source #
Specifies a character to use for escaping. This option is used only when
reading CSV files. The default value is none
. If enabled, the
character which immediately follows is used as-is, except for a small
set of well-known escapes (\n
, \r
, \t
, and \0
).
s3CsvSource_exclusions :: Lens' S3CsvSource (Maybe [Text]) Source #
A string containing a JSON list of Unix-style glob patterns to exclude. For example, "[\"**.pdf\"]" excludes all PDF files.
s3CsvSource_groupFiles :: Lens' S3CsvSource (Maybe Text) Source #
Grouping files is turned on by default when the input contains more than
50,000 files. To turn on grouping with fewer than 50,000 files, set this
parameter to "inPartition". To disable grouping when there are more
than 50,000 files, set this parameter to "none"
.
s3CsvSource_groupSize :: Lens' S3CsvSource (Maybe Text) Source #
The target group size in bytes. The default is computed based on the
input data size and the size of your cluster. When there are fewer than
50,000 input files, "groupFiles"
must be set to "inPartition"
for this to take effect.
s3CsvSource_maxBand :: Lens' S3CsvSource (Maybe Natural) Source #
This option controls the duration in milliseconds after which the s3 listing is likely to be consistent. Files with modification timestamps falling within the last maxBand milliseconds are tracked specially when using JobBookmarks to account for Amazon S3 eventual consistency. Most users don't need to set this option. The default is 900000 milliseconds, or 15 minutes.
s3CsvSource_maxFilesInBand :: Lens' S3CsvSource (Maybe Natural) Source #
This option specifies the maximum number of files to save from the last maxBand seconds. If this number is exceeded, extra files are skipped and only processed in the next job run.
s3CsvSource_multiline :: Lens' S3CsvSource (Maybe Bool) Source #
A Boolean value that specifies whether a single record can span multiple
lines. This can occur when a field contains a quoted new-line character.
You must set this option to True if any record spans multiple lines. The
default value is False
, which allows for more aggressive
file-splitting during parsing.
s3CsvSource_optimizePerformance :: Lens' S3CsvSource (Maybe Bool) Source #
A Boolean value that specifies whether to use the advanced SIMD CSV reader along with Apache Arrow based columnar memory formats. Only available in Glue version 3.0.
s3CsvSource_outputSchemas :: Lens' S3CsvSource (Maybe [GlueSchema]) Source #
Specifies the data schema for the S3 CSV source.
s3CsvSource_recurse :: Lens' S3CsvSource (Maybe Bool) Source #
If set to true, recursively reads files in all subdirectories under the specified paths.
s3CsvSource_skipFirst :: Lens' S3CsvSource (Maybe Bool) Source #
A Boolean value that specifies whether to skip the first data line. The
default value is False
.
s3CsvSource_withHeader :: Lens' S3CsvSource (Maybe Bool) Source #
A Boolean value that specifies whether to treat the first line as a
header. The default value is False
.
s3CsvSource_writeHeader :: Lens' S3CsvSource (Maybe Bool) Source #
A Boolean value that specifies whether to write the header to output.
The default value is True
.
s3CsvSource_name :: Lens' S3CsvSource Text Source #
The name of the data store.
s3CsvSource_paths :: Lens' S3CsvSource [Text] Source #
A list of the Amazon S3 paths to read from.
s3CsvSource_separator :: Lens' S3CsvSource Separator Source #
Specifies the delimiter character. The default is a comma: ",", but any other character can be specified.
s3CsvSource_quoteChar :: Lens' S3CsvSource QuoteChar Source #
Specifies the character to use for quoting. The default is a double
quote: '"'
. Set this to -1
to turn off quoting entirely.