Safe Haskell	None
Language	Haskell2010

Spark.IO.Inputs

Synopsis

Documentation

data SparkPath Source #

A path to some data that can be read by Spark.

Instances

Eq SparkPath Source #
Methods (==) :: SparkPath -> SparkPath -> Bool # (/=) :: SparkPath -> SparkPath -> Bool #
Show SparkPath Source #
Methods showsPrec :: Int -> SparkPath -> ShowS # show :: SparkPath -> String # showList :: [SparkPath] -> ShowS #
IsString SparkPath Source #
Methods fromString :: String -> SparkPath #
ToJSON SparkPath Source #
Methods toJSON :: SparkPath -> Value # toEncoding :: SparkPath -> Encoding #

data JsonMode Source #

data DataSchema Source #

The schema policty with respect to a data source. It should either request Spark to infer the schema from the source, or it should try to match the source against a schema provided by the user.

Instances

Eq DataSchema Source #
Methods (==) :: DataSchema -> DataSchema -> Bool # (/=) :: DataSchema -> DataSchema -> Bool #
Show DataSchema Source #
Methods showsPrec :: Int -> DataSchema -> ShowS # show :: DataSchema -> String # showList :: [DataSchema] -> ShowS #
ToJSON DataSchema Source #
Methods toJSON :: DataSchema -> Value # toEncoding :: DataSchema -> Encoding #

data JsonOptions Source #

The options for the json input.

data SourceDescription Source #

A description of a data source, following Spark's reader API version 2.

Eeach source constists in an input source (json, xml, etc.), an optional schema for this source, and a number of options specific to this source.

Since this descriptions is rather low-level, a number of wrappers of provided for each of the most popular sources that are already built into Spark.

Instances

Eq SourceDescription Source #
Methods (==) :: SourceDescription -> SourceDescription -> Bool # (/=) :: SourceDescription -> SourceDescription -> Bool #
Show SourceDescription Source #
Methods showsPrec :: Int -> SourceDescription -> ShowS # show :: SourceDescription -> String # showList :: [SourceDescription] -> ShowS #
ToJSON SourceDescription Source #
Methods toJSON :: SourceDescription -> Value # toEncoding :: SourceDescription -> Encoding #

json' :: DataType -> String -> DataFrame Source #

Declares a source of data of the given data type.

The source is not read at this point, it is just declared. It may be found to be invalid in subsequent computations.

json :: SQLTypeable a => String -> Dataset a Source #

Declares a source of data of the given data type.

The source is not read at this point, it is just declared.

jsonInfer :: SparkPath -> SparkState DataFrame Source #

Reads a source of data expected to be in the JSON format.

The schema is not required and Spark will infer the schema of the source. However, all the data contained in the source may end up being read in the process.