org.apache.spark.sql.sources.v2.reader

Interface DataReader<T>

All Superinterfaces:

AutoCloseable, java.io.Closeable

All Known Subinterfaces:

ContinuousDataReader<T>

@InterfaceStability.Evolving
public interface DataReader<T>
extends java.io.Closeable

A data reader returned by DataReaderFactory.createDataReader() and is responsible for outputting data for a RDD partition. Note that, Currently the type `T` can only be Row for normal data source readers, or UnsafeRow for data source readers that mix in SupportsScanUnsafeRow.

Method Summary

Modifier and Type	Method and Description
T	get() Return the current record.
boolean	next() Proceed to next record, returns false if there is no more records.

Methods inherited from interface java.io.Closeable

close

Method Detail

next

boolean next()
      throws java.io.IOException

Proceed to next record, returns false if there is no more records. If this method fails (by throwing an exception), the corresponding Spark task would fail and get retried until hitting the maximum retry times.

Throws:

java.io.IOException - if failure happens during disk/network IO like reading files.

get

T get()

Return the current record. This method should return same value until `next` is called. If this method fails (by throwing an exception), the corresponding Spark task would fail and get retried until hitting the maximum retry times.