org.apache.spark.sql.internal

Class CatalogImpl

Object

org.apache.spark.sql.catalog.Catalog

org.apache.spark.sql.internal.CatalogImpl

public class CatalogImpl
extends Catalog

Internal implementation of the user-facing Catalog.

Constructor Summary

Constructors

Constructor and Description
CatalogImpl(SparkSession sparkSession)

Method Summary

Methods

Modifier and Type	Method and Description
void	cacheTable(String tableName) Caches the specified table in-memory.
void	clearCache() Removes all cached tables from the in-memory cache.
Dataset<Row>	createExternalTable(String tableName, String path) :: Experimental :: Creates an external table from the given path and returns the corresponding DataFrame.
Dataset<Row>	createExternalTable(String tableName, String source, java.util.Map<String,String> options) :: Experimental :: Creates an external table from the given path based on a data source and a set of options.
Dataset<Row>	createExternalTable(String tableName, String source, scala.collection.immutable.Map<String,String> options) :: Experimental :: (Scala-specific) Creates an external table from the given path based on a data source and a set of options.
Dataset<Row>	createExternalTable(String tableName, String path, String source) :: Experimental :: Creates an external table from the given path based on a data source and returns the corresponding DataFrame.
Dataset<Row>	createExternalTable(String tableName, String source, StructType schema, java.util.Map<String,String> options) :: Experimental :: Create an external table from the given path based on a data source, a schema and a set of options.
Dataset<Row>	createExternalTable(String tableName, String source, StructType schema, scala.collection.immutable.Map<String,String> options) :: Experimental :: (Scala-specific) Create an external table from the given path based on a data source, a schema and a set of options.
String	currentDatabase() Returns the current default database in this session.
void	dropTempView(String viewName) Drops the temporary view with the given view name in the catalog.
boolean	isCached(String tableName) Returns true if the table is currently cached in-memory.
Dataset<Column>	listColumns(String tableName) Returns a list of columns for the given table in the current database.
Dataset<Column>	listColumns(String dbName, String tableName) Returns a list of columns for the given table in the specified database.
Dataset<Database>	listDatabases() Returns a list of databases available across all sessions.
Dataset<Function>	listFunctions() Returns a list of functions registered in the current database.
Dataset<Function>	listFunctions(String dbName) Returns a list of functions registered in the specified database.
Dataset<Table>	listTables() Returns a list of tables in the current database.
Dataset<Table>	listTables(String dbName) Returns a list of tables in the specified database.
static <T extends org.apache.spark.sql.catalyst.DefinedByConstructorParams> Dataset<T>	makeDataset(scala.collection.Seq<T> data, SparkSession sparkSession, scala.reflect.api.TypeTags.TypeTag<T> evidence$1)
void	refreshByPath(String resourcePath) Refresh the cache entry and the associated metadata for all dataframes (if any), that contain the given data source path.
void	refreshTable(String tableName) Refresh the cache entry for a table, if any.
void	setCurrentDatabase(String dbName) Sets the current default database in this session.
void	uncacheTable(String tableName) Removes the specified table from the in-memory cache.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CatalogImpl

public CatalogImpl(SparkSession sparkSession)

Method Detail

makeDataset

public static <T extends org.apache.spark.sql.catalyst.DefinedByConstructorParams> Dataset<T> makeDataset(scala.collection.Seq<T> data,
                                                                                          SparkSession sparkSession,
                                                                                          scala.reflect.api.TypeTags.TypeTag<T> evidence$1)

currentDatabase

public String currentDatabase()

Returns the current default database in this session.

Specified by:

currentDatabase in class Catalog

Returns:

(undocumented)

setCurrentDatabase

public void setCurrentDatabase(String dbName)
                        throws AnalysisException

Sets the current default database in this session.

Specified by:

setCurrentDatabase in class Catalog

Parameters:

dbName - (undocumented)

Throws:

AnalysisException

listDatabases

public Dataset<Database> listDatabases()

Returns a list of databases available across all sessions.

Specified by:

listDatabases in class Catalog

Returns:

(undocumented)

listTables

public Dataset<Table> listTables()

Returns a list of tables in the current database. This includes all temporary tables.

Specified by:

listTables in class Catalog

Returns:

(undocumented)

listTables

public Dataset<Table> listTables(String dbName)
                          throws AnalysisException

Returns a list of tables in the specified database. This includes all temporary tables.

Specified by:

listTables in class Catalog

Parameters:

dbName - (undocumented)

Returns:

(undocumented)

Throws:

AnalysisException

listFunctions

public Dataset<Function> listFunctions()

Returns a list of functions registered in the current database. This includes all temporary functions

Specified by:

listFunctions in class Catalog

Returns:

(undocumented)

listFunctions

public Dataset<Function> listFunctions(String dbName)
                                throws AnalysisException

Returns a list of functions registered in the specified database. This includes all temporary functions

Specified by:

listFunctions in class Catalog

Parameters:

dbName - (undocumented)

Returns:

(undocumented)

Throws:

AnalysisException

listColumns

public Dataset<Column> listColumns(String tableName)
                            throws AnalysisException

Returns a list of columns for the given table in the current database.

Specified by:

listColumns in class Catalog

Parameters:

tableName - (undocumented)

Returns:

(undocumented)

Throws:

AnalysisException

listColumns

public Dataset<Column> listColumns(String dbName,
                          String tableName)
                            throws AnalysisException

Returns a list of columns for the given table in the specified database.

Specified by:

listColumns in class Catalog

Parameters:

dbName - (undocumented)

tableName - (undocumented)

Returns:

(undocumented)

Throws:

AnalysisException

createExternalTable

public Dataset<Row> createExternalTable(String tableName,
                               String path)

:: Experimental :: Creates an external table from the given path and returns the corresponding DataFrame. It will use the default data source configured by spark.sql.sources.default.

Specified by:

createExternalTable in class Catalog

Parameters:

tableName - (undocumented)

path - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

createExternalTable

public Dataset<Row> createExternalTable(String tableName,
                               String path,
                               String source)

:: Experimental :: Creates an external table from the given path based on a data source and returns the corresponding DataFrame.

Specified by:

createExternalTable in class Catalog

Parameters:

tableName - (undocumented)

path - (undocumented)

source - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

createExternalTable

public Dataset<Row> createExternalTable(String tableName,
                               String source,
                               java.util.Map<String,String> options)

:: Experimental :: Creates an external table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.

Specified by:

createExternalTable in class Catalog

Parameters:

tableName - (undocumented)

source - (undocumented)

options - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

createExternalTable

public Dataset<Row> createExternalTable(String tableName,
                               String source,
                               scala.collection.immutable.Map<String,String> options)

:: Experimental :: (Scala-specific) Creates an external table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.

Specified by:

createExternalTable in class Catalog

Parameters:

tableName - (undocumented)

source - (undocumented)

options - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

createExternalTable

public Dataset<Row> createExternalTable(String tableName,
                               String source,
                               StructType schema,
                               java.util.Map<String,String> options)

:: Experimental :: Create an external table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

Specified by:

createExternalTable in class Catalog

Parameters:

tableName - (undocumented)

source - (undocumented)

schema - (undocumented)

options - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

createExternalTable

public Dataset<Row> createExternalTable(String tableName,
                               String source,
                               StructType schema,
                               scala.collection.immutable.Map<String,String> options)

:: Experimental :: (Scala-specific) Create an external table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

Specified by:

createExternalTable in class Catalog

Parameters:

tableName - (undocumented)

source - (undocumented)

schema - (undocumented)

options - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

dropTempView

public void dropTempView(String viewName)

Drops the temporary view with the given view name in the catalog. If the view has been cached/persisted before, it's also unpersisted.

Specified by:

dropTempView in class Catalog

Parameters:

viewName - the name of the view to be dropped.

Since:

2.0.0

isCached

public boolean isCached(String tableName)

Returns true if the table is currently cached in-memory.

Specified by:

isCached in class Catalog

Parameters:

tableName - (undocumented)

Returns:

(undocumented)

Since:

2.0.0

cacheTable

public void cacheTable(String tableName)

Caches the specified table in-memory.

Specified by:

cacheTable in class Catalog

Parameters:

tableName - (undocumented)

Since:

2.0.0

uncacheTable

public void uncacheTable(String tableName)

Removes the specified table from the in-memory cache.

Specified by:

uncacheTable in class Catalog

Parameters:

tableName - (undocumented)

Since:

2.0.0

clearCache

public void clearCache()

Removes all cached tables from the in-memory cache.

Specified by:

clearCache in class Catalog

Since:

2.0.0

refreshTable

public void refreshTable(String tableName)

Refresh the cache entry for a table, if any. For Hive metastore table, the metadata is refreshed.

Specified by:

refreshTable in class Catalog

Parameters:

tableName - (undocumented)

Since:

2.0.0

refreshByPath

public void refreshByPath(String resourcePath)

Refresh the cache entry and the associated metadata for all dataframes (if any), that contain the given data source path.

Specified by:

refreshByPath in class Catalog

Parameters:

resourcePath - (undocumented)

Since:

2.0.0