UnityCatalog

Namespace: Polars.FSharp

Databricks Unity Catalog Client

Constructors & Static Methods

new

new(workspaceUrl: string, bearerToken: string) : UnityCatalog

Parameters

Name	Type	Description
workspaceUrl	string	-
bearerToken	string	-

---

Instance Methods

AddFeature

Adds a feature to the Unity Catalog Delta table.

member AddFeature(
    catalogName: string,
    schemaName: string,
    tableName: string,
    featureName: string,
    ?allowProtocolIncrease: bool,
    ?cloudOptions: CloudOptions
) : unit

Parameters

Name	Type	Description
catalogName	string	The catalog name.
schemaName	string	The schema name.
tableName	string	The table name.
featureName	string	The feature name (e.g. "deletionVectors").
allowProtocolIncrease	bool (optional)	If true, allows the operation to upgrade the table's Protocol Version if the feature requires it.
cloudOptions	CloudOptions (optional)	Optional cloud storage options and retry policies.

---

CreateCatalogTable

Creates a new table registration within the Unity Catalog.

member CreateCatalogTable(
    catalogName: string,
    schemaName: string,
    tableName: string,
    polarsSchema: PolarsSchema,
    ?tableType: CatalogTableType,
    ?storageLocation: string
) : unit

Parameters

Name	Type	Description
catalogName	string	The name of the catalog where the table will be created.
schemaName	string	The name of the schema (database) within the catalog.
tableName	string	The name of the table to be created.
polarsSchema	PolarsSchema	The structural definition of the table columns and data types via PolarsSchema.
tableType	CatalogTableType (optional)	The management type of the table. CatalogTableType.Managed: Unity Catalog manages both metadata and physical data. CatalogTableType.External: Unity Catalog manages metadata, but data resides at a specific storageLocation.
storageLocation	string (optional)	The physical URI for the table data (e.g., s3://my-bucket/path/). Required if tableType is set to External; typically null for Managed tables.

---

DeleteCatalogRecords

Deletes records from a Unity Catalog table that match the specified predicate. This operation is ACID-compliant and leverages the underlying Delta Lake protocol. If Deletion Vectors (DVs) are enabled on the table, it performs a metadata-only delete; otherwise, it performs a physical rewrite of the affected data files.

member DeleteCatalogRecords(
    catalogName: string,
    schemaName: string,
    tableName: string,
    predicate: Expr,
    ?cloudOptions: CloudOptions
) : unit

Parameters

Name	Type	Description
catalogName	string	The name of the catalog containing the table.
schemaName	string	The name of the schema (database) containing the table.
tableName	string	The name of the table from which records will be removed.
predicate	Expr	A boolean Expr defining the condition for rows to be deleted.
cloudOptions	CloudOptions (optional)	Cloud-specific configurations including storage credentials and retry policies for concurrent conflicts.

---

DeleteCatalogTable

Deletes (drops) a table from the Unity Catalog. For Managed tables, this typically removes both the metadata and the underlying physical data. For External tables, only the metadata registration is removed, leaving the physical data intact in your cloud storage.

member DeleteCatalogTable(catalogName: string, schemaName: string, tableName: string) : unit

Parameters

Name	Type	Description
catalogName	string	The name of the catalog containing the table.
schemaName	string	The name of the schema (database) containing the table.
tableName	string	The name of the table to be deleted.

---

DeltaHistory

Retrieves the commit history (audit trail) for a Unity Catalog Delta table as a DataFrame. This includes metadata about who performed what operation, when it occurred, and what parameters were used.

member DeltaHistory(
    catalogName: string,
    schemaName: string,
    tableName: string,
    ?limit: int,
    ?cloudOptions: CloudOptions
) : DataFrame

Parameters

Name	Type	Description
catalogName	string	The name of the catalog (e.g., "main").
schemaName	string	The name of the schema/database (e.g., "default").
tableName	string	The name of the table to audit.
limit	int (optional)	The maximum number of history commits to retrieve. If 0, retrieves all available history.
cloudOptions	CloudOptions (optional)	Cloud-specific configurations and credentials resolved via Unity Catalog.

Returns: A DataFrame containing the flattened and formatted history of the table.

NOTE

Data Processing: The raw JSON history from the Delta log is automatically processed:

• Timestamps are cast to UTC DataType.Datetime.
• Complex fields like operationMetrics and operationParameters are unnested into flat columns.
• Columns are reordered to place essential information (version, timestamp, operation) at the beginning.

Note on Versioning: For standard DML operations, the version column might be null in the raw log; rows are sorted by version or timestamp in descending order (latest first) by default.

---

DeltaRestore

Restores a Unity Catalog Delta table to a previous state based on a version number or a timestamp. This operation is atomic and ACID-compliant.

member DeltaRestore(
    catalogName: string,
    schemaName: string,
    tableName: string,
    ?version: uint64,
    ?timestamp: DateTime,
    ?ignoreMissingFiles: bool,
    ?protocolDowngradeAllowed: bool,
    ?cloudOptions: CloudOptions
) : uint64

Parameters

Name	Type	Description
catalogName	string	The name of the catalog (e.g., "main").
schemaName	string	The name of the schema/database (e.g., "default").
tableName	string	The name of the table to restore.
version	uint64 (optional)	The specific Delta table version to restore to. Mutually exclusive with timestamp.
timestamp	DateTime (optional)	The specific DateTime to restore the table state to. Mutually exclusive with version.
ignoreMissingFiles	bool (optional)	If true, allows the restore to proceed even if some underlying data files are missing from storage. Use with caution as this may result in a partial restoration.
protocolDowngradeAllowed	bool (optional)	If true, allows the table to be restored to a version that requires a lower protocol reader/writer version than the current one.
cloudOptions	CloudOptions (optional)	Cloud-specific configurations and credentials resolved via Unity Catalog.

Returns: The new version number of the table after the restore operation is committed.

NOTE

How it works: Instead of physically deleting newer data, Restore creates a new commit in the transaction log that reverts the table's metadata and file references to the target state.

Safety: This operation will fail if any required data files have been physically removed (e.g., by a Vacuum operation) unless ignoreMissingFiles is set to true.

---

DeltaVacuum

Physically deletes data files that are no longer referenced by the Delta table and are older than the specified retention threshold. This operation helps reduce storage costs and ensures compliance by removing historical data that is no longer needed.

member DeltaVacuum(
    catalogName: string,
    schemaName: string,
    tableName: string,
    ?retentionHours: int,
    ?enforceRetention: bool,
    ?dryRun: bool,
    ?vacuumModeFull: bool,
    ?cloudOptions: CloudOptions
) : int64

Parameters

Name	Type	Description
catalogName	string	The name of the catalog (e.g., "main").
schemaName	string	The name of the schema/database (e.g., "default").
tableName	string	The name of the table to vacuum.
retentionHours	int (optional)	The number of hours to retain historical files. Files modified within this window will not be deleted. If null, the table's default retention setting is used.
enforceRetention	bool (optional)	If true, the operation will fail if retentionHours is less than the table's minimum retention (usually 168h). Set to false only for emergency cleanup, at the risk of corrupting active readers.
dryRun	bool (optional)	If true, the method returns the number of files that would be deleted without actually removing them.
vacuumModeFull	bool (optional)	If true, performs a full scan of the object store to find unreferenced files. If false (Lite mode), uses the Delta log to identify files to be removed.
cloudOptions	CloudOptions (optional)	Cloud-specific configurations and credentials resolved via Unity Catalog.

Returns: The total number of physical files deleted from storage (or identified for deletion if dryRun is true).

NOTE

Warning: This is a destructive operation. Once files are vacuumed, you can no longer "Time Travel" to versions that relied on those files.

Retention Period: By default, Delta Lake prevents deleting files younger than 168 hours (7 days) to protect active readers and concurrent transactions.

---

MergeCatalogRecords

Initializes a MERGE (Upsert) operation against a Unity Catalog table using a LazyFrame as the data source. This method provides an ACID-compliant way to perform atomic updates, inserts, or deletes by joining the source data with the target table.

Initializes a MERGE (Upsert) operation against a Unity Catalog table using an in-memory DataFrame. The sourceData is automatically converted to a LazyFrame to leverage Polars' query optimization.

member MergeCatalogRecords(
    catalogName: string,
    schemaName: string,
    tableName: string,
    sourceData: LazyFrame,
    mergeKeys: string[],
    ?canEvolve: bool,
    ?cloudOptions: CloudOptions
) : DeltaMergeBuilder

member MergeCatalogRecords(
    catalogName: string,
    schemaName: string,
    tableName: string,
    sourceData: DataFrame,
    mergeKeys: string[],
    ?canEvolve: bool,
    ?cloudOptions: CloudOptions
) : DeltaMergeBuilder

Parameters

Name	Type	Description
catalogName	string	-
schemaName	string	-
tableName	string	-
sourceData	LazyFrame / DataFrame	-
mergeKeys	string[]	-
canEvolve	bool (optional)	-
cloudOptions	CloudOptions (optional)	-

NOTE

This is a builder pattern. No data is modified until .Execute() is called on the resulting DeltaMergeBuilder. The operation performs a join between the source and target based on the mergeKeys. This is a builder pattern. No data is modified until .Execute() is called on the resulting DeltaMergeBuilder.

---

OptimizeCatalogTable

Optimizes the storage layout of a Unity Catalog Delta table by compacting small files and optionally applying Z-Order clustering. This operation significantly improves read performance by reducing file-open overhead and enabling efficient data skipping.

member OptimizeCatalogTable(
    catalogName: string,
    schemaName: string,
    tableName: string,
    ?targetSizeMb: int64,
    ?partitionFilters: Map<string, string>,
    ?zOrderColumns: seq<string>,
    ?cloudOptions: CloudOptions
) : int64

Parameters

Name	Type	Description
catalogName	string	The name of the catalog (e.g., "main").
schemaName	string	The name of the schema/database (e.g., "default").
tableName	string	The name of the table to optimize.
targetSizeMb	int64 (optional)	The target size for the compacted Parquet files in Megabytes. Default is 128 MB.
partitionFilters	Map<string, string> (optional)	Optional filters to limit the optimization to specific partitions. If null, the entire table is processed.
zOrderColumns	seq<string> (optional)	An optional collection of column names to use for Z-Order clustering.
cloudOptions	CloudOptions (optional)	Cloud-specific configurations, including storage credentials and retry logic.

Returns: The total number of files that were successfully compacted/optimized during the operation.

NOTE

Compaction: Groups small Parquet files into larger ones based on the target size.

Z-Ordering: If zOrderColumns are provided, the data is rearranged into multidimensional clusters.

Deletion Vector (DV) Materialization: If the table uses DVs, this operation physically removes any soft-deleted rows.

---

ReadChangeDataFeed

Reads the Change Data Feed (CDC) stream for a Unity Catalog Delta table between the specified versions. The table must have 'delta.enableChangeDataFeed' set to 'true'.

Reads the Change Data Feed (CDC) stream for a Unity Catalog Delta table between the specified datetimes. The table must have 'delta.enableChangeDataFeed' set to 'true'.

member ReadChangeDataFeed(
    catalogName: string,
    schemaName: string,
    tableName: string,
    startVersion: int64,
    endVersion: int64,
    ?cloudOptions: CloudOptions
) : LazyFrame

member ReadChangeDataFeed(
    catalogName: string,
    schemaName: string,
    tableName: string,
    startTimestamp: DateTime,
    ?endTimestamp: DateTime,
    ?cloudOptions: CloudOptions
) : LazyFrame

Parameters

Name	Type	Description
catalogName	string	The name of the catalog.
schemaName	string	The name of the schema/database.
tableName	string	The name of the table.
startVersion	int64	The starting version (inclusive).
endVersion	int64	The ending version (inclusive).
cloudOptions	CloudOptions (optional)	Optional cloud storage credentials and retry policies.
startTimestamp	DateTime	The starting datetime (inclusive).
endTimestamp	DateTime (optional)	Optional. The ending datetime (inclusive). Defaults to DateTime.UtcNow if null.

Returns: A LazyFrame containing the change data stream. / A LazyFrame containing the change data stream filtered by the exact milliseconds.

---

ScanCatalogTable

Scans a table managed by Unity Catalog into a LazyFrame. This method performs a two-step resolution: 1. It fetches the physical storage location and dynamic credentials from Unity Catalog. 2. It initializes a Polars scan on the underlying Delta files using the resolved credentials.

member ScanCatalogTable(
    catalogName: string,
    schemaName: string,
    tableName: string,
    ?version: uint64,
    ?datetime: string,
    ?nRows: uint64,
    ?parallelStrategy: ParallelStrategy,
    ?lowMemory: bool,
    ?useStatistics: bool,
    ?glob: bool,
    ?rechunk: bool,
    ?cache: bool,
    ?rowIndexName: string,
    ?rowIndexOffset: uint32,
    ?includePathColumn: string,
    ?schema: PolarsSchema,
    ?hivePartitioning: bool,
    ?hivePartitionSchema: PolarsSchema,
    ?tryParseHiveDates: bool,
    ?cloudOptions: CloudOptions
) : LazyFrame

Parameters

Name	Type	Description
catalogName	string	-
schemaName	string	-
tableName	string	-
version	uint64 (optional)	-
datetime	string (optional)	-
nRows	uint64 (optional)	-
parallelStrategy	ParallelStrategy (optional)	-
lowMemory	bool (optional)	-
useStatistics	bool (optional)	-
glob	bool (optional)	-
rechunk	bool (optional)	-
cache	bool (optional)	-
rowIndexName	string (optional)	-
rowIndexOffset	uint32 (optional)	-
includePathColumn	string (optional)	-
schema	PolarsSchema (optional)	-
hivePartitioning	bool (optional)	-
hivePartitionSchema	PolarsSchema (optional)	-
tryParseHiveDates	bool (optional)	-
cloudOptions	CloudOptions (optional)	-

---

SinkCatalogTable

Sink the LazyFrame to a Unity Catalog managed Delta Lake table with partition discovery.

member SinkCatalogTable(
    catalogName: string,
    schemaName: string,
    tableName: string,
    lf: LazyFrame,
    ?partitionBy: Selector,
    ?mode: DeltaSaveMode,
    ?canEvolve: bool,
    ?includeKeys: bool,
    ?keysPreGrouped: bool,
    ?maxRowsPerFile: int,
    ?approxBytesPerFile: int64,
    ?compression: ParquetCompression,
    ?compressionLevel: int,
    ?statistics: bool,
    ?rowGroupSize: uint32,
    ?dataPageSize: uint32,
    ?compatLevel: int,
    ?maintainOrder: bool,
    ?syncOnClose: SyncOnClose,
    ?mkdir: bool,
    ?cloudOptions: CloudOptions
) : unit

Parameters

Name	Type	Description
catalogName	string	-
schemaName	string	-
tableName	string	-
lf	LazyFrame	-
partitionBy	Selector (optional)	-
mode	DeltaSaveMode (optional)	-
canEvolve	bool (optional)	-
includeKeys	bool (optional)	-
keysPreGrouped	bool (optional)	-
maxRowsPerFile	int (optional)	-
approxBytesPerFile	int64 (optional)	-
compression	ParquetCompression (optional)	-
compressionLevel	int (optional)	-
statistics	bool (optional)	-
rowGroupSize	uint32 (optional)	-
dataPageSize	uint32 (optional)	-
compatLevel	int (optional)	-
maintainOrder	bool (optional)	-
syncOnClose	SyncOnClose (optional)	-
mkdir	bool (optional)	-
cloudOptions	CloudOptions (optional)	-

---