Delta

Namespace: Polars.FSharp

Methods for DeltaLake

Constructors & Static Methods

AddFeature

Enables a specific feature on the Delta Table.

WARNING: Enabling features may upgrade the Delta Protocol version (e.g., to Reader v3 / Writer v7). Older readers may not be able to read the table after this operation.

fsharp

static member AddFeature(
    path: string,
    featureName: string,
    ?allowProtocolIncrease: bool,
    ?cloudOptions: CloudOptions
) : unit

Parameters

Name	Type	Description
`path`	`string`	The root URI of the Delta table.
`featureName`	`string`	The name of the feature to enable (use `DeltaTableFeatures` constants).
`allowProtocolIncrease`	`bool (optional)`	If true, allows the operation to upgrade the table's Protocol Version if the feature requires it. If false, and the current protocol is too low, the operation will fail.
`cloudOptions`	`CloudOptions (optional)`	Cloud storage credentials.

Delete

Deletes rows from a Delta Lake table that match a given predicate.

By default, this operation performs a Copy-on-Write (CoW), meaning any underlying data files containing matching rows are entirely rewritten.

Note: If Deletion Vectors are enabled on the table, this operation automatically shifts to a Merge-on-Read (MoR) approach. Instead of expensive file rewrites, deleted rows are swiftly marked in a separate deletion vector file, which drastically improves deletion performance.

fsharp

static member Delete(path: string, predicate: Expr, ?cloudOptions: CloudOptions) : unit

Parameters

Name	Type	Description
`path`	`string`	Path to the Delta table.
`predicate`	`Expr`	Filter expression to identify rows to delete.
`cloudOptions`	`CloudOptions (optional)`	Cloud storage configuration.

History

Returns provenance information, including the operation, user, timestamp, etc., for each write to a table.

fsharp

static member History(path: string, ?limit: int, ?cloudOptions: CloudOptions) : DataFrame

Parameters

Name	Type	Description
`path`	`string`	The root URI of the Delta table.
`limit`	`int (optional)`	The number of latest commits to retrieve. Set to 0 (default) to retrieve all history.
`cloudOptions`	`CloudOptions (optional)`	Cloud storage credentials.

Returns: A DataFrame containing the commit history.

Optimize

Optimizes the layout of the Delta table by compacting small files (bin-packing) and optionally applying Z-Order clustering.

This operation significantly improves read performance by reducing the number of files and co-locating related data.

Note: If Deletion Vectors (DV) are enabled on the table, any soft-deleted rows tracked by the vectors will be physically removed (materialized) from the newly compacted Parquet files, effectively clearing the deletion vectors for the optimized partitions.

fsharp

static member Optimize(
    path: string,
    ?targetSizeMb: int64,
    ?partitionFilters: seq<(string * string)>,
    ?zOrderColumns: seq<string>,
    ?cloudOptions: CloudOptions
) : int64

Parameters

Name	Type	Description
`path`	`string`	The root URI of the Delta table.
`targetSizeMb`	`int64 (optional)`	The target file size in Megabytes for the compacted files (default: 128 MB).
`partitionFilters`	`seq<(string * string)> (optional)`	Optional sequence of partition key-value pairs to restrict optimization to specific partitions. Example: `[ ("date", "2024-01-01") ]`
`zOrderColumns`	`seq<string> (optional)`	Optional sequence of column names to apply Z-Order clustering. Z-Ordering co-locates data based on these columns, significantly speeding up queries that filter on them.
`cloudOptions`	`CloudOptions (optional)`	Cloud storage credentials and configuration.

Returns: The number of new data files created during the optimization process.

ReadChangeDataFeed

Reads the Change Data Feed (CDC) stream for a Delta table between the specified versions. The table must have 'delta.enableChangeDataFeed' set to 'true'.

Reads the Change Data Feed (CDC) stream for a Delta table between the specified datetimes. The table must have 'delta.enableChangeDataFeed' set to 'true'.

fsharp

static member ReadChangeDataFeed(
    path: string,
    startVersion: int64,
    endVersion: int64,
    ?cloudOptions: CloudOptions
) : LazyFrame

static member ReadChangeDataFeed(
    path: string,
    startTimestamp: DateTime,
    ?endTimestamp: DateTime,
    ?cloudOptions: CloudOptions
) : LazyFrame

Parameters

Name	Type	Description
`path`	`string`	The URI or local path of the Delta table.
`startVersion`	`int64`	The starting version (inclusive).
`endVersion`	`int64`	The ending version (inclusive).
`cloudOptions`	`CloudOptions (optional)`	Optional cloud storage credentials and retry policies.
`startTimestamp`	`DateTime`	The starting datetime (inclusive).
`endTimestamp`	`DateTime (optional)`	Optional. The ending datetime (inclusive). Defaults to DateTime.UtcNow if null.

Returns: A LazyFrame containing the change data stream. / A LazyFrame containing the change data stream filtered by the exact milliseconds.

Restore

Restores the Delta table to a previous state defined by a specific version or timestamp.

Note: This operation creates a new commit (version) that reflects the state of the table at the target point.

fsharp

static member Restore(
    path: string,
    ?version: uint64,
    ?timestamp: DateTime,
    ?ignoreMissingFiles: bool,
    ?protocolDowngradeAllowed: bool,
    ?cloudOptions: CloudOptions
) : uint64

Parameters

Name	Type	Description
`path`	`string`	The root URI of the Delta table.
`version`	`uint64 (optional)`	The specific version number to restore to. Set to null if restoring by timestamp.
`timestamp`	`DateTime (optional)`	The timestamp to restore to. Set to null if restoring by version.
`ignoreMissingFiles`	`bool (optional)`	If true, the restore operation will ignore missing files (e.g., those deleted by Vacuum). If false, it will fail if any required files are missing.
`protocolDowngradeAllowed`	`bool (optional)`	Whether to allow downgrading the Delta Protocol version during restore (rarely needed).
`cloudOptions`	`CloudOptions (optional)`	Cloud storage credentials.

Returns: The new version number of the table after the restore operation.

SetTableProperties

Sets or updates properties on the Delta Table.

fsharp

static member SetTableProperties(
    path: string,
    properties: seq<(string * string)>,
    ?raiseIfNotExists: bool,
    ?cloudOptions: CloudOptions
) : unit

Parameters

Name	Type	Description
`path`	`string`	The root URI of the Delta table.
`properties`	`seq<(string * string)>`	A sequence of key-value pairs to set.
`raiseIfNotExists`	`bool (optional)`	If true, the operation fails if you try to update a property that doesn't strictly exist (rarely used, default false).
`cloudOptions`	`CloudOptions (optional)`	Cloud storage credentials.

Source

Represents a column from the incoming Source LazyFrame. (Internally maps to "{name}_src_tmp")

fsharp

static member Source(columnName: string) : Expr

Parameters

Name	Type	Description
`columnName`	`string`	-

Target

Represents a column from the existing Target Delta Table. (Alias for standard Col(), adds semantic clarity)

fsharp

static member Target(columnName: string) : Expr

Parameters

Name	Type	Description
`columnName`	`string`	-

Vacuum

Recursively delete files and directories in the table that are no longer needed by the table for maintaining the transaction history.

fsharp

static member Vacuum(
    path: string,
    ?retentionHours: int,
    ?enforceRetention: bool,
    ?dryRun: bool,
    ?vacuumModeFull: bool,
    ?cloudOptions: CloudOptions
) : int64

Parameters

Name	Type	Description
`path`	`string`	The root URI of the Delta table (e.g., "s3://bucket/table").
`retentionHours`	`int (optional)`	The retention threshold in hours. Files removed before this threshold will be deleted. Set to -1 to use the default retention period configured in the table (usually 168 hours / 7 days).
`enforceRetention`	`bool (optional)`	If true, prevents the vacuum command from deleting files that have not yet reached the retention threshold. Setting this to false allows forcing deletion of recent files (use with caution).
`dryRun`	`bool (optional)`	If true, simply returns the number of files that would be deleted, without actually deleting them.
`vacuumModeFull`	`bool (optional)`	Specifies the vacuum mode: false (Lite): Only removes files that are explicitly referenced as removed in the transaction log. Faster. true (Full): Scans the entire storage directory to find and remove any files not referenced in the transaction log (including orphan files from failed writes). Slower but more thorough.
`cloudOptions`	`CloudOptions (optional)`	Cloud storage credentials and configuration (e.g. AWS Keys, Azure Secrets).

Returns: The number of files deleted (or selected for deletion in dry-run mode).

Delta ​

Constructors & Static Methods ​

AddFeature ​

Delete ​

History ​

Optimize ​

ReadChangeDataFeed ​

Restore ​

SetTableProperties ​

Source ​

Target ​

Vacuum ​

Delta

Constructors & Static Methods

AddFeature

Delete

History

Optimize

ReadChangeDataFeed

Restore

SetTableProperties

Source

Target

Vacuum