Polars.CSharp Idioms & Best Practices

1. Namespace & Static Imports

Two standard import idioms are used across Polars.CSharp:

// Style 1: Explicit Alias (Python-like: Pl.Col, Pl.Lit, Cs.Numeric)
using Polars.CSharp;
using Pl = Polars.CSharp.Polars;
using Cs = Polars.CSharp.Polars.Selectors;

// Style 2: Static Import (Rust-like: Col, Lit, When, Numeric)
using Polars.CSharp;
using static Polars.CSharp.Polars;
using static Polars.CSharp.Polars.Selectors;

---

2. Creating DataFrames & Series

// 1. Direct row ingestion from List<T> / IEnumerable<T> (Most idiomatic .NET pattern)
List<Employee> employees = GetEmployees();
using var df1 = DataFrame.FromRows(employees);

// Convert back to strongly-typed records/classes using high-performance stack enumerator:
List<Employee> list = df1.Rows<Employee>().ToList();

// 2. Direct LINQ queryable pipeline from any in-memory collection:
using Polars.CSharp.Linq;
var topEmployees = employees.AsPolarsQueryable()
    .Where(e => e.Score > 90.0)
    .OrderByDescending(e => e.Score)
    .ToDataFrame();

// Seamless in-memory Join, Concat, and Union without manual conversion:
List<Department> departments = GetDepartments();
using var enrichedDf = employees.AsPolarsQueryable()
    .Join(departments, e => e.DeptId, d => d.Id, (e, d) => new { e.Name, d.DeptName })
    .ToDataFrame();

// 3. From columnar anonymous object (Structure of Arrays)
using var df2 = DataFrame.FromColumns(new
{
    id = new[] { 1, 2, 3 },
    name = new[] { "Alice", "Bob", "Charlie" },
    score = new double?[] { 95.5, null, 88.0 }
});

// 4. From Series (supports spans, arrays, nullable arrays, tensors)
using var df3 = new DataFrame(
    Series.From("id", [1, 2, 3]),
    Series.From("val", [10.0, 20.0, 30.0])
);

// 4. Zero-copy Apache Arrow Interop
using var dfArrow = DataFrame.FromArrow(recordBatch);
using RecordBatch batch = dfArrow.ToArrow();

// 5. C# 12 Collection Expressions (via [CollectionBuilder])
using DataFrame df5 = [
    Series.From("id", [1, 2, 3]),
    Series.From("val", [10.0, 20.0, 30.0])
];

// 6. Direct Column Enumeration (DataFrame implements IEnumerable<Series>)
foreach (Series col in df5)
{
    Console.WriteLine($"Column '{col.Name}' ({col.DataType}): {col.Len()} rows");
}

// 7. Schema Dictionary Protocols (PolarsSchema implements IReadOnlyDictionary<string, DataType>)
PolarsSchema schema = df5.Schema;
DataType valType = schema["val"];
bool hasId = schema.ContainsKey("id");
foreach (var (colName, dt) in schema)
{
    Console.WriteLine($"{colName}: {dt}");
}

---

3. Expressive 2D Indexers (df[...])

DataFrame supports Python/NumPy-style 1D and 2D indexing with Range, Index (^1), boolean mask Series, index Series, column names, and IntoSelector (Cs.*, typeof(...)):

Series col = df["Age"];                                  // Single column by name
Series col0 = df[0];                                     // Single column by index
object? cell = df[^1, "Age"];                            // Single cell at last row
DataFrame subCols = df[["Name", "Score"]];               // Column subset
DataFrame numCols = df[Cs.Numeric()];                    // Selector subset
DataFrame floatCols = df[typeof(double)];                // By .NET Type selector
DataFrame sliced = df[1..10, Cs.Numeric()];              // Row range + Selector
DataFrame filtered = df[df["Age"] >= 18, ["Name", "Age"]]; // Boolean Series mask + columns

---

4. Expressions, Operators & Sub-namespaces

Operators on Expr

• Arithmetic: +, -, *, /, % (automatically broadcasts scalars: Pl.Col("x") * 2)
• Comparisons: ==, !=, >, >=, <, <=
  • Note: Pl.Col("a") == "foo" compares column "a" to literal "foo". To compare two columns, use Pl.Col("a") == Pl.Col("b").
• Logical / Bitwise: Use single &, |, ^, !, ~ on Expr (C# && and || cannot be overloaded to return Expr directly):

  df.Filter((Pl.Col("age") >= 18) & (Pl.Col("dept") == "IT"))

Conditionals (When / Then / Otherwise)

Pl.When(Pl.Col("score") >= 90).Then("A")
  .When(Pl.Col("score") >= 60).Then("B")
  .Otherwise("F")
  .Alias("grade")

Sub-namespaces on Expr and Series

• .Str (Expr.Str): Contains, ContainsAny, ReplaceAll, Extract, Split, StripChars, ToDatetime, JsonPathMatch, etc.
• .Dt (Expr.Dt): Year, Month, Day, Truncate, OffsetBy, ConvertTimeZone, TotalMilliseconds, AddBusinessDays, etc.
• .List (Expr.List): Len, Get, Slice, Eval, Explode, Join, Contains, Unique, Sum, Mean, etc.
• .Array (Expr.Array): fixed-size array operations (Width, Sum, Max, ToList, Get, etc.).
• .Struct (Expr.Struct): Field, RenameFields, JsonEncode, Unnest.
• .Name (Expr.Name): Prefix, Suffix, ToLowercase, ToUppercase, Keep.
• .Meta (Expr.Meta): FormatTree, OutputName, RootNames, Eq.

---

5. Selectors (Polars.Selectors / Cs)

Selectors allow selecting columns by type, pattern, or set algebra (|, &, -, ~):

using Cs = Polars.CSharp.Polars.Selectors;

// Select all numeric columns except "id", and fill nulls with 0
df.WithColumns((Cs.Numeric() - Cs.ByName("id")).FillNull(0));

// Select strings or columns matching regex
df.Select(Cs.String() | Cs.Matches("^meta_.*"));

---

6. GroupBy, Having & Sugar Aggregations

GroupByBuilder supports both full .Agg(...) and direct sugar methods (.Len(), .Sum(), .Mean(), .Min(), .Max(), .First(), .Last(), .Head(), .Tail()), plus SQL-style .Having(...):

using var res = df
    .GroupBy("dept")
    .Having(Pl.Col("salary").Sum() > 100_000)
    .Agg(
        Pl.Col("salary").Sum().Alias("total_salary"),
        Pl.Col("salary").Mean().Alias("avg_salary"),
        Pl.Len().Alias("headcount")
    )
    .Sort("total_salary", descending: true);

---

7. LazyFrame, Streaming & Sinks

Prefer LazyFrame (ScanParquet, ScanCsv, ScanNdjson, ScanDelta, or df.Lazy()) for query optimization, predicate pushdown, and streaming execution:

using var lf = LazyFrame.ScanParquet("s3://bucket/data/*.parquet", cloudOptions: CloudOptions.Aws("us-east-1"));
using var df = lf
    .Filter(Pl.Col("ts") >= new DateOnly(2025, 1, 1))
    .GroupBy("region")
    .Agg(Pl.Col("revenue").Sum())
    .Collect(engine: Engine.Streaming); // Or SinkParquet("output.parquet")

---

8. Polars.NET Exclusive: Merge Operator (SQL MERGE / Upsert)

The Merge operator is unique to Polars.NET (not present in Python Polars). It provides complete ANSI SQL MERGE INTO ... USING upsert semantics with unmatched speed across in-memory DataFrames and lazy plans.

Fluent API (on DataFrame and LazyFrame)

Matches rows using column names, composite keys, or Selectors (Cs.*):

using var resultDf = targetDf.Merge(sourceDf, "Id") // Or composite: new[] { "TenantId", "UserId" }, or Cs.StartsWith("Id")
    // 1. Delete matched records based on condition:
    .WhenMatchedDelete(m => m.Source("IsBanned"))
    // 2. Update matched records:
    .WhenMatchedUpdate(
        condition: null,
        set: s => s
            .Set(Cs.StartsWith("Stat_"), (m, colName) => m.Source(colName))
            .Set("Score", m => m.Target("Score") + m.Source("ScoreDelta"))
            .Set("Tag", Pl.Lit("Updated"))
    )
    // 3. Insert new records from source:
    .WhenNotMatchedInsert(
        condition: null,
        set: s => s
            .Set("Id", m => m.Source("Id"))
            .Set("Name", m => m.Source("Name"))
            .Set(Cs.StartsWith("Stat_"), (m, colName) => m.Source(colName))
            .Set("Tag", Pl.Lit("New"))
    )
    // 4. Delete target records that no longer exist in source:
    .WhenNotMatchedBySourceDelete(m => m.Target("IsTemporary"))
    .Execute(); // Or .ToLazyFrame() for LazyFrame!

(For strongly-typed LINQ Merge directly on DataFrames or in-memory List<T>, see LINQ & Query CE.)

---

9. High-Performance Typed Row Traversal (df.Rows<T>())

While procedural row-level loops should generally be avoided in favor of vectorized SIMD column expressions, certain workflows (such as sending individual records to legacy external services or gRPC streaming endpoints) genuinely require row iteration.

Instead of untyped df.Rows() (which incurs boxing and GC allocations), use df.Rows<T>():

public record ScoreRecord(int Id, string Student, double Score);

// Zero GC Allocation: Returns a stack-allocated `ref struct DataFrameRowEnumerator<T>`
// with pre-compiled IL fast-path field accessors:
foreach (ScoreRecord record in df.Rows<ScoreRecord>())
{
    await SendRecordToExternalServiceAsync(record);
}

// Or materialize directly into a typed List<T> without enumerator heap allocation:
List<ScoreRecord> allScores = df.Rows<ScoreRecord>().ToList();

---

10. Asynchronous Streaming (IAsyncEnumerable)

Polars.NET integrates cooperatively with modern C# asynchronous streams to process large datasets without starving the thread pool:

// 1. Cooperative row-level streaming:
await foreach (ScoreRecord r in df.ToAsyncEnumerable<ScoreRecord>(chunkSize: 10_000))
{
    await ProcessAsync(r);
}

// 2. Out-of-core batch streaming directly from LazyFrame:
await foreach (DataFrame batch in lf.CollectBatchesAsync(chunkSize: 50_000))
{
    using (batch)
    {
        Console.WriteLine($"Processed native batch of {batch.Height} rows.");
    }
}