{"id":10301,"date":"2026-05-03T20:33:00","date_gmt":"2026-05-04T04:33:00","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/python\/?p=10301"},"modified":"2026-05-03T20:33:00","modified_gmt":"2026-05-04T04:33:00","slug":"introducing-apache-arrow-support-in-mssql-python","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/python\/introducing-apache-arrow-support-in-mssql-python\/","title":{"rendered":"Introducing Apache Arrow Support in mssql-python"},"content":{"rendered":"

<\/span><\/p>\n

\n

\"c1014e61<\/a><\/span><\/p>\n

Reviewed by Sumit Sarabhai<\/span><\/em><\/p>\n

Fetching a million rows from SQL Server into a Polars<\/a> DataFrame used to mean a million Python objects, a million GC allocations, and then throwing it all away to build a DataFrame. Not anymore. mssql-python<\/a> now supports fetching SQL Server data directly as Apache Arrow structures – a faster and more memory-efficient path for anyone working with SQL Server data in Polars, Pandas, DuckDB, or any other Arrow-native library. This feature was contributed by community developer Felix Gra\u00dfl (@ffelixg)<\/strong>, and we are thrilled to ship it.<\/span><\/p>\n

<\/i>Key Terms<\/strong><\/p>API (Application Programming Interface):<\/strong> a source-code contract that defines how to call a function or library.<\/span><\/p>\n

ABI (Application Binary Interface):<\/strong> a binary-level contract that specifies how compiled code is laid out in memory. Two programs built in different languages can share an ABI and exchange data directly – no serialization is needed.<\/span><\/p>\n

Arrow C Data Interface:<\/strong> Apache Arrow’s ABI specification – the standard that makes zero-copy data exchange between languages possible.<\/div><\/span><\/p>\n

What Is Apache Arrow?<\/span><\/h2>\n

The key insight behind Apache Arrow<\/a> is zero-copy language interoperability<\/strong>. Arrow defines a stable shared-memory layout – the Arrow C Data Interface<\/strong>, a cross-language ABI – that any language can produce or consume by exchanging a pointer, with no serialization, no copies, and no re-parsing. A C++ database driver and a Python DataFrame library can work on the exact same memory without either one knowing about the other.<\/span><\/p>\n

Built on top of that, Arrow uses a columnar in-memory format<\/strong>: instead of representing a table as a list of rows, each row a collection of Python objects, Arrow stores all values for a column contiguously in a typed buffer. Nulls are tracked in a compact bitmap rather than per-cell None<\/code> objects.<\/span><\/p>\n

For a database driver, this means the entire fetch loop can run in C++ and write values directly into Arrow buffers – no Python object creation per row, no garbage-collector pressure. The DataFrame library receives a pointer to that memory and can begin operating on it immediately. Crucially, subsequent operations – filters, joins, aggregations – also work in-place on those same buffers. A Polars pipeline reading from mssql-python never needs to materialize intermediate Python objects at any stage, making Arrow the right foundation for high-throughput data processing pipelines.<\/span><\/p>\n

For users of mssql-python, this translates into four concrete benefits:<\/span><\/p>\n