mlx-data reads CSV, JSONL and text, and fetches from S3, but has no Parquet support and no prior issue about it. Parquet is where most tabular training data now lives (it is the format under Iceberg and Delta). Today you either convert to JSONL first — losing the column projection that makes Parquet worth using — or build the stream outside mlx-data and lose the pipeline ops.
I would like to contribute this. Two shapes, and I would rather agree on which before writing it.
A lazy Parquet stream already works with no changes to mlx-data:
def samples(path, columns=None, batch_size=1024):
for batch in pq.ParquetFile(path).iter_batches(batch_size, columns=columns):
for row in batch.to_pylist():
yield {k: v.encode() if isinstance(v, str) else v
for k, v in row.items() if v is not None}
dset = dx.stream_python_iterable(lambda: samples("data.parquet"))Option 1 — dx.stream_parquet_reader(...) in python/mlx/data/, wrapping the above. Small, and the dependency lands only on Python (pyarrow, an optional extra), never on the C++ core. Cost: to_pylist() builds a Python object per value and holds the GIL, so throughput is well below native and it interacts badly with prefetch.
Option 2 — a native mlx/data/stream/ParquetReader beside CSVReader and JSONLReader, via Arrow C++, producing Arrays with no Python objects in the path. Arrow does the format work, so it should be in the range of CSVReader (243 lines) and JSONLReader (588).
The ask: which option, and for option 2, is an optional Arrow C++ dependency behind MLX_HAS_PARQUET acceptable? I am happy to start with option 1 regardless, with correctness tests for the cases Parquet gets silently wrong rather than loudly — required columns carrying no definition-level data, encoding differing per row group within one file, "no dictionary page" not implying PLAIN, BYTE_ARRAY that is not UTF-8.
mlx-datareads CSV, JSONL and text, and fetches from S3, but has no Parquet support and no prior issue about it. Parquet is where most tabular training data now lives (it is the format under Iceberg and Delta). Today you either convert to JSONL first — losing the column projection that makes Parquet worth using — or build the stream outsidemlx-dataand lose the pipeline ops.I would like to contribute this. Two shapes, and I would rather agree on which before writing it.
A lazy Parquet stream already works with no changes to mlx-data:
Option 1 —
dx.stream_parquet_reader(...)inpython/mlx/data/, wrapping the above. Small, and the dependency lands only on Python (pyarrow, an optional extra), never on the C++ core. Cost:to_pylist()builds a Python object per value and holds the GIL, so throughput is well below native and it interacts badly withprefetch.Option 2 — a native
mlx/data/stream/ParquetReaderbesideCSVReaderandJSONLReader, via Arrow C++, producingArrays with no Python objects in the path. Arrow does the format work, so it should be in the range ofCSVReader(243 lines) andJSONLReader(588).The ask: which option, and for option 2, is an optional Arrow C++ dependency behind
MLX_HAS_PARQUETacceptable? I am happy to start with option 1 regardless, with correctness tests for the cases Parquet gets silently wrong rather than loudly — required columns carrying no definition-level data, encoding differing per row group within one file, "no dictionary page" not implying PLAIN,BYTE_ARRAYthat is not UTF-8.