Support larger TRX files

[36000]

I updated the code to handle tractograms with large numbers of vertices, fixing #50

[Copilot]

Pull request overview

This PR updates TRX reading/writing and in-memory handling to support tractograms with large vertex/streamline counts (and 64-bit offsets), addressing crashes when counts/offsets exceed 32-bit limits (issue #50).

Changes:

• Switch many count/shape variables from int to int64_t across TRX loading, saving, resizing, and mmap shapes.
• Introduce _json_int64() and replace several int_value() reads to avoid 32-bit truncation when consuming header counts.
• Update mmap/remap helpers and shard-merge header writing to better accommodate large datasets.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
src/trx.cpp	Widened header parsing/counts and mmap shape handling; updated shard merge header counts to avoid 32-bit truncation.
include/trx/trx.tpp	Propagated 64-bit counts/shapes through TrxFile construction, loading, resizing, saving, and subset operations.
include/trx/trx.h	Added _json_int64() and widened several public/internal count fields and APIs to int64_t.
include/trx/detail/dtype_helpers.h	Updated remap and length-computation helpers to accept 64-bit dimensions/counts.

Suppressed comments (1)

src/trx.cpp:916

• _create_memmap computes filesize via unchecked multiplication and, when the file is missing, calls allocate_file() which writes a std::string(filesize, '\0'). For large TRX datasets this can overflow size_t (producing a too-small mapping) and/or try to allocate enormous RAM just to preallocate the file, defeating the goal of supporting large files. Consider adding overflow/negative-shape checks and preallocating with filesystem resize (or seek+write) instead of materializing the full file in memory.

mio::shared_mmap_sink _create_memmap(std::string filename,
                                     const std::tuple<int64_t, int64_t> &shape,
                                     const std::string &mode,
                                     const std::string &dtype,
                                     long long offset) {
  static_cast<void>(mode);
  const std::size_t filesize = static_cast<std::size_t>(std::get<0>(shape)) *
                               static_cast<std::size_t>(std::get<1>(shape)) *
                               static_cast<std::size_t>(trx::detail::_sizeof_dtype(dtype));

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.