Skip to content

Add hex-native convolution support via keras-hexagdly - #286

Open
YugnatD wants to merge 2 commits into
ctlearn-project:mainfrom
YugnatD:add-hexagdly-hex-convolution-support
Open

Add hex-native convolution support via keras-hexagdly#286
YugnatD wants to merge 2 commits into
ctlearn-project:mainfrom
YugnatD:add-hexagdly-hex-convolution-support

Conversation

@YugnatD

@YugnatD YugnatD commented Aug 26, 2026

Copy link
Copy Markdown

Summary

IACT cameras (LST-1, MAGIC, MST NectarCam/FlashCam, etc.) have hexagonal
pixel grids, but CTLearn's model backbones only support square convolutions
(keras.layers.Conv2D), so dl1_data_handler's ImageMapper interpolates
every hex camera image onto an axis-aligned square grid (BilinearMapper
and friends) before it ever reaches the network.

This PR adds a hex-native alternative built on
keras-hexagdly:
HexagdlyMapper places each pixel at its exact grid cell (no interpolation)
on the addressing grid keras_hexagdly.layers.Conv2d/MaxPool2d expect,
and HexCNN is a CTLearnModel backbone built on those layers.

Both plug into CTLearn's existing factories
(image_mapper_type=HexagdlyMapper, model_type=HexCNN) with zero
changes
to dl1_data_handler or to CTLearn's config system --
ImageMapper.from_name/CTLearnModel.from_name resolve them via plain
subclass discovery once imported.

Why this matters beyond model architecture: FPGA-native inference

keras-hexagdly supports exporting Conv2d/MaxPool2d (io_stream)
directly to FPGA via hls4ml, synthesizing the hexagonal convolution itself
in hardware. This is not just a modeling nicety -- it removes a hard
blocker for CTA's real-time / edge-inference use cases:

  • Today, every hex camera goes through BilinearMapper (or another
    interpolation mapper) before a model ever sees it. That resampling step
    has no FPGA synthesis path, so an interpolate-to-square model cannot be
    deployed as a single FPGA pipeline from raw camera data.
  • With HexagdlyMapper, there is no interpolation step to begin with --
    HexCNN operates on the camera's native geometry, so the whole
    pipeline (mapping + convolution + pooling) becomes FPGA-synthesizable,
    opening the door to low-latency, on-site hex-native inference that
    the square path structurally cannot offer.

This is the main reason to care about this PR beyond "an alternative
architecture to try": it's a prerequisite for FPGA-based hex-native
inference in CTLearn at all.

What's added

  • ctlearn/core/hexagdly_geometry.py -- HexGridTransform: maps any
    hexagonal ctapipe.CameraGeometry (including rotated cameras like LST-1)
    onto keras-hexagdly's offset-column grid, with an auto de-rotation step
    and a neighbour-graph consistency check.
  • ctlearn/core/hexagdly_mapper.py -- HexagdlyMapper(ImageMapper).
  • ctlearn/core/hexagdly_model.py -- HexCNN(CTLearnModel), mirrors
    SingleCNN.
  • Guarded imports in train_model.py / predict_model.py /
    predict_LST1.py so the new classes are actually discoverable through the
    CLI tools once keras-hexagdly is installed -- a no-op otherwise.
  • New optional hexagdly extra in pyproject.toml
    (pip install ctlearn[hexagdly]) -- no new dependency for anyone who
    doesn't opt in.
  • Tests in ctlearn/core/tests/test_hexagdly_*.py: fast/offline using the
    bundled LST-1 resource, plus --remote-data-gated checks against
    MAGICCam/NectarCam/FlashCam/CHEC/SCTCam.
  • benchmarks/hexcnn_vs_singlecnn_lst1_energy.py +
    benchmarks/results/ -- see below.

Benchmark (honest caveat)

Ran HexCNN+HexagdlyMapper vs. SingleCNN+BilinearMapper on LST-1
energy regression through the real TrainCTLearnModel tool -- see
benchmarks/results/. Both train end-to-end with no special-casing, at
comparable wall-clock cost. The only dataset available in this environment
is ctapipe's small CI test file (~70 images from ~18-19 events), too small
for a real accuracy claim, and the raw loss numbers currently favor
SingleCNN for an explainable reason (an untuned initializer-scale
mismatch between the two libraries' defaults, detailed in the report -- not
evidence the hex path underperforms). A real accuracy comparison needs
production-scale data this environment doesn't have access to.

Also found, not fixed here

SingleCNN's default attention_mechanism ("Dual-SE") crashes
immediately with KeyError: 'ratio' -- ctlearn/core/model.py reads
self.attention["ratio"], but CTLearnModel.__init__ only ever sets
"reduction_ratio". Pre-existing and unrelated to this PR (HexCNN and
ResNet both correctly use "reduction_ratio"). Happy to open a separate
tiny fix for it if useful.

Testing

```
pytest ctlearn/core/tests/test_hexagdly_.py -v # offline
pytest ctlearn/core/tests/test_hexagdly_
.py --remote-data=any # + other cameras
```

HexagdlyMapper maps a hexagonal camera geometry directly onto the
addressed grid keras_hexagdly's Conv2d/MaxPool2d expect, instead of
interpolating onto an axis-aligned square grid (BilinearMapper etc.).
HexCNN is a CTLearnModel backbone built on those layers, mirroring
SingleCNN. Both register with ctapipe's existing Component factories
(image_mapper_type=HexagdlyMapper, model_type=HexCNN) with no changes
to dl1_data_handler or to CTLearn's config system.

New optional 'hexagdly' extra in pyproject.toml pulls in keras-hexagdly;
nothing changes for users who don't opt in.

Includes tests (offline, using the bundled LST-1 resource, plus
--remote-data-gated checks against other real CTA camera geometries)
and a benchmarks/ script comparing HexCNN+HexagdlyMapper against
SingleCNN+BilinearMapper -- see benchmarks/results/ for the current
run and its stated limitations (small CI test dataset, not yet a
production-scale accuracy comparison).
@tjarkmiener
tjarkmiener self-requested a review August 27, 2026 07:57

@tjarkmiener tjarkmiener left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @YugnatD! Thank you so much for this important and interesting work! I posted some comments. As said in the comments everything related to the geometry and mapping should be moved to the corresponding modules of the dl1dh. Besides currently the tools are missing a check that validates the configuration, i.e. when HexagdlyMapper is selected the Conv2D import has to be from hgly and vice versa.

Comment thread ctlearn/core/hexagdly_mapper.py Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is a new subclass of the ImageMapper, I'd prefer to move HexagdlyMapper to the dl1_data_handler.image_mapper modules.

Comment thread ctlearn/core/hexagdly_geometry.py Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same as HexagdlyMapper, I'd prefer to the HexGridTransform to the corresponding dl1dh module.

Comment thread ctlearn/core/hexagdly_model.py Outdated
zip(filters_list, kernel_sizes, numbers_list)
):
for nr in range(number):
x = hgly.Conv2d(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this line (import of a different Conv2D implementation) the only difference between HexCNN and SingleCNN? If so, I would suggestion to add a new config traits to the existing SingleCNN which specify the Conv2D import. That would remove quite a lot of code reputation. In principle the scheme can be also added to the ResNet model.

Comment thread ctlearn/tools/predict_LST1.py Outdated
Comment on lines +59 to +68
# Optional hex-native convolution support (the ``ctlearn[hexagdly]`` extra).
# Importing registers ``HexagdlyMapper`` with ``ImageMapper.from_name`` and
# makes ``keras_hexagdly``'s custom layers available when deserializing a
# saved ``HexCNN`` model. A no-op if ``keras-hexagdly`` isn't installed.
try:
import ctlearn.core.hexagdly_mapper # noqa: F401
import ctlearn.core.hexagdly_model # noqa: F401
except ImportError:
pass

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here and elsewhere: If we move it to the dl1dh we should bump the version and do not make it optional.

Comment thread pyproject.toml Outdated
Comment on lines +56 to +59
# Hexagonal convolution support (HexCNN + HexagdlyMapper) built on keras-hexagdly.
hexagdly = [
"keras-hexagdly>=0.3.0",
]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's available on pypi we can just add it to the normal dependencies I guess.

@YugnatD

YugnatD commented Aug 28, 2026

Copy link
Copy Markdown
Author

Hi @YugnatD! Thank you so much for this important and interesting work! I posted some comments. As said in the comments everything related to the geometry and mapping should be moved to the corresponding modules of the dl1dh. Besides currently the tools are missing a check that validates the configuration, i.e. when HexagdlyMapper is selected the Conv2D import has to be from hgly and vice versa.

Thanks @tjarkmiener for the review, im currently modiyfing it and will soon make a pr to dl1dh to split the implemetation as you asked.

But While working on this I had the thought that data augmentation could silently break things for the hex path, and noticed #282 ("Add data augmentation") is open. So I'm wondering: is augmentation actually planned/being worked on, and how should I take that into account here?

I dug a bit and found an old prototype (ctlearn/core/data_loader/base_loader.py, referenced from that issue but not on any current branch) that implements flip/rotate/translate augmentation via cv2.warpAffine directly on the post-mapping (H, W, C) image array.

That approach works fine for the square-mapped path, but it won't be safe for HexagdlyMapper's output once/if this PR lands: cell adjacency there depends on column parity, so rotating or flipping the mapped array scrambles which cells the hex convolution treats as physical neighbors — not just interpolation blur like it would be for a square grid, but a geometrically wrong input that still looks well-formed. A hex-lattice rotation/reflection needs to be applied before mapping (at the pixel/geometry level) or worked out in axial coordinates and re-converted through the same offset-column addressing, not applied to the mapped array directly.

Do you want me to guard against this in this PR (e.g. reject or restrict augmentation when conv_backend="hexagdly"), or is it fine to leave alone for now and handle it properly once augmentation actually gets implemented?

- HexGridTransform/HexagdlyMapper moved to dl1_data_handler
  (cta-observatory/dl1-data-handler#192) -- no longer ctlearn-local.
- Replace the standalone HexCNN class with a conv_backend trait
  ("square"/"hexagdly") on SingleCNN and ResNet directly, so hex support
  works with a ResNet backbone too, not just SingleCNN. Both
  residual_block_type variants covered.
- Add validate_conv_backend/model_conv_backend (ctlearn/utils.py): the
  missing image_mapper_type <-> model conv_backend consistency check,
  wired into train_model.py, predict_model.py, and predict_LST1.py.
- keras-hexagdly moves from the optional `hexagdly` extra to a normal
  dependency, bumped to >=0.4.1 (fixes a real default-initializer scaling
  bug found while benchmarking this).
- Benchmark reworked for the new SingleCNN(conv_backend=...) API.
- New tests requiring HexagdlyMapper are skipped (not failed) until
  dl1-data-handler#192 merges/releases -- they run for real once it does,
  no further change needed here.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants