Add hex-native convolution support via keras-hexagdly - #286
Conversation
HexagdlyMapper maps a hexagonal camera geometry directly onto the addressed grid keras_hexagdly's Conv2d/MaxPool2d expect, instead of interpolating onto an axis-aligned square grid (BilinearMapper etc.). HexCNN is a CTLearnModel backbone built on those layers, mirroring SingleCNN. Both register with ctapipe's existing Component factories (image_mapper_type=HexagdlyMapper, model_type=HexCNN) with no changes to dl1_data_handler or to CTLearn's config system. New optional 'hexagdly' extra in pyproject.toml pulls in keras-hexagdly; nothing changes for users who don't opt in. Includes tests (offline, using the bundled LST-1 resource, plus --remote-data-gated checks against other real CTA camera geometries) and a benchmarks/ script comparing HexCNN+HexagdlyMapper against SingleCNN+BilinearMapper -- see benchmarks/results/ for the current run and its stated limitations (small CI test dataset, not yet a production-scale accuracy comparison).
tjarkmiener
left a comment
There was a problem hiding this comment.
Hi @YugnatD! Thank you so much for this important and interesting work! I posted some comments. As said in the comments everything related to the geometry and mapping should be moved to the corresponding modules of the dl1dh. Besides currently the tools are missing a check that validates the configuration, i.e. when HexagdlyMapper is selected the Conv2D import has to be from hgly and vice versa.
There was a problem hiding this comment.
Since this is a new subclass of the ImageMapper, I'd prefer to move HexagdlyMapper to the dl1_data_handler.image_mapper modules.
There was a problem hiding this comment.
The same as HexagdlyMapper, I'd prefer to the HexGridTransform to the corresponding dl1dh module.
| zip(filters_list, kernel_sizes, numbers_list) | ||
| ): | ||
| for nr in range(number): | ||
| x = hgly.Conv2d( |
There was a problem hiding this comment.
Is this line (import of a different Conv2D implementation) the only difference between HexCNN and SingleCNN? If so, I would suggestion to add a new config traits to the existing SingleCNN which specify the Conv2D import. That would remove quite a lot of code reputation. In principle the scheme can be also added to the ResNet model.
| # Optional hex-native convolution support (the ``ctlearn[hexagdly]`` extra). | ||
| # Importing registers ``HexagdlyMapper`` with ``ImageMapper.from_name`` and | ||
| # makes ``keras_hexagdly``'s custom layers available when deserializing a | ||
| # saved ``HexCNN`` model. A no-op if ``keras-hexagdly`` isn't installed. | ||
| try: | ||
| import ctlearn.core.hexagdly_mapper # noqa: F401 | ||
| import ctlearn.core.hexagdly_model # noqa: F401 | ||
| except ImportError: | ||
| pass | ||
|
|
There was a problem hiding this comment.
Here and elsewhere: If we move it to the dl1dh we should bump the version and do not make it optional.
| # Hexagonal convolution support (HexCNN + HexagdlyMapper) built on keras-hexagdly. | ||
| hexagdly = [ | ||
| "keras-hexagdly>=0.3.0", | ||
| ] |
There was a problem hiding this comment.
If it's available on pypi we can just add it to the normal dependencies I guess.
Thanks @tjarkmiener for the review, im currently modiyfing it and will soon make a pr to dl1dh to split the implemetation as you asked. But While working on this I had the thought that data augmentation could silently break things for the hex path, and noticed #282 ("Add data augmentation") is open. So I'm wondering: is augmentation actually planned/being worked on, and how should I take that into account here? I dug a bit and found an old prototype (ctlearn/core/data_loader/base_loader.py, referenced from that issue but not on any current branch) that implements flip/rotate/translate augmentation via cv2.warpAffine directly on the post-mapping (H, W, C) image array. That approach works fine for the square-mapped path, but it won't be safe for HexagdlyMapper's output once/if this PR lands: cell adjacency there depends on column parity, so rotating or flipping the mapped array scrambles which cells the hex convolution treats as physical neighbors — not just interpolation blur like it would be for a square grid, but a geometrically wrong input that still looks well-formed. A hex-lattice rotation/reflection needs to be applied before mapping (at the pixel/geometry level) or worked out in axial coordinates and re-converted through the same offset-column addressing, not applied to the mapped array directly. Do you want me to guard against this in this PR (e.g. reject or restrict augmentation when conv_backend="hexagdly"), or is it fine to leave alone for now and handle it properly once augmentation actually gets implemented? |
- HexGridTransform/HexagdlyMapper moved to dl1_data_handler (cta-observatory/dl1-data-handler#192) -- no longer ctlearn-local. - Replace the standalone HexCNN class with a conv_backend trait ("square"/"hexagdly") on SingleCNN and ResNet directly, so hex support works with a ResNet backbone too, not just SingleCNN. Both residual_block_type variants covered. - Add validate_conv_backend/model_conv_backend (ctlearn/utils.py): the missing image_mapper_type <-> model conv_backend consistency check, wired into train_model.py, predict_model.py, and predict_LST1.py. - keras-hexagdly moves from the optional `hexagdly` extra to a normal dependency, bumped to >=0.4.1 (fixes a real default-initializer scaling bug found while benchmarking this). - Benchmark reworked for the new SingleCNN(conv_backend=...) API. - New tests requiring HexagdlyMapper are skipped (not failed) until dl1-data-handler#192 merges/releases -- they run for real once it does, no further change needed here.
Summary
IACT cameras (LST-1, MAGIC, MST NectarCam/FlashCam, etc.) have hexagonal
pixel grids, but CTLearn's model backbones only support square convolutions
(
keras.layers.Conv2D), sodl1_data_handler'sImageMapperinterpolatesevery hex camera image onto an axis-aligned square grid (
BilinearMapperand friends) before it ever reaches the network.
This PR adds a hex-native alternative built on
keras-hexagdly:
HexagdlyMapperplaces each pixel at its exact grid cell (no interpolation)on the addressing grid
keras_hexagdly.layers.Conv2d/MaxPool2dexpect,and
HexCNNis aCTLearnModelbackbone built on those layers.Both plug into CTLearn's existing factories
(
image_mapper_type=HexagdlyMapper,model_type=HexCNN) with zerochanges to
dl1_data_handleror to CTLearn's config system --ImageMapper.from_name/CTLearnModel.from_nameresolve them via plainsubclass discovery once imported.
Why this matters beyond model architecture: FPGA-native inference
keras-hexagdly supports exporting
Conv2d/MaxPool2d(io_stream)directly to FPGA via
hls4ml, synthesizing the hexagonal convolution itselfin hardware. This is not just a modeling nicety -- it removes a hard
blocker for CTA's real-time / edge-inference use cases:
BilinearMapper(or anotherinterpolation mapper) before a model ever sees it. That resampling step
has no FPGA synthesis path, so an interpolate-to-square model cannot be
deployed as a single FPGA pipeline from raw camera data.
HexagdlyMapper, there is no interpolation step to begin with --HexCNNoperates on the camera's native geometry, so the wholepipeline (mapping + convolution + pooling) becomes FPGA-synthesizable,
opening the door to low-latency, on-site hex-native inference that
the square path structurally cannot offer.
This is the main reason to care about this PR beyond "an alternative
architecture to try": it's a prerequisite for FPGA-based hex-native
inference in CTLearn at all.
What's added
ctlearn/core/hexagdly_geometry.py--HexGridTransform: maps anyhexagonal
ctapipe.CameraGeometry(including rotated cameras like LST-1)onto keras-hexagdly's offset-column grid, with an auto de-rotation step
and a neighbour-graph consistency check.
ctlearn/core/hexagdly_mapper.py--HexagdlyMapper(ImageMapper).ctlearn/core/hexagdly_model.py--HexCNN(CTLearnModel), mirrorsSingleCNN.train_model.py/predict_model.py/predict_LST1.pyso the new classes are actually discoverable through theCLI tools once
keras-hexagdlyis installed -- a no-op otherwise.hexagdlyextra inpyproject.toml(
pip install ctlearn[hexagdly]) -- no new dependency for anyone whodoesn't opt in.
ctlearn/core/tests/test_hexagdly_*.py: fast/offline using thebundled LST-1 resource, plus
--remote-data-gated checks againstMAGICCam/NectarCam/FlashCam/CHEC/SCTCam.
benchmarks/hexcnn_vs_singlecnn_lst1_energy.py+benchmarks/results/-- see below.Benchmark (honest caveat)
Ran
HexCNN+HexagdlyMappervs.SingleCNN+BilinearMapperon LST-1energyregression through the realTrainCTLearnModeltool -- seebenchmarks/results/. Both train end-to-end with no special-casing, atcomparable wall-clock cost. The only dataset available in this environment
is ctapipe's small CI test file (~70 images from ~18-19 events), too small
for a real accuracy claim, and the raw loss numbers currently favor
SingleCNNfor an explainable reason (an untuned initializer-scalemismatch between the two libraries' defaults, detailed in the report -- not
evidence the hex path underperforms). A real accuracy comparison needs
production-scale data this environment doesn't have access to.
Also found, not fixed here
SingleCNN's defaultattention_mechanism("Dual-SE") crashesimmediately with
KeyError: 'ratio'--ctlearn/core/model.pyreadsself.attention["ratio"], butCTLearnModel.__init__only ever sets"reduction_ratio". Pre-existing and unrelated to this PR (HexCNNandResNetboth correctly use"reduction_ratio"). Happy to open a separatetiny fix for it if useful.
Testing
```
pytest ctlearn/core/tests/test_hexagdly_.py -v # offline
pytest ctlearn/core/tests/test_hexagdly_.py --remote-data=any # + other cameras
```