Is your feature request related to a problem? Please describe.
The current MQTT-based mechanism for fins is fragile and requires an extra broker dependency on MQTT. It also has no mature implementation in production yet, so the transport/wire protocol is still free to change.
My main issues is that it needs a lot of state on both the side of soarca and on the side of the client. Soarca and the client both need to connect to the broker and channels names are very important for routing the fin calls.
The existing python library is also not very simple to use (clone and modify vs a simple pypi library you can import). Note that this is from my last understanding of the fin library, I started fresh and did not look at the old implementation since it's focussed on MQTT.
Describe the solution you'd like
Introduce a new HTTP/JSON pull-based protocol, modeled on how GitLab Runner fetches jobs: Fins periodically long-poll SOARCA for work, claim a job, execute it, and submit the result back over plain HTTP. No broker or persistent connection management, implementable in any language with just an HTTP client and some JSON reading and writing.
This was heavily inspired by how gitlab workers work. I've had a look at the gitlab API and read about the long polling mechanism, which is very cheap compared to naive polling every second.
The whole ack/nack and liveliness can piggyback on HTTP calls and status updates on individual step handling.
I've implemented this in my personal fork:
hidde-jan#2
hidde-jan#4
Note that I did use copilot and the resulting code is overly wordy in some places because it uses a lot of comments.
The analysis I did before creating the two PRs above can be found here:
hidde-jan#1
For an idea of what a fin library for python can look like, have a look at this repo: https://github.com/hidde-jan/soarca-fin
Describe alternatives you've considered
First I was thinking about some kind of webhook setup, but this means fins need to listen to incoming HTTP requests, and need to be reachable by soarca. This can lead to firewall/NAT problems.
I also considered replacing MQTT by (an internal) NATS server, which is easily embedded in go and provides request/response primitives, which is what soarca actually used to communicate with fins, just hidden behind a message bus. But it isn't necessary at all.
Additional context
This also makes is much easier to implement #344, since status is just "has this fin recently checked in", which can be derived from a database field (last checked in). This is also how gitlab checks worker health.
Is your feature request related to a problem? Please describe.
The current MQTT-based mechanism for fins is fragile and requires an extra broker dependency on MQTT. It also has no mature implementation in production yet, so the transport/wire protocol is still free to change.
My main issues is that it needs a lot of state on both the side of soarca and on the side of the client. Soarca and the client both need to connect to the broker and channels names are very important for routing the fin calls.
The existing python library is also not very simple to use (clone and modify vs a simple pypi library you can import). Note that this is from my last understanding of the fin library, I started fresh and did not look at the old implementation since it's focussed on MQTT.
Describe the solution you'd like
Introduce a new HTTP/JSON pull-based protocol, modeled on how GitLab Runner fetches jobs: Fins periodically long-poll SOARCA for work, claim a job, execute it, and submit the result back over plain HTTP. No broker or persistent connection management, implementable in any language with just an HTTP client and some JSON reading and writing.
This was heavily inspired by how gitlab workers work. I've had a look at the gitlab API and read about the long polling mechanism, which is very cheap compared to naive polling every second.
The whole ack/nack and liveliness can piggyback on HTTP calls and status updates on individual step handling.
I've implemented this in my personal fork:
hidde-jan#2
hidde-jan#4
Note that I did use copilot and the resulting code is overly wordy in some places because it uses a lot of comments.
The analysis I did before creating the two PRs above can be found here:
hidde-jan#1
For an idea of what a fin library for python can look like, have a look at this repo: https://github.com/hidde-jan/soarca-fin
Describe alternatives you've considered
First I was thinking about some kind of webhook setup, but this means fins need to listen to incoming HTTP requests, and need to be reachable by soarca. This can lead to firewall/NAT problems.
I also considered replacing MQTT by (an internal) NATS server, which is easily embedded in go and provides request/response primitives, which is what soarca actually used to communicate with fins, just hidden behind a message bus. But it isn't necessary at all.
Additional context
This also makes is much easier to implement #344, since status is just "has this fin recently checked in", which can be derived from a database field (last checked in). This is also how gitlab checks worker health.