feat: implement worker functionality and queue management - #406
Merged
Merged
Conversation
- Added a new Worker class to manage job execution, including reserving jobs, executing handlers, and persisting outcomes. - Introduced methods for starting, completing, and failing job executions in the QueueStore interface. - Enhanced the CLI to support building and starting a worker with specified options. - Implemented tests for the Worker class to ensure correct job handling and error management. - Updated documentation to reflect the new Worker class and its usage. closes: #14
There was a problem hiding this comment.
Pull request overview
Implements a first-class worker runtime and execution lifecycle in Karya, including CLI wiring, queue-store execution APIs, and supporting tests/docs (closes #14).
Changes:
- Added
Karya::Workerwith a polling run loop, handler dispatch, and configuration/runtime validation. - Extended the
QueueStorecontract (andInMemoryQueueStore) with explicit execution lifecycle methods (start_execution,complete_execution,fail_execution). - Added CLI support for bootstrapping a worker with handler mappings, plus RSpec coverage and docs updates.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/pages/runtime/workers.md | Updates worker runtime usage example to the new Karya::Worker API |
| core/karya/lib/karya/worker.rb | Introduces worker implementation (reserve → execute → finalize) with config/runtime helpers |
| core/karya/lib/karya/queue_store.rb | Adds execution lifecycle methods to the queue store interface |
| core/karya/lib/karya/in_memory_queue_store.rb | Implements execution lifecycle transitions and refactors internal state handling |
| core/karya/lib/karya/job.rb | Extends transition_to to support updating attempt during state transitions |
| core/karya/lib/karya/cli.rb | Adds worker command, handler parsing, and --require support |
| core/karya/lib/karya/constant_resolver.rb | Adds constant resolution utility used by CLI handler parsing |
| core/karya/lib/karya.rb | Requires new runtime components (constant_resolver, worker) |
| core/karya/spec/karya/worker_spec.rb | Adds worker behavior tests (success/failure paths, validation, run loop behavior) |
| core/karya/spec/karya/queue_store_spec.rb | Adds interface-contract specs for new queue store methods |
| core/karya/spec/karya/in_memory_queue_store_spec.rb | Adds execution lifecycle tests and adjusts to new internal state object |
| core/karya/spec/karya/cli_spec.rb | Adds CLI coverage for worker bootstrap and handler loading |
| core/karya/README.md | Documents worker bootstrap via CLI |
- Introduced a new configuration method for queue stores to ensure proper initialization before worker execution. - Updated the CLI to require a configured queue store, improving error handling for missing configurations. - Enhanced the InMemoryQueueStore to manage execution tokens more effectively, including requeuing expired jobs. - Added error handling for constant resolution failures, providing clearer feedback for unresolved constants. - Improved worker execution logic to handle lease expirations gracefully without crashing. - Expanded test coverage for worker and queue store functionalities, ensuring robustness against edge cases.
- Added a check to reject mixed positional and keyword signatures in the keyword dispatch method. - This change ensures that handler methods conform to the expected argument structure, preventing configuration errors.
- Updated job lifecycle to allow transitioning from running to queued state. - Refactored to_queued_job method to use transition_to for better clarity. - Introduced MutableGraphCopy class to ensure mutable copies of arguments are passed to handlers. - Added tests for requeuing jobs and validating lifecycle transitions. - Enhanced worker to handle mutable copies of arguments for positional-hash handlers.
- Updated constant resolution to handle blank strings more effectively. - Enhanced error messaging for empty constant paths. - Adjusted execution activation logic to ensure it only occurs when the job transition is successful. - Added a test to verify that execution is not activated when the job cannot be persisted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes: #14