Sync with upstream, cherry-pick 15 clean community PRs, verify paperless-ngx v3 compat - #1
Merged
Conversation
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…icereed#833) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
* feat: add Anthropic/Claude API as LLM provider Add support for using Claude models via the Anthropic API for both normal text prompts and OCR vision tasks * test(anthropic): add E2E tests for anthropic client
…#852) * refactor: cleanup to make var naming a little clearer * fix: deleted code that was removing the wrong tags in an auto ocr + non-ocr context * fix: handle auto ocr the same as other tags when it is the last one * test: refactored RemovingLastTag test to more closely mimic the behavior of the core system * test: added clarifying comment based on coderabbitai
* feat: Add Document Type detection via LLM - Add document_type_prompt.tmpl for LLM-based document type suggestion - Add GenerateDocumentTypes to request/response types - Add getSuggestedDocumentType() function in app_llm.go - Wire document type generation into suggestion flow - Update paperless.go to patch document_type field (restrict to existing types) - Add AUTO_GENERATE_DOCUMENT_TYPE env var (default: true) - Add frontend toggle and input field for document type - Update README with new environment variable * ci: Enable GitHub Actions workflow for forks - Update workflow to push images to GHCR for forks - Remove icereed/paperless-gpt repository check for push - Add fork-specific tag generation (latest, commit SHA) - Update manifest creation to use GHCR digests for forks - Update E2E tests to use correct image for forks * fix: Add document_type_prompt.tmpl to test setup The test setup was missing the new document_type_prompt.tmpl in the list of dummy prompt files, causing loadTemplates() to fail. --------- Co-authored-by: Icereed <git@icereed.net>
* Initial plan * Fix e2e test parallelism and add missing API key - Remove --workers 4 from package.json to use config default (workers: 1) - Add ANTHROPIC_API_KEY to CI workflow environment variables - Increase test timeout from 2min to 5min for OCR processing tests Co-authored-by: icereed <444269+icereed@users.noreply.github.com> * Remove redundant page.close() calls and unused browser - Remove page.close() in afterEach - Playwright manages page lifecycle - Remove unused browser creation from test environment setup - Remove unused waitForElement helper function - Improves test reliability and reduces resource usage Co-authored-by: icereed <444269+icereed@users.noreply.github.com> * Add comprehensive documentation of e2e test fixes - Document all 5 issues identified and resolved - Include detailed analysis and impact for each issue - Provide verification checklist and testing recommendations - Add architectural overview of test environment Co-authored-by: icereed <444269+icereed@users.noreply.github.com> * Fix E2E tests Docker image authentication issue - Change E2E tests to use GHCR images instead of Docker Hub - Add Docker login step to authenticate with GHCR using GITHUB_TOKEN - Ensures TestContainers can pull the built image for testing Co-authored-by: icereed <444269+icereed@users.noreply.github.com> * Fix Mistral OCR test to handle punctuation variations - Update test assertion to accept text with or without period after "de" - OCR extraction may have minor punctuation differences - Makes test more resilient to OCR output variations Co-authored-by: icereed <444269+icereed@users.noreply.github.com> * Skip Anthropic tests when API key is not available - Use test.skip() to skip entire test file when ANTHROPIC_API_KEY is missing - Prevents beforeAll hooks from running when tests will be skipped - Removes redundant skip checks from individual tests Co-authored-by: icereed <444269+icereed@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: icereed <444269+icereed@users.noreply.github.com> Co-authored-by: Icereed <git@icereed.net>
* Initial plan * Implement fork PR E2E testing workflow with label-based approval Co-authored-by: icereed <444269+icereed@users.noreply.github.com> * Fix fork PR detection to skip E2E tests without label Co-authored-by: icereed <444269+icereed@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: icereed <444269+icereed@users.noreply.github.com>
* Fix paperless documents API call The api call does not support multiple tags. This commit reduces the call to a single tag. * Use paperless tags API call to check for documents to process Also do not get tags/correspondents if there is no document to process. This reduces the processing time, cpu usage significantly. Example: 1000 documents, 200 tags, 100 correspondents and no document to process 10-50 ms instead of 500-1000 ms * Fix DocumentSuggestion fmt
Added Trendshift badge to README for repository tracking.
Replace hardcoded constants (max pixel dimension, max total pixels, max render DPI, max file bytes) with configurable environment variables. Co-authored-by: Cristian Filipov <5324653+cfilipov@users.noreply.github.com>
* add Gemini as Vision provider * fix: skip thinking parts in GoogleAI response parsing When thinking budget is enabled, Gemini returns internal reasoning as Part objects with Thought=true. Filter these out in both GenerateText and GenerateContent to avoid leaking thinking content into OCR output. * fix: error on empty response after filtering thinking parts GenerateContent now returns an error instead of silently returning an empty string when all response parts are thinking-only.
Adjust docs (icereed#758)
* Initial plan * Fix URL encoding for emoji tags in GetDocumentsByTag - Add url.QueryEscape() to properly encode tag parameter in API query - Add test case TestGetDocumentsByTagWithEmoji to verify emoji/Unicode tag handling - Resolves 400 error when querying documents with emoji tags like "🤖 AI-Queue" Co-authored-by: icereed <444269+icereed@users.noreply.github.com> * Improve test maintainability: use url.QueryEscape in test expectations - Replace hardcoded encoded string with url.QueryEscape() for clarity - Add net/url import to test file - Makes test more maintainable and self-documenting Co-authored-by: icereed <444269+icereed@users.noreply.github.com> * Final progress update for emoji tag URL encoding fix Co-authored-by: icereed <444269+icereed@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: icereed <444269+icereed@users.noreply.github.com>
* Initial plan
* feat: add version display in WebUI footer
- Added GET /api/version endpoint to expose version, commit, and build date
- Frontend fetches version info on mount and displays in footer
- Version appears as "paperless-gpt {version} ({short-commit})" in an unobtrusive gray text
- Commit hash is only shown if it's not the default "devCommit" value
Co-authored-by: icereed <444269+icereed@users.noreply.github.com>
* test: add test for version API endpoint
Co-authored-by: icereed <444269+icereed@users.noreply.github.com>
* fix: address code review feedback for version display
- Use absolute path for API fetch (/api/version instead of ./api/version)
- Add length check before slicing commit hash to prevent errors
- Use slice() instead of substring() for safer string operations
Co-authored-by: icereed <444269+icereed@users.noreply.github.com>
* fix: use relative path for version API fetch to support reverse proxy base paths
Changed fetch('/api/version') to fetch('./api/version') to match the
convention used by all other API calls in the codebase (DocumentProcessor,
AdhocAnalysis, Sidebar, History). The relative path resolves correctly
when the app is served under a base path like /paperless-gpt/ through
a reverse proxy.
Co-authored-by: icereed <444269+icereed@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: icereed <444269+icereed@users.noreply.github.com>
) * Skip tag update on deleted document when PDF_REPLACE=true When PDF_REPLACE is enabled, ProcessDocumentOCR uploads the OCR'd PDF as a new document and deletes the original. The subsequent UpdateDocuments call in processAutoOcrTagDocuments then fails with a 404 since the original document no longer exists. Skip the update when ReplaceOriginal is true, since the original document was already deleted and the replacement will be processed as a new document on the next cycle. Fixes icereed#910 * Check actual replacement outcome before skipping tag update Instead of relying solely on options.ReplaceOriginal (which reflects config, not outcome), add a ReplacedOriginal field to ProcessedDocument that is set to true only when uploadProcessedPDF succeeds with ReplaceOriginal enabled. The guard in processAutoOcrTagDocuments now checks this field, ensuring autoOcrTag is still removed when replacement was configured but didn't actually happen (e.g., no hOCR capability, page count mismatch, PDF generation failure, or upload failure).
…pdf mode (icereed#909) In the whole_pdf processing branch, the short variable declaration `:=` on the DownloadDocumentAsPDF call created a new local `totalPdfPages` variable that shadowed the outer one declared at function scope. After the if-block exited, the outer `totalPdfPages` remained 0, causing the safety check `processedPageCount != totalPdfPages` to always fail and skip PDF generation. The fix declares `pdfBytes` and `err` separately, then uses plain assignment `=` so that `totalPdfPages` refers to the outer variable. Fixes icereed#874
…new tags (icereed#936) * Initial plan * feat: add CREATE_NEW_TAGS env var to allow LLM to suggest new tags When CREATE_NEW_TAGS=true: - LLM prompt is updated to allow suggesting new tags beyond the available list - New tags suggested by the LLM are no longer filtered out - New tags are automatically created in paperless-ngx when applying suggestions Default behavior (CREATE_NEW_TAGS=false) is unchanged. Co-authored-by: icereed <444269+icereed@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: icereed <444269+icereed@users.noreply.github.com>
Alpine 3.21's repo GC'd 1.2.5-r9, breaking every fresh image build.
Wires langchaingo's ollama.WithThink() option to an OLLAMA_THINK environment variable in the Ollama LLM provider case of createLLM(). When the variable is unset, behavior is unchanged from current main — the model's own default applies. When set to "false", reasoning mode is disabled, which is required for thinking-mode Ollama models (Gemma 4 family, Qwen 3 family) on tasks where strict output-format compliance matters more than chain-of-thought: - Closed-list correspondent extraction (the entire premise of correspondent_prompt.tmpl) - 3-line classification output (tag prompt) - Any JSON-structured response Without this, thinking-mode models running through Ollama's OpenAI-style or langchaingo paths produce empty `content` while reasoning fills a separate field, and at production token budgets they never finish and emit the actual answer. Pattern mirrors the existing GOOGLEAI_THINKING_BUDGET handling already in this file (createLLM() googleai case): read env var, parse, append option. Adds no new dependency — langchaingo v0.1.14 (current go.mod pin) already exposes ollama.WithThink. Tested locally with gemma4:12b serving correspondent extraction on a closed list of 105 entries; without this env var, content is empty because /api/chat reasoning consumes the response budget; with OLLAMA_THINK=false, correspondent names parse cleanly from the closed list (7/8 exact match vs 6/8 for qwen2.5:14b-instruct on the same test set).
The LLM frequently emits monetary values in formats paperless-ngx
rejects (e.g. "USD1,053.52" with US thousands separators), so the
PATCH to /api/documents/{id}/ returns 400. The document is never
cleared from the AUTO_TAG queue and background.go's exponential
backoff loops on the same poison document at the 1h cap forever
(looks like a hang; it is a retry loop).
Normalize at the boundary, just before sending to Paperless:
- normalizeMonetary + normalizeCustomFieldValue helper in
normalize_monetary.go disambiguate US vs EU separators
(US "1,053.52" -> 1053.52; EU "1.053,52" -> 1053.52) and
emit Paperless-canonical form: optional 3-letter code +
plain number + exactly two decimals, no thousands separator.
- UpdateDocuments builds a field_id -> data_type map once per
call (lazily, only when at least one document has suggested
custom fields) and normalizes at all three write modes
(replace / update / append). Only data_type == "monetary"
string values are touched; everything else passes through
byte-for-byte.
Tests cover US and EU formats, currency code/symbol variants,
padding/truncation of decimals, and pathological inputs.
Refs icereed#894
When paperless-ngx rejected an UpdateDocuments PATCH with a 400
(e.g. an LLM-suggested value that fails server-side validation,
such as a malformed date like "2023-01-79"), paperless-gpt would:
- Log the error
- NOT remove the auto tag
- Return, leaving the next poll cycle to re-run the full LLM
pipeline against the same document — indefinitely.
For documents tagged with paperless-gpt-auto and processed via a
paid LLM, this billed fresh LLM calls (~6 per cycle, every ~9s)
until the tag was manually removed.
This change introduces:
1. FAIL_TAG env var (default: paperless-gpt-failed). Auto-created
in paperless-ngx at startup so the user doesn't have to.
2. Strip-and-retry in UpdateDocuments: on a 400, parse the
validation response, identify the rejected fields/custom_field
entries, drop them, and retry. The valid fields land instead
of being discarded with the bad ones. Bounded to 3 retries.
3. PartialUpdateError sentinel so the caller can apply FAIL_TAG
to a document whose update partially succeeded.
4. recoverFromFailedUpdate helper: when an update cannot be
salvaged (unparseable response, tag-related errors, retries
exhausted), explicitly remove the auto tag and add FAIL_TAG
in a tag-only PATCH. The same handling applies to the OCR
auto-tag pipeline.
Adds unit tests for the validation-error parser, the strip
helper, and the loop-break recovery path.
The previous regex `^\d{4}-\d{2}-\d{2}$` only validates the string
format, not whether the digits form a real calendar date. It accepted
impossible values like "2023-01-79" (day 79 does not exist), which
were passed through to paperless-ngx and rejected with a 400.
Replacing the regex with time.Parse("2006-01-02", ...) catches these
values before the PATCH is sent. The field is dropped and added to
partialDroppedFields so the caller applies the fail tag — same
user-visible outcome as the strip-and-retry path for post-PATCH
rejections, but with one fewer HTTP round-trip.
Adds a test verifying that UpdateDocuments returns a PartialUpdateError
with created_date in DroppedFields when given an impossible date, and
that the PATCH payload does not include the invalid field.
Added the config volume per icereed#849 Without this, custom field options would not save.
processAutoTagDocuments reads settings.CustomFieldsEnable from a background goroutine without holding settingsMutex. Other callsites (app_llm.go) already guard the same field correctly. Add the matching RLock/RUnlock guard for consistency. Closes icereed#927 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Red phase: tests for SetPrompt/GetPrompt on LLMProvider and renderOCRPrompt function that renders OCR templates per-document with existing document content. Ref: icereed#882
Moves OCR template rendering from startup-only to per-document,
following the same pattern used by title, tag, correspondent, and
all other prompt templates. Enables the OCR prompt to reference the
document's existing text via {{.Content}}.
Changes:
- Add renderOCRPrompt() for per-document template rendering
- Add SetPrompt/GetPrompt on LLMProvider for per-document overrides
- Add ExistingContent to OCROptions, passed from background processor
- Update default ocr_prompt.tmpl with conditional {{.Content}} block
- Startup rendering passes Content="" as fallback (backward compatible)
Templates without {{.Content}} work identically to before. Users with
custom prompts opt in by adding {{if .Content}}...{{end}} to their
template.
Ref: icereed#882
- Replace SetPrompt with WithPrompt to avoid mutating the shared ocrProvider singleton (concurrency safety) - Cap existing content to 8000 chars before injecting into OCR prompt to avoid blowing vision model context on long documents - Tighten language test assertion with t.Setenv and assert.Equal Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…m debug log - Remove empty-string guard so intentionally-empty templates take effect - Log prompt length instead of full content in debug output Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Introduce sanitize package to strip configured patterns from content before sending to LLMs. Supports literal string removal via REMOVE_FROM_CONTENT env var and regex pattern removal via REMOVE_FROM_CONTENT_REGEX env var. Apply sanitization to document content processing and OCR prompts to prevent sensitive information from being sent to external LLM APIs.
…or unclosed <think> tags. update tests
Add AUTO_TAG_COMPLETE environment variable to tag documents after auto-processing is complete, similar to PDF_OCR_COMPLETE_TAG for OCR processing. Default value: paperless-gpt-auto-complete
…abling The autoTagComplete tag was unconditionally added in generateDocumentSuggestions, which meant documents processed via manual review also received it. Additionally, validateOrDefaultEnvVars always defaulted the tag, making it impossible to disable. - Add IsAutoProcessing field to GenerateSuggestionsRequest to distinguish flows - Set IsAutoProcessing=true only in the background auto-processing caller - Gate auto-complete tag on both non-empty value and IsAutoProcessing flag - Use os.LookupEnv so AUTO_TAG_COMPLETE="" explicitly disables the feature - Update README to document the disable mechanism
Match the existing logging pattern used by manualTag and autoTag.
internal/textsanitize (introduced by the <think>-tag stripping fix, originally PR icereed#915) wasn't in the Dockerfile's COPY list, breaking image builds.
Ran the full tag-polling/title/correspondent/custom-field update flow against ghcr.io/paperless-ngx/paperless-ngx:beta (v3.0.0-beta.rc1) with no code changes required.
Alongside the existing upstream Docker Hub/GHCR references.
…label pull_request_target fires on ANY label being applied to a fork PR, but only the e2e-tests-local job's run_e2e_local gate actually checked for "safe-to-test". The test, build-amd64, build-arm64, and merge-manifests jobs had no such gate: they checked out the fork's untrusted head SHA and ran with repo secrets (GITHUB_TOKEN, Docker Hub creds) regardless of which label was applied, and would push an attacker-built image to the real registries. Add the same label check to all four jobs.
setupTestEnvironment started network/redis/postgres/paperlessNgx/ paperlessGpt sequentially and only returned a cleanup() once every step succeeded. If a later container failed (e.g. paperless-gpt exits because an API key env var is missing), earlier containers were never stopped, and afterAll(() => testEnv.cleanup()) crashed with "Cannot read properties of undefined (reading 'cleanup')" since testEnv was never assigned — masking the real error behind a confusing second one. Track started resources and stop them (best-effort, reverse order) on any failure before rethrowing; guard the afterAll calls with optional chaining so a setup failure surfaces as a single clean error.
Forks without LLM API key secrets set up (the default for a fresh fork) were failing E2E on every push/PR instead of just skipping it. Gate both e2e-tests and e2e-tests-local on the secret actually being present so CI goes green until/unless the secrets are added.
secrets.* isn't a valid context in a job-level if condition (only needs/vars/inputs/github are), so the previous commit's "secrets.OPENAI_API_KEY != ''" broke the whole workflow file (0 jobs scheduled, "likely failed because of a workflow file issue"). Compute it inside determine-workflow's step instead (where secrets IS available via env:) and expose it as a job output, then reference that output from the two E2E jobs' if: conditions.
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Brings this fork up to date with upstream
icereed/paperless-gptand bundles in a curated set of clean, self-contained community PRs that have been sitting unreviewed upstream. Also verifies paperless-ngx 3.0 beta compatibility and prepares GHCR image publishing for this fork.Upstream (
icereed/paperless-gpt) hasn't merged PRs since mid-April; this branch is our own base going forward so we can keep moving and ship images ourselves.What's included
Upstream sync — merges the
rootlessbranch (18 upstream commits: Anthropic/Claude provider, Gemini vision provider, Document Type Assignment,CREATE_NEW_TAGS, relative-URL fix for reverse proxies, etc.) plus the non-root Docker user change (our own PR icereed#722) plus a fork-aware CI workflow that already publishes to GHCR only for non-icereed/paperless-gptrepos.Cherry-picked upstream PRs (original authorship preserved via
git cherry-pick):OLLAMA_THINKenv varownerfield from correspondent POST when nilOLLAMA_HEADERSsupportsettings.CustomFieldsEnablewith RLock (data race fix)<think>tagsAUTO_TAG_COMPLETEenv varThese were selected out of ~50 open upstream PRs as small, self-contained, and low-risk (see full triage rationale in the session — happy to share if useful). A handful of merge conflicts (mostly two PRs touching the same function) were resolved by hand and re-verified with the full test suite.
Infra fixes found along the way:
musl-devAlpine package (upstream pin1.2.5-r9was GC'd from the Alpine 3.21 repo, breaking every fresh build).COPY internal ./internalto the Dockerfile (the newinternal/textsanitizepackage wasn't in the build context).paperless-ngx v3 compatibility — ran the full tag-polling / title / correspondent / custom-field (monetary + select) update flow against
ghcr.io/paperless-ngx/paperless-ngx:beta(v3.0.0-beta.rc1). No code changes were needed; documented in the README.GHCR publishing — the CI workflow (inherited from
rootless) already publishesghcr.io/hensing/paperless-gpton push tomain, no extra secrets required. Added this fork's GHCR image as a docker-compose option in the README.Test plan
go build ./...andgo test ./...green (Docker Go 1.25.5 container, since Go isn't installed on this host)main— confirmghcr.io/hensing/paperless-gptimage appears