diff --git a/.claude/skills/fess-doc-review/refs/source-mapping.md b/.claude/skills/fess-doc-review/refs/source-mapping.md index a119e56..c02c44c 100644 --- a/.claude/skills/fess-doc-review/refs/source-mapping.md +++ b/.claude/skills/fess-doc-review/refs/source-mapping.md @@ -2,15 +2,21 @@ Map each doc file to its primary source reference. +15.9 moved the four SSO authenticators into `fess-sso-*` repositories and the S3/GCS storage clients +into `fess-storage-*`; the rows below point at those. Reviewing 15.8 or earlier docs, read the same +classes under `repos/fess/src/main/java/org/codelibs/fess/sso/` and `.../storage/` on the `15.8.x` +branch instead. + | Doc Topic | Primary Source | |-----------|---------------| | Core config properties | `repos/fess/src/main/resources/fess_config.properties` | | Config constants/defaults | `repos/fess/src/main/java/org/codelibs/fess/mylasta/direction/FessConfig.java` | | Application constants | `repos/fess/src/main/java/org/codelibs/fess/Constants.java` | -| SSO (SAML) | `repos/fess/src/main/java/org/codelibs/fess/sso/saml/SamlAuthenticator.java`, `SamlCredential.java` (attribute mapping), `FessProp.java` (permission/behavior properties) | -| SSO (OIDC) | `repos/fess/src/main/java/org/codelibs/fess/sso/oic/OpenIdConnectAuthenticator.java`, `repos/fess/src/main/java/org/codelibs/fess/app/web/base/login/OpenIdConnectCredential.java` (user ID/groups/roles extraction), `FessProp.java` (permission/behavior properties) | -| SSO (Entra ID) | `repos/fess/src/main/java/org/codelibs/fess/sso/entraid/EntraIdAuthenticator.java`, `FessProp.java` (permission/behavior properties e.g. `entraid.permission.fields`, `entraid.use.ds`) | -| SSO (SPNEGO) | `repos/fess/src/main/java/org/codelibs/fess/sso/spnego/SpnegoAuthenticator.java`, `FessProp.java` (permission/behavior properties) | +| SSO (type dispatch, missing plugin) | `repos/fess/src/main/java/org/codelibs/fess/sso/SsoManager.java` — resolves the authenticator as `Authenticator`, maps the legacy `aad` to `entraid`, and warns which plugin to install; `FessProp.java` holds `sso.type` | +| SSO (SAML) | `repos/fess-sso-saml/src/main/java/org/codelibs/fess/sso/saml/SamlAuthenticator.java`, `SamlCredential.java` (attribute mapping); `saml.*` keys come from `WEB-INF/conf/system.properties` | +| SSO (OIDC) | `repos/fess-sso-oidc/src/main/java/org/codelibs/fess/sso/oic/OpenIdConnectAuthenticator.java`, `OpenIdConnectCredential.java` (user ID/groups/roles extraction — the credential moved out of `org.codelibs.fess.app.web.base.login`). The type stays `sso.type=oic` | +| SSO (Entra ID) | `repos/fess-sso-entraid/src/main/java/org/codelibs/fess/sso/entraid/EntraIdAuthenticator.java`, `EntraIdCredential.java`; `FessProp.java` still owns `entraid.permission.fields` and `entraid.use.ds`, each with an `aad.*` fallback | +| SSO (SPNEGO) | `repos/fess-sso-spnego/src/main/java/org/codelibs/fess/sso/spnego/SpnegoAuthenticator.java` — the `SpnegoConfig.getInitParameter()` switch is the authoritative list of `spnego.*` keys and their defaults | | API endpoints | `repos/fess/src/main/java/org/codelibs/fess/api/` | | LLM/RAG chat (core) | `repos/fess/src/main/java/org/codelibs/fess/llm/AbstractLlmClient.java` | | LLM/RAG chat (session/history) | `repos/fess/src/main/java/org/codelibs/fess/chat/ChatClient.java`, `ChatSessionManager.java` | @@ -23,6 +29,9 @@ Map each doc file to its primary source reference. | Crawler clients (S3) | `repos/fess-crawler/fess-crawler/src/main/java/org/codelibs/fess/crawler/client/s3/S3Client.java` | | Crawler clients (GCS) | `repos/fess-crawler/fess-crawler/src/main/java/org/codelibs/fess/crawler/client/gcs/GcsClient.java` | | Crawler clients (Storage) | `repos/fess-crawler/fess-crawler/src/main/java/org/codelibs/fess/crawler/client/storage/StorageClient.java` | +| Storage clients (registry, config) | `repos/fess/src/main/java/org/codelibs/fess/storage/StorageClientFactory.java` — resolves `StorageClient`; `StorageType.java` (endpoint auto-detection), `FessProp.java` (`storage.*` accessors), `AdminStorageAction.java` (admin UI) | +| Storage clients (S3) | `repos/fess-storage-s3/src/main/java/org/codelibs/fess/storage/s3/S3StorageClient.java`; `fess_storage++.xml` registers `s3StorageClient` and `s3_compatStorageClient`, and `crawler/client++.xml` registers the `s3:` crawler client via `S3ClientCreator` | +| Storage clients (GCS) | `repos/fess-storage-gcs/src/main/java/org/codelibs/fess/storage/gcs/GcsStorageClient.java`; `fess_storage++.xml` registers `gcsStorageClient`, and `crawler/client++.xml` registers the `gcs:` crawler client via `GcsClientCreator` | | Datastore connectors (implementations) | `repos/fess-ds-*/src/main/java/` — each plugin's DataStore subclass(es) | | Datastore connectors (client/auth params) | `repos/fess-ds-*/src/main/java/` — plugin's `*Client.java` or `*Helper.java` classes (e.g., `Microsoft365Client.java`, `BoxClient.java`) define shared authentication and connection parameters (`proxy_*`, `cache_size`, `max_content_length`, etc.) that are common to all connectors in the plugin but not declared in the DataStore classes | | Datastore handler registration | `repos/fess-ds-*/src/main/resources/fess_ds++.xml` — `` entries define registered handler names | diff --git a/.claude/skills/fess-tech-doc/SKILL.md b/.claude/skills/fess-tech-doc/SKILL.md index 7b58b0f..d8dae2b 100644 --- a/.claude/skills/fess-tech-doc/SKILL.md +++ b/.claude/skills/fess-tech-doc/SKILL.md @@ -20,6 +20,8 @@ Understand the Fess features to cover by reading source code — never write abo - Data store plugins: `repos/fess-ds-*/` — each repo = one data source connector - LLM plugins: `repos/fess-llm-*/` — RAG/AI integration (OpenAI, Gemini, Ollama) - Webapp plugins: `repos/fess-webapp-*/` — web extensions (MCP server, etc.) +- SSO plugins: `repos/fess-sso-*/` — one authenticator each (`sso.type=saml`, `spnego`, `entraid`, `oic`) +- Storage plugins: `repos/fess-storage-*/` — object storage clients (`storage.type`, `s3:`/`gcs:` crawling) - Existing docs: `repos/fess-docs/ja/{version}/admin/`, `repos/fess-docs/ja/{version}/config/` **Verification checklist:** diff --git a/.claude/skills/fess-version-bump/SKILL.md b/.claude/skills/fess-version-bump/SKILL.md index 3130e72..0eb3341 100644 --- a/.claude/skills/fess-version-bump/SKILL.md +++ b/.claude/skills/fess-version-bump/SKILL.md @@ -26,8 +26,9 @@ Wave 1 : fess-parent (bumps project version + fess*.vers Wave 2 : fess-suggest, fess-crawler (independent of each other; both consumed by downstream) Wave 3 : fess-crawler-playwright (depends on fess-crawler) Wave 4 : fess (depends on parent + suggest + crawler + crawler-playwright) -Wave 5 : every remaining plugin (fess-ds-*, fess-llm-*, fess-theme-*, fess-webapp-*, - fess-script-*, fess-ingest-*, fess-thumbnail-playwright) +Wave 5 : every remaining plugin (fess-ds-*, fess-llm-*, fess-sso-*, fess-storage-*, + fess-theme-*, fess-webapp-*, fess-script-*, + fess-ingest-*, fess-thumbnail-playwright) ``` After opening the PRs for a wave, tell the user which PRs are open, then **wait**. The user merges them and deploys SNAPSHOTs; they will signal when to proceed. diff --git a/CLAUDE.md b/CLAUDE.md index fe58e74..9b8f9b3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -54,6 +54,22 @@ docker-fess -> fessctl -> fess-test-ui, fess-docs 1. Change `repos/fess/pom.xml` packaging to `jar` 2. `cd repos/fess && mvn clean install -DskipTests` 3. Revert packaging back to `war` +- **15.9 moves the S3/GCS storage clients and the four SSO authenticators out of `repos/fess`** + into `fess-storage-{s3,gcs}` and `fess-sso-{saml,spnego,entraid,oidc}`. The storage half is + already on `master`, so `S3StorageClient` and `GcsStorageClient` are only on `15.8.x` and earlier; + the SSO half is fess#3430, so until that merges `repos/fess` carries the four authenticators too. + Core keeps the extension point and the admin UI: `StorageClientFactory` resolves + `StorageClient`, `SsoManager` resolves `Authenticator`, `FessProp` keeps + `sso.type` and the `storage.*` accessors, and the General screen still offers all four SSO types + (a plugin cannot supply a JSP). Each plugin reads its own `saml.*` / `spnego.*` / + `entraid.*` / `oic.*` keys from `WEB-INF/conf/system.properties`. `sso.type=oic` is served by + `fess-sso-oidc` - the one place a type and its repo name differ. +- **Build a `fess-sso-*` / `fess-storage-*` plugin against the same Fess minor version it is + installed into.** An SSO plugin in an older war is not a silent mismatch: core ships its own + `fess_sso++.xml` up to 15.8, LastaDi merges every `++` file on the class path, and the pair + registers the same component name twice - `container.init()` still succeeds and `getComponent()` + then throws `TooManyRegistrationComponentException`. Storage has no counterpart before 15.9: + `fess_storage.xml` and the `StorageClient` lookup are both new in 15.9. - Run `mvn formatter:format && mvn license:format` in each repo separately - Custom repo sets: copy `sets/custom.yaml.example` to `sets/my-set.yaml`, then pass `my-set` to any script (e.g. `./scripts/build.sh my-set`) @@ -61,7 +77,8 @@ docker-fess -> fessctl -> fess-test-ui, fess-docs - **Default branch is `master`**, not `main` (`defaults.branch` in `sets/*.yaml`). `main` is a per-repo override - incl. fess-parent, fess-themes, java-saml, jcifs, fesen-httpclient, - fess-crawler-playwright. `repos/fess` has no `main` branch at all. + fess-crawler-playwright, and every `fess-sso-*` / `fess-storage-*`. `repos/fess` has no `main` + branch at all. - `FESS_WORKSPACE_GIT_SSH=true` switches clone/sync remotes from HTTPS to SSH. - `build.sh` writes `logs/build/.log`; without `--verbose` Maven output goes only there. Check that file first when a build fails. @@ -73,8 +90,10 @@ docker-fess -> fessctl -> fess-test-ui, fess-docs ## Code Reference Repo naming: `fess` (app), `fess-crawler*` (crawlers), `fess-suggest`, `fess-parent` (dependency -BOM), `fess-ds-*` (data stores), `fess-llm-*`, `fess-webapp-*` (plugins), `fess-theme*` / -`fess-themes` (themes); the rest are CodeLibs libraries. +BOM), `fess-ds-*` (data stores), `fess-llm-*`, `fess-webapp-*` (plugins), `fess-sso-*` (SSO +authenticators), `fess-storage-*` (object storage clients), `fess-theme*` / `fess-themes` (themes); +the rest are CodeLibs libraries. A plugin repo's prefix is also its `PluginHelper.ArtifactType`, so +a new plugin family means a new entry there (`STORAGE` and `SSO` were added in 15.9). See `sets/*.yaml` for full repository listings (`all.yaml`, `core.yaml`, `plugins.yaml`). diff --git a/README.md b/README.md index 5ba32c2..b1210fe 100644 --- a/README.md +++ b/README.md @@ -40,7 +40,7 @@ Cloning defaults to HTTPS. Export `FESS_WORKSPACE_GIT_SSH=true` to use SSH remot ``` fess-workspace/ ├── sets/ # Repository set definitions -│ ├── all.yaml # Complete ecosystem (48 repositories) +│ ├── all.yaml # Complete ecosystem (54 repositories) │ ├── core.yaml # Core components only │ ├── plugins.yaml # core.yaml + data store / theme plugins │ └── custom.yaml.example # Template for custom sets @@ -117,6 +117,28 @@ fess-workspace/ | **fess-llm-openai** | main | OpenAI chat / embedding integration | | **fess-llm-gemini** | main | Google Gemini chat / embedding integration | +#### Storage Plugins (`build_order` 10) + +| Repository | Branch | Description | +|------------|--------|-------------| +| **fess-storage-s3** | main | Amazon S3 and S3-compatible object storage (`storage.type=s3`, `s3:` crawling) | +| **fess-storage-gcs** | main | Google Cloud Storage (`storage.type=gcs`, `gcs:` crawling) | + +#### SSO Plugins (`build_order` 10) + +| Repository | Branch | Description | +|------------|--------|-------------| +| **fess-sso-saml** | main | SAML 2.0 (`sso.type=saml`) | +| **fess-sso-spnego** | main | Integrated Windows Authentication (`sso.type=spnego`) | +| **fess-sso-entraid** | main | Microsoft Entra ID (`sso.type=entraid`, and the legacy `aad`) | +| **fess-sso-oidc** | main | OpenID Connect — the type is `sso.type=oic`, not `oidc` | + +Both groups carry features that were part of the Fess distribution up to 15.8. From 15.9 core keeps +only the extension point — `SsoManager` resolving `Authenticator`, `StorageClientFactory` +resolving `StorageClient` — and the implementation, plus the SDK it needs, ships as +one of these plugins. `fess-storage-*` additionally registers the matching `s3:` / `gcs:` crawler +client through `crawlerClientCreator`; that client class itself stays in `fess-crawler`. + #### WebApp and Other Plugins (`build_order` 10) | Repository | Branch | Description | @@ -163,7 +185,8 @@ Minimal set for core development: `fess-parent`, `fess-crawler`, `fess-crawler-p `includes: core.yaml` plus data store connectors (`fess-ds-db`, `-csv`, `-json`, `-office365`, `-salesforce`, `-slack`, `-box`, `-dropbox`, `-s3`, `-gsuite`, `-atlassian`, `-gitbucket`, `-git`) -and `fess-theme-simple`. +and `fess-theme-simple`. It covers data stores only — the LLM, webapp, script, storage and SSO +plugins are in `all.yaml`. ## Script Reference @@ -264,6 +287,11 @@ Build order: libraries (`corelib`, `curl4j`, `java-saml`, `jcifs`, `jhighlight`, with `NoClassDefFoundError: jakarta/annotation/PostConstruct`. - `repos/fess` uses `war`. To install it as a jar for plugin compilation, temporarily switch the packaging to `jar`, run `mvn clean install -DskipTests`, then revert. +- **A `fess-sso-*` or `fess-storage-*` plugin has to match the Fess minor version it is installed + into.** Up to 15.8 core declares the four authenticators in its own `fess_sso++.xml`, and LastaDi + merges every `++` file on the class path, so a 15.9 SSO plugin on an older war registers the same + component name twice: the container still initializes and the lookup then throws + `TooManyRegistrationComponentException`. - Run `mvn formatter:format && mvn license:format` in each repository before committing. ## Custom Sets diff --git a/sets/all.yaml b/sets/all.yaml index 48d5c08..53f48cb 100644 --- a/sets/all.yaml +++ b/sets/all.yaml @@ -111,6 +111,30 @@ repositories: branch: main build_order: 10 + - name: fess-sso-entraid + branch: main + build_order: 10 + + - name: fess-sso-oidc + branch: main + build_order: 10 + + - name: fess-sso-saml + branch: main + build_order: 10 + + - name: fess-sso-spnego + branch: main + build_order: 10 + + - name: fess-storage-gcs + branch: main + build_order: 10 + + - name: fess-storage-s3 + branch: main + build_order: 10 + - name: fess-themes branch: main build_order: 10