Skip to content

feat: async remote storage upload with tracking, QR indicator and env… - #1603

Open
fmiccolis wants to merge 3 commits into
PhotoboothProject:devfrom
fmiccolis:feature/async-remote-upload
Open

feat: async remote storage upload with tracking, QR indicator and env…#1603
fmiccolis wants to merge 3 commits into
PhotoboothProject:devfrom
fmiccolis:feature/async-remote-upload

Conversation

@fmiccolis

Copy link
Copy Markdown
Contributor

… debug tab

Prerequisites checklist

What is the purpose of this pull request? (put an "x" next to an item)

  • Documentation update
  • Bug fix
  • New feature
  • Other, please explain:

What changes did you make? (Give an overview)

This PR makes the remote storage (FTP/SFTP) upload asynchronous, so the transition
from capture to the result screen is no longer blocked by a slow uplink, and adds
upload tracking plus two new admin debug panel tabs.

Asynchronous upload with tracking

  • applyEffects.php no longer uploads synchronously: files are enqueued in a persistent
    JSON queue (var/run/remotestorage_queue.json, new RemoteStorageQueueService) and the
    response returns immediately.
  • After rendering the result, the frontend fires a non-blocking request to the new
    api/remoteStorageUpload.php, which drains the queue in the background:
    ignore_user_abort(), single-drainer lock (flock), session_write_close() (critical:
    without it the PHP session lock would freeze the booth UI during uploads), priority for
    the just-captured photo, and a circuit breaker after 2 consecutive failures so an
    unreachable server does not pile up ~90s FTP timeouts.
  • Per-file status (pending/uploading/done/failed), attempts and last error are persisted.
    Retries happen on later drains (max 3 attempts); failed uploads can be listed/retried/
    cleared via the new admin-protected api/remoteStorageAdmin.php.
  • The QR code still appears immediately on the result screen (its URL never depended on the
    upload); a small spinner next to it turns into a check/error icon, driven by a cheap
    status-polling mode of the same endpoint (same pattern as gallery.php?status).
  • deletePhoto.php cancels queued uploads and skips remote round-trips for photos that
    were never uploaded.

createWebpage() fixes/improvements

  • Fixed the existence check (config.phpconfig.inc.php): before this, the early-return
    condition could never be true, so the webpage template + config were re-uploaded on
    every single photo.
  • The webpage creation now runs at most once per storage folder (flag cached in the queue
    file) instead of doing remote fileExists round-trips per photo.
  • New ftp.template_config_location option (default private/template/config.inc.php):
    an optional PHP file returning an array that is merged over the generated
    config.inc.php defaults — the counterpart of the already-customizable
    template_location for people using a custom remote gallery template. Also applied to
    the test/remote-storage-template.php preview.

New debug panel tabs

  • Environment & network: OS, PHP version, all network interfaces with IP, netmask and
    CIDR network (via net_get_interfaces()), Wi-Fi SSID (best-effort via iwgetid), and
    one QR code per IPv4 address encoding the photobooth URL on that network — so an
    operator can quickly open the booth (or SSH/VNC into it) from another device.
  • Remote storage queue: live table of the upload queue (status, attempts, errors,
    timestamps), compatible with the existing auto-refresh toggle.
  • Added the missing translation label for the existing remote storage log tab.

Behavior changes to be aware of

  • Chroma captures with keying.show_all = false are no longer uploaded to remote storage
    (previously they were uploaded and then deleted locally; the remote gallery now mirrors
    local visibility).
  • After 3 failed attempts an upload stays failed until retried via the admin endpoint —
    deliberately not retried on every capture so an offline server cannot slow down an event.
  • With useForQr + append_filename the QR deep link is now <gallery>/?photo=<file>
    instead of <gallery>/images/<file> (see review notes below).

Is there anything you'd like reviewers to focus on?

  • Session lock: session_write_close() in the drain/status endpoints is load-bearing;
    please double-check I haven't missed a path that would hold the session while uploading.
  • Queue concurrency: single-drainer lock, the enqueue-vs-drainer-exit race (mitigated by
    a 1.5s linger re-check, a one-shot re-trigger from the status poller, and the next
    capture's trigger), and stale uploading recovery (entries reset after 10 min).
  • QR deep link change (/?photo= instead of /images/): the bundled gallery template
    currently ignores the photo query parameter, so with the default template the QR opens
    the gallery page rather than the raw image file. I use a custom template that supports
    it. If you prefer, I can split this into a separate PR or extend the bundled template to
    honor ?photo=.
  • On Apache/mod_php the drain request occupies one worker while a backlog is uploading;
    set_time_limit(0) cannot override server-level timeouts — leftover entries are simply
    picked up by the next drain.

Tested on a live installation during a real event (FTP over a slow uplink): the result
screen now appears immediately, the queue drains in the background, and the
failure/recovery paths were exercised by taking the FTP server offline mid-event.

AI used to create this Pull Request?

Yes. This PR was developed with Claude Code (Anthropic). The AI wrote the initial
implementation (queue service, endpoints, frontend changes, debug tabs) based on a design
we discussed and I decided on; I reviewed every change and tested the full flow on a real
installation at a live event (successful uploads, failure/retry paths, burst captures,
offline-server recovery). npm run build and npm run eslint were run (clean), and all
touched PHP files were syntax-checked.

fmiccolis and others added 3 commits July 15, 2026 18:17
- Environment & network tab: one QR code per IPv4 address encoding the
  photobooth URL on that network, to open it from another device
- New remote storage queue tab showing per-file upload status, attempts,
  errors and timestamps (works with the existing auto-refresh toggle)
- Add missing label for the remote storage log tab

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The build workflow runs "npm run build", which sorts en.json with
localeCompare and then requires a clean working tree. The new
remoteUpload* keys were placed after "rembglog" but localeCompare
orders them after "remoteStorageTemplate", so the build modified the
file and the status check failed. it.json is aligned for consistency.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@flacoonb

Copy link
Copy Markdown
Contributor

Hi, thanks for working on this. This PR is very similar in scope to my earlier PR #1463, which also implements asynchronous FTP/SFTP uploads with a persistent queue, retries and upload tracking.

The main difference seems to be the architecture: #1603 uses a JSON queue processed through a background HTTP request, while #1463 uses SQLite and a dedicated Symfony/systemd worker. It may be worth comparing the approaches and seeing whether parts can be consolidated.

@fmiccolis

Copy link
Copy Markdown
Contributor Author

Hi @flacoonb and sorry for not digging in the previous PRs before submitting mine.
I was the original creator of the FTP upload feature but I never had the time to review its fallacies (one above all the synchronous upload).
One of the things that I would like to keep of your approach is the semplification of the admin settings regarding folder, baseFolder, website ecc because it can be just one property.
I want to know why you used SQLite/systemd, what are the strengths of this choice?
I preferred to not introduce others services along the existing ones in the photobooth.

@flacoonb

Copy link
Copy Markdown
Contributor

Hi @fmiccolis, no problem at all — it is easy to miss an older open PR, especially in a project with many ongoing contributions. I am glad you brought your approach forward, as the overlap gives us a good opportunity to compare both implementations and potentially combine the strongest parts.

I chose SQLite because it provides a persistent transactional queue without requiring an external database server. It allows atomic job claiming, reliable recovery after crashes or restarts, and structured tracking of attempts, errors and filename mappings.

The systemd worker keeps uploads fully independent from PHP and the web request lifecycle. This avoids request timeouts, session locks and occupied web-server workers during slow or unavailable FTP connections. It also gives automatic startup and restart on failure.

The downside is additional installation complexity and a stronger dependency on Linux/systemd. Your HTTP-based approach is simpler and integrates more naturally into the existing Photobooth setup, so I can understand that choice.

I agree that the remote-storage configuration should be simplified, and it may be useful to combine the strongest parts of both PRs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants