diff --git a/README.md b/README.md index 502cab8..917fda3 100644 --- a/README.md +++ b/README.md @@ -1,112 +1,79 @@ -

- MThread Draw — a cat's portrait assembling itself out of white touch points -

- -

- Every white circle is one touch point, in the order it is sent to the phone. Nothing about that picture is illustrated: - it is a real tools/make_banner.py run over examples/cat.jpg. -

- -

- Play it in your browser - Download -

- -

- CI - Latest release - Downloads - Licence: AGPL-3.0 - Python 3.9+ - Windows | macOS | Linux - Stars -

- -

- A colour photograph of a guitar being traced into touch strokes and drawn on a phone screen -

- -

- An ordinary photograph in, 57 strokes and 478 touch points out: the exact path list mthread sends to the device, in the order it draws them.
- On a Pixel 8 Pro that draws in under two seconds. Rendered from a real run by tools/make_demo.py; playback speed here is arbitrary. -

+
---- + + MThread Draw — a motorcycle assembling itself out of touch strokes, one stroke at a time + -> ### Licence in one line -> -> **AGPL-3.0, with a commercial licence available.** Use it, change it, share -> it — but a version you distribute, or run as a service other people use, has -> to publish its source under the AGPL too. To put it inside a product whose -> source stays closed, [ask for a commercial licence](https://github.com/MAXAWER/MThread-Draw/issues). -> Full explanation, in English and Russian: **[TERMS.md](TERMS.md)**. -> -> **AGPL-3.0 плюс коммерческая лицензия.** Пользуйтесь, меняйте, делитесь — но -> распространяемая версия и сервис на её основе обязаны публиковать исходники -> под AGPL. Чтобы встроить в продукт с закрытым кодом, нужна коммерческая -> лицензия. Подробно: [TERMS.md](TERMS.md). +The drawing above draws itself, and it is not an illustration: those are the 232 strokes
+tools/make_hero.py gets out of examples/motorcycle.jpg, in the order the program sends them to a phone. ---- +

-## Try it before you install it +Download for Windows +Download for macOS +Try it in your browser -**[maxawer.github.io/MThread-Draw](https://maxawer.github.io/MThread-Draw/)** puts -three of these drawings in your browser and asks you to trace one by hand. The -lines are not a mock-up: they are the exact strokes this program sends to a -phone, exported from a real run over the photographs in [`examples/`](examples/), -and the time to beat is the time the program takes. +
-A guitar is 294 points and lands in about a second. Tracing it yourself takes -most people twenty. +CI +Latest release +Downloads +Licence: AGPL-3.0 +Native front ends +По-русски -## Get it +

-| | | -|---|---| -| **Windows** | [**Download the installer**](https://github.com/MAXAWER/MThread-Draw/releases/latest) — `MThreadDraw-x.y.z-x64.msi`. Installs like any other program, Start Menu entry and uninstaller included. | -| **macOS** | [**Download the app**](https://github.com/MAXAWER/MThread-Draw/releases/latest) — `.dmg` for Apple Silicon or Intel. Drag it to Applications. | -| **Linux** | The command line, from source; three commands, [below](#from-source). | +### It draws pictures on an Android screen by touching it,
and it records and replays gestures. Nothing is installed on the phone. -Both windows are native: WinUI 3 on Windows, SwiftUI on macOS, and each drives -the same engine over a pipe. Linux has the command line and the library, which -is everything except a window. +USB or wireless ADB · no root on most devices · no Android SDK · one download, everything inside -**Nothing else to install.** Python, OpenCV and **adb** all travel inside the -application — no Android SDK, no platform-tools download, no `PATH` to edit. +
-The builds are not code-signed, because certificates cost money this project -does not take. Windows SmartScreen says "unknown publisher" once — *More info* → -*Run anyway*. macOS wants a right-click → **Open** on the first launch. +A colour photograph of a guitar being traced into touch strokes and drawn on a phone screen -## Then, in three steps +A photograph in, 57 strokes and 478 touch points out — the exact path list sent to the device.
+On a Pixel 8 Pro that draws in under two seconds. -1. **Turn on USB debugging** on the phone: Settings → About phone → tap *Build - number* seven times → Developer options → *USB debugging*. -2. **Plug it in.** Or go wireless: `adb connect 192.168.1.42:5555`. -3. **Open MThread Draw.** It finds the phone by itself, shows its screen live, - and lays the drawing over exactly where it will land. Drag to move it, wheel - to resize, Shift and wheel to turn — then **START DRAWING**. +
-### What you can do to a drawing before it is drawn +--- + +## In about a minute -| | | +| | | |---|---| -| **Move, resize, turn, flip** | Drag it over the live view; wheel resizes, Shift and wheel turns, and `Flip` mirrors it. `Fit` puts it back. | -| **Layers** | Load several pictures and arrange them against each other. Each keeps its own position and its own tracer settings; hide one, reorder them, remove one. | -| **Erase single strokes** | Turn the eraser on and drag across the lines you do not want. `Undo erase` brings them back. | -| **Change the tracing** | The *detail* slider re-traces what is already loaded. There is no need to open the file again. | -| **No live view?** | If capture fails, *Use a screenshot…* takes a picture you copied off the phone by hand. It does not update, but its proportions are what placement needs. | +| **1 · Install** | [**Windows**](https://github.com/MAXAWER/MThread-Draw/releases/latest) — `MThreadDraw-x.y.z-x64.msi`, Start Menu entry and uninstaller included. [**macOS**](https://github.com/MAXAWER/MThread-Draw/releases/latest) — `.dmg` for Apple Silicon or Intel, drag it to Applications. **Linux** — the command line, [from source](#from-source). | +| **2 · Wake the phone** | Settings → About phone → tap **Build number** seven times → Developer options → **USB debugging**. Plug it in, or `adb connect 192.168.1.42:5555`. | +| **3 · Draw** | It finds the phone, shows the screen live, and lays the drawing over exactly where it will land. Drag to move, wheel to resize, Shift and wheel to turn — then **START DRAWING**. | -Nothing to install and nothing to draw with a mouse: +**Nothing else to install.** Python, OpenCV and **adb** all travel inside the +application. Both windows are native — **WinUI 3** on Windows, **SwiftUI** on +macOS — and each drives the same engine over a pipe, so neither has its own idea +of how anything works. The builds are not signed: SmartScreen says "unknown +publisher" once, and macOS wants a right-click → **Open** the first time. + +--- + +## What the window lets you do + +| | | +|---|---| +| **Place it exactly** | Drag the drawing across the live view; wheel resizes, Shift and wheel turns, `Flip` mirrors, `Fit` starts over. Held in fractions of the screen, so it survives the phone being turned. | +| **Layers** | Several pictures arranged against each other, each with its own placement and its own tracer settings. Hide one, reorder them, remove one. | +| **Erase single strokes** | Drag across the lines you do not want with the eraser on; `Undo erase` brings them back. | +| **Re-trace in place** | The detail slider re-traces what is already loaded. No need to open the file again. | +| **Record and replay** | Press record, do something on the phone, press stop. The file holds fractions of the screen, so it **replays on a different phone**, at any speed, any number of times. | +| **No live view?** | If capture fails, a screenshot you copied off the phone by hand will do: it does not update, but its proportions are what placement needs. | + +Nothing to prepare, from the command line or from code: ```bash mthread shape heart # a heart, fitted to the screen -mthread shape star --points 7 # a seven-pointed star, turned however you like mthread text "hello" --y 0.35 # words, in any font this machine has +mthread record -o login.json # then: mthread play login.json --speed 2 ``` -Or from code: - ```python from mthread import Device Device().draw_paths([[(100, 200), (400, 200), (400, 600)]]) @@ -114,190 +81,115 @@ Device().draw_paths([[(100, 200), (400, 200), (400, 600)]]) --- -## How an image becomes touches +## How a photograph becomes touches -

- Source photograph, the lines the tracer finds, and the resulting stroke paths -

+
+Source photograph, the lines the tracer finds, and the resulting stroke paths +
-Feed it an ordinary photograph. A tracer decides where the lines are, the result -is thinned to one pixel of width, and each line is then walked into a single -stroke — not an outline *around* the line, which is what draws everything twice. -The picture above is `examples/guitar.jpg`, untouched: 57 strokes, 478 points. +A tracer decides where the lines are, the result is thinned to one pixel wide, +and each line is walked into a single stroke — not an outline *around* the line, +which is what draws everything twice. Above is `examples/guitar.jpg`, untouched: +57 strokes, 478 points. -There are two tracers, and the app picks between them by asking what is in the -picture rather than which algorithm you would like: +The app asks what is in the picture rather than which algorithm you would like: | What you say is in it | What runs | Why that one | |---|---|---| -| **Buildings, machines, objects** | Canny edges, thinned, then walked into strokes | Keeps every bit of structure an edge detector sees — which is what a machine, a tower or a building is made of. | -| **Portraits, animals, nature** | Flow-based coherent lines, after Kang, Lee and Chui | Works out the direction each line runs in and filters along it. Calmer, longer strokes, and a face stays a face instead of becoming film grain. | +| **Buildings, machines, objects** | Canny edges, thinned, walked into strokes | Keeps every bit of structure an edge detector sees — which is what a machine or a building is made of. | +| **Portraits, animals, nature** | Flow-based coherent lines, after Kang, Lee and Chui | Works out the direction each line runs in and filters along it: calmer, longer strokes, and a face stays a face. | -Neither wins everywhere, which is why both are still here. There is also an -opt-in third method, `method="neural"`, which asks a trained model which edges -matter; it needs a 46 MB download and a few seconds, and it is better than both -on some photographs and worse than both on grainy ones. +Neither wins everywhere, which is why both are here. Colour is what gets lost — +a finger draws one black line, so the output is always a line drawing. -
-The whole path, from a JPEG to a finger on the glass - -```mermaid -flowchart LR - A["photograph"] --> B["tracer
Canny or flow"] - B --> C["thin to one pixel"] - C --> D["walk each line
into one stroke"] - D --> E["join, simplify,
drop specks"] - E --> F{"how should
it draw?"} - F -->|instantly| G["1 ms a point"] - F -->|like a hand| H["rounded corners,
velocity, tremor"] - G --> I{"does the device
allow raw touch?"} - H --> I - I -->|yes| J["sendevent
into /dev/input"] - I -->|no| K["injector jar
through app_process"] - J --> L["the phone draws"] - K --> L -``` +
+Four photographs and the line drawings traced from them: a guitar, a motorcycle, a cat and a lighthouse -Every box is a module: `mthread.vectorize`, `mthread.trace`, `mthread.paths`, -`mthread.hand`, `mthread.injector`. The branch at the bottom is the one that -matters in practice — see [Why this exists](#why-this-exists). +Nothing prepared, retouched or masked — the files in examples/, resized and otherwise untouched.
+The only thing that differs between the columns is the two sliders every user has. +
-
+--- -### The same pipeline, four photographs +## Why it exists -

- Four photographs and the line drawings traced from them: a guitar, a motorcycle, a cat and a lighthouse -

+`adb shell input tap` spawns a process on the device for every call. At +100–300 ms each, anything continuous — a gesture, a line, a test script — is +unusably slow. `mthread` gets the whole drawing onto the device in one go +instead: kernel events through a single pushed script where the phone allows it, +and a 3 KB injector run through `app_process` where it does not. A stroke that +takes 40 seconds through `input swipe` finishes in well under a second. -Nothing was prepared, retouched or masked — these are the files in -[`examples/`](examples/), resized and otherwise untouched. The only thing that -differs between the columns is the two sliders every user has: the cat gets one -notch more *Detail*, and the lighthouse one notch less *Edge sensitivity*, -because it stands under a sky full of stars and every star is an edge. -`python tools/make_demo.py` reproduces the whole picture. +Two pieces, and either works without the other: **`mthread`**, a Python library +for synthetic touch input whose core has no dependencies at all, and **MThread +Draw**, the desktop application on top of it. -Colour is what gets lost: a finger draws one black line, so the output is always -a line drawing. Illustrations and line art come out closest to the original, -photographs come out as their edges — the *Edge sensitivity* and *Detail* sliders -decide how much detail survives, and an optional background remover (`rembg`) -helps with portraits and product shots. +**[How it works, in detail →](docs/INTERNALS.md)** — the digitizer's own +coordinate space, the three ways into a device and why a recent Pixel refuses +the fast one, what "instant" costs, and how it draws like a hand. --- ## What it works with -| | | +| | | |---|---| -| **Devices** | Any Android phone or tablet that `adb devices` lists — over USB, or wireless ADB (`adb connect :5555`). Root is not needed on most devices. | -| **Emulators** | Anything exposing an ADB port: Android Studio AVD, BlueStacks (`:5555`), LDPlayer (`:5555`), Nox (`:62001`), MEmu (`:21503`). Raw `/dev/input` support differs between builds — `mthread info` tells you in one line, and [device reports](https://github.com/MAXAWER/MThread-Draw/issues/new?template=device_report.md) are welcome. | -| **Image formats** | PNG, JPEG, BMP, WebP. Raster only for now; SVG input is [an open task](https://github.com/MAXAWER/MThread-Draw/issues). | +| **Devices** | Anything `adb devices` lists, over USB or wireless ADB. Root is not needed on most devices. | +| **Emulators** | Android Studio AVD, BlueStacks (`:5555`), LDPlayer (`:5555`), Nox (`:62001`), MEmu (`:21503`). Raw `/dev/input` support differs between builds — `mthread info` says which path yours gets, and [device reports](https://github.com/MAXAWER/MThread-Draw/issues/new?template=device_report.md) are welcome. | +| **Images** | PNG, JPEG, BMP, WebP. Raster only for now. | | **Host** | Windows, macOS, Linux. Python 3.9+. | -## What people use it for - -- **Drawing games and canvases on the phone** — Gartic Phone, Skribbl.io, sketch - chats, whiteboards, notes apps: anything where the picture has to be produced by - an actual finger on the glass. -- **Signatures and stamps** you would otherwise redraw by hand every time. -- **QA and regression passes** — record a login flow once, replay it against every - build, at 2x, ten times in a row. -- **Repetitive tapping** in apps that offer no other automation hook. - -Whether automating a particular game is allowed is between you and that game's -rules; this is a general-purpose input tool. - ---- - -## Why this exists - -`adb shell input tap` spawns a process on the device for every single call. At -100–300 ms each, anything continuous — a gesture, a drawn line, a test script — -is unusably slow. - -`mthread` writes raw kernel input events into `/dev/input` through **one** pushed -shell script instead. A stroke that takes 40 seconds through `input swipe` -finishes in well under a second. That single difference is what makes both -gesture replay and image drawing practical. - -This repository is two things: - -- **`mthread`** — a small Python library for fast synthetic touch input on Android - over ADB. Records gestures, replays them, drives raw `/dev/input` events. Pure - standard library; the core has no dependencies at all. -- **`MThread Draw`** — a desktop app built on it, for people who would rather click - buttons than write code. There are two front ends: a portable one in Python - that runs anywhere, and a native WinUI 3 one for Windows. Both drive the same - engine over a pipe, so neither has its own idea of how anything works. - ---- - -## Record and replay gestures - -Press record, do something on the phone, press stop. You get a JSON file of -strokes and their timing, and you can replay it whenever you want, at whatever -speed. - -```bash -mthread record -o login.json # do the thing on the phone, press Enter -mthread play login.json --speed 2 --repeat 5 -``` - -**The file replays on a different phone.** It stores each touch as a fraction of -the screen rather than as a digitizer coordinate, so playing it back on another -device scales it to that screen. This was not true of the old format, and could -not be: a digitizer's range has little to do with any display, which is why -replaying one elsewhere used to be refused rather than attempted. Nor could the -old format be replayed on a *current* phone at all, since it went through -`/dev/input`, which every recent Pixel denies to the shell. Playback now goes -through the injector, like drawing. - -Useful for regression passes, for reproducing a bug reliably, or for any -repetitive tapping you would rather not do by hand. +Used for drawing games and canvases, signatures and stamps, QA passes that +replay a recorded flow against every build, and repetitive tapping in apps with +no other automation hook. Whether automating a particular game is allowed is +between you and that game's rules; this is a general-purpose input tool. +
+From source, and building the applications yourself -## From source +
[`run.bat`](run.bat) on Windows and [`run.sh`](run.sh) elsewhere do the whole thing: virtual environment, dependencies, and `adb` if the machine has none. -Otherwise, by hand: +Otherwise: ```bash git clone https://github.com/MAXAWER/MThread-Draw.git cd MThread-Draw pip install -e . # library only - no dependencies at all -pip install -e ".[draw]" # + image vectorisation (OpenCV, NumPy, Pillow) -pip install -e ".[gui]" # + the desktop app +pip install -e ".[draw]" # + image tracing (OpenCV, NumPy, Pillow) pip install -e ".[bg]" # + rembg background removal ``` `adb` is found in this order: `ADB_PATH`, the copy inside a packaged build, your `PATH`, a `platform-tools` directory beside the working directory, then the usual -Android SDK locations. If you have none of those: +Android SDK locations. If you have none of those, +`python tools/fetch_platform_tools.py` fetches it — 7 MB, straight from Google. ```bash -python tools/fetch_platform_tools.py # ~7 MB, straight from Google +pip install pyinstaller +python tools/build_app.py --msi # Windows: engine, WinUI front end, installer +python tools/build_macos.py --dmg # macOS: the bundle and its disk image ``` -To build the packaged application and its installer yourself: +The installer needs WiX: `dotnet tool install --global wix --version 5.0.2`. +[Releasing](docs/RELEASING.md) covers the rest. -```bash -pip install pyinstaller -python tools/build_app.py --msi # Windows, needs `dotnet tool install --global wix --version 5.0.2` -python tools/build_app.py --dmg # macOS -``` +
+ +
+Command line — every command, and the options they share -## Command line +
```bash mthread devices # what is attached mthread info # screen size and digitizer ranges -mthread shape heart # draw a shape, nothing to prepare +mthread shape heart # heart, star, circle, square, polygon, spiral, wave mthread shape star --points 7 --rotate 20 -mthread shape circle --scale 0.4 --x 0.25 --y 0.3 mthread text "hello world" # any font the machine has mthread text "signed" --font arial.ttf --scale 0.5 --y 0.8 @@ -305,16 +197,20 @@ mthread record -o session.json # record until Enter mthread play session.json --speed 0.5 --repeat 3 ``` -Every drawing command takes the same placement options: `--scale`, `--rotate`, -`--flip-x`, `--flip-y`, `--x`, `--y`, `--margin`, and `--speed`/`--human` for -how it is drawn. The shapes are `heart`, `star`, `circle`, `square`, `polygon`, -`spiral` and `wave`. +Every drawing command takes the same placement options — `--scale`, `--rotate`, +`--flip-x`, `--flip-y`, `--x`, `--y`, `--margin` — and `--speed`/`--human` for +how it draws. + +Text is rendered with a real font and then traced, which is why any font works +and why letters come out as outlines: a filled glyph is a shape with an inside +and an outside, and this draws with one finger. -Text is rendered with a real font and then traced, which is why any font on the -machine works and why the letters come out as outlines: a filled glyph is a -shape with an inside and an outside, and this draws with one finger. +
-## Library +
+Library — the whole API in ten lines + +
```python from mthread import Device, Recorder, Session, replay @@ -330,300 +226,54 @@ recorder.stop().save("flow.json") replay(device, Session.load("flow.json"), speed=2.0, repeat=10) ``` ---- - -## How it works - -**Batched events.** Every stroke becomes a list of `sendevent` lines, written to a -temporary script, pushed once to `/data/local/tmp`, executed, and deleted. One ADB -round trip instead of thousands. - -**Coordinate translation.** The touchscreen digitizer has its own coordinate -space, and on many phones it is *not* the display resolution — a 1080-pixel-wide -screen commonly sits on a 4096-step digitizer. Sending display pixels straight to -`sendevent` puts the touch in the wrong place. `mthread` reads the real axis -ranges from `getevent -pl` and rescales. Run `mthread info` to see yours. - -**Three ways in, picked automatically.** Writing kernel events is fastest, but a -recent Android refuses it: SELinux denies the shell domain write access to -`/dev/input` however the file permissions read, so `sendevent` fails per line -while the script exits cleanly. Where that happens, `mthread` streams points to -a small injector it runs on the device instead - one process for a whole -drawing, with the wait between points under our control. Failing even that, it -shells out to `input` once per point, which needs nothing installed and costs -about 110 ms each. `mthread info` says which path your device gets. - -**Drawing like a hand.** Timing is what gives a machine away, and the injector is -what makes timing ours to choose. `mthread.hand` rounds corners, varies pen -speed along a stroke, adds a slow tremor, overshoots stroke ends and reorders -strokes the way a person would; `Pacing` decides how long each point takes. Set -`human=0` and it draws as fast as the receiving app can sample - about 6 ms a -point, since anything faster arrives between frames and is never seen. - -**Retrace removal.** `findContours` walks the *boundary* of a region, and Canny -turns one pen stroke into two parallel edges — so the naive path traces up one -side of every line and back down the other, drawing everything twice. -`dedupe_retrace` detects when a contour's two halves are the same stroke and keeps -one of them, while leaving genuine closed shapes like circles intact. +
---- +
+Known limits, said plainly -## Known limits +
-- **Recordings are portable now**, holding fractions of a screen rather than - digitizer coordinates. A file written by an older version is not, and says so - rather than misfiring. - **A recording does not know which way up the phone was.** Drawing follows the - orientation - the engine reads it from the captured frame, since `wm size` does - not change when a phone is turned - but a recording holds fractions of the - screen it was made on, so replaying a portrait one in landscape lands sideways. - Replay in the orientation you recorded in. -- **Replay timing carries a fixed overhead.** Starting and stopping the on-device - injector costs a second or two, so a two-second recording takes longer than two - seconds. The strokes and the gaps between them are faithful; the total is not, - and that matters if you are timing something. + orientation; a recording holds fractions of the screen it was made on, so a + portrait recording replayed in landscape lands sideways. +- **Replay carries a fixed overhead.** Starting and stopping the on-device + injector costs a second or two: the strokes and the gaps are faithful, the + total is not. +- **Stop is not instantaneous.** It cancels what has not been queued yet and the + device finishes what it already has — about two seconds' worth. +- **Recordings from before 1.2 are not portable** and say so rather than + misfiring. - **`mthread info` is the first thing to check** when touches land in the wrong place. ---- - -## Open ends - -Contributions welcome — see [CONTRIBUTING.md](CONTRIBUTING.md). Issues labelled -[`good first issue`](https://github.com/MAXAWER/MThread-Draw/labels/good%20first%20issue) -are the easiest way in. Things worth doing: - -- SVG input, so line art skips edge detection entirely. -- Auto-detect swapped X/Y axes (the `swap_xy` flag exists but nothing sets it). -- Rotation in recordings: store which way up the phone was, and turn a replay to - match. Drawing already follows the orientation; replay does not. -- Take the fixed second or two out of replay by keeping the injector alive - between runs. -- Trim recordings visually in the app; cut dead time at the start and end. -- Assertions during replay — wait for a screenshot to match before continuing, - which is what turns this into a real test runner. -- Pressure-sensitive strokes from image darkness. -- Shed OpenCV. Five of its functions are used and it is half the download. +
--- -## If something does not work +## Help, and helping -Open an issue — there are templates for +Something not working? Open an issue — there are templates for [bugs](https://github.com/MAXAWER/MThread-Draw/issues/new?template=bug_report.md) -and for [device reports](https://github.com/MAXAWER/MThread-Draw/issues/new?template=device_report.md). -Touches landing in the wrong place, a phone that refuses to connect, an emulator -behaving differently — paste the output of `mthread info` and it is usually a -short fix. Digitizer ranges differ wildly between panels, and only what people -report can be handled. +and [device reports](https://github.com/MAXAWER/MThread-Draw/issues/new?template=device_report.md). +Paste the output of `mthread info`; digitizer ranges differ wildly between +panels, and only what people report can be handled. -**If this saved you an afternoon, a ⭐ costs nothing and is how anyone else finds -it.** +Contributions welcome — [CONTRIBUTING.md](CONTRIBUTING.md), and +[`good first issue`](https://github.com/MAXAWER/MThread-Draw/labels/good%20first%20issue) +is the easiest way in. [What is worth doing next →](docs/INTERNALS.md#open-ends) ## Licence -**AGPL-3.0** — see [LICENSE](LICENSE). A **commercial licence** is available from -the author for use in products that will not publish their source. Both are -explained in plain English and Russian in **[TERMS.md](TERMS.md)**. - ---- - -
-По-русски - -## Что это - -Инструмент, который **рисует картинки на экране Android** и **записывает и -повторяет жесты** — по USB или беспроводному ADB, без установки чего-либо на сам -телефон. - -Две части в одном репозитории: - -- **`mthread`** — библиотека для быстрого синтетического ввода касаний на Android - через ADB. Записывает жесты, воспроизводит их, работает с событиями - `/dev/input` напрямую. Ядро не требует зависимостей. -- **`MThread Draw`** — десктопное приложение поверх неё, для тех, кто предпочитает - кнопки коду. - -## Установка - -| | | -|---|---| -| **Windows** | [**Скачать установщик**](https://github.com/MAXAWER/MThread-Draw/releases/latest) — `MThreadDraw-x.y.z-x64.msi`. Ставится как обычная программа, с ярлыком в меню «Пуск» и деинсталлятором. | -| **macOS** | [**Скачать приложение**](https://github.com/MAXAWER/MThread-Draw/releases/latest) — `.dmg` для Apple Silicon или Intel, перетащить в Applications. | -| **Linux** | Командная строка из исходников. Окна для Linux нет: оба интерфейса нативные — WinUI 3 и SwiftUI. | -| **Linux** | Из исходников, три команды — [ниже](#из-исходников). | - -**Больше ничего ставить не нужно.** Python, OpenCV и **adb** лежат внутри самого -приложения: ни Android SDK, ни platform-tools скачивать не придётся, `PATH` -трогать тоже. - -Сборки не подписаны — сертификаты стоят денег, которых у проекта нет. Windows -один раз скажет «неизвестный издатель»: *Подробнее* → *Выполнить в любом случае*. -На macOS первый запуск — правой кнопкой → **Открыть**. +**AGPL-3.0, with a commercial licence available from the author.** Use it, change +it, share it, free of charge — but a version you **distribute**, or **run as a +service** other people use, has to publish its complete source under the AGPL +too, a rebranded copy included. To put it inside a product whose source stays +closed, [ask for a commercial licence](https://github.com/MAXAWER/MThread-Draw/issues/new?title=Licence%20request). -## Дальше три шага +Binding text: [LICENSE](LICENSE). Plain language, English and Russian: +**[TERMS.md](TERMS.md)**. -1. **Включить отладку по USB**: Настройки → О телефоне → семь раз по «Номер - сборки» → Для разработчиков → Отладка по USB. -2. **Подключить телефон.** Или по Wi-Fi: `adb connect 192.168.1.42:5555`. -3. **Открыть MThread Draw.** Телефон находится сам, его экран показывается - живьём, а рисунок ложится поверх того места, куда он попадёт. Тащите мышью, - колесо меняет размер, Shift с колесом поворачивает — затем **START DRAWING**. - -Что можно сделать с рисунком до отрисовки: - -| | | -|---|---| -| **Двигать, размер, поворот, отражение** | Перетаскивание по живому экрану; колесо — размер, Shift с колесом — поворот, `Flip` — отражение, `Fit` — вписать заново. | -| **Слои** | Несколько картинок, расставленных друг относительно друга. У каждой своё положение и свои настройки трассировки; слой можно спрятать, переставить, удалить. | -| **Ластик** | Включите ластик и проведите по ненужным линиям. `Undo erase` возвращает их. | -| **Пересчёт** | Ползунок детализации перетрассирует уже загруженное. Открывать файл заново не нужно. | -| **Нет живого экрана?** | Если захват не работает, *Use a screenshot…* берёт снимок, снятый на телефоне вручную. Он не обновляется, но его пропорции — это то, что нужно для размещения. | - - - -### Из исходников - -```bash -git clone https://github.com/MAXAWER/MThread-Draw.git -cd MThread-Draw -pip install -e ".[gui]" -mthread_draw -``` - -Проще запустить [`run.bat`](run.bat) на Windows или [`run.sh`](run.sh) на macOS и -Linux — они сами создадут виртуальное окружение, поставят зависимости и скачают -`adb`, если своего на машине нет. - -`adb` ищется по порядку: `ADB_PATH`, копия внутри собранного приложения, ваш -`PATH`, папка `platform-tools` рядом с рабочим каталогом, затем обычные пути -Android SDK. Если ничего из этого нет: - -```bash -python tools/fetch_platform_tools.py # ~7 МБ, прямо от Google -``` - -## Как картинка превращается в касания - -Подаёте обычную фотографию, готовить её заранее не нужно. Трассировщик находит -линии, результат утончается до одного пикселя, и каждая линия проходится одним -штрихом — не обводится по контуру, иначе всё рисовалось бы дважды. Гитара выше -это `examples/guitar.jpg` без единой правки: 57 штрихов, 478 точек. - -Трассировщика два, и приложение спрашивает не про алгоритм, а про то, что на -фотографии: - -| Что на фотографии | Что работает | Почему именно это | -|---|---|---| -| **Здания, техника, объекты** | Границы по Кэнни, утончение, обход в штрихи | Сохраняет всю структуру, которую видит детектор границ, — а машина или башня из неё и состоит. | -| **Портреты, животные, природа** | Когерентные линии по направлению потока | Считает, куда идёт каждая линия, и фильтрует вдоль неё: штрихи длиннее и спокойнее, лицо остаётся лицом, а не зерном плёнки. | - -Ни один не выигрывает везде — поэтому остались оба. Есть и третий, по желанию: -`method="neural"` спрашивает у обученной модели, какие границы важны; ему нужны -46 МБ модели и несколько секунд. - -

- Четыре фотографии и штриховые рисунки, полученные из них -

- -Ничего не готовилось и не ретушировалось: это файлы из [`examples/`](examples/), -только уменьшенные. Между колонками отличаются лишь те два ползунка, что есть у -любого пользователя: коту добавлена одна ступень *Detail*, маяку убрана одна -ступень *Edge sensitivity* — он стоит под звёздным небом, а каждая звезда это -граница. Всю картинку целиком собирает `python tools/make_demo.py`. - -Теряется цвет: палец рисует одну чёрную линию, поэтому результат всегда штриховой. -Ближе всего к оригиналу выходят иллюстрации и контурные рисунки, из фотографии -получатся её границы. Ползунки *Edge sensitivity* и *Detail* решают, сколько -деталей останется, а опциональное удаление фона (`rembg`) помогает с портретами и -предметной съёмкой. - -## С чем работает - -| | | -|---|---| -| **Устройства** | Любой телефон или планшет, который виден в `adb devices` — по USB или по Wi-Fi (`adb connect :5555`). На большинстве устройств root не нужен. | -| **Эмуляторы** | Всё, что открывает порт ADB: Android Studio AVD, BlueStacks (`:5555`), LDPlayer (`:5555`), Nox (`:62001`), MEmu (`:21503`). Поддержка сырого `/dev/input` отличается от сборки к сборке — `mthread info` покажет за одну строку. Отчёты о конкретных устройствах приветствуются. | -| **Форматы** | PNG, JPEG, BMP, WebP. Пока только растр; SVG — в списке задач. | -| **Хост** | Windows, macOS, Linux. Python 3.9+. | - -## Зачем это нужно - -`adb shell input tap` запускает отдельный процесс на устройстве при каждом -вызове — 100–300 мс на команду. Для чего-либо непрерывного это неприемлемо -медленно. `mthread` пишет события ядра напрямую через **один** сценарий, -загруженный на устройство. Штрих, который через `input swipe` рисуется 40 секунд, -здесь занимает меньше секунды. - -## Что с этим делают - -- **Рисовалки на телефоне** — Gartic Phone, Skribbl.io, скетч-чаты, заметки и - доски: всё, где картинку нужно вывести пальцем по стеклу. -- **Подписи и штампы**, которые иначе приходится перерисовывать вручную. -- **Тестирование**: записали сценарий логина один раз — прогоняете на каждой - сборке, на удвоенной скорости, десять раз подряд. -- **Однообразные нажатия** в приложениях, где других способов автоматизации нет. - -Допустимо ли автоматизировать конкретную игру — вопрос её правил; это инструмент -ввода общего назначения. - -## Командная строка - -```bash -mthread devices # какие устройства подключены -mthread info # разрешение экрана и диапазоны тачскрина - -mthread shape heart # нарисовать фигуру, ничего не готовя -mthread shape star --points 7 --rotate 20 -mthread text "привет" # текст любым шрифтом, что есть в системе - -mthread record -o session.json # запись до нажатия Enter -mthread play session.json --speed 2 --repeat 5 -``` - -У всех команд рисования одни и те же параметры размещения: `--scale`, -`--rotate`, `--flip-x`, `--flip-y`, `--x`, `--y`, `--margin`, а также -`--speed`/`--human` — как рисовать. Фигуры: `heart`, `star`, `circle`, -`square`, `polygon`, `spiral`, `wave`. - -Текст рисуется настоящим шрифтом и затем трассируется — поэтому доступен любой -шрифт системы, и поэтому буквы выходят контурами: залитая глифа это фигура с -внутренней и внешней границей, а здесь рисует один палец. - -Если `adb` установлен в нестандартное место — укажите путь в переменной -окружения `ADB_PATH`. - -## Ограничения - -- Записи **переносятся между телефонами**: внутри доли экрана, а не сырые - координаты дигитайзера. Файлы старого формата не переносятся и честно об этом - сообщают, а не рисуют мимо. -- **Запись не знает, как был повёрнут телефон.** Рисование ориентацию учитывает — - движок читает её из снятого кадра, поскольку `wm size` при повороте не - меняется, — но запись хранит доли того экрана, на котором сделана, поэтому - портретная запись в горизонтальной ориентации ляжет набок. -- **У воспроизведения есть постоянная накладная стоимость** — секунда-две на - запуск и остановку инжектора. Штрихи и паузы между ними точны, а общая - длительность нет. Старые записи — сырые координаты - тачскрина. Попытка воспроизвести запись на панели другого размера будет - отклонена, а не выполнена криво. -- Поворот экрана не учитывается: записывайте и воспроизводите в одной ориентации. -- На части устройств тачскрин недоступен для `sendevent`. Тогда рисование - завершится понятной ошибкой, а не кривым результатом; `Device.swipe()` - продолжает работать, автоматический откат — в списке задач. -- Если касания попадают не туда — начните с `mthread info`. - -## Если что-то не работает - -Заведите issue — есть шаблоны для -[багов](https://github.com/MAXAWER/MThread-Draw/issues/new?template=bug_report.md) -и для [отчётов об устройстве](https://github.com/MAXAWER/MThread-Draw/issues/new?template=device_report.md). -Касания не туда, телефон не подключается, эмулятор ведёт себя иначе — приложите -вывод `mthread info`, обычно это чинится быстро. Диапазоны координат тачскрина у -разных панелей разные, и починить можно только то, что видно. - -**Если инструмент сэкономил вам вечер — звезда ⭐ ничего не стоит, а найти проект -другим людям помогает.** - -
+
+
+If this saved you an afternoon, a ⭐ costs nothing and is how anyone else finds it. +
diff --git a/README.ru.md b/README.ru.md new file mode 100644 index 0000000..764a7b4 --- /dev/null +++ b/README.ru.md @@ -0,0 +1,274 @@ +
+ + + MThread Draw — мотоцикл, собирающийся из штрихов касаний + + +Рисунок выше рисует себя сам, и это не иллюстрация: это те самые 232 штриха, которые
+tools/make_hero.py получает из examples/motorcycle.jpg — в том порядке, в котором программа отправляет их на телефон. + +

+ +Скачать для Windows +Скачать для macOS +Попробовать в браузере + +
+ +CI +Последний релиз +Лицензия AGPL-3.0 +Нативные интерфейсы +In English + +

+ +### Рисует картинки на экране Android, касаясь его,
записывает и повторяет жесты. На телефон ничего не ставится. + +USB или ADB по Wi-Fi · на большинстве устройств без root · без Android SDK · одна загрузка, всё внутри + +
+ +Фотография гитары, превращаемая в штрихи и рисуемая на экране телефона + +На входе фотография, на выходе 57 штрихов и 478 точек касания — ровно тот список путей, что уходит на устройство.
+На Pixel 8 Pro это рисуется меньше чем за две секунды. + +
+ +--- + +## Примерно за минуту + +| | | +|---|---| +| **1 · Установить** | [**Windows**](https://github.com/MAXAWER/MThread-Draw/releases/latest) — `MThreadDraw-x.y.z-x64.msi`, с ярлыком в «Пуске» и деинсталлятором. [**macOS**](https://github.com/MAXAWER/MThread-Draw/releases/latest) — `.dmg` для Apple Silicon или Intel, перетащить в Applications. **Linux** — командная строка, [из исходников](#из-исходников). | +| **2 · Разбудить телефон** | Настройки → О телефоне → семь раз по «Номер сборки» → Для разработчиков → **Отладка по USB**. Подключить кабель или `adb connect 192.168.1.42:5555`. | +| **3 · Рисовать** | Телефон находится сам, экран показывается живьём, рисунок ложится поверх того места, куда попадёт. Тащить мышью, колесо — размер, Shift с колесом — поворот, затем **START DRAWING**. | + +**Больше ничего ставить не нужно.** Python, OpenCV и **adb** лежат внутри самого +приложения. Оба окна нативные — **WinUI 3** на Windows и **SwiftUI** на macOS — и +оба разговаривают с одним и тем же движком по каналу, так что у них не может +быть разных представлений о происходящем. Сборки не подписаны: Windows один раз +скажет «неизвестный издатель», а macOS попросит первый запуск через правую +кнопку → **Открыть**. + +--- + +## Что можно сделать в окне + +| | | +|---|---| +| **Разместить точно** | Перетаскивание по живому экрану; колесо — размер, Shift с колесом — поворот, `Flip` — отражение, `Fit` — вписать заново. Положение хранится в долях экрана и переживает поворот телефона. | +| **Слои** | Несколько картинок друг относительно друга, у каждой своё положение и свои настройки трассировки. Слой можно спрятать, переставить, удалить. | +| **Ластик по штрихам** | Включить ластик и провести по ненужным линиям; `Undo erase` возвращает их. | +| **Пересчёт на месте** | Ползунок детализации перетрассирует уже загруженное — открывать файл заново не нужно. | +| **Запись и повтор** | Нажать запись, поделать что-нибудь на телефоне, остановить. Файл хранит доли экрана, поэтому **воспроизводится на другом телефоне**, с любой скоростью и сколько угодно раз. | +| **Нет живого экрана?** | Если захват не работает, подойдёт снимок, снятый на телефоне вручную: он не обновляется, но его пропорции — это то, что нужно для размещения. | + +Ничего не готовя — из командной строки или из кода: + +```bash +mthread shape heart # сердце, вписанное в экран +mthread text "привет" --y 0.35 # текст любым шрифтом, что есть в системе +mthread record -o login.json # затем: mthread play login.json --speed 2 +``` + +```python +from mthread import Device +Device().draw_paths([[(100, 200), (400, 200), (400, 600)]]) +``` + +--- + +## Как фотография превращается в касания + +
+Исходная фотография, найденные линии и итоговые штрихи +
+ +Трассировщик находит линии, результат утончается до одного пикселя, и каждая +линия проходится одним штрихом — не обводится по контуру, иначе всё рисовалось бы +дважды. Выше — `examples/guitar.jpg` без единой правки: 57 штрихов, 478 точек. + +Приложение спрашивает не про алгоритм, а про то, что на фотографии: + +| Что на фотографии | Что работает | Почему именно это | +|---|---|---| +| **Здания, техника, объекты** | Границы по Кэнни, утончение, обход в штрихи | Сохраняет всю структуру, которую видит детектор границ, — а машина или башня из неё и состоит. | +| **Портреты, животные, природа** | Когерентные линии по направлению потока | Считает, куда идёт каждая линия, и фильтрует вдоль неё: штрихи длиннее и спокойнее, лицо остаётся лицом, а не зерном плёнки. | + +Ни один не выигрывает везде — поэтому остались оба. Теряется цвет: палец рисует +одну чёрную линию, поэтому результат всегда штриховой. + +
+Четыре фотографии и штриховые рисунки, полученные из них + +Ничего не готовилось и не ретушировалось — это файлы из examples/, только уменьшенные.
+Между колонками отличаются лишь те два ползунка, что есть у любого пользователя. +
+ +--- + +## Зачем это нужно + +`adb shell input tap` запускает отдельный процесс на устройстве при каждом +вызове — 100–300 мс на команду, и для чего-либо непрерывного это неприемлемо +медленно. `mthread` отправляет весь рисунок на устройство целиком: события ядра +одним загруженным сценарием там, где телефон это позволяет, и инжектор на 3 КБ +через `app_process` там, где нет. Штрих, который через `input swipe` рисуется +40 секунд, здесь занимает меньше секунды. + +Две части, и любая работает без другой: **`mthread`** — библиотека +синтетического ввода касаний, ядру которой не нужно вообще никаких зависимостей, +и **MThread Draw** — приложение поверх неё. + +**[Как это устроено, подробно →](docs/INTERNALS.md)** (по-английски) — своя +система координат у тачскрина, три пути внутрь устройства и почему свежий Pixel +отказывает в самом быстром, чего стоит «мгновенно» и как это рисует по-человечески. + +--- + +## С чем работает + +| | | +|---|---| +| **Устройства** | Всё, что видно в `adb devices` — по USB или по Wi-Fi. На большинстве устройств root не нужен. | +| **Эмуляторы** | Android Studio AVD, BlueStacks (`:5555`), LDPlayer (`:5555`), Nox (`:62001`), MEmu (`:21503`). Поддержка сырого `/dev/input` отличается от сборки к сборке — `mthread info` покажет за одну строку. | +| **Картинки** | PNG, JPEG, BMP, WebP. Пока только растр. | +| **Хост** | Windows, macOS, Linux. Python 3.9+. | + +Используют для рисовалок и досок на телефоне, подписей и штампов, тестовых +прогонов, где записанный сценарий повторяют на каждой сборке, и однообразных +нажатий там, где других способов автоматизации нет. Допустимо ли автоматизировать +конкретную игру — вопрос её правил; это инструмент ввода общего назначения. + + +
+Из исходников и как собрать приложения самому + +
+ +[`run.bat`](run.bat) на Windows и [`run.sh`](run.sh) на macOS и Linux делают всё +сами: окружение, зависимости и `adb`, если своего нет. Иначе вручную: + +```bash +git clone https://github.com/MAXAWER/MThread-Draw.git +cd MThread-Draw + +pip install -e . # только библиотека - зависимостей нет вообще +pip install -e ".[draw]" # + трассировка картинок (OpenCV, NumPy, Pillow) +pip install -e ".[bg]" # + удаление фона (rembg) +``` + +`adb` ищется по порядку: `ADB_PATH`, копия внутри собранного приложения, ваш +`PATH`, папка `platform-tools` рядом с рабочим каталогом, затем обычные пути +Android SDK. Если ничего из этого нет, `python tools/fetch_platform_tools.py` +скачает его — 7 МБ, прямо от Google. + +```bash +pip install pyinstaller +python tools/build_app.py --msi # Windows: движок, интерфейс WinUI, установщик +python tools/build_macos.py --dmg # macOS: бандл и образ диска +``` + +Установщику нужен WiX: `dotnet tool install --global wix --version 5.0.2`. + +
+ +
+Командная строка — все команды и общие параметры + +
+ +```bash +mthread devices # какие устройства подключены +mthread info # разрешение экрана и диапазоны тачскрина + +mthread shape heart # heart, star, circle, square, polygon, spiral, wave +mthread shape star --points 7 --rotate 20 +mthread text "привет" # текст любым шрифтом, что есть в системе +mthread text "подпись" --font arial.ttf --scale 0.5 --y 0.8 + +mthread record -o session.json # запись до нажатия Enter +mthread play session.json --speed 2 --repeat 5 +``` + +У всех команд рисования одни и те же параметры размещения — `--scale`, +`--rotate`, `--flip-x`, `--flip-y`, `--x`, `--y`, `--margin` — и +`--speed`/`--human`, отвечающие за то, как рисовать. + +Текст рисуется настоящим шрифтом и затем трассируется — поэтому доступен любой +шрифт системы, и поэтому буквы выходят контурами: залитая глифа это фигура с +внутренней и внешней границей, а здесь рисует один палец. + +
+ +
+Библиотека — весь интерфейс в десяти строках + +
+ +```python +from mthread import Device, Recorder, Session, replay + +device = Device() +print(device.screen_size, device.touch_device.path) + +recorder = Recorder(device) +recorder.start() +input("Сделайте что-нибудь на телефоне и нажмите Enter...") +recorder.stop().save("flow.json") + +replay(device, Session.load("flow.json"), speed=2.0, repeat=10) +``` + +
+ +
+Ограничения, честно + +
+ +- **Запись не знает, как был повёрнут телефон.** Рисование ориентацию учитывает, + а запись хранит доли того экрана, на котором сделана, поэтому портретная + запись в горизонтальной ориентации ляжет набок. +- **У воспроизведения есть постоянная накладная стоимость** — секунда-две на + запуск и остановку инжектора. Штрихи и паузы точны, общая длительность нет. +- **Стоп не мгновенный.** Он отменяет то, что ещё не отправлено, а устройство + дорисовывает уже полученное — около двух секунд. +- **Записи старше 1.2 не переносятся** между телефонами и честно об этом + сообщают, а не рисуют мимо. +- **Начинайте с `mthread info`**, если касания попадают не туда. + +
+ +--- + +## Помощь и участие + +Что-то не работает — заведите issue, есть шаблоны для +[багов](https://github.com/MAXAWER/MThread-Draw/issues/new?template=bug_report.md) +и [отчётов об устройстве](https://github.com/MAXAWER/MThread-Draw/issues/new?template=device_report.md). +Приложите вывод `mthread info`: диапазоны координат тачскрина у разных панелей +разные, и починить можно только то, что видно. + +Участие приветствуется — [CONTRIBUTING.md](CONTRIBUTING.md), а метка +[`good first issue`](https://github.com/MAXAWER/MThread-Draw/labels/good%20first%20issue) +самый простой вход. + +## Лицензия + +**AGPL-3.0 плюс коммерческая лицензия от автора.** Пользуйтесь, меняйте, +делитесь, бесплатно — но распространяемая версия и сервис на её основе обязаны +опубликовать полный исходный код под AGPL, включая перекрашенную копию. Чтобы +встроить в продукт с закрытым кодом, нужна +[коммерческая лицензия](https://github.com/MAXAWER/MThread-Draw/issues/new?title=Licence%20request). + +Юридический текст: [LICENSE](LICENSE). Человеческим языком, по-русски и +по-английски: **[TERMS.md](TERMS.md)**. + +
+
+Если инструмент сэкономил вам вечер — звезда ⭐ ничего не стоит, а найти проект другим людям помогает. +
diff --git a/docs/INTERNALS.md b/docs/INTERNALS.md new file mode 100644 index 0000000..ae9dc82 --- /dev/null +++ b/docs/INTERNALS.md @@ -0,0 +1,134 @@ +# How it works + +The parts that are not obvious from the code, and the measurements that decided +them. The [README](../README.md) says what the program does; this says why it +does it that way. + +## From a JPEG to a finger on the glass + +```mermaid +flowchart LR + A["photograph"] --> B["tracer
Canny or flow"] + B --> C["thin to one pixel"] + C --> D["walk each line
into one stroke"] + D --> E["join, simplify,
drop specks"] + E --> F{"how should
it draw?"} + F -->|instantly| G["1 ms a point"] + F -->|like a hand| H["rounded corners,
velocity, tremor"] + G --> I{"does the device
allow raw touch?"} + H --> I + I -->|yes| J["sendevent
into /dev/input"] + I -->|no| K["injector jar
through app_process"] + J --> L["the phone draws"] + K --> L +``` + +Every box is a module: `mthread.vectorize`, `mthread.trace`, `mthread.paths`, +`mthread.hand`, `mthread.injector`. The branch at the bottom is the one that +matters in practice. + +## The touchscreen has its own coordinate space + +On many phones it is *not* the display resolution: a 1080-pixel-wide screen +commonly sits on a 4096-step digitizer. Sending display pixels straight to +`sendevent` puts the touch somewhere else entirely. `mthread` reads the real axis +ranges out of `getevent -pl` and rescales. `mthread info` prints yours, and it is +the first thing to look at when touches land in the wrong place. + +## Three ways into a device, picked automatically + +`Device.draw_paths` chooses; `mthread info` says which one this device gets. + +| | | | +|---|---|---| +| **`raw`** | `sendevent` into `/dev/input` | Fastest, and refused by any recent Pixel. SELinux denies the shell domain write access whatever the file mode says, and `sendevent` then fails per line while the script exits cleanly — so it looks like it worked. `Device.supports_raw_touch` probes for it rather than trusting the permissions. | +| **`injector`** | A 3 KB jar run once through `app_process`, fed points over stdin | Works everywhere, and it is the only path where the time between points is ours. That is what makes both instant drawing and hand-like drawing possible. Built by `tools/build_injector.py`. | +| **`input`** | `input motionevent`, one process per point | About 110 ms each. The last resort, and it needs nothing installed. | + +Two things about the injector that are easy to break: + +- Events sharing a millisecond get coalesced, so it forces event time forward + for every event. +- The host loop is bounded by a look-ahead. The injector sleeps on the device, + so an unbounded loop queues the whole drawing in one breath and **Stop** has + nothing left to cancel. + +## "Instant" is not zero delay + +The receiving app samples input once a frame, so a stroke delivered in under a +millisecond arrives as a press and a release with nothing in between. Measured +on a Pixel 8 Pro against a 1,679-point drawing: + +| Delay between points | What arrived | +|---|---| +| 0 ms | Two thirds of the points lost | +| **1 ms** | **All of them, in 5.0 seconds** | +| 6 ms | All of them, in 19 seconds | + +1 ms is the default for that reason. + +## Drawing like a hand + +Timing is what gives a machine away, and the injector is what makes timing ours +to choose. `mthread.hand` rounds corners, varies pen speed along a stroke, adds a +slow tremor, overshoots stroke ends and reorders strokes the way a person would. +`Pacing` decides how long each point takes; `human=0` skips all of it. + +## Retrace removal + +`findContours` walks the *boundary* of a region, and Canny turns one pen stroke +into two parallel edges — so the naive path traces up one side of every line and +back down the other, drawing everything twice. `dedupe_retrace` detects when a +contour's two halves are the same stroke and keeps one of them, while leaving +genuine closed shapes like circles intact. + +## Recordings + +`mthread.gestures` decodes raw touch events into strokes of `(time, x, y)` with +the coordinates as fractions of the screen, so `Device.play_gestures` can scale +them to whatever screen it is given. + +The old format stored digitizer coordinates, which is why replaying one +elsewhere was refused: a digitizer's range has little to do with any display. +Nor could it be replayed on a *current* phone at all, since it went through +`/dev/input`. Playback now goes through the injector, like drawing. + +Two things had to be learned to record anything at all: + +- `adb shell` needs **two** `-t` flags to force a pty. Without one, libc buffers + `getevent` output at 4 KB and a short recording produces nothing. A single + `-t` is refused when stdin is not a terminal. +- The recorder listens to **every** input device by default. The node Android + calls the touchscreen is not always the one touches arrive on. + +## Screen mirroring + +`mthread.mirror` drives a small on-device class over `adb shell -T` and reads +base64 frames. Not `exec-out`, which has no stdin; not a plain `shell`, whose +pty expands `\n` to `\r\n` and corrupts every frame after the first. + + + +## Open ends + +Contributions welcome — see [CONTRIBUTING.md](../CONTRIBUTING.md). Issues +labelled [`good first issue`](https://github.com/MAXAWER/MThread-Draw/labels/good%20first%20issue) +are the easiest way in. + +- SVG input, so line art skips edge detection entirely. +- Auto-detect swapped X/Y axes: the `swap_xy` flag exists but nothing sets it. +- Rotation in recordings: store which way up the phone was and turn a replay to + match. Drawing already follows the orientation; replay does not. +- Take the fixed second or two out of replay by keeping the injector alive + between runs. +- Trim recordings visually in the app; cut dead time at the start and end. +- Assertions during replay — wait for a screenshot to match before continuing, + which is what turns this into a real test runner. +- Pressure-sensitive strokes from image darkness. +- Shed OpenCV. Five of its functions are used and it is half the download. + +## See also + +- [RELEASING.md](RELEASING.md) — how a tag becomes an installer, and what each + front end needs to build. +- [DEMO.md](DEMO.md) — how the pictures in the README are generated. diff --git a/docs/RELEASING.md b/docs/RELEASING.md index 6208008..90db613 100644 --- a/docs/RELEASING.md +++ b/docs/RELEASING.md @@ -16,15 +16,18 @@ | Artifact | Built on | |---|---| -| `MThread Draw--x64.msi`, and the same app as a `.zip` | windows-latest | +| `MThreadDraw--x64.msi`, and the same application as a `.zip` | windows-latest | | `MThreadDraw--arm64.dmg` | macos-latest | | `MThreadDraw--x64.dmg` | macos-15-intel | | sdist and wheel | ubuntu-latest | Tags must start with `v`. Anything else is ignored by the workflow. The same -build jobs also run on any pull request that touches `packaging/` or the build -scripts, without publishing - packaging breaks quietly, and only on the platform -you are not developing on. +build jobs also run on any pull request that touches `packaging/`, the build +scripts or either front end, without publishing - packaging breaks quietly, and +only on the platform you are not developing on. For `macos/` that is not a +safety net but the only compiler it ever meets: there is no Mac on the +maintainer's desk, and the first commit of the Swift front end reached CI with +two errors in it. ## The Windows front end @@ -53,15 +56,43 @@ Two things about it are not obvious and cost an afternoon each: targets load MSBuild tasks that only ship with VS. Version 2.x does not, which is why the reference is pinned there. +## The macOS front end + +`macos/` is a SwiftUI application, built as a Swift package rather than an Xcode +project so that `swift build` on a runner is the whole build with no project file +to keep in sync. `tools/build_macos.py` assembles the `.app` around the binary +and folds the engine into `Contents/Resources/engine`. + +The glass is `NSVisualEffectView` with `behindWindow` blending, not SwiftUI's +`.ultraThinMaterial`: the material frosts what is behind it *inside* the window, +which on a plain background is a grey panel. Only the AppKit view samples the +desktop under the window, which is what the word glass means here. + +Two things to know before touching it: + +- **The deployment target is macOS 13**, and a modern SDK will happily compile + macOS 14 API without a word until it reaches `swift build` on the runner. The + argument-less `onChange(of:)` is the one that got in. +- **No file may be called `main.swift`**, because a file by that name is + top-level code and cannot coexist with the `@main` attribute the app entry + point uses. + ## Building locally ```bash pip install pyinstaller -python tools/build_app.py --msi # Windows -python tools/build_app.py --dmg # macOS -python tools/build_app.py --archive # a plain zip / tar.gz anywhere +python tools/build_app.py # the engine alone +python tools/build_app.py --msi # engine, WinUI front end, installer +python tools/build_macos.py --dmg # macOS: the bundle and its disk image ``` +Everything is built in a scratch directory outside the checkout, and only the +finished installer and zip are copied into `dist/`. This tree lives in OneDrive +on one machine, and a sync client holds handles on files while it uploads them: +writing several thousand of them into a synced folder failed at a different step +every time - deleting the last build, zipping a DLL, harvesting for the +installer - always with an access denied that named a file rather than a cause. + Each build runs the packaged app's own self-test (`MThread Draw --selftest`) before it is considered finished: it imports the whole application and resolves and runs the bundled adb. An incomplete bundle fails the build instead of failing on @@ -121,7 +152,7 @@ Once the first upload lands, the install line in the README can become: ```bash pip install mthread # library -pip install "mthread-draw[gui]" # + the desktop app +pip install mthread-draw # + the engine the windows drive ``` ## Version numbers diff --git a/docs/hero.svg b/docs/hero.svg new file mode 100644 index 0000000..2447b60 --- /dev/null +++ b/docs/hero.svg @@ -0,0 +1,293 @@ + + MThread Draw + + + + + + + + + + + + + + + + + + + + + + + + + + + ANDROID · ADB + MThread Draw + Draw any picture on a phone's screen, + by touching it. Nothing installed on the phone. + + + 1,679 points in 5.0 seconds + Records gestures · replays them on another phone + Windows · macOS · Linux · AGPL-3.0 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/tools/make_hero.py b/tools/make_hero.py new file mode 100644 index 0000000..91ad5dc --- /dev/null +++ b/tools/make_hero.py @@ -0,0 +1,229 @@ +"""Render the README's hero: a drawing that draws itself, in real strokes. + + python tools/make_hero.py # -> docs/hero.svg + +Nothing here is illustration. The lines are what :class:`mthread.Vectorizer` +produces from `examples/motorcycle.jpg`, in the order the program would send +them to a phone, and the animation is the same order played out over eleven +seconds. A reader watching the top of the README is watching the product work. + +SVG rather than the GIF the banner uses, for three reasons. It is a tenth of the +bytes at four times the resolution; it stays sharp on a display of any density; +and the timing is declarative, so a stroke can begin the moment the one before +it ends without a frame budget to spend. + +Two things had to be measured rather than assumed. + +GitHub serves images through a proxy that passes SVG through unchanged, so SMIL +animation inside one plays - but only inside an ``. Inline SVG in Markdown +is stripped, so the file must be referenced, never pasted. + +The whole thing has to hold together at about 900 px wide, which is what a +README column actually gets. Everything here is laid out for that and scales +from it. +""" + +from __future__ import annotations + +import argparse +from pathlib import Path + +from mthread.vectorize import VectorizeSettings, Vectorizer + +ROOT = Path(__file__).resolve().parent.parent +SOURCE = ROOT / "examples" / "motorcycle.jpg" +OUT = ROOT / "docs" / "hero.svg" + +WIDTH, HEIGHT = 1280, 520 + +#: The picture sits on the right; the words have the left. +ART = (700, 60, 520, 400) + +#: Total time for every stroke to be drawn, and the pause before it repeats. +DRAW_SECONDS = 11.0 +HOLD_SECONDS = 2.6 + +#: A stroke shorter than this is a speck. Dropping them keeps the file small +#: and, more to the point, keeps the animation from spending its time on dust. +MINIMUM_POINTS = 6 + +#: Points closer together than this add bytes and nothing else at this size. +MINIMUM_STEP = 1.6 + + +#: Sensitivity low and detail high: the low threshold keeps the long contours of +#: the tank and the rider whole rather than breaking them into dashes, and the +#: detail fills in the engine. Rendered at every combination that seemed +#: plausible, this is the one where the motorcycle is a motorcycle. +SENSITIVITY, DETAIL = 4.0, 9.5 + + +def trace() -> list[list[tuple[float, float]]]: + settings = VectorizeSettings.from_sliders(SENSITIVITY, DETAIL, + target_width=900, method="canny") + vectorizer = Vectorizer() + vectorizer.load_image(str(SOURCE)) + _, paths = vectorizer.process(settings) + return [path for path in paths if len(path) >= MINIMUM_POINTS] + + +def fit(paths, box): + """Scale and centre the ink into *box* = (x, y, w, h).""" + points = [point for path in paths for point in path] + min_x = min(x for x, _ in points) + max_x = max(x for x, _ in points) + min_y = min(y for _, y in points) + max_y = max(y for _, y in points) + scale = min(box[2] / max(max_x - min_x, 1), box[3] / max(max_y - min_y, 1)) + dx = box[0] + (box[2] - (max_x - min_x) * scale) / 2 - min_x * scale + dy = box[1] + (box[3] - (max_y - min_y) * scale) / 2 - min_y * scale + return [[(x * scale + dx, y * scale + dy) for x, y in path] for path in paths] + + +def thin(path): + """Drop points that land on top of the ones before them.""" + kept = [path[0]] + for point in path[1:]: + last = kept[-1] + if abs(point[0] - last[0]) + abs(point[1] - last[1]) >= MINIMUM_STEP: + kept.append(point) + if len(kept) < 2: + kept.append(path[-1]) + return kept + + +def length_of(path) -> float: + total = 0.0 + for (x1, y1), (x2, y2) in zip(path, path[1:]): + total += ((x2 - x1) ** 2 + (y2 - y1) ** 2) ** 0.5 + return total + + +def polyline(path) -> str: + return " ".join(f"{x:.1f},{y:.1f}" for x, y in path) + + +def strokes_svg(paths) -> str: + """Every stroke, each waiting its turn. + + The reveal is a dash the length of the stroke, offset out of sight and + animated back to zero: the standard trick, and the only one that works + without scripting. Each stroke's start is its share of the total ink, so + a long stroke takes longer than a short one - the way drawing works. + """ + total = sum(length_of(path) for path in paths) or 1.0 + cycle = DRAW_SECONDS + HOLD_SECONDS + lines = [] + elapsed = 0.0 + for path in paths: + length = length_of(path) + start = DRAW_SECONDS * (elapsed / total) + # Clamped: a two-point stroke drawing for eight milliseconds reads as a + # pop rather than a stroke, and below about forty the eye cannot see a + # direction anyway. + span = max(DRAW_SECONDS * (length / total), 0.04) + elapsed += length + + begin = max(start / cycle, 0.001) + end = min((start + span) / cycle, 0.999) + if end <= begin: + end = min(begin + 0.001, 0.999) + + lines.append( + f'' + f'' + f"") + return "\n ".join(lines) + + +def build(paths) -> str: + ink = strokes_svg([thin(path) for path in fit(paths, ART)]) + cycle = DRAW_SECONDS + HOLD_SECONDS + + return f""" + MThread Draw + + + + + + + + + + + + + + + + + + + + + + + + + + + ANDROID · ADB + MThread Draw + Draw any picture on a phone's screen, + by touching it. Nothing installed on the phone. + + + 1,679 points in 5.0 seconds + Records gestures · replays them on another phone + Windows · macOS · Linux · AGPL-3.0 + + + + + + + + + {ink} + + + + + + + + + +""" + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--out", type=Path, default=OUT) + args = parser.parse_args() + + paths = trace() + svg = build(paths) + args.out.write_text(svg, encoding="utf-8", newline="\n") + kilobytes = args.out.stat().st_size / 1024 + strokes = len(paths) + points = sum(len(path) for path in paths) + print(f"wrote {args.out} - {strokes} strokes, {points} points, {kilobytes:.0f} KB") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main())