vsync aligned operations?

[markg85]

Hi,

I'm developing an app where mouse and keyboard events flow in through a socket and are passed to this library.
It's possible for these events to be many within the time of a single frame.

As a disclaimer: I don't actually have this code yet but i'm working on it and finding the right libraries to use for my usecase.

In that case i'm wondering what the best strategy is.
Like if you receive many X/Y movements within a frame's time (say 16ms for 60 fps) then only the last movement is going to matter anyhow, anything before it within that frame's time is probably just added noise. (this is just an assumption)

I can see a couple possible strategies for this:

1. It doesn't matter. A very valid strategy and easy :)
2. libinput already does it (does it?) so still doing nothing is the way to go
3. the library would need to do vsync aligned movements
4. the application using this library needs to do vsync alignment to only send events to this library when it matters

What are your thoughts on this?

[positiveway]

Hi, I made similar application for remote control and another one to translate Steam Controller Track-pads and buttons to mouse+keyboard events.

The tricks I learned:

1. Mouse movement should be done in a separate thread within equal intervals.
2. Just waiting (1 ms for example) is not enough. It's better to store current time in the beginning of each loop and then measure it again in the end of the loop when all operations are done. Subtract it from waiting interval and make thread sleep the remaining time. This result in great consistency.
3. The most demanding operation is writing events in system virtual file. Because such operation (no matter what library or language you use) requires syscall to kernel. During syscall processor switches context to a different Ring (https://en.wikipedia.org/wiki/Protection_ring).

To solve this problem I added buffered functions (same functions with prefix 'buffered_'). You can find them in the main branch on github. These function create Vector of events which you can combine in even bigger Vector and then write it all at once in one syscall operation.

Every event requires synchronization operation so writing it naive way (like all other libraries do) causes lags and high CPU consumption.

4. Moving a mouse instantly by x,y pixels will result in jitter. To avoid it I added (again in main branch of github, not published to Cargo yet) function buffered_gradual_move_mouse. This function generate events to move mouse pixel by pixel instead of jumping x,y pixels at once. It returns you a buffer of events which you can then write with function write_buffer

5. It all may sound confusing but I already have it all working perfectly in my project for SteamConroller to Mouse/Keyboard emulation. The link is https://github.com/positiveway/JoystickFullRust/blob/63dfed42f9b20bfb90dd78855bba0e58bfbd3a09/src/writing_thread.rs

6. As for vsync problem not sure I understand exactly what you need but intermediate mouse events can be discarded easily if you send absolute position of the mouse. (Like SteamController or Remote Controll app on phone send absolute position on track-pad or screen). When sending relative movement each event matters and should be processed.

[positiveway]

I also have tricks how to optimize sending events over network.

Send events to different ports (open several sockets, each in its own thread). Usually at one moment of a time mouse moves only vertically or horizontally. So instead of sending messages with size of two bytes (x,y) to port 1000 (for example). Send x events to port 1000 and y events to port 1001. Each event will be the size of one byte then.

Same with button press, release events. One port for sending buttons which needed to be pressed and another one for sending buttons to release. This saves you one byte as well.

Commonly used buttons can be coded as u8 (look at key_codes.rs file) and mouse events can be coded as i8. In my app I'm sending events every 1 millisecond so it's rear to mouse move more than 127 pixels at once but if it happens I just clamp it to 127 or -127 ( max and min values for i8).

Hope it helped!

[markg85]

Thank you so much for your detailed description!

I still have some questions though :)

What i'm essentially making is desktop streaming.
1 side sends the desktop (captures the framebuffer and puts it in a video container)
1 side receives it and plays it.

The receiving side is - technically - just 1 window, a video player.
If the user moves their mouse inside that window i want that position to be send to the sender side (the side where the video is captured).

I have all this working.

What complicates things here is the relative X and Y position that your library wants. It would be much easier for me to just send the absolute X and Y position as they match exactly between. Sure, i can make a diff between previous send position and current position which would be a new relative position... but i rather not do that. Also, this is getting more complex when scaling is involved. It's not at the moment but will be later on.

Is there a reason why your library doesn't accept absolute positions?

[positiveway]

I based this library on another older unsupported library (have a link in readme.md). And if I remember correctly author of that library have comments in the code saying about some problems with mouse movement by absolute position. It was of no use to me at that time so I decided not to implement it. Maybe I will implement it later. Or you can fork my library and port missing parts of the code from the old parent library.

[markg85]

Could you explain what relative positioning means in your library?

An example.
Say you have a 1920x1080.
The first time you pass move_mouse(10, 10) it should be at 10, 10 high up in the left upper corner.
Now what happens if you again pass move_mouse(10, 10)?

If it's relative then it should not move at all and stay at 10,10, right?

I'm getting the feeling your library does "relative" based positioning compared to where the cursor currently is, not relative compared to the parent. I'm not entirely sure because I'm having trouble getting other parts to work nicely.

Your own example is why i think this:

        // move cursor 50 pixels up and 50 pixels to the right from the current position
        device.move_mouse(50, 50).unwrap();

note the move .. from the current position.

My understanding of what relative (compared to what) means could also be wrong. My current understanding is this:
If you have 1 root window and only that, relative and absolute are the same.
If you have "sub windows" then relative is compared to the position the window has in the parent.

[positiveway]

"Relative" means to the current cursor position. We are writing to uinput, it has no information about windows, tabs, programs, or what on the screen. The only thing it knows is current cursor position. When you start your PC let's say cursor is in the middle of the screen. So (960,540) (y is measured from top to bottom), when you perform device.move_mouse(50, 50).unwrap() cursor will be in position (1010, 490).

If user moves the real mouse, the root position will change and device.move_mouse(x, y) will move cursor by (x, y) from the current cursor position on screen.

[positiveway]

I can temporarily recommend you similar library that supports absolute mouse movements. https://github.com/indianakernick/The-Fat-Controller

It utilizes the same logic and mechanisms as my library (when check-x11 feature is disabled), just doesn't have optimization for maximizing performance / lowering latency.

I'm currently preoccupied at my job but I'll try to add missing functionality (abs_mouse_move) soon to this library.

[markg85]

@positiveway Thank you so much for your detailed and super helpful response!

While i was aware of the concept of absolute and relative, i had not even considered uinput to work differently. I just, apparently ignorantly, assumed it would work on the concept of the window stack too. Good to know that things work differently. I should have known, lol.

I'm not in a rush but if you are willing to add a abs_mouse_move, that would be awesome! I can live with relative (compared to the current mouse position).

The issue i see with relative is that you don't know where to start and can't easily correct for mistakes either. For relative to be stable you's need to (occasionally) sync your mouse position. What i mean by that is let the side sending the video stream know where exactly the mouse is so that the side receiving it can either "jump" to mouse to that location or add in that as offset. Relative is an absolute pain ;) (pardon the intentional play on words, I couldn't resist). Therefore absolute positioning (or relative but to a certain fixed non-moving anchor point) is much easier. You just tell from the side that processes the video "here are my mouse coordinates, put your mouse there!" upon which the sending side jumps the mouse to that position. And you're done. Easy :)

[markg85]

Any update on your end?

I tried adding it but i can't get it to work. I lack X11 knowledge and how the protocol works for this.
Here's what i tried, it might help you or you might have a pointer for me to get it working.

All my changes are in virtual devices. Sorry for it not being a diff.

In the register_mouse function i added (it has both the register_relative loop as you know it and this new register_absolute loop):

        for code in [ABS_X, ABS_Y] {
            self.register_absolute(code)?
        }

I added the function register_absolute which looks like this:

    fn register_absolute(&self, code: u16) -> EmptyResult {
        unsafe {
            Errno::result(ui_set_evbit(self.file.as_raw_fd(), EV_ABS as i32))?;
            Errno::result(ui_set_absbit(self.file.as_raw_fd(), code as i32))?;
        }
        Ok(())
    }

And then i have a move_mouse_abs which looks like this:

    pub fn move_mouse_abs(&mut self, x: Coord, y: Coord) -> EmptyResult {
        self.write_batch(vec![
            (EV_ABS, ABS_X, x),
            (EV_ABS, ABS_Y, y),
            SYN_PARAMS
        ])
    }

Making these changes all compiles just fine but no movement happens. Relative movement is also not working anymore 😛 so yeah, this breaks something somewhere but i don't know where.

[positiveway]

I'll try looking into it tomorrow

[markg85]

Ping :)

[markg85]

Any update on your end?
I've tried many hours to get it working but i just can't.. Apparently.

It would already help a lot if you could point me to some documentation or tips on how to get debug messages out of uinput. I can't find much of that.. I'm guessing, once i know how to get that, i'll probably see that something is wrong in the device setup and be able to fix it. Hopefully :)

Edit.

I found an interesting bit of extra information.
https://github.com/vmware/open-vm-tools/blob/bd6418528907fc327dfe83983c540fb6d7dd0572/open-vm-tools/services/plugins/dndcp/fakeMouseWayland/fakeMouseWayland.cpp
Specifically this part:

   /*
    * The uinput old interface is used for compatibility.
    * For more information please refer to:
    * https://www.kernel.org/doc/html/v4.12/input/uinput.html
    */
   struct uinput_user_dev dev;
   memset(&dev, 0, sizeof(dev));
   snprintf(dev.name, UINPUT_MAX_NAME_SIZE, UINPUT_DND_POINTER_NAME);

   dev.absmin[ABS_X] = 0;
   dev.absmax[ABS_X] = width - 1;
   dev.absmin[ABS_Y] = 0;
   dev.absmax[ABS_Y] = height - 1;

When googling a bit more on EV_ABS i find this helpful page: https://www.kernel.org/doc/html/v4.12/input/input-programming.html?highlight=ev_abs
Where this part is quite vital:

However EV_ABS requires a little special care. Before calling input_register_device, you have to fill additional fields in the input_dev struct for each absolute axis your device has. If our button device had also the ABS_X axis:

button_dev.absmin[ABS_X] = 0;
button_dev.absmax[ABS_X] = 255;
button_dev.absfuzz[ABS_X] = 4;
button_dev.absflat[ABS_X] = 8;
Or, you can just say:

input_set_abs_params(button_dev, ABS_X, 0, 255, 4, 8);
This setting would be appropriate for a joystick X axis, with the minimum of 0, maximum of 255 (which the joystick must be able to reach, no problem if it sometimes reports more, but it must be able to always reach the min and max values), with noise in the data up to +- 4, and with a center flat position of size 8.

If you don?t need absfuzz and absflat, you can set them to zero, which mean that the thing is precise and always returns to exactly the center position (if it has any).

So to summarize, your library needs a way to set absmin, absmax, absfuzz and absflat in the device creation step. I'm quite sure that is the missing piece to get absolute positioning working. Would you be able to add this?

Edit 2
Your library has ways to set those (they are in def). Defining them still doesn't work properly though. Could the current git version be a bit broken in the bits that it sets? Also, at least this example first writes to uinput, checking them before it writes the other properties.