Poor Man's Multi-Touch: using multiple mice with Xorg
Since Jan. 13th, 2010 Xorg version 7.5 has landed to Debian unstable; one of the most notable additions to it was the XInput2 system, which incorporates the MPX efforts. So I hooked up a second USB mouse to my machine and started playing with it.
Pointless personal note: I have been training myself to use the mouse with the left hand, and this is how I am currently using it, but the instructions below still assume you are using a right handed setup.
And some nomenclature: a pointer is what (normally) moves on the screen when you move your mouse, a cursor is the visual appearance of a pointer.
I am referring to multi-pointer interaction here, which is a particular case of multi-touch. In a more general meaning, a multi-touch input system can detect surfaces too, like briefly explained by Peter Hutterer in his blog post Multi-touch? or multi-point?.
Setup
This is how you setup multiple pointers in recent Xorg (basically stolen from here):
-
Plug you second mouse in and list your input devices:
$ xinput list ⎡ Virtual core pointer id=2 [master pointer (3)] ⎜ ↳ Virtual core XTEST pointer id=4 [slave pointer (2)] ⎜ ↳ "Logitech USB-PS/2 Optical Mouse" id=10 [slave pointer (2)] ⎜ ↳ "Topro USB Mouse" id=12 [slave pointer (2)] ⎣ Virtual core keyboard id=3 [master keyboard (2)] ↳ Virtual core XTEST keyboard id=5 [slave keyboard (3)] ↳ "Power Button" id=6 [slave keyboard (3)] ↳ "Power Button" id=7 [slave keyboard (3)] ↳ "Sleep Button" id=8 [slave keyboard (3)] ↳ "DVB on-card IR receiver" id=9 [slave keyboard (3)] ↳ "AT Translated Set 2 keyboard" id=11 [slave keyboard (3)]I am going to use the"Topro USB Mouse"withid=12as the auxiliary (left) mouse. -
Create a new master input device and list the devices again:
$ xinput create-master Auxiliary $ xinput list ⎡ Virtual core pointer id=2 [master pointer (3)] ⎜ ↳ Virtual core XTEST pointer id=4 [slave pointer (2)] ⎜ ↳ "Logitech USB-PS/2 Optical Mouse" id=10 [slave pointer (2)] ⎜ ↳ "Topro USB Mouse" id=12 [slave pointer (2)] ⎣ Virtual core keyboard id=3 [master keyboard (2)] ↳ Virtual core XTEST keyboard id=5 [slave keyboard (3)] ↳ "Power Button" id=6 [slave keyboard (3)] ↳ "Power Button" id=7 [slave keyboard (3)] ↳ "Sleep Button" id=8 [slave keyboard (3)] ↳ "DVB on-card IR receiver" id=9 [slave keyboard (3)] ↳ "AT Translated Set 2 keyboard" id=11 [slave keyboard (3)] ⎡ Auxiliary pointer id=13 [master pointer (14)] ⎜ ↳ Auxiliary XTEST pointer id=15 [slave pointer (13)] ⎣ Auxiliary keyboard id=14 [master keyboard (13)] ↳ Auxiliary XTEST keyboard id=16 [slave keyboard (14)] -
Attach the mouse to it and check out how the device hierarchy changed:
$ xinput reattach 12 "Auxiliary pointer" $ xinput list ⎡ Virtual core pointer id=2 [master pointer (3)] ⎜ ↳ Virtual core XTEST pointer id=4 [slave pointer (2)] ⎜ ↳ "Logitech USB-PS/2 Optical Mouse" id=10 [slave pointer (2)] ⎣ Virtual core keyboard id=3 [master keyboard (2)] ↳ Virtual core XTEST keyboard id=5 [slave keyboard (3)] ↳ "Power Button" id=6 [slave keyboard (3)] ↳ "Power Button" id=7 [slave keyboard (3)] ↳ "Sleep Button" id=8 [slave keyboard (3)] ↳ "DVB on-card IR receiver" id=9 [slave keyboard (3)] ↳ "AT Translated Set 2 keyboard" id=11 [slave keyboard (3)] ⎡ Auxiliary pointer id=13 [master pointer (14)] ⎜ ↳ "Topro USB Mouse" id=12 [slave pointer (13)] ⎜ ↳ Auxiliary XTEST pointer id=15 [slave pointer (13)] ⎣ Auxiliary keyboard id=14 [master keyboard (13)] ↳ Auxiliary XTEST keyboard id=16 [slave keyboard (14)]
Now you have two independent pointers working already, but let's have some extra touches.
-
Change the buttons mapping for your left mouse with
id=12, in order to have the buttons orientation symmetric between the two mice.$ xinput set-button-map 12 3 2 1
-
We also want pointers to be recognizable, we can change the cursors with xicursorset, a little tool I wrote for the purpose:
$ sudo aptitude install git-core $ git clone git://git.ao2.it/xicursorset.git $ cd xicursorset $ sudo aptitude install libxi-dev libxcursor-dev $ make
We set the cursor for the left hand pointer first (remember, xicursorset expects the master pointer id from xinput):
$ ./xicursorset 13 right_ptr redglass
And for the right hand pointer too optionally:
$ ./xicursorset 2 left_ptr whiteglass
Unfortunately I haven't found a way to change the whole cursor theme per-pointer yet, I asked on the xorg mailing list, and it looks like there's no way to do this transparently by only using Xcursor for now.
Tests and observations
Let's try a multi-pointer aware application first, from vinput:
$ git clone git://gitorious.org/vinput/vinput.git $ cd vinput/demo $ gcc -o demo-paint demo-paint.c `pkg-config --cflags --libs xi` $ ./demo-paint
The result is pretty much like in this video about mpx-paint from Quentin Casasnovas.
Now, what is the situation for general use on a standard (GNOME) desktop?
- Gtk+ doesn't support MPX yet, so GIMP, Inkscape and the like (which I guess rely on Gtk+ input system) will only allow you to use one pointer at time when drawing, and you cannot assign different tools to different pointers either. However, there is some work undergoing to support multiple pointers in Gtk+3.
- Metacity is still being dumb here, I can't grab and move two windows using two pointers, let alone gestures. There is a very old guide about bringing multi-pointer to it, but the dedicated git branch is more than two years old.
- Anyhow, using two pointers on two distinct apps works quite well, I can move a window (Metacity) and drag and drop and icon on the desktop (Nautilus) at the same time, but this only gives a loose sensation of a multi-touch experience on the desktop.
- Qt seems to support multi-touch since version 4.6, here's a demo. I haven't tried myself, tho.
- Finally, an issue independent from MPX: Firefox keeps using the default cursors stubbornly ignoring the ones I set with xicursorset.
Multi-pointers setups are interesting from an HCI point of view even without full support: I tried using GIMP with two mice: with the left one I was changing tool or color, and with the right hand I was drawing, it wasn't a bad experience at all.
But having input from multiple pointers is also interesting from a Software Programming perspective, as this opens a whole new class of bugs in applications and toolkits, which assume that only one pointer exists; let me greet you with this weird one: it looks like menus are bound to pointers and not to windows: if a menu is opened from the menu bar of a window and kept active with one pointer, and the same window is moved with another pointer, the menu does not follow the window; is this really the expected behavior?
![[Valid RSS]](https://ao2.it/sites/default/files/valid-rss-rogers.png "Validate my RSS feed")
Comments
it looks like menus are bound
it looks like menus are bound to pointers and not to windows, is this really the expected behavior?
Debatable, I guess. For menus opened from a menu bar, probably not - it's too confusing for them to become detached from the menubar, as that video clip shows. On the other hand, it might make sense for context menus. Of course, it might not make sense either for one pointer to be able to move the menu away from the other pointer which opened it. Debatable...
Yes Simon, I agree this is
Yes Simon, I agree this is debatable, and it is debatable just because the behavior of menus and other UI elements is not well specified in the case of multiple pointers, that was the point.
It would be interesting to see how other systems (OSes, toolkits) behave in such a scenario.
This was always normal
This was always normal behavior for X, as menus were not considered part of the application. (Which is why they were part of the window frame/decoration, not of the application drawing space.)
Traditionally, menus and tool boxes/bars were the same thing, as any menu could simply be detached, and dragged to an arbitrary location, to be used as a tool box/bax.
In a sane world, that would be the only way an application would do such things, and all functionality of the application would be presented via such menus, with each item containing the icon and the keyboard shortcut too, and calling a function that can also be called via its console, a CLI, or dbus. (Think popular 3D/CAD tools with consoles.)
Of course, the newer generations have forgotten all the good things of Unix-likes, then raped them all to death, and then some. (See Linux ignoring “everything is a file”, and Firefox/What[TheFuck]WG having gone batshit insane with their UIs.)
Compiz supports MPX (you
Compiz supports MPX (you might need to use a version from their SVN and explicitly enable it when compiling). It does make me wonder what sort of havoc you could cause by switching virtual desktop when the other user is just about to click on something!
Heh, I actually knew about
Heh, I actually knew about the menus being bound to the mouse for quite a while. You can get the same effect by clicking the menubar and then moving your mouse while the system is thrashing the swap file enough to induce significant latency.
EFL(Enlightenment Foundation
EFL(Enlightenment Foundation Libraries) and e17(Enlightenment 0.17) desktop environment support multi-touch nicely :P i can easily move two windows at one time
quaker66, thanks for
quaker66, thanks for reporting that, very interesting.
What about the issue of menus detaching from menubars in EFL?
:( that isnt working for me.
:( that isnt working for me. Did you set it up for tablet?
@Corne what's the part which
@Corne what's the part which is not working? And what about the mention to the tablet? Do you use a mouse with the tablet?
Hi, I'd like to share a
Hi,
I'd like to share a script I wrote to automatize what explained above, i.e. attaching a new mouse, detaching it, and testing with vinput, which the script will try to download, compile and run. The compilation process needs an extra library in Fedora compared to Ubuntu. The script worked for me on both these platforms. (Fedora 13)
There's a command line interface and a simple GUI written in bash too, using zenity.
I posted the script here. (in the comments ... ough ugly)
http://el64.blogspot.com/2010/02/mpx-multipuntero-para-la-proxima-ubuntu...
If you have hard times with the few words in portuguese let me know (lazy to translate a useless stuff :p )
hope someone will enjoy.
How do I undo this? I did
How do I undo this? I did everything up until the "extra touches" section. I unplugged the other mouse, but it's cursor didn't go away.
First you set back the button
First you set back the button map of the mouse: xinput set-button-map 12 1 2 3
and after this modification the "xinput get-button-map 12" command should show the following (or similar): 1 2 3 4 5 6 7
Secondly you should reattach the second mouse to the first "Virtual core pointer", for example: xinput reattach 12 2
You should remove the new master pointer: xinput remove-master 13
More help: xinput man
Instructions are confusing!
Instructions are confusing! :(
Great tutorial, how do I set
Great tutorial, how do I set per user keyboard layout?
thx
Thanks. I am trying this out
Thanks. I am trying this out in 13.10 gnome. Works okay but a little buggy. Maybe I should just wait and see if it works out of the box when 14.04 goes to Mir.
This is a most-interesting
This is a most-interesting effort, which I'm going to have to come back to and try and work through when I have some spare time.
My 'wished-for' use case is not-dissimilar, whereas virtually all effort into multitouch looks to be based on gestures. I'm currently building a system for use as a Digital Audio Workstation (DAW), and managed to pick up a reasonably-priced ten-touchpoint monitor. Of course, that then ends up running Ubuntu Studio (currently 13.10, but I'm contemplating just taking the plunge and trying to do a Gentoo build).
The really obvious use for ten touchpoints is controlling multiple faders in the DAW. I know someone will get round to implementing that, ... eventually; since I first used a touchscreen about 25 year ago, I'm hoping it doesn't take quite that long.
Here we are 7 years later...
Here we are 7 years later... I too have 10 point touch monitors that I want to use to simply slide faders simultaneously in ardour. Who knew such a simple concept was so... far from reality.
Thank you for posting this.
Thank you for posting this. I've been fiddling around with this with the intent of making multi-person games using the nintendo wii as separate mouse cursors. Unfortunately your tool, after downloading and compiling, runs but does not change the cursor for me and reading further on the maillinglist it seems that proper support for multiple pointers is still quite a far way off.
Hi Jonas, the examples from
Hi Jonas,
the examples from the README file of xicursorset work just fine for me after installing the xcursor-themes package.
Tell me the exact error you get maybe we can figure this out.
OTOH you are right in that my tool is just a proof of concept, even if you get it to work it may not be the definitive solution you are looking for.
Wow, for the mouse it really
Wow, for the mouse it really works well! This is so cool! I have been looking for this, partly because I'm a twin looking to hot-seat a single workstation, partly because my work frequently involves pair programming.
Unfortunately, this does wonky thinks to keyboard input ... at least, I've found little logic in how 'A' and 'B' clicks (for lack of a better term) affect keyboard focus.
Also, I can't build your xicursorset project ... I get:
xicursorset/xicursorset.c:53: undefined reference to `XcursorLibraryShape'xicursorset/xicursorset.c:59: undefined reference to `XOpenDisplay'
xicursorset/xicursorset.c:68: undefined reference to `XcursorGetDefaultSize'
xicursorset/xicursorset.c:78: undefined reference to `XcursorGetTheme'
xicursorset/xicursorset.c:84: undefined reference to `XcursorShapeLoadImage'
xicursorset/xicursorset.c:90: undefined reference to `XcursorImageLoadCursor'
xicursorset/xicursorset.c:92: undefined reference to `XIDefineCursor'
xicursorset/xicursorset.c:93: undefined reference to `XFlush'
xicursorset/xicursorset.c:95: undefined reference to `XCloseDisplay'
This is on Ubuntu MATE 14.04, by the way.
Hi Jan, you need to install
Hi Jan, you need to install
pkg-config, and thelibxcursor-devandlibxi-devpackages, which will bring in alsolibx11-dev.I added this info to the README file in the repository.
NOTE that xicursorset can change the theme only for one state of the cursor, it does not change the whole cursor theme.
And yes, UI interaction with multiple cursors can be weird.
Who said anything about
Who said anything about multiple users though?
The idea is that you, as a human, usually have two hands, and ten fingers, yet only use one hand and one finger at a time. That is quite insane.
Multi-touch “solved” this … for people who accept gorilla arms and, frankly, ridiculously coarse pointing.
Multi-pointer is the start of solving this for precise work, associated with actually creating things and doing work on a computer.
X has done a good job, fully implementing what is required, to achieve this.
Now it is the DEs’ turn, and then that of applications like Krita/Gimp/Inkscape, Ardour, Kdenlive, FreeCAD, etc, to finally get us there.
Oh, and I want foot-mice too!
Not just buttons, but actual flip-flop-likes with optical sensors below them, and three buttons, one for each big toe, one for the rest, one for pinching between them.
Hmm, I have a spare mouse and a spare flip (or flop?) … This should be a fun project! :D
Post new comment