Poor Man's Multi-Touch: using multiple mice with Xorg

Since Jan. 13th, 2010 Xorg version 7.5 has landed to Debian unstable; one of the most notable additions to it was the XInput2 system, which incorporates the MPX efforts. So I hooked up a second USB mouse to my machine and started playing with it.

Pointless personal note: I have been training myself to use the mouse with the left hand, and this is how I am currently using it, but the instructions below still assume you are using a right handed setup.

And some nomenclature: a pointer is what (normally) moves on the screen when you move your mouse, a cursor is the visual appearance of a pointer.

I am referring to multi-pointer interaction here, which is a particular case of multi-touch. In a more general meaning, a multi-touch input system can detect surfaces too, like briefly explained by Peter Hutterer in his blog post Multi-touch? or multi-point?.

Setup

This is how you setup multiple pointers in recent Xorg (basically stolen from here):

  • Plug you second mouse in and list your input devices:
    $ xinput list
    ⎡ Virtual core pointer                          id=2    [master pointer  (3)]
    ⎜   ↳ Virtual core XTEST pointer                id=4    [slave  pointer  (2)]
    ⎜   ↳ "Logitech USB-PS/2 Optical Mouse"         id=10   [slave  pointer  (2)]
    ⎜   ↳ "Topro USB Mouse"                         id=12   [slave  pointer  (2)]
    ⎣ Virtual core keyboard                         id=3    [master keyboard (2)]
        ↳ Virtual core XTEST keyboard               id=5    [slave  keyboard (3)]
        ↳ "Power Button"                            id=6    [slave  keyboard (3)]
        ↳ "Power Button"                            id=7    [slave  keyboard (3)]
        ↳ "Sleep Button"                            id=8    [slave  keyboard (3)]
        ↳ "DVB on-card IR receiver"                 id=9    [slave  keyboard (3)]
        ↳ "AT Translated Set 2 keyboard"            id=11   [slave  keyboard (3)]
    
    I am going to use the "Topro USB Mouse" with id=12 as the auxiliary (left) mouse.
  • Create a new master input device and list the devices again:
    $ xinput create-master Auxiliary
    $ xinput list
    ⎡ Virtual core pointer                          id=2    [master pointer  (3)]
    ⎜   ↳ Virtual core XTEST pointer                id=4    [slave  pointer  (2)]
    ⎜   ↳ "Logitech USB-PS/2 Optical Mouse"         id=10   [slave  pointer  (2)]
    ⎜   ↳ "Topro USB Mouse"                         id=12   [slave  pointer  (2)]
    ⎣ Virtual core keyboard                         id=3    [master keyboard (2)]
        ↳ Virtual core XTEST keyboard               id=5    [slave  keyboard (3)]
        ↳ "Power Button"                            id=6    [slave  keyboard (3)]
        ↳ "Power Button"                            id=7    [slave  keyboard (3)]
        ↳ "Sleep Button"                            id=8    [slave  keyboard (3)]
        ↳ "DVB on-card IR receiver"                 id=9    [slave  keyboard (3)]
        ↳ "AT Translated Set 2 keyboard"            id=11   [slave  keyboard (3)]
    ⎡ Auxiliary pointer                             id=13   [master pointer  (14)]
    ⎜   ↳ Auxiliary XTEST pointer                   id=15   [slave  pointer  (13)]
    ⎣ Auxiliary keyboard                            id=14   [master keyboard (13)]
        ↳ Auxiliary XTEST keyboard                  id=16   [slave  keyboard (14)]
    
  • Attach the mouse to it and check out how the device hierarchy changed:
    $ xinput reattach 12 "Auxiliary pointer"
    $ xinput list
    ⎡ Virtual core pointer                          id=2    [master pointer  (3)]
    ⎜   ↳ Virtual core XTEST pointer                id=4    [slave  pointer  (2)]
    ⎜   ↳ "Logitech USB-PS/2 Optical Mouse"         id=10   [slave  pointer  (2)]
    ⎣ Virtual core keyboard                         id=3    [master keyboard (2)]
        ↳ Virtual core XTEST keyboard               id=5    [slave  keyboard (3)]
        ↳ "Power Button"                            id=6    [slave  keyboard (3)]
        ↳ "Power Button"                            id=7    [slave  keyboard (3)]
        ↳ "Sleep Button"                            id=8    [slave  keyboard (3)]
        ↳ "DVB on-card IR receiver"                 id=9    [slave  keyboard (3)]
        ↳ "AT Translated Set 2 keyboard"            id=11   [slave  keyboard (3)]
    ⎡ Auxiliary pointer                             id=13   [master pointer  (14)]
    ⎜   ↳ "Topro USB Mouse"                         id=12   [slave  pointer  (13)]
    ⎜   ↳ Auxiliary XTEST pointer                   id=15   [slave  pointer  (13)]
    ⎣ Auxiliary keyboard                            id=14   [master keyboard (13)]
        ↳ Auxiliary XTEST keyboard                  id=16   [slave  keyboard (14)]
    

Now you have two independent pointers working already, but let's have some extra touches.

  • Change the buttons mapping for your left mouse with id=12, in order to have the buttons orientation symmetric between the two mice.
    $ xinput set-button-map 12 3 2 1
    
  • We also want pointers to be recognizable, we can change the cursors with xicursorset, a little tool I wrote for the purpose:
    $ sudo aptitude install git-core
    $ git clone git://git.ao2.it/xicursorset.git
    $ cd xicursorset
    $ sudo aptitude install libxi-dev libxcursor-dev
    $ make
    

    We set the cursor for the left hand pointer first (remember, xicursorset expects the master pointer id from xinput):

    $ ./xicursorset 13 right_ptr redglass
    

    And for the right hand pointer too optionally:

    $ ./xicursorset 2 left_ptr whiteglass
    
    Unfortunately I haven't found a way to change the whole cursor theme per-pointer yet, I asked on the xorg mailing list, and it looks like there's no way to do this transparently by only using Xcursor for now.

Tests and observations

Let's try a multi-pointer aware application first, from vinput:

$ git clone git://gitorious.org/vinput/vinput.git
$ cd vinput/demo
$ gcc -o demo-paint demo-paint.c `pkg-config --cflags --libs xi`
$ ./demo-paint

The result is pretty much like in this video about mpx-paint from Quentin Casasnovas.

Now, what is the situation for general use on a standard (GNOME) desktop?

  • Gtk+ doesn't support MPX yet, so GIMP, Inkscape and the like (which I guess rely on Gtk+ input system) will only allow you to use one pointer at time when drawing, and you cannot assign different tools to different pointers either. However, there is some work undergoing to support multiple pointers in Gtk+3.
  • Metacity is still being dumb here, I can't grab and move two windows using two pointers, let alone gestures. There is a very old guide about bringing multi-pointer to it, but the dedicated git branch is more than two years old.
  • Anyhow, using two pointers on two distinct apps works quite well, I can move a window (Metacity) and drag and drop and icon on the desktop (Nautilus) at the same time, but this only gives a loose sensation of a multi-touch experience on the desktop.
  • Qt seems to support multi-touch since version 4.6, here's a demo. I haven't tried myself, tho.
  • Finally, an issue independent from MPX: Firefox keeps using the default cursors stubbornly ignoring the ones I set with xicursorset.

Multi-pointers setups are interesting from an HCI point of view even without full support: I tried using GIMP with two mice: with the left one I was changing tool or color, and with the right hand I was drawing, it wasn't a bad experience at all.

But having input from multiple pointers is also interesting from a Software Programming perspective, as this opens a whole new class of bugs in applications and toolkits, which assume that only one pointer exists; let me greet you with this weird one: it looks like menus are bound to pointers and not to windows, is this really the expected behavior?


CommentsSyndicate content

it looks like menus are bound

Simon's picture

it looks like menus are bound to pointers and not to windows, is this really the expected behavior?

Debatable, I guess. For menus opened from a menu bar, probably not - it's too confusing for them to become detached from the menubar, as that video clip shows. On the other hand, it might make sense for context menus. Of course, it might not make sense either for one pointer to be able to move the menu away from the other pointer which opened it. Debatable...

Yes Simon, I agree this is

ao2's picture

Yes Simon, I agree this is debatable, and it is debatable just because the behavior of menus and other UI elements is not well specified in the case of multiple pointers, that was the point.

It would be interesting to see how other systems (OSes, toolkits) behave in such a scenario.

Compiz supports MPX (you

Chris Lees's picture

Compiz supports MPX (you might need to use a version from their SVN and explicitly enable it when compiling). It does make me wonder what sort of havoc you could cause by switching virtual desktop when the other user is just about to click on something!

Heh, I actually knew about

Stephan Sokolow's picture

Heh, I actually knew about the menus being bound to the mouse for quite a while. You can get the same effect by clicking the menubar and then moving your mouse while the system is thrashing the swap file enough to induce significant latency.

EFL(Enlightenment Foundation

quaker66's picture

EFL(Enlightenment Foundation Libraries) and e17(Enlightenment 0.17) desktop environment support multi-touch nicely :P i can easily move two windows at one time

quaker66, thanks for

ao2's picture

quaker66, thanks for reporting that, very interesting.

What about the issue of menus detaching from menubars in EFL?

:( that isnt working for me.

Corne's picture

:( that isnt working for me. Did you set it up for tablet?

@Corne what's the part which

ao2's picture

@Corne what's the part which is not working? And what about the mention to the tablet? Do you use a mouse with the tablet?

Hi, I'd like to share a

vicente's picture

Hi,
I'd like to share a script I wrote to automatize what explained above, i.e. attaching a new mouse, detaching it, and testing with vinput, which the script will try to download, compile and run. The compilation process needs an extra library in Fedora compared to Ubuntu. The script worked for me on both these platforms. (Fedora 13)
There's a command line interface and a simple GUI written in bash too, using zenity.

I posted the script here. (in the comments ... ough ugly)
http://el64.blogspot.com/2010/02/mpx-multipuntero-para-la-proxima-ubuntu...
If you have hard times with the few words in portuguese let me know (lazy to translate a useless stuff :p )

hope someone will enjoy.

How do I undo this? I did

Anonymous's picture

How do I undo this? I did everything up until the "extra touches" section. I unplugged the other mouse, but it's cursor didn't go away.

First you set back the button

janovicst's picture

First you set back the button map of the mouse: xinput set-button-map 12 1 2 3
and after this modification the "xinput get-button-map 12" command should show the following (or similar): 1 2 3 4 5 6 7
Secondly you should reattach the second mouse to the first "Virtual core pointer", for example: xinput reattach 12 2
You should remove the new master pointer: xinput remove-master 13
More help: xinput man

Instructions are confusing!

Anonymous's picture

Instructions are confusing! :(

Great tutorial, how do I set

Sébastien's picture

Great tutorial, how do I set per user keyboard layout?

thx

Thanks. I am trying this out

Anonymous's picture

Thanks. I am trying this out in 13.10 gnome. Works okay but a little buggy. Maybe I should just wait and see if it works out of the box when 14.04 goes to Mir.

This is a most-interesting

BrianMc's picture

This is a most-interesting effort, which I'm going to have to come back to and try and work through when I have some spare time.

My 'wished-for' use case is not-dissimilar, whereas virtually all effort into multitouch looks to be based on gestures. I'm currently building a system for use as a Digital Audio Workstation (DAW), and managed to pick up a reasonably-priced ten-touchpoint monitor. Of course, that then ends up running Ubuntu Studio (currently 13.10, but I'm contemplating just taking the plunge and trying to do a Gentoo build).

The really obvious use for ten touchpoints is controlling multiple faders in the DAW. I know someone will get round to implementing that, ... eventually; since I first used a touchscreen about 25 year ago, I'm hoping it doesn't take quite that long.

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account associated with the e-mail address you provide, it will be used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

More information about formatting options

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
H
2
P
T
W
U
Enter the code without spaces.