| 1 | .. include:: <isonum.txt>
|
| 2 |
|
| 3 | =========================
|
| 4 | Multi-touch (MT) Protocol
|
| 5 | =========================
|
| 6 |
|
| 7 | :Copyright: |copy| 2009-2010 Henrik Rydberg <[email protected]>
|
| 8 |
|
| 9 |
|
| 10 | Introduction
|
| 11 | ------------
|
| 12 |
|
| 13 | In order to utilize the full power of the new multi-touch and multi-user
|
| 14 | devices, a way to report detailed data from multiple contacts, i.e.,
|
| 15 | objects in direct contact with the device surface, is needed. This
|
| 16 | document describes the multi-touch (MT) protocol which allows kernel
|
| 17 | drivers to report details for an arbitrary number of contacts.
|
| 18 |
|
| 19 | The protocol is divided into two types, depending on the capabilities of the
|
| 20 | hardware. For devices handling anonymous contacts (type A), the protocol
|
| 21 | describes how to send the raw data for all contacts to the receiver. For
|
| 22 | devices capable of tracking identifiable contacts (type B), the protocol
|
| 23 | describes how to send updates for individual contacts via event slots.
|
| 24 |
|
| 25 | .. note::
|
| 26 | MT protocol type A is obsolete, all kernel drivers have been
|
| 27 | converted to use type B.
|
| 28 |
|
| 29 | Protocol Usage
|
| 30 | --------------
|
| 31 |
|
| 32 | Contact details are sent sequentially as separate packets of ABS_MT
|
| 33 | events. Only the ABS_MT events are recognized as part of a contact
|
| 34 | packet. Since these events are ignored by current single-touch (ST)
|
| 35 | applications, the MT protocol can be implemented on top of the ST protocol
|
| 36 | in an existing driver.
|
| 37 |
|
| 38 | Drivers for type A devices separate contact packets by calling
|
| 39 | input_mt_sync() at the end of each packet. This generates a SYN_MT_REPORT
|
| 40 | event, which instructs the receiver to accept the data for the current
|
| 41 | contact and prepare to receive another.
|
| 42 |
|
| 43 | Drivers for type B devices separate contact packets by calling
|
| 44 | input_mt_slot(), with a slot as argument, at the beginning of each packet.
|
| 45 | This generates an ABS_MT_SLOT event, which instructs the receiver to
|
| 46 | prepare for updates of the given slot.
|
| 47 |
|
| 48 | All drivers mark the end of a multi-touch transfer by calling the usual
|
| 49 | input_sync() function. This instructs the receiver to act upon events
|
| 50 | accumulated since last EV_SYN/SYN_REPORT and prepare to receive a new set
|
| 51 | of events/packets.
|
| 52 |
|
| 53 | The main difference between the stateless type A protocol and the stateful
|
| 54 | type B slot protocol lies in the usage of identifiable contacts to reduce
|
| 55 | the amount of data sent to userspace. The slot protocol requires the use of
|
| 56 | the ABS_MT_TRACKING_ID, either provided by the hardware or computed from
|
| 57 | the raw data .
|
| 58 |
|
| 59 | For type A devices, the kernel driver should generate an arbitrary
|
| 60 | enumeration of the full set of anonymous contacts currently on the
|
| 61 | surface. The order in which the packets appear in the event stream is not
|
| 62 | important. Event filtering and finger tracking is left to user space .
|
| 63 |
|
| 64 | For type B devices, the kernel driver should associate a slot with each
|
| 65 | identified contact, and use that slot to propagate changes for the contact.
|
| 66 | Creation, replacement and destruction of contacts is achieved by modifying
|
| 67 | the ABS_MT_TRACKING_ID of the associated slot. A non-negative tracking id
|
| 68 | is interpreted as a contact, and the value -1 denotes an unused slot. A
|
| 69 | tracking id not previously present is considered new, and a tracking id no
|
| 70 | longer present is considered removed. Since only changes are propagated,
|
| 71 | the full state of each initiated contact has to reside in the receiving
|
| 72 | end. Upon receiving an MT event, one simply updates the appropriate
|
| 73 | attribute of the current slot.
|
| 74 |
|
| 75 | Some devices identify and/or track more contacts than they can report to the
|
| 76 | driver. A driver for such a device should associate one type B slot with each
|
| 77 | contact that is reported by the hardware. Whenever the identity of the
|
| 78 | contact associated with a slot changes, the driver should invalidate that
|
| 79 | slot by changing its ABS_MT_TRACKING_ID. If the hardware signals that it is
|
| 80 | tracking more contacts than it is currently reporting, the driver should use
|
| 81 | a BTN_TOOL_*TAP event to inform userspace of the total number of contacts
|
| 82 | being tracked by the hardware at that moment. The driver should do this by
|
| 83 | explicitly sending the corresponding BTN_TOOL_*TAP event and setting
|
| 84 | use_count to false when calling input_mt_report_pointer_emulation().
|
| 85 | The driver should only advertise as many slots as the hardware can report.
|
| 86 | Userspace can detect that a driver can report more total contacts than slots
|
| 87 | by noting that the largest supported BTN_TOOL_*TAP event is larger than the
|
| 88 | total number of type B slots reported in the absinfo for the ABS_MT_SLOT axis.
|
| 89 |
|
| 90 | The minimum value of the ABS_MT_SLOT axis must be 0.
|
| 91 |
|
| 92 | Protocol Example A
|
| 93 | ------------------
|
| 94 |
|
| 95 | Here is what a minimal event sequence for a two-contact touch would look
|
| 96 | like for a type A device::
|
| 97 |
|
| 98 | ABS_MT_POSITION_X x[0]
|
| 99 | ABS_MT_POSITION_Y y[0]
|
| 100 | SYN_MT_REPORT
|
| 101 | ABS_MT_POSITION_X x[1]
|
| 102 | ABS_MT_POSITION_Y y[1]
|
| 103 | SYN_MT_REPORT
|
| 104 | SYN_REPORT
|
| 105 |
|
| 106 | The sequence after moving one of the contacts looks exactly the same; the
|
| 107 | raw data for all present contacts are sent between every synchronization
|
| 108 | with SYN_REPORT.
|
| 109 |
|
| 110 | Here is the sequence after lifting the first contact::
|
| 111 |
|
| 112 | ABS_MT_POSITION_X x[1]
|
| 113 | ABS_MT_POSITION_Y y[1]
|
| 114 | SYN_MT_REPORT
|
| 115 | SYN_REPORT
|
| 116 |
|
| 117 | And here is the sequence after lifting the second contact::
|
| 118 |
|
| 119 | SYN_MT_REPORT
|
| 120 | SYN_REPORT
|
| 121 |
|
| 122 | If the driver reports one of BTN_TOUCH or ABS_PRESSURE in addition to the
|
| 123 | ABS_MT events, the last SYN_MT_REPORT event may be omitted. Otherwise, the
|
| 124 | last SYN_REPORT will be dropped by the input core, resulting in no
|
| 125 | zero-contact event reaching userland.
|
| 126 |
|
| 127 |
|
| 128 | Protocol Example B
|
| 129 | ------------------
|
| 130 |
|
| 131 | Here is what a minimal event sequence for a two-contact touch would look
|
| 132 | like for a type B device::
|
| 133 |
|
| 134 | ABS_MT_SLOT 0
|
| 135 | ABS_MT_TRACKING_ID 45
|
| 136 | ABS_MT_POSITION_X x[0]
|
| 137 | ABS_MT_POSITION_Y y[0]
|
| 138 | ABS_MT_SLOT 1
|
| 139 | ABS_MT_TRACKING_ID 46
|
| 140 | ABS_MT_POSITION_X x[1]
|
| 141 | ABS_MT_POSITION_Y y[1]
|
| 142 | SYN_REPORT
|
| 143 |
|
| 144 | Here is the sequence after moving contact 45 in the x direction::
|
| 145 |
|
| 146 | ABS_MT_SLOT 0
|
| 147 | ABS_MT_POSITION_X x[0]
|
| 148 | SYN_REPORT
|
| 149 |
|
| 150 | Here is the sequence after lifting the contact in slot 0::
|
| 151 |
|
| 152 | ABS_MT_TRACKING_ID -1
|
| 153 | SYN_REPORT
|
| 154 |
|
| 155 | The slot being modified is already 0, so the ABS_MT_SLOT is omitted. The
|
| 156 | message removes the association of slot 0 with contact 45, thereby
|
| 157 | destroying contact 45 and freeing slot 0 to be reused for another contact.
|
| 158 |
|
| 159 | Finally, here is the sequence after lifting the second contact::
|
| 160 |
|
| 161 | ABS_MT_SLOT 1
|
| 162 | ABS_MT_TRACKING_ID -1
|
| 163 | SYN_REPORT
|
| 164 |
|
| 165 |
|
| 166 | Event Usage
|
| 167 | -----------
|
| 168 |
|
| 169 | A set of ABS_MT events with the desired properties is defined. The events
|
| 170 | are divided into categories, to allow for partial implementation. The
|
| 171 | minimum set consists of ABS_MT_POSITION_X and ABS_MT_POSITION_Y, which
|
| 172 | allows for multiple contacts to be tracked. If the device supports it, the
|
| 173 | ABS_MT_TOUCH_MAJOR and ABS_MT_WIDTH_MAJOR may be used to provide the size
|
| 174 | of the contact area and approaching tool, respectively.
|
| 175 |
|
| 176 | The TOUCH and WIDTH parameters have a geometrical interpretation; imagine
|
| 177 | looking through a window at someone gently holding a finger against the
|
| 178 | glass. You will see two regions, one inner region consisting of the part
|
| 179 | of the finger actually touching the glass, and one outer region formed by
|
| 180 | the perimeter of the finger. The center of the touching region (a) is
|
| 181 | ABS_MT_POSITION_X/Y and the center of the approaching finger (b) is
|
| 182 | ABS_MT_TOOL_X/Y. The touch diameter is ABS_MT_TOUCH_MAJOR and the finger
|
| 183 | diameter is ABS_MT_WIDTH_MAJOR. Now imagine the person pressing the finger
|
| 184 | harder against the glass. The touch region will increase, and in general,
|
| 185 | the ratio ABS_MT_TOUCH_MAJOR / ABS_MT_WIDTH_MAJOR, which is always smaller
|
| 186 | than unity, is related to the contact pressure. For pressure-based devices,
|
| 187 | ABS_MT_PRESSURE may be used to provide the pressure on the contact area
|
| 188 | instead. Devices capable of contact hovering can use ABS_MT_DISTANCE to
|
| 189 | indicate the distance between the contact and the surface.
|
| 190 |
|
| 191 | ::
|
| 192 |
|
| 193 |
|
| 194 | Linux MT Win8
|
| 195 | __________ _______________________
|
| 196 | / \ | |
|
| 197 | / \ | |
|
| 198 | / ____ \ | |
|
| 199 | / / \ \ | |
|
| 200 | \ \ a \ \ | a |
|
| 201 | \ \____/ \ | |
|
| 202 | \ \ | |
|
| 203 | \ b \ | b |
|
| 204 | \ \ | |
|
| 205 | \ \ | |
|
| 206 | \ \ | |
|
| 207 | \ / | |
|
| 208 | \ / | |
|
| 209 | \ / | |
|
| 210 | \__________/ |_______________________|
|
| 211 |
|
| 212 |
|
| 213 | In addition to the MAJOR parameters, the oval shape of the touch and finger
|
| 214 | regions can be described by adding the MINOR parameters, such that MAJOR
|
| 215 | and MINOR are the major and minor axis of an ellipse. The orientation of
|
| 216 | the touch ellipse can be described with the ORIENTATION parameter, and the
|
| 217 | direction of the finger ellipse is given by the vector (a - b).
|
| 218 |
|
| 219 | For type A devices, further specification of the touch shape is possible
|
| 220 | via ABS_MT_BLOB_ID.
|
| 221 |
|
| 222 | The ABS_MT_TOOL_TYPE may be used to specify whether the touching tool is a
|
| 223 | finger or a pen or something else. Finally, the ABS_MT_TRACKING_ID event
|
| 224 | may be used to track identified contacts over time .
|
| 225 |
|
| 226 | In the type B protocol, ABS_MT_TOOL_TYPE and ABS_MT_TRACKING_ID are
|
| 227 | implicitly handled by input core; drivers should instead call
|
| 228 | input_mt_report_slot_state().
|
| 229 |
|
| 230 |
|
| 231 | Event Semantics
|
| 232 | ---------------
|
| 233 |
|
| 234 | ABS_MT_TOUCH_MAJOR
|
| 235 | The length of the major axis of the contact. The length should be given in
|
| 236 | surface units. If the surface has an X times Y resolution, the largest
|
| 237 | possible value of ABS_MT_TOUCH_MAJOR is sqrt(X^2 + Y^2), the diagonal .
|
| 238 |
|
| 239 | ABS_MT_TOUCH_MINOR
|
| 240 | The length, in surface units, of the minor axis of the contact. If the
|
| 241 | contact is circular, this event can be omitted .
|
| 242 |
|
| 243 | ABS_MT_WIDTH_MAJOR
|
| 244 | The length, in surface units, of the major axis of the approaching
|
| 245 | tool. This should be understood as the size of the tool itself. The
|
| 246 | orientation of the contact and the approaching tool are assumed to be the
|
| 247 | same .
|
| 248 |
|
| 249 | ABS_MT_WIDTH_MINOR
|
| 250 | The length, in surface units, of the minor axis of the approaching
|
| 251 | tool. Omit if circular .
|
| 252 |
|
| 253 | The above four values can be used to derive additional information about
|
| 254 | the contact. The ratio ABS_MT_TOUCH_MAJOR / ABS_MT_WIDTH_MAJOR approximates
|
| 255 | the notion of pressure. The fingers of the hand and the palm all have
|
| 256 | different characteristic widths.
|
| 257 |
|
| 258 | ABS_MT_PRESSURE
|
| 259 | The pressure, in arbitrary units, on the contact area. May be used instead
|
| 260 | of TOUCH and WIDTH for pressure-based devices or any device with a spatial
|
| 261 | signal intensity distribution.
|
| 262 |
|
| 263 | If the resolution is zero, the pressure data is in arbitrary units.
|
| 264 | If the resolution is non-zero, the pressure data is in units/gram. See
|
| 265 | :ref:`input-event-codes` for details.
|
| 266 |
|
| 267 | ABS_MT_DISTANCE
|
| 268 | The distance, in surface units, between the contact and the surface. Zero
|
| 269 | distance means the contact is touching the surface. A positive number means
|
| 270 | the contact is hovering above the surface.
|
| 271 |
|
| 272 | ABS_MT_ORIENTATION
|
| 273 | The orientation of the touching ellipse. The value should describe a signed
|
| 274 | quarter of a revolution clockwise around the touch center. The signed value
|
| 275 | range is arbitrary, but zero should be returned for an ellipse aligned with
|
| 276 | the Y axis (north) of the surface, a negative value when the ellipse is
|
| 277 | turned to the left, and a positive value when the ellipse is turned to the
|
| 278 | right. When aligned with the X axis in the positive direction, the range
|
| 279 | max should be returned; when aligned with the X axis in the negative
|
| 280 | direction, the range -max should be returned.
|
| 281 |
|
| 282 | Touch ellipses are symmetrical by default. For devices capable of true 360
|
| 283 | degree orientation, the reported orientation must exceed the range max to
|
| 284 | indicate more than a quarter of a revolution. For an upside-down finger,
|
| 285 | range max * 2 should be returned.
|
| 286 |
|
| 287 | Orientation can be omitted if the touch area is circular, or if the
|
| 288 | information is not available in the kernel driver. Partial orientation
|
| 289 | support is possible if the device can distinguish between the two axes, but
|
| 290 | not (uniquely) any values in between. In such cases, the range of
|
| 291 | ABS_MT_ORIENTATION should be [0, 1] .
|
| 292 |
|
| 293 | ABS_MT_POSITION_X
|
| 294 | The surface X coordinate of the center of the touching ellipse.
|
| 295 |
|
| 296 | ABS_MT_POSITION_Y
|
| 297 | The surface Y coordinate of the center of the touching ellipse.
|
| 298 |
|
| 299 | ABS_MT_TOOL_X
|
| 300 | The surface X coordinate of the center of the approaching tool. Omit if
|
| 301 | the device cannot distinguish between the intended touch point and the
|
| 302 | tool itself.
|
| 303 |
|
| 304 | ABS_MT_TOOL_Y
|
| 305 | The surface Y coordinate of the center of the approaching tool. Omit if the
|
| 306 | device cannot distinguish between the intended touch point and the tool
|
| 307 | itself.
|
| 308 |
|
| 309 | The four position values can be used to separate the position of the touch
|
| 310 | from the position of the tool. If both positions are present, the major
|
| 311 | tool axis points towards the touch point . Otherwise, the tool axes are
|
| 312 | aligned with the touch axes.
|
| 313 |
|
| 314 | ABS_MT_TOOL_TYPE
|
| 315 | The type of approaching tool. A lot of kernel drivers cannot distinguish
|
| 316 | between different tool types, such as a finger or a pen. In such cases, the
|
| 317 | event should be omitted. The protocol currently mainly supports
|
| 318 | MT_TOOL_FINGER, MT_TOOL_PEN, and MT_TOOL_PALM .
|
| 319 | For type B devices, this event is handled by input core; drivers should
|
| 320 | instead use input_mt_report_slot_state(). A contact's ABS_MT_TOOL_TYPE may
|
| 321 | change over time while still touching the device, because the firmware may
|
| 322 | not be able to determine which tool is being used when it first appears.
|
| 323 |
|
| 324 | ABS_MT_BLOB_ID
|
| 325 | The BLOB_ID groups several packets together into one arbitrarily shaped
|
| 326 | contact. The sequence of points forms a polygon which defines the shape of
|
| 327 | the contact. This is a low-level anonymous grouping for type A devices, and
|
| 328 | should not be confused with the high-level trackingID . Most type A
|
| 329 | devices do not have blob capability, so drivers can safely omit this event.
|
| 330 |
|
| 331 | ABS_MT_TRACKING_ID
|
| 332 | The TRACKING_ID identifies an initiated contact throughout its life cycle
|
| 333 | . The value range of the TRACKING_ID should be large enough to ensure
|
| 334 | unique identification of a contact maintained over an extended period of
|
| 335 | time. For type B devices, this event is handled by input core; drivers
|
| 336 | should instead use input_mt_report_slot_state().
|
| 337 |
|
| 338 |
|
| 339 | Event Computation
|
| 340 | -----------------
|
| 341 |
|
| 342 | The flora of different hardware unavoidably leads to some devices fitting
|
| 343 | better to the MT protocol than others. To simplify and unify the mapping,
|
| 344 | this section gives recipes for how to compute certain events.
|
| 345 |
|
| 346 | For devices reporting contacts as rectangular shapes, signed orientation
|
| 347 | cannot be obtained. Assuming X and Y are the lengths of the sides of the
|
| 348 | touching rectangle, here is a simple formula that retains the most
|
| 349 | information possible::
|
| 350 |
|
| 351 | ABS_MT_TOUCH_MAJOR := max(X, Y)
|
| 352 | ABS_MT_TOUCH_MINOR := min(X, Y)
|
| 353 | ABS_MT_ORIENTATION := bool(X > Y)
|
| 354 |
|
| 355 | The range of ABS_MT_ORIENTATION should be set to [0, 1], to indicate that
|
| 356 | the device can distinguish between a finger along the Y axis (0) and a
|
| 357 | finger along the X axis (1).
|
| 358 |
|
| 359 | For Win8 devices with both T and C coordinates, the position mapping is::
|
| 360 |
|
| 361 | ABS_MT_POSITION_X := T_X
|
| 362 | ABS_MT_POSITION_Y := T_Y
|
| 363 | ABS_MT_TOOL_X := C_X
|
| 364 | ABS_MT_TOOL_Y := C_Y
|
| 365 |
|
| 366 | Unfortunately, there is not enough information to specify both the touching
|
| 367 | ellipse and the tool ellipse, so one has to resort to approximations. One
|
| 368 | simple scheme, which is compatible with earlier usage, is::
|
| 369 |
|
| 370 | ABS_MT_TOUCH_MAJOR := min(X, Y)
|
| 371 | ABS_MT_TOUCH_MINOR := <not used>
|
| 372 | ABS_MT_ORIENTATION := <not used>
|
| 373 | ABS_MT_WIDTH_MAJOR := min(X, Y) + distance(T, C)
|
| 374 | ABS_MT_WIDTH_MINOR := min(X, Y)
|
| 375 |
|
| 376 | Rationale: We have no information about the orientation of the touching
|
| 377 | ellipse, so approximate it with an inscribed circle instead. The tool
|
| 378 | ellipse should align with the vector (T - C), so the diameter must
|
| 379 | increase with distance(T, C). Finally, assume that the touch diameter is
|
| 380 | equal to the tool thickness, and we arrive at the formulas above.
|
| 381 |
|
| 382 | Finger Tracking
|
| 383 | ---------------
|
| 384 |
|
| 385 | The process of finger tracking, i.e., to assign a unique trackingID to each
|
| 386 | initiated contact on the surface, is a Euclidean Bipartite Matching
|
| 387 | problem. At each event synchronization, the set of actual contacts is
|
| 388 | matched to the set of contacts from the previous synchronization. A full
|
| 389 | implementation can be found in .
|
| 390 |
|
| 391 |
|
| 392 | Gestures
|
| 393 | --------
|
| 394 |
|
| 395 | In the specific application of creating gesture events, the TOUCH and WIDTH
|
| 396 | parameters can be used to, e.g., approximate finger pressure or distinguish
|
| 397 | between index finger and thumb. With the addition of the MINOR parameters,
|
| 398 | one can also distinguish between a sweeping finger and a pointing finger,
|
| 399 | and with ORIENTATION, one can detect twisting of fingers.
|
| 400 |
|
| 401 |
|
| 402 | Notes
|
| 403 | -----
|
| 404 |
|
| 405 | In order to stay compatible with existing applications, the data reported
|
| 406 | in a finger packet must not be recognized as single-touch events.
|
| 407 |
|
| 408 | For type A devices, all finger data bypasses input filtering, since
|
| 409 | subsequent events of the same type refer to different fingers.
|
| 410 |
|
| 411 |
|
| 412 |
|
| 413 |
|
| 414 |
|
| 415 |
|