| 1 | |
| 2 |
|
| 3 | ==================================
|
| 4 | relay interface (formerly relayfs)
|
| 5 | ==================================
|
| 6 |
|
| 7 | The relay interface provides a means for kernel applications to
|
| 8 | efficiently log and transfer large quantities of data from the kernel
|
| 9 | to userspace via user-defined 'relay channels'.
|
| 10 |
|
| 11 | A 'relay channel' is a kernel->user data relay mechanism implemented
|
| 12 | as a set of per-cpu kernel buffers ('channel buffers'), each
|
| 13 | represented as a regular file ('relay file') in user space. Kernel
|
| 14 | clients write into the channel buffers using efficient write
|
| 15 | functions; these automatically log into the current cpu's channel
|
| 16 | buffer. User space applications mmap() or read() from the relay files
|
| 17 | and retrieve the data as it becomes available. The relay files
|
| 18 | themselves are files created in a host filesystem, e.g. debugfs, and
|
| 19 | are associated with the channel buffers using the API described below.
|
| 20 |
|
| 21 | The format of the data logged into the channel buffers is completely
|
| 22 | up to the kernel client; the relay interface does however provide
|
| 23 | hooks which allow kernel clients to impose some structure on the
|
| 24 | buffer data. The relay interface doesn't implement any form of data
|
| 25 | filtering - this also is left to the kernel client. The purpose is to
|
| 26 | keep things as simple as possible.
|
| 27 |
|
| 28 | This document provides an overview of the relay interface API. The
|
| 29 | details of the function parameters are documented along with the
|
| 30 | functions in the relay interface code - please see that for details.
|
| 31 |
|
| 32 | Semantics
|
| 33 | =========
|
| 34 |
|
| 35 | Each relay channel has one buffer per CPU; each buffer has one or more
|
| 36 | sub-buffers. Messages are written to the first sub-buffer until it is
|
| 37 | too full to contain a new message, in which case it is written to
|
| 38 | the next (if available). Messages are never split across sub-buffers.
|
| 39 | At this point, userspace can be notified so it empties the first
|
| 40 | sub-buffer, while the kernel continues writing to the next.
|
| 41 |
|
| 42 | When notified that a sub-buffer is full, the kernel knows how many
|
| 43 | bytes of it are padding, i.e., unused space occurring because a complete
|
| 44 | message couldn't fit into a sub-buffer. Userspace can use this
|
| 45 | knowledge to copy only valid data.
|
| 46 |
|
| 47 | After copying it, userspace can notify the kernel that a sub-buffer
|
| 48 | has been consumed.
|
| 49 |
|
| 50 | A relay channel can operate in a mode where it will overwrite data not
|
| 51 | yet collected by userspace, and not wait for it to be consumed.
|
| 52 |
|
| 53 | The relay channel itself does not provide for communication of such
|
| 54 | data between userspace and kernel, allowing the kernel side to remain
|
| 55 | simple and not impose a single interface on userspace. It does
|
| 56 | provide a set of examples and a separate helper though, described
|
| 57 | below.
|
| 58 |
|
| 59 | The read() interface both removes padding and internally consumes the
|
| 60 | read sub-buffers; thus in cases where read(2) is being used to drain
|
| 61 | the channel buffers, special-purpose communication between kernel and
|
| 62 | user isn't necessary for basic operation.
|
| 63 |
|
| 64 | One of the major goals of the relay interface is to provide a low
|
| 65 | overhead mechanism for conveying kernel data to userspace. While the
|
| 66 | read() interface is easy to use, it's not as efficient as the mmap()
|
| 67 | approach; the example code attempts to make the tradeoff between the
|
| 68 | two approaches as small as possible.
|
| 69 |
|
| 70 | klog and relay-apps example code
|
| 71 | ================================
|
| 72 |
|
| 73 | The relay interface itself is ready to use, but to make things easier,
|
| 74 | a couple of simple utility functions and a set of examples are provided.
|
| 75 |
|
| 76 | The relay-apps example tarball, available on the relay sourceforge
|
| 77 | site, contains a set of self-contained examples, each consisting of a
|
| 78 | pair of .c files containing boilerplate code for each of the user and
|
| 79 | kernel sides of a relay application. When combined these two sets of
|
| 80 | boilerplate code provide glue to easily stream data to disk, without
|
| 81 | having to bother with mundane housekeeping chores.
|
| 82 |
|
| 83 | The 'klog debugging functions' patch (klog.patch in the relay-apps
|
| 84 | tarball) provides a couple of high-level logging functions to the
|
| 85 | kernel which allow writing formatted text or raw data to a channel,
|
| 86 | regardless of whether a channel to write into exists or not, or even
|
| 87 | whether the relay interface is compiled into the kernel or not. These
|
| 88 | functions allow you to put unconditional 'trace' statements anywhere
|
| 89 | in the kernel or kernel modules; only when there is a 'klog handler'
|
| 90 | registered will data actually be logged (see the klog and kleak
|
| 91 | examples for details).
|
| 92 |
|
| 93 | It is of course possible to use the relay interface from scratch,
|
| 94 | i.e., without using any of the relay-apps example code or klog, but
|
| 95 | you'll have to implement communication between userspace and kernel,
|
| 96 | allowing both to convey the state of buffers (full, empty, amount of
|
| 97 | padding). The read() interface both removes padding and internally
|
| 98 | consumes the read sub-buffers; thus in cases where read(2) is being
|
| 99 | used to drain the channel buffers, special-purpose communication
|
| 100 | between kernel and user isn't necessary for basic operation. Things
|
| 101 | such as buffer-full conditions would still need to be communicated via
|
| 102 | some channel though.
|
| 103 |
|
| 104 | klog and the relay-apps examples can be found in the relay-apps
|
| 105 | tarball on http://relayfs.sourceforge.net
|
| 106 |
|
| 107 | The relay interface user space API
|
| 108 | ==================================
|
| 109 |
|
| 110 | The relay interface implements basic file operations for user space
|
| 111 | access to relay channel buffer data. Here are the file operations
|
| 112 | that are available and some comments regarding their behavior:
|
| 113 |
|
| 114 | =========== ============================================================
|
| 115 | open() enables user to open an _existing_ channel buffer.
|
| 116 |
|
| 117 | mmap() results in channel buffer being mapped into the caller's
|
| 118 | memory space. Note that you can't do a partial mmap - you
|
| 119 | must map the entire file, which is NRBUF * SUBBUFSIZE.
|
| 120 |
|
| 121 | read() read the contents of a channel buffer. The bytes read are
|
| 122 | 'consumed' by the reader, i.e., they won't be available
|
| 123 | again to subsequent reads. If the channel is being used
|
| 124 | in no-overwrite mode (the default), it can be read at any
|
| 125 | time even if there's an active kernel writer. If the
|
| 126 | channel is being used in overwrite mode and there are
|
| 127 | active channel writers, results may be unpredictable -
|
| 128 | users should make sure that all logging to the channel has
|
| 129 | ended before using read() with overwrite mode. Sub-buffer
|
| 130 | padding is automatically removed and will not be seen by
|
| 131 | the reader.
|
| 132 |
|
| 133 | sendfile() transfer data from a channel buffer to an output file
|
| 134 | descriptor. Sub-buffer padding is automatically removed
|
| 135 | and will not be seen by the reader.
|
| 136 |
|
| 137 | poll() POLLIN/POLLRDNORM/POLLERR supported. User applications are
|
| 138 | notified when sub-buffer boundaries are crossed.
|
| 139 |
|
| 140 | close() decrements the channel buffer's refcount. When the refcount
|
| 141 | reaches 0, i.e., when no process or kernel client has the
|
| 142 | buffer open, the channel buffer is freed.
|
| 143 | =========== ============================================================
|
| 144 |
|
| 145 | In order for a user application to make use of relay files, the
|
| 146 | host filesystem must be mounted. For example::
|
| 147 |
|
| 148 | mount -t debugfs debugfs /sys/kernel/debug
|
| 149 |
|
| 150 | .. Note::
|
| 151 |
|
| 152 | The host filesystem doesn't need to be mounted for kernel
|
| 153 | clients to create or use channels - it only needs to be
|
| 154 | mounted when user space applications need access to the buffer
|
| 155 | data.
|
| 156 |
|
| 157 |
|
| 158 | The relay interface kernel API
|
| 159 | ==============================
|
| 160 |
|
| 161 | Here's a summary of the API the relay interface provides to in-kernel clients:
|
| 162 |
|
| 163 | TBD(curr. line MT:/API/)
|
| 164 | channel management functions::
|
| 165 |
|
| 166 | relay_open(base_filename, parent, subbuf_size, n_subbufs,
|
| 167 | callbacks, private_data)
|
| 168 | relay_close(chan)
|
| 169 | relay_flush(chan)
|
| 170 | relay_reset(chan)
|
| 171 |
|
| 172 | channel management typically called on instigation of userspace::
|
| 173 |
|
| 174 | relay_subbufs_consumed(chan, cpu, subbufs_consumed)
|
| 175 |
|
| 176 | write functions::
|
| 177 |
|
| 178 | relay_write(chan, data, length)
|
| 179 | __relay_write(chan, data, length)
|
| 180 | relay_reserve(chan, length)
|
| 181 |
|
| 182 | callbacks::
|
| 183 |
|
| 184 | subbuf_start(buf, subbuf, prev_subbuf, prev_padding)
|
| 185 | buf_mapped(buf, filp)
|
| 186 | buf_unmapped(buf, filp)
|
| 187 | create_buf_file(filename, parent, mode, buf, is_global)
|
| 188 | remove_buf_file(dentry)
|
| 189 |
|
| 190 | helper functions::
|
| 191 |
|
| 192 | relay_buf_full(buf)
|
| 193 | subbuf_start_reserve(buf, length)
|
| 194 |
|
| 195 |
|
| 196 | Creating a channel
|
| 197 | ------------------
|
| 198 |
|
| 199 | relay_open() is used to create a channel, along with its per-cpu
|
| 200 | channel buffers. Each channel buffer will have an associated file
|
| 201 | created for it in the host filesystem, which can be and mmapped or
|
| 202 | read from in user space. The files are named basename0...basenameN-1
|
| 203 | where N is the number of online cpus, and by default will be created
|
| 204 | in the root of the filesystem (if the parent param is NULL). If you
|
| 205 | want a directory structure to contain your relay files, you should
|
| 206 | create it using the host filesystem's directory creation function,
|
| 207 | e.g. debugfs_create_dir(), and pass the parent directory to
|
| 208 | relay_open(). Users are responsible for cleaning up any directory
|
| 209 | structure they create, when the channel is closed - again the host
|
| 210 | filesystem's directory removal functions should be used for that,
|
| 211 | e.g. debugfs_remove().
|
| 212 |
|
| 213 | In order for a channel to be created and the host filesystem's files
|
| 214 | associated with its channel buffers, the user must provide definitions
|
| 215 | for two callback functions, create_buf_file() and remove_buf_file().
|
| 216 | create_buf_file() is called once for each per-cpu buffer from
|
| 217 | relay_open() and allows the user to create the file which will be used
|
| 218 | to represent the corresponding channel buffer. The callback should
|
| 219 | return the dentry of the file created to represent the channel buffer.
|
| 220 | remove_buf_file() must also be defined; it's responsible for deleting
|
| 221 | the file(s) created in create_buf_file() and is called during
|
| 222 | relay_close().
|
| 223 |
|
| 224 | Here are some typical definitions for these callbacks, in this case
|
| 225 | using debugfs::
|
| 226 |
|
| 227 | /*
|
| 228 | * create_buf_file() callback. Creates relay file in debugfs.
|
| 229 | */
|
| 230 | static struct dentry *create_buf_file_handler(const char *filename,
|
| 231 | struct dentry *parent,
|
| 232 | umode_t mode,
|
| 233 | struct rchan_buf *buf,
|
| 234 | int *is_global)
|
| 235 | {
|
| 236 | return debugfs_create_file(filename, mode, parent, buf,
|
| 237 | &relay_file_operations);
|
| 238 | }
|
| 239 |
|
| 240 | /*
|
| 241 | * remove_buf_file() callback. Removes relay file from debugfs.
|
| 242 | */
|
| 243 | static int remove_buf_file_handler(struct dentry *dentry)
|
| 244 | {
|
| 245 | debugfs_remove(dentry);
|
| 246 |
|
| 247 | return 0;
|
| 248 | }
|
| 249 |
|
| 250 | /*
|
| 251 | * relay interface callbacks
|
| 252 | */
|
| 253 | static struct rchan_callbacks relay_callbacks =
|
| 254 | {
|
| 255 | .create_buf_file = create_buf_file_handler,
|
| 256 | .remove_buf_file = remove_buf_file_handler,
|
| 257 | };
|
| 258 |
|
| 259 | And an example relay_open() invocation using them::
|
| 260 |
|
| 261 | chan = relay_open("cpu", NULL, SUBBUF_SIZE, N_SUBBUFS, &relay_callbacks, NULL);
|
| 262 |
|
| 263 | If the create_buf_file() callback fails, or isn't defined, channel
|
| 264 | creation and thus relay_open() will fail.
|
| 265 |
|
| 266 | The total size of each per-cpu buffer is calculated by multiplying the
|
| 267 | number of sub-buffers by the sub-buffer size passed into relay_open().
|
| 268 | The idea behind sub-buffers is that they're basically an extension of
|
| 269 | double-buffering to N buffers, and they also allow applications to
|
| 270 | easily implement random-access-on-buffer-boundary schemes, which can
|
| 271 | be important for some high-volume applications. The number and size
|
| 272 | of sub-buffers is completely dependent on the application and even for
|
| 273 | the same application, different conditions will warrant different
|
| 274 | values for these parameters at different times. Typically, the right
|
| 275 | values to use are best decided after some experimentation; in general,
|
| 276 | though, it's safe to assume that having only 1 sub-buffer is a bad
|
| 277 | idea - you're guaranteed to either overwrite data or lose events
|
| 278 | depending on the channel mode being used.
|
| 279 |
|
| 280 | The create_buf_file() implementation can also be defined in such a way
|
| 281 | as to allow the creation of a single 'global' buffer instead of the
|
| 282 | default per-cpu set. This can be useful for applications interested
|
| 283 | mainly in seeing the relative ordering of system-wide events without
|
| 284 | the need to bother with saving explicit timestamps for the purpose of
|
| 285 | merging/sorting per-cpu files in a postprocessing step.
|
| 286 |
|
| 287 | To have relay_open() create a global buffer, the create_buf_file()
|
| 288 | implementation should set the value of the is_global outparam to a
|
| 289 | non-zero value in addition to creating the file that will be used to
|
| 290 | represent the single buffer. In the case of a global buffer,
|
| 291 | create_buf_file() and remove_buf_file() will be called only once. The
|
| 292 | normal channel-writing functions, e.g. relay_write(), can still be
|
| 293 | used - writes from any cpu will transparently end up in the global
|
| 294 | buffer - but since it is a global buffer, callers should make sure
|
| 295 | they use the proper locking for such a buffer, either by wrapping
|
| 296 | writes in a spinlock, or by copying a write function from relay.h and
|
| 297 | creating a local version that internally does the proper locking.
|
| 298 |
|
| 299 | The private_data passed into relay_open() allows clients to associate
|
| 300 | user-defined data with a channel, and is immediately available
|
| 301 | (including in create_buf_file()) via chan->private_data or
|
| 302 | buf->chan->private_data.
|
| 303 |
|
| 304 | Channel 'modes'
|
| 305 | ---------------
|
| 306 |
|
| 307 | relay channels can be used in either of two modes - 'overwrite' or
|
| 308 | 'no-overwrite'. The mode is entirely determined by the implementation
|
| 309 | of the subbuf_start() callback, as described below. The default if no
|
| 310 | subbuf_start() callback is defined is 'no-overwrite' mode. If the
|
| 311 | default mode suits your needs, and you plan to use the read()
|
| 312 | interface to retrieve channel data, you can ignore the details of this
|
| 313 | section, as it pertains mainly to mmap() implementations.
|
| 314 |
|
| 315 | In 'overwrite' mode, also known as 'flight recorder' mode, writes
|
| 316 | continuously cycle around the buffer and will never fail, but will
|
| 317 | unconditionally overwrite old data regardless of whether it's actually
|
| 318 | been consumed. In no-overwrite mode, writes will fail, i.e., data will
|
| 319 | be lost, if the number of unconsumed sub-buffers equals the total
|
| 320 | number of sub-buffers in the channel. It should be clear that if
|
| 321 | there is no consumer or if the consumer can't consume sub-buffers fast
|
| 322 | enough, data will be lost in either case; the only difference is
|
| 323 | whether data is lost from the beginning or the end of a buffer.
|
| 324 |
|
| 325 | As explained above, a relay channel is made of up one or more
|
| 326 | per-cpu channel buffers, each implemented as a circular buffer
|
| 327 | subdivided into one or more sub-buffers. Messages are written into
|
| 328 | the current sub-buffer of the channel's current per-cpu buffer via the
|
| 329 | write functions described below. Whenever a message can't fit into
|
| 330 | the current sub-buffer, because there's no room left for it, the
|
| 331 | client is notified via the subbuf_start() callback that a switch to a
|
| 332 | new sub-buffer is about to occur. The client uses this callback to 1)
|
| 333 | initialize the next sub-buffer if appropriate 2) finalize the previous
|
| 334 | sub-buffer if appropriate and 3) return a boolean value indicating
|
| 335 | whether or not to actually move on to the next sub-buffer.
|
| 336 |
|
| 337 | To implement 'no-overwrite' mode, the userspace client provides
|
| 338 | an implementation of the subbuf_start() callback something like the
|
| 339 | following::
|
| 340 |
|
| 341 | static int subbuf_start(struct rchan_buf *buf,
|
| 342 | void *subbuf,
|
| 343 | void *prev_subbuf,
|
| 344 | unsigned int prev_padding)
|
| 345 | {
|
| 346 | if (prev_subbuf)
|
| 347 | *((unsigned *)prev_subbuf) = prev_padding;
|
| 348 |
|
| 349 | if (relay_buf_full(buf))
|
| 350 | return 0;
|
| 351 |
|
| 352 | subbuf_start_reserve(buf, sizeof(unsigned int));
|
| 353 |
|
| 354 | return 1;
|
| 355 | }
|
| 356 |
|
| 357 | If the current buffer is full, i.e., all sub-buffers remain unconsumed,
|
| 358 | the callback returns 0 to indicate that the buffer switch should not
|
| 359 | occur yet, i.e., until the consumer has had a chance to read the
|
| 360 | current set of ready sub-buffers. For the relay_buf_full() function
|
| 361 | to make sense, the consumer is responsible for notifying the relay
|
| 362 | interface when sub-buffers have been consumed via
|
| 363 | relay_subbufs_consumed(). Any subsequent attempts to write into the
|
| 364 | buffer will again invoke the subbuf_start() callback with the same
|
| 365 | parameters; only when the consumer has consumed one or more of the
|
| 366 | ready sub-buffers will relay_buf_full() return 0, in which case the
|
| 367 | buffer switch can continue.
|
| 368 |
|
| 369 | The implementation of the subbuf_start() callback for 'overwrite' mode
|
| 370 | would be very similar::
|
| 371 |
|
| 372 | static int subbuf_start(struct rchan_buf *buf,
|
| 373 | void *subbuf,
|
| 374 | void *prev_subbuf,
|
| 375 | size_t prev_padding)
|
| 376 | {
|
| 377 | if (prev_subbuf)
|
| 378 | *((unsigned *)prev_subbuf) = prev_padding;
|
| 379 |
|
| 380 | subbuf_start_reserve(buf, sizeof(unsigned int));
|
| 381 |
|
| 382 | return 1;
|
| 383 | }
|
| 384 |
|
| 385 | In this case, the relay_buf_full() check is meaningless and the
|
| 386 | callback always returns 1, causing the buffer switch to occur
|
| 387 | unconditionally. It's also meaningless for the client to use the
|
| 388 | relay_subbufs_consumed() function in this mode, as it's never
|
| 389 | consulted.
|
| 390 |
|
| 391 | The default subbuf_start() implementation, used if the client doesn't
|
| 392 | define any callbacks, or doesn't define the subbuf_start() callback,
|
| 393 | implements the simplest possible 'no-overwrite' mode, i.e., it does
|
| 394 | nothing but return 0.
|
| 395 |
|
| 396 | Header information can be reserved at the beginning of each sub-buffer
|
| 397 | by calling the subbuf_start_reserve() helper function from within the
|
| 398 | subbuf_start() callback. This reserved area can be used to store
|
| 399 | whatever information the client wants. In the example above, room is
|
| 400 | reserved in each sub-buffer to store the padding count for that
|
| 401 | sub-buffer. This is filled in for the previous sub-buffer in the
|
| 402 | subbuf_start() implementation; the padding value for the previous
|
| 403 | sub-buffer is passed into the subbuf_start() callback along with a
|
| 404 | pointer to the previous sub-buffer, since the padding value isn't
|
| 405 | known until a sub-buffer is filled. The subbuf_start() callback is
|
| 406 | also called for the first sub-buffer when the channel is opened, to
|
| 407 | give the client a chance to reserve space in it. In this case the
|
| 408 | previous sub-buffer pointer passed into the callback will be NULL, so
|
| 409 | the client should check the value of the prev_subbuf pointer before
|
| 410 | writing into the previous sub-buffer.
|
| 411 |
|
| 412 | Writing to a channel
|
| 413 | --------------------
|
| 414 |
|
| 415 | Kernel clients write data into the current cpu's channel buffer using
|
| 416 | relay_write() or __relay_write(). relay_write() is the main logging
|
| 417 | function - it uses local_irqsave() to protect the buffer and should be
|
| 418 | used if you might be logging from interrupt context. If you know
|
| 419 | you'll never be logging from interrupt context, you can use
|
| 420 | __relay_write(), which only disables preemption. These functions
|
| 421 | don't return a value, so you can't determine whether or not they
|
| 422 | failed - the assumption is that you wouldn't want to check a return
|
| 423 | value in the fast logging path anyway, and that they'll always succeed
|
| 424 | unless the buffer is full and no-overwrite mode is being used, in
|
| 425 | which case you can detect a failed write in the subbuf_start()
|
| 426 | callback by calling the relay_buf_full() helper function.
|
| 427 |
|
| 428 | relay_reserve() is used to reserve a slot in a channel buffer which
|
| 429 | can be written to later. This would typically be used in applications
|
| 430 | that need to write directly into a channel buffer without having to
|
| 431 | stage data in a temporary buffer beforehand. Because the actual write
|
| 432 | may not happen immediately after the slot is reserved, applications
|
| 433 | using relay_reserve() can keep a count of the number of bytes actually
|
| 434 | written, either in space reserved in the sub-buffers themselves or as
|
| 435 | a separate array. See the 'reserve' example in the relay-apps tarball
|
| 436 | at http://relayfs.sourceforge.net for an example of how this can be
|
| 437 | done. Because the write is under control of the client and is
|
| 438 | separated from the reserve, relay_reserve() doesn't protect the buffer
|
| 439 | at all - it's up to the client to provide the appropriate
|
| 440 | synchronization when using relay_reserve().
|
| 441 |
|
| 442 | Closing a channel
|
| 443 | -----------------
|
| 444 |
|
| 445 | The client calls relay_close() when it's finished using the channel.
|
| 446 | The channel and its associated buffers are destroyed when there are no
|
| 447 | longer any references to any of the channel buffers. relay_flush()
|
| 448 | forces a sub-buffer switch on all the channel buffers, and can be used
|
| 449 | to finalize and process the last sub-buffers before the channel is
|
| 450 | closed.
|
| 451 |
|
| 452 | Misc
|
| 453 | ----
|
| 454 |
|
| 455 | Some applications may want to keep a channel around and reuse it
|
| 456 | rather than open and close a new channel for each use. relay_reset()
|
| 457 | can be used for this purpose - it resets a channel to its initial
|
| 458 | state without reallocating channel buffer memory or destroying
|
| 459 | existing mappings. It should however only be called when it's safe to
|
| 460 | do so, i.e., when the channel isn't currently being written to.
|
| 461 |
|
| 462 | Finally, there are a couple of utility callbacks that can be used for
|
| 463 | different purposes. buf_mapped() is called whenever a channel buffer
|
| 464 | is mmapped from user space and buf_unmapped() is called when it's
|
| 465 | unmapped. The client can use this notification to trigger actions
|
| 466 | within the kernel application, such as enabling/disabling logging to
|
| 467 | the channel.
|
| 468 |
|
| 469 |
|
| 470 | Resources
|
| 471 | =========
|
| 472 |
|
| 473 | For news, example code, mailing list, etc. see the relay interface homepage:
|
| 474 |
|
| 475 | http://relayfs.sourceforge.net
|
| 476 |
|
| 477 |
|
| 478 | Credits
|
| 479 | =======
|
| 480 |
|
| 481 | The ideas and specs for the relay interface came about as a result of
|
| 482 | discussions on tracing involving the following:
|
| 483 |
|
| 484 | Michel Dagenais <[email protected]>
|
| 485 | Richard Moore <[email protected]>
|
| 486 | Bob Wisniewski <[email protected]>
|
| 487 | Karim Yaghmour <[email protected]>
|
| 488 | Tom Zanussi <[email protected]>
|
| 489 |
|
| 490 | Also thanks to Hubertus Franke for a lot of useful suggestions and bug
|
| 491 | reports.
|