| 1 | |
| 2 |
|
| 3 | ==========
|
| 4 | Netconsole
|
| 5 | ==========
|
| 6 |
|
| 7 |
|
| 8 | started by Ingo Molnar <[email protected]>, 2001.09.17
|
| 9 |
|
| 10 | 2.6 port and netpoll api by Matt Mackall <[email protected]>, Sep 9 2003
|
| 11 |
|
| 12 | IPv6 support by Cong Wang <[email protected]>, Jan 1 2013
|
| 13 |
|
| 14 | Extended console support by Tejun Heo <[email protected]>, May 1 2015
|
| 15 |
|
| 16 | Release prepend support by Breno Leitao <[email protected]>, Jul 7 2023
|
| 17 |
|
| 18 | Userdata append support by Matthew Wood <[email protected]>, Jan 22 2024
|
| 19 |
|
| 20 | Sysdata append support by Breno Leitao <[email protected]>, Jan 15 2025
|
| 21 |
|
| 22 | Introduction:
|
| 23 | =============
|
| 24 |
|
| 25 | This module logs kernel printk messages over UDP allowing debugging of
|
| 26 | problem where disk logging fails and serial consoles are impractical.
|
| 27 |
|
| 28 | It can be used either built-in or as a module. As a built-in,
|
| 29 | netconsole initializes immediately after NIC cards and will bring up
|
| 30 | the specified interface as soon as possible. While this doesn't allow
|
| 31 | capture of early kernel panics, it does capture most of the boot
|
| 32 | process.
|
| 33 |
|
| 34 | Sender and receiver configuration:
|
| 35 | ==================================
|
| 36 |
|
| 37 | It takes a string configuration parameter "netconsole" in the
|
| 38 | following format::
|
| 39 |
|
| 40 | netconsole=[+][r][src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr]
|
| 41 |
|
| 42 | where
|
| 43 | + if present, enable extended console support
|
| 44 | r if present, prepend kernel version (release) to the message
|
| 45 | src-port source for UDP packets (defaults to 6665)
|
| 46 | src-ip source IP to use (interface address)
|
| 47 | dev network interface name (eth0) or MAC address
|
| 48 | tgt-port port for logging agent (6666)
|
| 49 | tgt-ip IP address for logging agent
|
| 50 | tgt-macaddr ethernet MAC address for logging agent (broadcast)
|
| 51 |
|
| 52 | Examples::
|
| 53 |
|
| 54 | linux [email protected]/eth1,[email protected]/12:34:56:78:9a:bc
|
| 55 |
|
| 56 | or::
|
| 57 |
|
| 58 | insmod netconsole netconsole=@/,@10.0.0.2/
|
| 59 |
|
| 60 | or using IPv6::
|
| 61 |
|
| 62 | insmod netconsole netconsole=@/,@fd00:1:2:3::1/
|
| 63 |
|
| 64 | or using a MAC address to select the egress interface::
|
| 65 |
|
| 66 | linux [email protected]/22:33:44:55:66:77,[email protected]/12:34:56:78:9a:bc
|
| 67 |
|
| 68 | It also supports logging to multiple remote agents by specifying
|
| 69 | parameters for the multiple agents separated by semicolons and the
|
| 70 | complete string enclosed in "quotes", thusly::
|
| 71 |
|
| 72 | modprobe netconsole netconsole="@/,@10.0.0.2/;@/eth1,[email protected]/"
|
| 73 |
|
| 74 | Built-in netconsole starts immediately after the TCP stack is
|
| 75 | initialized and attempts to bring up the supplied dev at the supplied
|
| 76 | address.
|
| 77 |
|
| 78 | The remote host has several options to receive the kernel messages,
|
| 79 | for example:
|
| 80 |
|
| 81 | 1) syslogd
|
| 82 |
|
| 83 | 2) netcat
|
| 84 |
|
| 85 | On distributions using a BSD-based netcat version (e.g. Fedora,
|
| 86 | openSUSE and Ubuntu) the listening port must be specified without
|
| 87 | the -p switch::
|
| 88 |
|
| 89 | nc -u -l -p <port>' / 'nc -u -l <port>
|
| 90 |
|
| 91 | or::
|
| 92 |
|
| 93 | netcat -u -l -p <port>' / 'netcat -u -l <port>
|
| 94 |
|
| 95 | 3) socat
|
| 96 |
|
| 97 | ::
|
| 98 |
|
| 99 | socat udp-recv:<port> -
|
| 100 |
|
| 101 | Dynamic reconfiguration:
|
| 102 | ========================
|
| 103 |
|
| 104 | Dynamic reconfigurability is a useful addition to netconsole that enables
|
| 105 | remote logging targets to be dynamically added, removed, or have their
|
| 106 | parameters reconfigured at runtime from a configfs-based userspace interface.
|
| 107 |
|
| 108 | To include this feature, select CONFIG_NETCONSOLE_DYNAMIC when building the
|
| 109 | netconsole module (or kernel, if netconsole is built-in).
|
| 110 |
|
| 111 | Some examples follow (where configfs is mounted at the /sys/kernel/config
|
| 112 | mountpoint).
|
| 113 |
|
| 114 | To add a remote logging target (target names can be arbitrary)::
|
| 115 |
|
| 116 | cd /sys/kernel/config/netconsole/
|
| 117 | mkdir target1
|
| 118 |
|
| 119 | Note that newly created targets have default parameter values (as mentioned
|
| 120 | above) and are disabled by default -- they must first be enabled by writing
|
| 121 | "1" to the "enabled" attribute (usually after setting parameters accordingly)
|
| 122 | as described below.
|
| 123 |
|
| 124 | To remove a target::
|
| 125 |
|
| 126 | rmdir /sys/kernel/config/netconsole/othertarget/
|
| 127 |
|
| 128 | The interface exposes these parameters of a netconsole target to userspace:
|
| 129 |
|
| 130 | =============== ================================= ============
|
| 131 | enabled Is this target currently enabled? (read-write)
|
| 132 | extended Extended mode enabled (read-write)
|
| 133 | release Prepend kernel release to message (read-write)
|
| 134 | dev_name Local network interface name (read-write)
|
| 135 | local_port Source UDP port to use (read-write)
|
| 136 | remote_port Remote agent's UDP port (read-write)
|
| 137 | local_ip Source IP address to use (read-write)
|
| 138 | remote_ip Remote agent's IP address (read-write)
|
| 139 | local_mac Local interface's MAC address (read-only)
|
| 140 | remote_mac Remote agent's MAC address (read-write)
|
| 141 | transmit_errors Number of packet send errors (read-only)
|
| 142 | =============== ================================= ============
|
| 143 |
|
| 144 | The "enabled" attribute is also used to control whether the parameters of
|
| 145 | a target can be updated or not -- you can modify the parameters of only
|
| 146 | disabled targets (i.e. if "enabled" is 0).
|
| 147 |
|
| 148 | To update a target's parameters::
|
| 149 |
|
| 150 | cat enabled # check if enabled is 1
|
| 151 | echo 0 > enabled # disable the target (if required)
|
| 152 | echo eth2 > dev_name # set local interface
|
| 153 | echo 10.0.0.4 > remote_ip # update some parameter
|
| 154 | echo cb:a9:87:65:43:21 > remote_mac # update more parameters
|
| 155 | echo 1 > enabled # enable target again
|
| 156 |
|
| 157 | You can also update the local interface dynamically. This is especially
|
| 158 | useful if you want to use interfaces that have newly come up (and may not
|
| 159 | have existed when netconsole was loaded / initialized).
|
| 160 |
|
| 161 | Netconsole targets defined at boot time (or module load time) with the
|
| 162 | `netconsole=` param are assigned the name `cmdline<index>`. For example, the
|
| 163 | first target in the parameter is named `cmdline0`. You can control and modify
|
| 164 | these targets by creating configfs directories with the matching name.
|
| 165 |
|
| 166 | Let's suppose you have two netconsole targets defined at boot time::
|
| 167 |
|
| 168 | [email protected]/eth1,[email protected]/12:34:56:78:9a:bc;[email protected]/eth1,[email protected]/12:34:56:78:9a:bc
|
| 169 |
|
| 170 | You can modify these targets in runtime by creating the following targets::
|
| 171 |
|
| 172 | mkdir cmdline0
|
| 173 | cat cmdline0/remote_ip
|
| 174 | 10.0.0.2
|
| 175 |
|
| 176 | mkdir cmdline1
|
| 177 | cat cmdline1/remote_ip
|
| 178 | 10.0.0.3
|
| 179 |
|
| 180 | Append User Data
|
| 181 | ----------------
|
| 182 |
|
| 183 | Custom user data can be appended to the end of messages with netconsole
|
| 184 | dynamic configuration enabled. User data entries can be modified without
|
| 185 | changing the "enabled" attribute of a target.
|
| 186 |
|
| 187 | Directories (keys) under `userdata` are limited to 53 character length, and
|
| 188 | data in `userdata/<key>/value` are limited to 200 bytes::
|
| 189 |
|
| 190 | cd /sys/kernel/config/netconsole && mkdir cmdline0
|
| 191 | cd cmdline0
|
| 192 | mkdir userdata/foo
|
| 193 | echo bar > userdata/foo/value
|
| 194 | mkdir userdata/qux
|
| 195 | echo baz > userdata/qux/value
|
| 196 |
|
| 197 | Messages will now include this additional user data::
|
| 198 |
|
| 199 | echo "This is a message" > /dev/kmsg
|
| 200 |
|
| 201 | Sends::
|
| 202 |
|
| 203 | 12,607,22085407756,-;This is a message
|
| 204 | foo=bar
|
| 205 | qux=baz
|
| 206 |
|
| 207 | Preview the userdata that will be appended with::
|
| 208 |
|
| 209 | cd /sys/kernel/config/netconsole/cmdline0/userdata
|
| 210 | for f in `ls userdata`; do echo $f=$(cat userdata/$f/value); done
|
| 211 |
|
| 212 | If a `userdata` entry is created but no data is written to the `value` file,
|
| 213 | the entry will be omitted from netconsole messages::
|
| 214 |
|
| 215 | cd /sys/kernel/config/netconsole && mkdir cmdline0
|
| 216 | cd cmdline0
|
| 217 | mkdir userdata/foo
|
| 218 | echo bar > userdata/foo/value
|
| 219 | mkdir userdata/qux
|
| 220 |
|
| 221 | The `qux` key is omitted since it has no value::
|
| 222 |
|
| 223 | echo "This is a message" > /dev/kmsg
|
| 224 | 12,607,22085407756,-;This is a message
|
| 225 | foo=bar
|
| 226 |
|
| 227 | Delete `userdata` entries with `rmdir`::
|
| 228 |
|
| 229 | rmdir /sys/kernel/config/netconsole/cmdline0/userdata/qux
|
| 230 |
|
| 231 | .. warning::
|
| 232 | When writing strings to user data values, input is broken up per line in
|
| 233 | configfs store calls and this can cause confusing behavior::
|
| 234 |
|
| 235 | mkdir userdata/testing
|
| 236 | printf "val1\nval2" > userdata/testing/value
|
| 237 | # userdata store value is called twice, first with "val1\n" then "val2"
|
| 238 | # so "val2" is stored, being the last value stored
|
| 239 | cat userdata/testing/value
|
| 240 | val2
|
| 241 |
|
| 242 | It is recommended to not write user data values with newlines.
|
| 243 |
|
| 244 | Task name auto population in userdata
|
| 245 | -------------------------------------
|
| 246 |
|
| 247 | Inside the netconsole configfs hierarchy, there is a file called
|
| 248 | `taskname_enabled` under the `userdata` directory. This file is used to enable
|
| 249 | or disable the automatic task name population feature. This feature
|
| 250 | automatically populates the current task name that is scheduled in the CPU
|
| 251 | sneding the message.
|
| 252 |
|
| 253 | To enable task name auto-population::
|
| 254 |
|
| 255 | echo 1 > /sys/kernel/config/netconsole/target1/userdata/taskname_enabled
|
| 256 |
|
| 257 | When this option is enabled, the netconsole messages will include an additional
|
| 258 | line in the userdata field with the format `taskname=<task name>`. This allows
|
| 259 | the receiver of the netconsole messages to easily find which application was
|
| 260 | currently scheduled when that message was generated, providing extra context
|
| 261 | for kernel messages and helping to categorize them.
|
| 262 |
|
| 263 | Example::
|
| 264 |
|
| 265 | echo "This is a message" > /dev/kmsg
|
| 266 | 12,607,22085407756,-;This is a message
|
| 267 | taskname=echo
|
| 268 |
|
| 269 | In this example, the message was generated while "echo" was the current
|
| 270 | scheduled process.
|
| 271 |
|
| 272 | Kernel release auto population in userdata
|
| 273 | ------------------------------------------
|
| 274 |
|
| 275 | Within the netconsole configfs hierarchy, there is a file named `release_enabled`
|
| 276 | located in the `userdata` directory. This file controls the kernel release
|
| 277 | (version) auto-population feature, which appends the kernel release information
|
| 278 | to userdata dictionary in every message sent.
|
| 279 |
|
| 280 | To enable the release auto-population::
|
| 281 |
|
| 282 | echo 1 > /sys/kernel/config/netconsole/target1/userdata/release_enabled
|
| 283 |
|
| 284 | Example::
|
| 285 |
|
| 286 | echo "This is a message" > /dev/kmsg
|
| 287 | 12,607,22085407756,-;This is a message
|
| 288 | release=6.14.0-rc6-01219-g3c027fbd941d
|
| 289 |
|
| 290 | .. note::
|
| 291 |
|
| 292 | This feature provides the same data as the "release prepend" feature.
|
| 293 | However, in this case, the release information is appended to the userdata
|
| 294 | dictionary rather than being included in the message header.
|
| 295 |
|
| 296 |
|
| 297 | CPU number auto population in userdata
|
| 298 | --------------------------------------
|
| 299 |
|
| 300 | Inside the netconsole configfs hierarchy, there is a file called
|
| 301 | `cpu_nr` under the `userdata` directory. This file is used to enable or disable
|
| 302 | the automatic CPU number population feature. This feature automatically
|
| 303 | populates the CPU number that is sending the message.
|
| 304 |
|
| 305 | To enable the CPU number auto-population::
|
| 306 |
|
| 307 | echo 1 > /sys/kernel/config/netconsole/target1/userdata/cpu_nr
|
| 308 |
|
| 309 | When this option is enabled, the netconsole messages will include an additional
|
| 310 | line in the userdata field with the format `cpu=<cpu_number>`. This allows the
|
| 311 | receiver of the netconsole messages to easily differentiate and demultiplex
|
| 312 | messages originating from different CPUs, which is particularly useful when
|
| 313 | dealing with parallel log output.
|
| 314 |
|
| 315 | Example::
|
| 316 |
|
| 317 | echo "This is a message" > /dev/kmsg
|
| 318 | 12,607,22085407756,-;This is a message
|
| 319 | cpu=42
|
| 320 |
|
| 321 | In this example, the message was sent by CPU 42.
|
| 322 |
|
| 323 | .. note::
|
| 324 |
|
| 325 | If the user has set a conflicting `cpu` key in the userdata dictionary,
|
| 326 | both keys will be reported, with the kernel-populated entry appearing after
|
| 327 | the user one. For example::
|
| 328 |
|
| 329 | # User-defined CPU entry
|
| 330 | mkdir -p /sys/kernel/config/netconsole/target1/userdata/cpu
|
| 331 | echo "1" > /sys/kernel/config/netconsole/target1/userdata/cpu/value
|
| 332 |
|
| 333 | Output might look like::
|
| 334 |
|
| 335 | 12,607,22085407756,-;This is a message
|
| 336 | cpu=1
|
| 337 | cpu=42 # kernel-populated value
|
| 338 |
|
| 339 |
|
| 340 | Message ID auto population in userdata
|
| 341 | --------------------------------------
|
| 342 |
|
| 343 | Within the netconsole configfs hierarchy, there is a file named `msgid_enabled`
|
| 344 | located in the `userdata` directory. This file controls the message ID
|
| 345 | auto-population feature, which assigns a numeric id to each message sent to a
|
| 346 | given target and appends the ID to userdata dictionary in every message sent.
|
| 347 |
|
| 348 | The message ID is generated using a per-target 32 bit counter that is
|
| 349 | incremented for every message sent to the target. Note that this counter will
|
| 350 | eventually wrap around after reaching uint32_t max value, so the message ID is
|
| 351 | not globally unique over time. However, it can still be used by the target to
|
| 352 | detect if messages were dropped before reaching the target by identifying gaps
|
| 353 | in the sequence of IDs.
|
| 354 |
|
| 355 | It is important to distinguish message IDs from the message <sequnum> field.
|
| 356 | Some kernel messages may never reach netconsole (for example, due to printk
|
| 357 | rate limiting). Thus, a gap in <sequnum> cannot be solely relied upon to
|
| 358 | indicate that a message was dropped during transmission, as it may never have
|
| 359 | been sent via netconsole. The message ID, on the other hand, is only assigned
|
| 360 | to messages that are actually transmitted via netconsole.
|
| 361 |
|
| 362 | Example::
|
| 363 |
|
| 364 | echo "This is message #1" > /dev/kmsg
|
| 365 | echo "This is message #2" > /dev/kmsg
|
| 366 | 13,434,54928466,-;This is message #1
|
| 367 | msgid=1
|
| 368 | 13,435,54934019,-;This is message #2
|
| 369 | msgid=2
|
| 370 |
|
| 371 |
|
| 372 | Extended console:
|
| 373 | =================
|
| 374 |
|
| 375 | If '+' is prefixed to the configuration line or "extended" config file
|
| 376 | is set to 1, extended console support is enabled. An example boot
|
| 377 | param follows::
|
| 378 |
|
| 379 | linux [email protected]/eth1,[email protected]/12:34:56:78:9a:bc
|
| 380 |
|
| 381 | Log messages are transmitted with extended metadata header in the
|
| 382 | following format which is the same as /dev/kmsg::
|
| 383 |
|
| 384 | <level>,<sequnum>,<timestamp>,<contflag>;<message text>
|
| 385 |
|
| 386 | If 'r' (release) feature is enabled, the kernel release version is
|
| 387 | prepended to the start of the message. Example::
|
| 388 |
|
| 389 | 6.4.0,6,444,501151268,-;netconsole: network logging started
|
| 390 |
|
| 391 | Non printable characters in <message text> are escaped using "\xff"
|
| 392 | notation. If the message contains optional dictionary, verbatim
|
| 393 | newline is used as the delimiter.
|
| 394 |
|
| 395 | If a message doesn't fit in certain number of bytes (currently 1000),
|
| 396 | the message is split into multiple fragments by netconsole. These
|
| 397 | fragments are transmitted with "ncfrag" header field added::
|
| 398 |
|
| 399 | ncfrag=<byte-offset>/<total-bytes>
|
| 400 |
|
| 401 | For example, assuming a lot smaller chunk size, a message "the first
|
| 402 | chunk, the 2nd chunk." may be split as follows::
|
| 403 |
|
| 404 | 6,416,1758426,-,ncfrag=0/31;the first chunk,
|
| 405 | 6,416,1758426,-,ncfrag=16/31; the 2nd chunk.
|
| 406 |
|
| 407 | Miscellaneous notes:
|
| 408 | ====================
|
| 409 |
|
| 410 | .. Warning::
|
| 411 |
|
| 412 | the default target ethernet setting uses the broadcast
|
| 413 | ethernet address to send packets, which can cause increased load on
|
| 414 | other systems on the same ethernet segment.
|
| 415 |
|
| 416 | .. Tip::
|
| 417 |
|
| 418 | some LAN switches may be configured to suppress ethernet broadcasts
|
| 419 | so it is advised to explicitly specify the remote agents' MAC addresses
|
| 420 | from the config parameters passed to netconsole.
|
| 421 |
|
| 422 | .. Tip::
|
| 423 |
|
| 424 | to find out the MAC address of, say, 10.0.0.2, you may try using::
|
| 425 |
|
| 426 | ping -c 1 10.0.0.2 ; /sbin/arp -n | grep 10.0.0.2
|
| 427 |
|
| 428 | .. Tip::
|
| 429 |
|
| 430 | in case the remote logging agent is on a separate LAN subnet than
|
| 431 | the sender, it is suggested to try specifying the MAC address of the
|
| 432 | default gateway (you may use /sbin/route -n to find it out) as the
|
| 433 | remote MAC address instead.
|
| 434 |
|
| 435 | .. note::
|
| 436 |
|
| 437 | the network device (eth1 in the above case) can run any kind
|
| 438 | of other network traffic, netconsole is not intrusive. Netconsole
|
| 439 | might cause slight delays in other traffic if the volume of kernel
|
| 440 | messages is high, but should have no other impact.
|
| 441 |
|
| 442 | .. note::
|
| 443 |
|
| 444 | if you find that the remote logging agent is not receiving or
|
| 445 | printing all messages from the sender, it is likely that you have set
|
| 446 | the "console_loglevel" parameter (on the sender) to only send high
|
| 447 | priority messages to the console. You can change this at runtime using::
|
| 448 |
|
| 449 | dmesg -n 8
|
| 450 |
|
| 451 | or by specifying "debug" on the kernel command line at boot, to send
|
| 452 | all kernel messages to the console. A specific value for this parameter
|
| 453 | can also be set using the "loglevel" kernel boot option. See the
|
| 454 | dmesg(8) man page and Documentation/admin-guide/kernel-parameters.rst
|
| 455 | for details.
|
| 456 |
|
| 457 | Netconsole was designed to be as instantaneous as possible, to
|
| 458 | enable the logging of even the most critical kernel bugs. It works
|
| 459 | from IRQ contexts as well, and does not enable interrupts while
|
| 460 | sending packets. Due to these unique needs, configuration cannot
|
| 461 | be more automatic, and some fundamental limitations will remain:
|
| 462 | only IP networks, UDP packets and ethernet devices are supported.
|