| 1 | |
| 2 |
|
| 3 | ======================================================
|
| 4 | Discovering Linux kernel subsystems used by a workload
|
| 5 | ======================================================
|
| 6 |
|
| 7 | :Authors: - Shuah Khan <[email protected]>
|
| 8 | - Shefali Sharma <[email protected]>
|
| 9 | :maintained-by: Shuah Khan <[email protected]>
|
| 10 |
|
| 11 | Key Points
|
| 12 | ==========
|
| 13 |
|
| 14 | * Understanding system resources necessary to build and run a workload
|
| 15 | is important.
|
| 16 | * Linux tracing and strace can be used to discover the system resources
|
| 17 | in use by a workload. The completeness of the system usage information
|
| 18 | depends on the completeness of coverage of a workload.
|
| 19 | * Performance and security of the operating system can be analyzed with
|
| 20 | the help of tools such as:
|
| 21 | `perf <https://man7.org/linux/man-pages/man1/perf.1.html>`_,
|
| 22 | `stress-ng <https://www.mankier.com/1/stress-ng>`_,
|
| 23 | `paxtest <https://github.com/opntr/paxtest-freebsd>`_.
|
| 24 | * Once we discover and understand the workload needs, we can focus on them
|
| 25 | to avoid regressions and use it to evaluate safety considerations.
|
| 26 |
|
| 27 | Methodology
|
| 28 | ===========
|
| 29 |
|
| 30 | `strace <https://man7.org/linux/man-pages/man1/strace.1.html>`_ is a
|
| 31 | diagnostic, instructional, and debugging tool and can be used to discover
|
| 32 | the system resources in use by a workload. Once we discover and understand
|
| 33 | the workload needs, we can focus on them to avoid regressions and use it
|
| 34 | to evaluate safety considerations. We use strace tool to trace workloads.
|
| 35 |
|
| 36 | This method of tracing using strace tells us the system calls invoked by
|
| 37 | the workload and doesn't include all the system calls that can be invoked
|
| 38 | by it. In addition, this tracing method tells us just the code paths within
|
| 39 | these system calls that are invoked. As an example, if a workload opens a
|
| 40 | file and reads from it successfully, then the success path is the one that
|
| 41 | is traced. Any error paths in that system call will not be traced. If there
|
| 42 | is a workload that provides full coverage of a workload then the method
|
| 43 | outlined here will trace and find all possible code paths. The completeness
|
| 44 | of the system usage information depends on the completeness of coverage of a
|
| 45 | workload.
|
| 46 |
|
| 47 | The goal is tracing a workload on a system running a default kernel without
|
| 48 | requiring custom kernel installs.
|
| 49 |
|
| 50 | How do we gather fine-grained system information?
|
| 51 | =================================================
|
| 52 |
|
| 53 | strace tool can be used to trace system calls made by a process and signals
|
| 54 | it receives. System calls are the fundamental interface between an
|
| 55 | application and the operating system kernel. They enable a program to
|
| 56 | request services from the kernel. For instance, the open() system call in
|
| 57 | Linux is used to provide access to a file in the file system. strace enables
|
| 58 | us to track all the system calls made by an application. It lists all the
|
| 59 | system calls made by a process and their resulting output.
|
| 60 |
|
| 61 | You can generate profiling data combining strace and perf record tools to
|
| 62 | record the events and information associated with a process. This provides
|
| 63 | insight into the process. "perf annotate" tool generates the statistics of
|
| 64 | each instruction of the program. This document goes over the details of how
|
| 65 | to gather fine-grained information on a workload's usage of system resources.
|
| 66 |
|
| 67 | We used strace to trace the perf, stress-ng, paxtest workloads to illustrate
|
| 68 | our methodology to discover resources used by a workload. This process can
|
| 69 | be applied to trace other workloads.
|
| 70 |
|
| 71 | Getting the system ready for tracing
|
| 72 | ====================================
|
| 73 |
|
| 74 | Before we can get started we will show you how to get your system ready.
|
| 75 | We assume that you have a Linux distribution running on a physical system
|
| 76 | or a virtual machine. Most distributions will include strace command. Let’s
|
| 77 | install other tools that aren’t usually included to build Linux kernel.
|
| 78 | Please note that the following works on Debian based distributions. You
|
| 79 | might have to find equivalent packages on other Linux distributions.
|
| 80 |
|
| 81 | Install tools to build Linux kernel and tools in kernel repository.
|
| 82 | scripts/ver_linux is a good way to check if your system already has
|
| 83 | the necessary tools::
|
| 84 |
|
| 85 | sudo apt-get install build-essential flex bison yacc
|
| 86 | sudo apt install libelf-dev systemtap-sdt-dev libslang2-dev libperl-dev libdw-dev
|
| 87 |
|
| 88 | cscope is a good tool to browse kernel sources. Let's install it now::
|
| 89 |
|
| 90 | sudo apt-get install cscope
|
| 91 |
|
| 92 | Install stress-ng and paxtest::
|
| 93 |
|
| 94 | apt-get install stress-ng
|
| 95 | apt-get install paxtest
|
| 96 |
|
| 97 | Workload overview
|
| 98 | =================
|
| 99 |
|
| 100 | As mentioned earlier, we used strace to trace perf bench, stress-ng and
|
| 101 | paxtest workloads to show how to analyze a workload and identify Linux
|
| 102 | subsystems used by these workloads. Let's start with an overview of these
|
| 103 | three workloads to get a better understanding of what they do and how to
|
| 104 | use them.
|
| 105 |
|
| 106 | perf bench (all) workload
|
| 107 | -------------------------
|
| 108 |
|
| 109 | The perf bench command contains multiple multi-threaded microkernel
|
| 110 | benchmarks for executing different subsystems in the Linux kernel and
|
| 111 | system calls. This allows us to easily measure the impact of changes,
|
| 112 | which can help mitigate performance regressions. It also acts as a common
|
| 113 | benchmarking framework, enabling developers to easily create test cases,
|
| 114 | integrate transparently, and use performance-rich tooling subsystems.
|
| 115 |
|
| 116 | Stress-ng netdev stressor workload
|
| 117 | ----------------------------------
|
| 118 |
|
| 119 | stress-ng is used for performing stress testing on the kernel. It allows
|
| 120 | you to exercise various physical subsystems of the computer, as well as
|
| 121 | interfaces of the OS kernel, using "stressor-s". They are available for
|
| 122 | CPU, CPU cache, devices, I/O, interrupts, file system, memory, network,
|
| 123 | operating system, pipelines, schedulers, and virtual machines. Please refer
|
| 124 | to the `stress-ng man-page <https://www.mankier.com/1/stress-ng>`_ to
|
| 125 | find the description of all the available stressor-s. The netdev stressor
|
| 126 | starts specified number (N) of workers that exercise various netdevice
|
| 127 | ioctl commands across all the available network devices.
|
| 128 |
|
| 129 | paxtest kiddie workload
|
| 130 | -----------------------
|
| 131 |
|
| 132 | paxtest is a program that tests buffer overflows in the kernel. It tests
|
| 133 | kernel enforcements over memory usage. Generally, execution in some memory
|
| 134 | segments makes buffer overflows possible. It runs a set of programs that
|
| 135 | attempt to subvert memory usage. It is used as a regression test suite for
|
| 136 | PaX, but might be useful to test other memory protection patches for the
|
| 137 | kernel. We used paxtest kiddie mode which looks for simple vulnerabilities.
|
| 138 |
|
| 139 | What is strace and how do we use it?
|
| 140 | ====================================
|
| 141 |
|
| 142 | As mentioned earlier, strace which is a useful diagnostic, instructional,
|
| 143 | and debugging tool and can be used to discover the system resources in use
|
| 144 | by a workload. It can be used:
|
| 145 |
|
| 146 | * To see how a process interacts with the kernel.
|
| 147 | * To see why a process is failing or hanging.
|
| 148 | * For reverse engineering a process.
|
| 149 | * To find the files on which a program depends.
|
| 150 | * For analyzing the performance of an application.
|
| 151 | * For troubleshooting various problems related to the operating system.
|
| 152 |
|
| 153 | In addition, strace can generate run-time statistics on times, calls, and
|
| 154 | errors for each system call and report a summary when program exits,
|
| 155 | suppressing the regular output. This attempts to show system time (CPU time
|
| 156 | spent running in the kernel) independent of wall clock time. We plan to use
|
| 157 | these features to get information on workload system usage.
|
| 158 |
|
| 159 | strace command supports basic, verbose, and stats modes. strace command when
|
| 160 | run in verbose mode gives more detailed information about the system calls
|
| 161 | invoked by a process.
|
| 162 |
|
| 163 | Running strace -c generates a report of the percentage of time spent in each
|
| 164 | system call, the total time in seconds, the microseconds per call, the total
|
| 165 | number of calls, the count of each system call that has failed with an error
|
| 166 | and the type of system call made.
|
| 167 |
|
| 168 | * Usage: strace <command we want to trace>
|
| 169 | * Verbose mode usage: strace -v <command>
|
| 170 | * Gather statistics: strace -c <command>
|
| 171 |
|
| 172 | We used the “-c” option to gather fine-grained run-time statistics in use
|
| 173 | by three workloads we have chose for this analysis.
|
| 174 |
|
| 175 | * perf
|
| 176 | * stress-ng
|
| 177 | * paxtest
|
| 178 |
|
| 179 | What is cscope and how do we use it?
|
| 180 | ====================================
|
| 181 |
|
| 182 | Now let’s look at `cscope <https://cscope.sourceforge.net/>`_, a command
|
| 183 | line tool for browsing C, C++ or Java code-bases. We can use it to find
|
| 184 | all the references to a symbol, global definitions, functions called by a
|
| 185 | function, functions calling a function, text strings, regular expression
|
| 186 | patterns, files including a file.
|
| 187 |
|
| 188 | We can use cscope to find which system call belongs to which subsystem.
|
| 189 | This way we can find the kernel subsystems used by a process when it is
|
| 190 | executed.
|
| 191 |
|
| 192 | Let’s checkout the latest Linux repository and build cscope database::
|
| 193 |
|
| 194 | git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git linux
|
| 195 | cd linux
|
| 196 | cscope -R -p10 # builds cscope.out database before starting browse session
|
| 197 | cscope -d -p10 # starts browse session on cscope.out database
|
| 198 |
|
| 199 | Note: Run "cscope -R -p10" to build the database and "cscope -d -p10" to
|
| 200 | enter into the browsing session. cscope by default uses the cscope.out
|
| 201 | database. To get out of this mode press ctrl+d. -p option is used to
|
| 202 | specify the number of file path components to display. -p10 is optimal
|
| 203 | for browsing kernel sources.
|
| 204 |
|
| 205 | Alternatively, the kernel build system can generate the cscope database::
|
| 206 |
|
| 207 | make cscope
|
| 208 |
|
| 209 | To exclude directories from the generated database, pass IGNORE_DIRS to
|
| 210 | the cscope target. For example, to exclude Documentation/, run::
|
| 211 |
|
| 212 | make IGNORE_DIRS="Documentation" cscope
|
| 213 |
|
| 214 | What is perf and how do we use it?
|
| 215 | ==================================
|
| 216 |
|
| 217 | Perf is an analysis tool based on Linux 2.6+ systems, which abstracts the
|
| 218 | CPU hardware difference in performance measurement in Linux, and provides
|
| 219 | a simple command line interface. Perf is based on the perf_events interface
|
| 220 | exported by the kernel. It is very useful for profiling the system and
|
| 221 | finding performance bottlenecks in an application.
|
| 222 |
|
| 223 | If you haven't already checked out the Linux mainline repository, you can do
|
| 224 | so and then build kernel and perf tool::
|
| 225 |
|
| 226 | git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git linux
|
| 227 | cd linux
|
| 228 | make -j3 all
|
| 229 | cd tools/perf
|
| 230 | make
|
| 231 |
|
| 232 | Note: The perf command can be built without building the kernel in the
|
| 233 | repository and can be run on older kernels. However matching the kernel
|
| 234 | and perf revisions gives more accurate information on the subsystem usage.
|
| 235 |
|
| 236 | We used "perf stat" and "perf bench" options. For a detailed information on
|
| 237 | the perf tool, run "perf -h".
|
| 238 |
|
| 239 | perf stat
|
| 240 | ---------
|
| 241 | The perf stat command generates a report of various hardware and software
|
| 242 | events. It does so with the help of hardware counter registers found in
|
| 243 | modern CPUs that keep the count of these activities. "perf stat cal" shows
|
| 244 | stats for cal command.
|
| 245 |
|
| 246 | Perf bench
|
| 247 | ----------
|
| 248 | The perf bench command contains multiple multi-threaded microkernel
|
| 249 | benchmarks for executing different subsystems in the Linux kernel and
|
| 250 | system calls. This allows us to easily measure the impact of changes,
|
| 251 | which can help mitigate performance regressions. It also acts as a common
|
| 252 | benchmarking framework, enabling developers to easily create test cases,
|
| 253 | integrate transparently, and use performance-rich tooling.
|
| 254 |
|
| 255 | "perf bench all" runs all available benchmarks in the perf bench
|
| 256 | framework. The exact set of benchmarks depends on the perf version and on
|
| 257 | the features enabled when perf was built.
|
| 258 |
|
| 259 | To list the benchmark collections available on the current system, run::
|
| 260 |
|
| 261 | perf bench
|
| 262 |
|
| 263 | To list benchmarks in a collection, run::
|
| 264 |
|
| 265 | perf bench <collection>
|
| 266 |
|
| 267 | For example, to list the benchmarks in the mem collection, run::
|
| 268 |
|
| 269 | perf bench mem
|
| 270 |
|
| 271 | What is stress-ng and how do we use it?
|
| 272 | =======================================
|
| 273 |
|
| 274 | As mentioned earlier, stress-ng is used for performing stress testing on
|
| 275 | the kernel. It allows you to exercise various physical subsystems of the
|
| 276 | computer, as well as interfaces of the OS kernel, using stressor-s. They
|
| 277 | are available for CPU, CPU cache, devices, I/O, interrupts, file system,
|
| 278 | memory, network, operating system, pipelines, schedulers, and virtual
|
| 279 | machines.
|
| 280 |
|
| 281 | The netdev stressor starts N workers that exercise various netdevice ioctl
|
| 282 | commands across all the available network devices. The following ioctls are
|
| 283 | exercised:
|
| 284 |
|
| 285 | * SIOCGIFCONF, SIOCGIFINDEX, SIOCGIFNAME, SIOCGIFFLAGS
|
| 286 | * SIOCGIFADDR, SIOCGIFNETMASK, SIOCGIFMETRIC, SIOCGIFMTU
|
| 287 | * SIOCGIFHWADDR, SIOCGIFMAP, SIOCGIFTXQLEN
|
| 288 |
|
| 289 | The following command runs the stressor::
|
| 290 |
|
| 291 | stress-ng --netdev 1 -t 60 --metrics
|
| 292 |
|
| 293 | We can use the perf record command to record the events and information
|
| 294 | associated with a process. This command records the profiling data in the
|
| 295 | perf.data file in the same directory.
|
| 296 |
|
| 297 | Using the following commands you can record the events associated with the
|
| 298 | netdev stressor, view the generated report perf.data and annotate the output
|
| 299 | to view the statistics of each instruction of the program::
|
| 300 |
|
| 301 | perf record -- stress-ng --netdev 1 -t 60 --metrics
|
| 302 | perf report
|
| 303 | perf annotate
|
| 304 |
|
| 305 | What is paxtest and how do we use it?
|
| 306 | =====================================
|
| 307 |
|
| 308 | paxtest is a program that tests buffer overflows in the kernel. It tests
|
| 309 | kernel enforcements over memory usage. Generally, execution in some memory
|
| 310 | segments makes buffer overflows possible. It runs a set of programs that
|
| 311 | attempt to subvert memory usage. It is used as a regression test suite for
|
| 312 | PaX, and will be useful to test other memory protection patches for the
|
| 313 | kernel.
|
| 314 |
|
| 315 | paxtest provides kiddie and blackhat modes. The paxtest kiddie mode runs
|
| 316 | in normal mode, whereas the blackhat mode tries to get around the protection
|
| 317 | of the kernel testing for vulnerabilities. We focus on the kiddie mode here
|
| 318 | and combine "paxtest kiddie" run with "perf record" to collect CPU stack
|
| 319 | traces for the paxtest kiddie run to see which function is calling other
|
| 320 | functions in the performance profile. Then the "dwarf" (DWARF's Call Frame
|
| 321 | Information) mode can be used to unwind the stack.
|
| 322 |
|
| 323 | The following command can be used to view resulting report in call-graph
|
| 324 | format::
|
| 325 |
|
| 326 | perf record --call-graph dwarf paxtest kiddie
|
| 327 | perf report --stdio
|
| 328 |
|
| 329 | Tracing workloads
|
| 330 | =================
|
| 331 |
|
| 332 | Now that we understand the workloads, let's start tracing them.
|
| 333 |
|
| 334 | Tracing perf bench all workload
|
| 335 | -------------------------------
|
| 336 |
|
| 337 | Run the following command to trace perf bench all workload::
|
| 338 |
|
| 339 | strace -c perf bench all
|
| 340 |
|
| 341 | **System Calls made by the workload**
|
| 342 |
|
| 343 | The below table shows the system calls invoked by the workload, number of
|
| 344 | times each system call is invoked, and the corresponding Linux subsystem.
|
| 345 |
|
| 346 | +-------------------+-----------+-----------------+-------------------------+
|
| 347 | | System Call | # calls | Linux Subsystem | System Call (API) |
|
| 348 | +===================+===========+=================+=========================+
|
| 349 | | getppid | 10000001 | Process Mgmt | sys_getpid() |
|
| 350 | +-------------------+-----------+-----------------+-------------------------+
|
| 351 | | clone | 1077 | Process Mgmt. | sys_clone() |
|
| 352 | +-------------------+-----------+-----------------+-------------------------+
|
| 353 | | prctl | 23 | Process Mgmt. | sys_prctl() |
|
| 354 | +-------------------+-----------+-----------------+-------------------------+
|
| 355 | | prlimit64 | 7 | Process Mgmt. | sys_prlimit64() |
|
| 356 | +-------------------+-----------+-----------------+-------------------------+
|
| 357 | | getpid | 10 | Process Mgmt. | sys_getpid() |
|
| 358 | +-------------------+-----------+-----------------+-------------------------+
|
| 359 | | uname | 3 | Process Mgmt. | sys_uname() |
|
| 360 | +-------------------+-----------+-----------------+-------------------------+
|
| 361 | | sysinfo | 1 | Process Mgmt. | sys_sysinfo() |
|
| 362 | +-------------------+-----------+-----------------+-------------------------+
|
| 363 | | getuid | 1 | Process Mgmt. | sys_getuid() |
|
| 364 | +-------------------+-----------+-----------------+-------------------------+
|
| 365 | | getgid | 1 | Process Mgmt. | sys_getgid() |
|
| 366 | +-------------------+-----------+-----------------+-------------------------+
|
| 367 | | geteuid | 1 | Process Mgmt. | sys_geteuid() |
|
| 368 | +-------------------+-----------+-----------------+-------------------------+
|
| 369 | | getegid | 1 | Process Mgmt. | sys_getegid() |
|
| 370 | +-------------------+-----------+-----------------+-------------------------+
|
| 371 | | close | 49951 | Filesystem | sys_close() |
|
| 372 | +-------------------+-----------+-----------------+-------------------------+
|
| 373 | | pipe | 604 | Filesystem | sys_pipe() |
|
| 374 | +-------------------+-----------+-----------------+-------------------------+
|
| 375 | | openat | 48560 | Filesystem | sys_openat() |
|
| 376 | +-------------------+-----------+-----------------+-------------------------+
|
| 377 | | fstat | 8338 | Filesystem | sys_fstat() |
|
| 378 | +-------------------+-----------+-----------------+-------------------------+
|
| 379 | | stat | 1573 | Filesystem | sys_stat() |
|
| 380 | +-------------------+-----------+-----------------+-------------------------+
|
| 381 | | pread64 | 9646 | Filesystem | sys_pread64() |
|
| 382 | +-------------------+-----------+-----------------+-------------------------+
|
| 383 | | getdents64 | 1873 | Filesystem | sys_getdents64() |
|
| 384 | +-------------------+-----------+-----------------+-------------------------+
|
| 385 | | access | 3 | Filesystem | sys_access() |
|
| 386 | +-------------------+-----------+-----------------+-------------------------+
|
| 387 | | lstat | 1880 | Filesystem | sys_lstat() |
|
| 388 | +-------------------+-----------+-----------------+-------------------------+
|
| 389 | | lseek | 6 | Filesystem | sys_lseek() |
|
| 390 | +-------------------+-----------+-----------------+-------------------------+
|
| 391 | | ioctl | 3 | Filesystem | sys_ioctl() |
|
| 392 | +-------------------+-----------+-----------------+-------------------------+
|
| 393 | | dup2 | 1 | Filesystem | sys_dup2() |
|
| 394 | +-------------------+-----------+-----------------+-------------------------+
|
| 395 | | execve | 2 | Filesystem | sys_execve() |
|
| 396 | +-------------------+-----------+-----------------+-------------------------+
|
| 397 | | fcntl | 8779 | Filesystem | sys_fcntl() |
|
| 398 | +-------------------+-----------+-----------------+-------------------------+
|
| 399 | | statfs | 1 | Filesystem | sys_statfs() |
|
| 400 | +-------------------+-----------+-----------------+-------------------------+
|
| 401 | | epoll_create | 2 | Filesystem | sys_epoll_create() |
|
| 402 | +-------------------+-----------+-----------------+-------------------------+
|
| 403 | | epoll_ctl | 64 | Filesystem | sys_epoll_ctl() |
|
| 404 | +-------------------+-----------+-----------------+-------------------------+
|
| 405 | | newfstatat | 8318 | Filesystem | sys_newfstatat() |
|
| 406 | +-------------------+-----------+-----------------+-------------------------+
|
| 407 | | eventfd2 | 192 | Filesystem | sys_eventfd2() |
|
| 408 | +-------------------+-----------+-----------------+-------------------------+
|
| 409 | | mmap | 243 | Memory Mgmt. | sys_mmap() |
|
| 410 | +-------------------+-----------+-----------------+-------------------------+
|
| 411 | | mprotect | 32 | Memory Mgmt. | sys_mprotect() |
|
| 412 | +-------------------+-----------+-----------------+-------------------------+
|
| 413 | | brk | 21 | Memory Mgmt. | sys_brk() |
|
| 414 | +-------------------+-----------+-----------------+-------------------------+
|
| 415 | | munmap | 128 | Memory Mgmt. | sys_munmap() |
|
| 416 | +-------------------+-----------+-----------------+-------------------------+
|
| 417 | | set_mempolicy | 156 | Memory Mgmt. | sys_set_mempolicy() |
|
| 418 | +-------------------+-----------+-----------------+-------------------------+
|
| 419 | | set_tid_address | 1 | Process Mgmt. | sys_set_tid_address() |
|
| 420 | +-------------------+-----------+-----------------+-------------------------+
|
| 421 | | set_robust_list | 1 | Futex | sys_set_robust_list() |
|
| 422 | +-------------------+-----------+-----------------+-------------------------+
|
| 423 | | futex | 341 | Futex | sys_futex() |
|
| 424 | +-------------------+-----------+-----------------+-------------------------+
|
| 425 | | sched_getaffinity | 79 | Scheduler | sys_sched_getaffinity() |
|
| 426 | +-------------------+-----------+-----------------+-------------------------+
|
| 427 | | sched_setaffinity | 223 | Scheduler | sys_sched_setaffinity() |
|
| 428 | +-------------------+-----------+-----------------+-------------------------+
|
| 429 | | socketpair | 202 | Network | sys_socketpair() |
|
| 430 | +-------------------+-----------+-----------------+-------------------------+
|
| 431 | | rt_sigprocmask | 21 | Signal | sys_rt_sigprocmask() |
|
| 432 | +-------------------+-----------+-----------------+-------------------------+
|
| 433 | | rt_sigaction | 36 | Signal | sys_rt_sigaction() |
|
| 434 | +-------------------+-----------+-----------------+-------------------------+
|
| 435 | | rt_sigreturn | 2 | Signal | sys_rt_sigreturn() |
|
| 436 | +-------------------+-----------+-----------------+-------------------------+
|
| 437 | | wait4 | 889 | Time | sys_wait4() |
|
| 438 | +-------------------+-----------+-----------------+-------------------------+
|
| 439 | | clock_nanosleep | 37 | Time | sys_clock_nanosleep() |
|
| 440 | +-------------------+-----------+-----------------+-------------------------+
|
| 441 | | capget | 4 | Capability | sys_capget() |
|
| 442 | +-------------------+-----------+-----------------+-------------------------+
|
| 443 |
|
| 444 | Tracing stress-ng netdev stressor workload
|
| 445 | ------------------------------------------
|
| 446 |
|
| 447 | Run the following command to trace stress-ng netdev stressor workload::
|
| 448 |
|
| 449 | strace -c stress-ng --netdev 1 -t 60 --metrics
|
| 450 |
|
| 451 | **System Calls made by the workload**
|
| 452 |
|
| 453 | The below table shows the system calls invoked by the workload, number of
|
| 454 | times each system call is invoked, and the corresponding Linux subsystem.
|
| 455 |
|
| 456 | +-------------------+-----------+-----------------+-------------------------+
|
| 457 | | System Call | # calls | Linux Subsystem | System Call (API) |
|
| 458 | +===================+===========+=================+=========================+
|
| 459 | | openat | 74 | Filesystem | sys_openat() |
|
| 460 | +-------------------+-----------+-----------------+-------------------------+
|
| 461 | | close | 75 | Filesystem | sys_close() |
|
| 462 | +-------------------+-----------+-----------------+-------------------------+
|
| 463 | | read | 58 | Filesystem | sys_read() |
|
| 464 | +-------------------+-----------+-----------------+-------------------------+
|
| 465 | | fstat | 20 | Filesystem | sys_fstat() |
|
| 466 | +-------------------+-----------+-----------------+-------------------------+
|
| 467 | | flock | 10 | Filesystem | sys_flock() |
|
| 468 | +-------------------+-----------+-----------------+-------------------------+
|
| 469 | | write | 7 | Filesystem | sys_write() |
|
| 470 | +-------------------+-----------+-----------------+-------------------------+
|
| 471 | | getdents64 | 8 | Filesystem | sys_getdents64() |
|
| 472 | +-------------------+-----------+-----------------+-------------------------+
|
| 473 | | pread64 | 8 | Filesystem | sys_pread64() |
|
| 474 | +-------------------+-----------+-----------------+-------------------------+
|
| 475 | | lseek | 1 | Filesystem | sys_lseek() |
|
| 476 | +-------------------+-----------+-----------------+-------------------------+
|
| 477 | | access | 2 | Filesystem | sys_access() |
|
| 478 | +-------------------+-----------+-----------------+-------------------------+
|
| 479 | | getcwd | 1 | Filesystem | sys_getcwd() |
|
| 480 | +-------------------+-----------+-----------------+-------------------------+
|
| 481 | | execve | 1 | Filesystem | sys_execve() |
|
| 482 | +-------------------+-----------+-----------------+-------------------------+
|
| 483 | | mmap | 61 | Memory Mgmt. | sys_mmap() |
|
| 484 | +-------------------+-----------+-----------------+-------------------------+
|
| 485 | | munmap | 3 | Memory Mgmt. | sys_munmap() |
|
| 486 | +-------------------+-----------+-----------------+-------------------------+
|
| 487 | | mprotect | 20 | Memory Mgmt. | sys_mprotect() |
|
| 488 | +-------------------+-----------+-----------------+-------------------------+
|
| 489 | | mlock | 2 | Memory Mgmt. | sys_mlock() |
|
| 490 | +-------------------+-----------+-----------------+-------------------------+
|
| 491 | | brk | 3 | Memory Mgmt. | sys_brk() |
|
| 492 | +-------------------+-----------+-----------------+-------------------------+
|
| 493 | | rt_sigaction | 21 | Signal | sys_rt_sigaction() |
|
| 494 | +-------------------+-----------+-----------------+-------------------------+
|
| 495 | | rt_sigprocmask | 1 | Signal | sys_rt_sigprocmask() |
|
| 496 | +-------------------+-----------+-----------------+-------------------------+
|
| 497 | | sigaltstack | 1 | Signal | sys_sigaltstack() |
|
| 498 | +-------------------+-----------+-----------------+-------------------------+
|
| 499 | | rt_sigreturn | 1 | Signal | sys_rt_sigreturn() |
|
| 500 | +-------------------+-----------+-----------------+-------------------------+
|
| 501 | | getpid | 8 | Process Mgmt. | sys_getpid() |
|
| 502 | +-------------------+-----------+-----------------+-------------------------+
|
| 503 | | prlimit64 | 5 | Process Mgmt. | sys_prlimit64() |
|
| 504 | +-------------------+-----------+-----------------+-------------------------+
|
| 505 | | arch_prctl | 2 | Process Mgmt. | sys_arch_prctl() |
|
| 506 | +-------------------+-----------+-----------------+-------------------------+
|
| 507 | | sysinfo | 2 | Process Mgmt. | sys_sysinfo() |
|
| 508 | +-------------------+-----------+-----------------+-------------------------+
|
| 509 | | getuid | 2 | Process Mgmt. | sys_getuid() |
|
| 510 | +-------------------+-----------+-----------------+-------------------------+
|
| 511 | | uname | 1 | Process Mgmt. | sys_uname() |
|
| 512 | +-------------------+-----------+-----------------+-------------------------+
|
| 513 | | setpgid | 1 | Process Mgmt. | sys_setpgid() |
|
| 514 | +-------------------+-----------+-----------------+-------------------------+
|
| 515 | | getrusage | 1 | Process Mgmt. | sys_getrusage() |
|
| 516 | +-------------------+-----------+-----------------+-------------------------+
|
| 517 | | geteuid | 1 | Process Mgmt. | sys_geteuid() |
|
| 518 | +-------------------+-----------+-----------------+-------------------------+
|
| 519 | | getppid | 1 | Process Mgmt. | sys_getppid() |
|
| 520 | +-------------------+-----------+-----------------+-------------------------+
|
| 521 | | sendto | 3 | Network | sys_sendto() |
|
| 522 | +-------------------+-----------+-----------------+-------------------------+
|
| 523 | | connect | 1 | Network | sys_connect() |
|
| 524 | +-------------------+-----------+-----------------+-------------------------+
|
| 525 | | socket | 1 | Network | sys_socket() |
|
| 526 | +-------------------+-----------+-----------------+-------------------------+
|
| 527 | | clone | 1 | Process Mgmt. | sys_clone() |
|
| 528 | +-------------------+-----------+-----------------+-------------------------+
|
| 529 | | set_tid_address | 1 | Process Mgmt. | sys_set_tid_address() |
|
| 530 | +-------------------+-----------+-----------------+-------------------------+
|
| 531 | | wait4 | 2 | Time | sys_wait4() |
|
| 532 | +-------------------+-----------+-----------------+-------------------------+
|
| 533 | | alarm | 1 | Time | sys_alarm() |
|
| 534 | +-------------------+-----------+-----------------+-------------------------+
|
| 535 | | set_robust_list | 1 | Futex | sys_set_robust_list() |
|
| 536 | +-------------------+-----------+-----------------+-------------------------+
|
| 537 |
|
| 538 | Tracing paxtest kiddie workload
|
| 539 | -------------------------------
|
| 540 |
|
| 541 | Run the following command to trace paxtest kiddie workload::
|
| 542 |
|
| 543 | strace -c paxtest kiddie
|
| 544 |
|
| 545 | **System Calls made by the workload**
|
| 546 |
|
| 547 | The below table shows the system calls invoked by the workload, number of
|
| 548 | times each system call is invoked, and the corresponding Linux subsystem.
|
| 549 |
|
| 550 | +-------------------+-----------+-----------------+----------------------+
|
| 551 | | System Call | # calls | Linux Subsystem | System Call (API) |
|
| 552 | +===================+===========+=================+======================+
|
| 553 | | read | 3 | Filesystem | sys_read() |
|
| 554 | +-------------------+-----------+-----------------+----------------------+
|
| 555 | | write | 11 | Filesystem | sys_write() |
|
| 556 | +-------------------+-----------+-----------------+----------------------+
|
| 557 | | close | 41 | Filesystem | sys_close() |
|
| 558 | +-------------------+-----------+-----------------+----------------------+
|
| 559 | | stat | 24 | Filesystem | sys_stat() |
|
| 560 | +-------------------+-----------+-----------------+----------------------+
|
| 561 | | fstat | 2 | Filesystem | sys_fstat() |
|
| 562 | +-------------------+-----------+-----------------+----------------------+
|
| 563 | | pread64 | 6 | Filesystem | sys_pread64() |
|
| 564 | +-------------------+-----------+-----------------+----------------------+
|
| 565 | | access | 1 | Filesystem | sys_access() |
|
| 566 | +-------------------+-----------+-----------------+----------------------+
|
| 567 | | pipe | 1 | Filesystem | sys_pipe() |
|
| 568 | +-------------------+-----------+-----------------+----------------------+
|
| 569 | | dup2 | 24 | Filesystem | sys_dup2() |
|
| 570 | +-------------------+-----------+-----------------+----------------------+
|
| 571 | | execve | 1 | Filesystem | sys_execve() |
|
| 572 | +-------------------+-----------+-----------------+----------------------+
|
| 573 | | fcntl | 26 | Filesystem | sys_fcntl() |
|
| 574 | +-------------------+-----------+-----------------+----------------------+
|
| 575 | | openat | 14 | Filesystem | sys_openat() |
|
| 576 | +-------------------+-----------+-----------------+----------------------+
|
| 577 | | rt_sigaction | 7 | Signal | sys_rt_sigaction() |
|
| 578 | +-------------------+-----------+-----------------+----------------------+
|
| 579 | | rt_sigreturn | 38 | Signal | sys_rt_sigreturn() |
|
| 580 | +-------------------+-----------+-----------------+----------------------+
|
| 581 | | clone | 38 | Process Mgmt. | sys_clone() |
|
| 582 | +-------------------+-----------+-----------------+----------------------+
|
| 583 | | wait4 | 44 | Time | sys_wait4() |
|
| 584 | +-------------------+-----------+-----------------+----------------------+
|
| 585 | | mmap | 7 | Memory Mgmt. | sys_mmap() |
|
| 586 | +-------------------+-----------+-----------------+----------------------+
|
| 587 | | mprotect | 3 | Memory Mgmt. | sys_mprotect() |
|
| 588 | +-------------------+-----------+-----------------+----------------------+
|
| 589 | | munmap | 1 | Memory Mgmt. | sys_munmap() |
|
| 590 | +-------------------+-----------+-----------------+----------------------+
|
| 591 | | brk | 3 | Memory Mgmt. | sys_brk() |
|
| 592 | +-------------------+-----------+-----------------+----------------------+
|
| 593 | | getpid | 1 | Process Mgmt. | sys_getpid() |
|
| 594 | +-------------------+-----------+-----------------+----------------------+
|
| 595 | | getuid | 1 | Process Mgmt. | sys_getuid() |
|
| 596 | +-------------------+-----------+-----------------+----------------------+
|
| 597 | | getgid | 1 | Process Mgmt. | sys_getgid() |
|
| 598 | +-------------------+-----------+-----------------+----------------------+
|
| 599 | | geteuid | 2 | Process Mgmt. | sys_geteuid() |
|
| 600 | +-------------------+-----------+-----------------+----------------------+
|
| 601 | | getegid | 1 | Process Mgmt. | sys_getegid() |
|
| 602 | +-------------------+-----------+-----------------+----------------------+
|
| 603 | | getppid | 1 | Process Mgmt. | sys_getppid() |
|
| 604 | +-------------------+-----------+-----------------+----------------------+
|
| 605 | | arch_prctl | 2 | Process Mgmt. | sys_arch_prctl() |
|
| 606 | +-------------------+-----------+-----------------+----------------------+
|
| 607 |
|
| 608 | Conclusion
|
| 609 | ==========
|
| 610 |
|
| 611 | This document is intended to be used as a guide on how to gather fine-grained
|
| 612 | information on the resources in use by workloads using strace.
|
| 613 |
|
| 614 | References
|
| 615 | ==========
|
| 616 |
|
| 617 | * `Discovery Linux Kernel Subsystems used by OpenAPS <https://elisa.tech/blog/2022/02/02/discovery-linux-kernel-subsystems-used-by-openaps>`_
|
| 618 | * `ELISA-White-Papers-Discovering Linux kernel subsystems used by a workload <https://github.com/elisa-tech/ELISA-White-Papers/blob/master/Processes/Discovering_Linux_kernel_subsystems_used_by_a_workload.md>`_
|
| 619 | * `strace <https://man7.org/linux/man-pages/man1/strace.1.html>`_
|
| 620 | * `perf <https://man7.org/linux/man-pages/man1/perf.1.html>`_
|
| 621 | * `paxtest README <https://github.com/opntr/paxtest-freebsd/blob/hardenedbsd/0.9.14-hbsd/README>`_
|
| 622 | * `stress-ng <https://www.mankier.com/1/stress-ng>`_
|
| 623 | * `Monitoring and managing system status and performance <https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/monitoring_and_managing_system_status_and_performance/index>`_
|