projects/linux·blob master · refs

325 lines · 14.6 KiB

1.. _perf_security:
2
3Perf events and tool security
4=============================
5
6Overview
7--------
8
9Usage of Performance Counters for Linux (perf_events) [1]_ , [2]_ , [3]_
10can impose a considerable risk of leaking sensitive data accessed by
11monitored processes. The data leakage is possible both in scenarios of
12direct usage of perf_events system call API [2]_ and over data files
13generated by Perf tool user mode utility (Perf) [3]_ , [4]_ . The risk
14depends on the nature of data that perf_events performance monitoring
15units (PMU) [2]_ and Perf collect and expose for performance analysis.
16Collected system and performance data may be split into several
17categories:
18
191. System hardware and software configuration data, for example: a CPU
20 model and its cache configuration, an amount of available memory and
21 its topology, used kernel and Perf versions, performance monitoring
22 setup including experiment time, events configuration, Perf command
23 line parameters, etc.
24
252. User and kernel module paths and their load addresses with sizes,
26 process and thread names with their PIDs and TIDs, timestamps for
27 captured hardware and software events.
28
293. Content of kernel software counters (e.g., for context switches, page
30 faults, CPU migrations), architectural hardware performance counters
31 (PMC) [8]_ and machine specific registers (MSR) [9]_ that provide
32 execution metrics for various monitored parts of the system (e.g.,
33 memory controller (IMC), interconnect (QPI/UPI) or peripheral (PCIe)
34 uncore counters) without direct attribution to any execution context
35 state.
36
374. Content of architectural execution context registers (e.g., RIP, RSP,
38 RBP on x86_64), process user and kernel space memory addresses and
39 data, content of various architectural MSRs that capture data from
40 this category.
41
42Data that belong to the fourth category can potentially contain
43sensitive process data. If PMUs in some monitoring modes capture values
44of execution context registers or data from process memory then access
45to such monitoring modes requires to be ordered and secured properly.
46So, perf_events performance monitoring and observability operations are
47the subject for security access control management [5]_ .
48
49perf_events access control
50-------------------------------
51
52To perform security checks, the Linux implementation splits processes
53into two categories [6]_ : a) privileged processes (whose effective user
54ID is 0, referred to as superuser or root), and b) unprivileged
55processes (whose effective UID is nonzero). Privileged processes bypass
56all kernel security permission checks so perf_events performance
57monitoring is fully available to privileged processes without access,
58scope and resource restrictions.
59
60Unprivileged processes are subject to a full security permission check
61based on the process's credentials [5]_ (usually: effective UID,
62effective GID, and supplementary group list).
63
64Linux divides the privileges traditionally associated with superuser
65into distinct units, known as capabilities [6]_ , which can be
66independently enabled and disabled on per-thread basis for processes and
67files of unprivileged users.
68
69Unprivileged processes with enabled CAP_PERFMON capability are treated
70as privileged processes with respect to perf_events performance
71monitoring and observability operations, thus, bypass *scope* permissions
72checks in the kernel. CAP_PERFMON implements the principle of least
73privilege [13]_ (POSIX 1003.1e: 2.2.2.39) for performance monitoring and
74observability operations in the kernel and provides a secure approach to
75performance monitoring and observability in the system.
76
77For backward compatibility reasons the access to perf_events monitoring and
78observability operations is also open for CAP_SYS_ADMIN privileged
79processes but CAP_SYS_ADMIN usage for secure monitoring and observability
80use cases is discouraged with respect to the CAP_PERFMON capability.
81If system audit records [14]_ for a process using perf_events system call
82API contain denial records of acquiring both CAP_PERFMON and CAP_SYS_ADMIN
83capabilities then providing the process with CAP_PERFMON capability singly
84is recommended as the preferred secure approach to resolve double access
85denial logging related to usage of performance monitoring and observability.
86
87Prior Linux v5.9 unprivileged processes using perf_events system call
88are also subject for PTRACE_MODE_READ_REALCREDS ptrace access mode check
89[7]_ , whose outcome determines whether monitoring is permitted.
90So unprivileged processes provided with CAP_SYS_PTRACE capability are
91effectively permitted to pass the check. Starting from Linux v5.9
92CAP_SYS_PTRACE capability is not required and CAP_PERFMON is enough to
93be provided for processes to make performance monitoring and observability
94operations.
95
96Other capabilities being granted to unprivileged processes can
97effectively enable capturing of additional data required for later
98performance analysis of monitored processes or a system. For example,
99CAP_SYSLOG capability permits reading kernel space memory addresses from
100/proc/kallsyms file.
101
102Privileged Perf users groups
103---------------------------------
104
105Mechanisms of capabilities, privileged capability-dumb files [6]_,
106file system ACLs [10]_ and sudo [15]_ utility can be used to create
107dedicated groups of privileged Perf users who are permitted to execute
108performance monitoring and observability without limits. The following
109steps can be taken to create such groups of privileged Perf users.
110
1111. Create perf_users group of privileged Perf users, assign perf_users
112 group to Perf tool executable and limit access to the executable for
113 other users in the system who are not in the perf_users group:
114
115::
116
117 # groupadd perf_users
118 # ls -alhF
119 -rwxr-xr-x 2 root root 11M Oct 19 15:12 perf
120 # chgrp perf_users perf
121 # ls -alhF
122 -rwxr-xr-x 2 root perf_users 11M Oct 19 15:12 perf
123 # chmod o-rwx perf
124 # ls -alhF
125 -rwxr-x--- 2 root perf_users 11M Oct 19 15:12 perf
126
1272. Assign the required capabilities to the Perf tool executable file and
128 enable members of perf_users group with monitoring and observability
129 privileges [6]_ :
130
131::
132
133 # setcap "cap_perfmon,cap_sys_ptrace,cap_syslog=ep" perf
134 # setcap -v "cap_perfmon,cap_sys_ptrace,cap_syslog=ep" perf
135 perf: OK
136 # getcap perf
137 perf = cap_sys_ptrace,cap_syslog,cap_perfmon+ep
138
139If the libcap [16]_ installed doesn't yet support "cap_perfmon", use "38" instead,
140i.e.:
141
142::
143
144 # setcap "38,cap_ipc_lock,cap_sys_ptrace,cap_syslog=ep" perf
145
146Note that you may need to have 'cap_ipc_lock' in the mix for tools such as
147'perf top', alternatively use 'perf top -m N', to reduce the memory that
148it uses for the perf ring buffer, see the memory allocation section below.
149
150Using a libcap without support for CAP_PERFMON will make cap_get_flag(caps, 38,
151CAP_EFFECTIVE, &val) fail, which will lead the default event to be 'cycles:u',
152so as a workaround explicitly ask for the 'cycles' event, i.e.:
153
154::
155
156 # perf top -e cycles
157
158To get kernel and user samples with a perf binary with just CAP_PERFMON.
159
160As a result, members of perf_users group are capable of conducting
161performance monitoring and observability by using functionality of the
162configured Perf tool executable that, when executes, passes perf_events
163subsystem scope checks.
164
165In case Perf tool executable can't be assigned required capabilities (e.g.
166file system is mounted with nosuid option or extended attributes are
167not supported by the file system) then creation of the capabilities
168privileged environment, naturally shell, is possible. The shell provides
169inherent processes with CAP_PERFMON and other required capabilities so that
170performance monitoring and observability operations are available in the
171environment without limits. Access to the environment can be open via sudo
172utility for members of perf_users group only. In order to create such
173environment:
174
1751. Create shell script that uses capsh utility [16]_ to assign CAP_PERFMON
176 and other required capabilities into ambient capability set of the shell
177 process, lock the process security bits after enabling SECBIT_NO_SETUID_FIXUP,
178 SECBIT_NOROOT and SECBIT_NO_CAP_AMBIENT_RAISE bits and then change
179 the process identity to sudo caller of the script who should essentially
180 be a member of perf_users group:
181
182::
183
184 # ls -alh /usr/local/bin/perf.shell
185 -rwxr-xr-x. 1 root root 83 Oct 13 23:57 /usr/local/bin/perf.shell
186 # cat /usr/local/bin/perf.shell
187 exec /usr/sbin/capsh --iab=^cap_perfmon --secbits=239 --user=$SUDO_USER -- -l
188
1892. Extend sudo policy at /etc/sudoers file with a rule for perf_users group:
190
191::
192
193 # grep perf_users /etc/sudoers
194 %perf_users ALL=/usr/local/bin/perf.shell
195
1963. Check that members of perf_users group have access to the privileged
197 shell and have CAP_PERFMON and other required capabilities enabled
198 in permitted, effective and ambient capability sets of an inherent process:
199
200::
201
202 $ id
203 uid=1003(capsh_test) gid=1004(capsh_test) groups=1004(capsh_test),1000(perf_users) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
204 $ sudo perf.shell
205 [sudo] password for capsh_test:
206 $ grep Cap /proc/self/status
207 CapInh: 0000004000000000
208 CapPrm: 0000004000000000
209 CapEff: 0000004000000000
210 CapBnd: 000000ffffffffff
211 CapAmb: 0000004000000000
212 $ capsh --decode=0000004000000000
213 0x0000004000000000=cap_perfmon
214
215As a result, members of perf_users group have access to the privileged
216environment where they can use tools employing performance monitoring APIs
217governed by CAP_PERFMON Linux capability.
218
219This specific access control management is only available to superuser
220or root running processes with CAP_SETPCAP, CAP_SETFCAP [6]_
221capabilities.
222
223Unprivileged users
224-----------------------------------
225
226perf_events *scope* and *access* control for unprivileged processes
227is governed by perf_event_paranoid [2]_ setting:
228
229-1:
230 Impose no *scope* and *access* restrictions on using perf_events
231 performance monitoring. Per-user per-cpu perf_event_mlock_kb [2]_
232 locking limit is ignored when allocating memory buffers for storing
233 performance data. This is the least secure mode since allowed
234 monitored *scope* is maximized and no perf_events specific limits
235 are imposed on *resources* allocated for performance monitoring.
236
237>=0:
238 *scope* includes per-process and system wide performance monitoring
239 but excludes raw tracepoints and ftrace function tracepoints
240 monitoring. CPU and system events happened when executing either in
241 user or in kernel space can be monitored and captured for later
242 analysis. Per-user per-cpu perf_event_mlock_kb locking limit is
243 imposed but ignored for unprivileged processes with CAP_IPC_LOCK
244 [6]_ capability.
245
246>=1:
247 *scope* includes per-process performance monitoring only and
248 excludes system wide performance monitoring. CPU and system events
249 happened when executing either in user or in kernel space can be
250 monitored and captured for later analysis. Per-user per-cpu
251 perf_event_mlock_kb locking limit is imposed but ignored for
252 unprivileged processes with CAP_IPC_LOCK capability.
253
254>=2:
255 *scope* includes per-process performance monitoring only. CPU and
256 system events happened when executing in user space only can be
257 monitored and captured for later analysis. Per-user per-cpu
258 perf_event_mlock_kb locking limit is imposed but ignored for
259 unprivileged processes with CAP_IPC_LOCK capability.
260
261Resource control
262---------------------------------
263
264Open file descriptors
265+++++++++++++++++++++
266
267The perf_events system call API [2]_ allocates file descriptors for
268every configured PMU event. Open file descriptors are a per-process
269accountable resource governed by the RLIMIT_NOFILE [11]_ limit
270(ulimit -n), which is usually derived from the login shell process. When
271configuring Perf collection for a long list of events on a large server
272system, this limit can be easily hit preventing required monitoring
273configuration. RLIMIT_NOFILE limit can be increased on per-user basis
274modifying content of the limits.conf file [12]_ . Ordinarily, a Perf
275sampling session (perf record) requires an amount of open perf_event
276file descriptors that is not less than the number of monitored events
277multiplied by the number of monitored CPUs.
278
279Memory allocation
280+++++++++++++++++
281
282The amount of memory available to user processes for capturing
283performance monitoring data is governed by the perf_event_mlock_kb [2]_
284setting. This perf_event specific resource setting defines overall
285per-cpu limits of memory allowed for mapping by the user processes to
286execute performance monitoring. The setting essentially extends the
287RLIMIT_MEMLOCK [11]_ limit, but only for memory regions mapped
288specifically for capturing monitored performance events and related data.
289
290For example, if a machine has eight cores and perf_event_mlock_kb limit
291is set to 516 KiB, then a user process is provided with 516 KiB * 8 =
2924128 KiB of memory above the RLIMIT_MEMLOCK limit (ulimit -l) for
293perf_event mmap buffers. In particular, this means that, if the user
294wants to start two or more performance monitoring processes, the user is
295required to manually distribute the available 4128 KiB between the
296monitoring processes, for example, using the --mmap-pages Perf record
297mode option. Otherwise, the first started performance monitoring process
298allocates all available 4128 KiB and the other processes will fail to
299proceed due to the lack of memory.
300
301RLIMIT_MEMLOCK and perf_event_mlock_kb resource constraints are ignored
302for processes with the CAP_IPC_LOCK capability. Thus, perf_events/Perf
303privileged users can be provided with memory above the constraints for
304perf_events/Perf performance monitoring purpose by providing the Perf
305executable with CAP_IPC_LOCK capability.
306
307Bibliography
308------------
309
310.. [1] `<https://lwn.net/Articles/337493/>`_
311.. [2] `<http://man7.org/linux/man-pages/man2/perf_event_open.2.html>`_
312.. [3] `<http://web.eece.maine.edu/~vweaver/projects/perf_events/>`_
313.. [4] `<https://perf.wiki.kernel.org/index.php/Main_Page>`_
314.. [5] `<https://www.kernel.org/doc/html/latest/security/credentials.html>`_
315.. [6] `<http://man7.org/linux/man-pages/man7/capabilities.7.html>`_
316.. [7] `<http://man7.org/linux/man-pages/man2/ptrace.2.html>`_
317.. [8] `<https://en.wikipedia.org/wiki/Hardware_performance_counter>`_
318.. [9] `<https://en.wikipedia.org/wiki/Model-specific_register>`_
319.. [10] `<http://man7.org/linux/man-pages/man5/acl.5.html>`_
320.. [11] `<http://man7.org/linux/man-pages/man2/getrlimit.2.html>`_
321.. [12] `<http://man7.org/linux/man-pages/man5/limits.conf.5.html>`_
322.. [13] `<https://sites.google.com/site/fullycapable>`_
323.. [14] `<http://man7.org/linux/man-pages/man8/auditd.8.html>`_
324.. [15] `<https://man7.org/linux/man-pages/man8/sudo.8.html>`_
325.. [16] `<https://git.kernel.org/pub/scm/libs/libcap/libcap.git/>`_