projects/linux·commit b1e00ffa

Merge tag 'trace-v7.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace

b1e00ffa · Linus Torvalds <[email protected]> · 2026-09-06 14:21:24 -07:00

Pull tracing fixes from Steven Rostedt:

 - Fix several tracefs files that did not take the trace_array reference

   A trace instance can be created and destroyed in the tracefs
   "instances" directory via mkdir and rmdir respectively. The instance
   is represented by a trace_array descriptor.

   Most tracefs files pass the trace_array as the private data of the
   inode to the open/read/write functions. Since there is no locking
   between the time a task opens a file and the deletion of the instance
   (and the freeing of the trace_array), each open needs to get a
   reference to the trace_array and each close must remove it.

   An instance can't be removed if there's any reference taken on its
   trace_array. The open function uses trace_array_get() that takes a
   lock (preventing removal of instances) and iterates the list of all
   existing trace_arrays and if it finds a match, it takes the reference
   and releases the lock. If it doesn't find a match, it causes the open
   to return -ENODEV.

   There were some added files that did not take the trace_array
   reference on open that needed to be fixed. Sashiko also correctly
   pointed out that there were some files that took an address of an
   field or element of the trace_array which had a pointer back to the
   trace_array to take its reference on open. But this leaves a slight
   race between referencing this element to get the trace_array as the
   element itself could be freed. To solve this, some helper functions
   were created to look for trace_arrays with this field or element in
   the search so that the element did not have to be dereferenced before
   the trace_array's reference was taken.

 - Add a lock around ftrace_ops initialization

   When a ftrace_ops is first used by ftrace, some internal
   initialization is performed on the ops. But if multiple tasks were
   calling functions that did this initialization, it could race and
   perform doing the initialization more than once, corrupting the
   internal data. Add a lock in the initialization code to prevent this
   from happening.

 - Fix splice reads on mmapped buffers

   The logic in the ring buffer splice code for mmapped buffers is
   supposed to do a copy of the memory as the mapped buffers can't be
   given to splice. But there was an if statement within the copy code
   that would return a -1 if a request for a full page was done and it
   wasn't a partial read. This is because this logic was written before
   mmapped buffers existed and this case didn't make sense at the time.
   For mmapped buffers it makes perfect sense and by returning early can
   drop a lot of pages unnecessarily.

 - Have the persistent ring buffer validation check nr_subbufs

   Sashiko reported that the validation code was relying on the saved
   nr_subbufs to match the calculated nr_pages + 1 and if they were off,
   that the code could cause corruption. Sashiko is correct, and the
   saved nr_subbufs should be validated before assuming it is correct.

 - Do not allow more than one instance with the same name on cmdline

   If an admin were to add more than one trace instances with the same
   name they all would be created, but only the first one would be
   accessible via tracefs. This used to not be allowed but some
   restructuring of code has since made it possible.

 - Fix the race between subbuf resize and trace_pipe_raw readers

   If a task was reading trace_pipe_raw while another task was changing
   the ring buffer subbuf size, it could crash the reader. The
   trace_pipe_raw readers do get their own copy of the page from the
   buffer, but the code needs some restructuring to not have the resize
   of the subbuffers cause issues.

 - Cap the size of the mapped (static) ring buffer nr_pages

   The meta data used for ring buffer mapped buffers is 32 bit in size.
   A normal ring buffer could (in theory) have more than 4 billion
   pages. But this is not allowed by mapped buffers, so enforce it.

* tag 'trace-v7.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace:
  ring-buffer: Use a macro for static buffer bits
  tracing: Fix comment in tracing_buffers_splice_read()
  ring-buffer: Prevent truncation of nr_pages / nr_subbufs
  ring-buffer: Cap static ring buffer nr_pages
  tracing: Fix subbuf resize races with trace_pipe_raw readers
  tracing: Fix to avoid creating trace instances with duplicate names
  ring-buffer: Add checking nr_subbufs to persistent ring buffer validation
  ring-buffer: Allow splice reads on static buffers
  tracing: Take trace_array reference when opening options file
  ftrace: Synchronize the initialization of ftrace_ops
  ftrace: Take trace_array reference before accessing its ftrace_ops
  tracing: Have show_event_filters/triggers files take trace array ref
Commit
b1e00ffaf91c41eb752a1c200295c9ab7abfae1d
Parents
2beb1b31 · d80e1215

Merge commit. Changes below are compared with the first parent: 2beb1b31.

Changes

10 files changed · +357 −185

Only the first 600 diff lines are shown. Download the complete patch.

include/linux/ftrace.h

View file
diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h
index 02bc502..bd76a16 100644
--- a/include/linux/ftrace.h
+++ b/include/linux/ftrace.h
@@ -866,8 +866,9 @@ unsigned long ftrace_get_addr_new(struct dyn_ftrace *rec);
unsigned long ftrace_get_addr_curr(struct dyn_ftrace *rec);
extern ftrace_func_t ftrace_trace_function;
+struct trace_array;
-int ftrace_regex_open(struct ftrace_ops *ops, int flag,
+int ftrace_regex_open(struct trace_array *tr, struct ftrace_ops *ops, int flag,
struct inode *inode, struct file *file);
ssize_t ftrace_filter_write(struct file *file, const char __user *ubuf,
size_t cnt, loff_t *ppos);
@@ -1077,7 +1078,7 @@ static inline unsigned long ftrace_location(unsigned long ip)
* have them defined when ftrace is not enabled, but these
* functions may still be called. Use a macro instead of inline.
*/
-#define ftrace_regex_open(ops, flag, inod, file) ({ -ENODEV; })
+#define ftrace_regex_open(tr, ops, flag, inode, file) ({ -ENODEV; })
#define ftrace_set_early_filter(ops, buf, enable) do { } while (0)
#define ftrace_set_filter_ip(ops, ip, remove, reset) ({ -ENODEV; })
#define ftrace_set_filter_ips(ops, ips, cnt, remove, reset) ({ -ENODEV; })

include/linux/ring_buffer.h

View file
diff --git a/include/linux/ring_buffer.h b/include/linux/ring_buffer.h
index 0670742..afc7daa 100644
--- a/include/linux/ring_buffer.h
+++ b/include/linux/ring_buffer.h
@@ -218,14 +218,15 @@ bool ring_buffer_time_stamp_abs(struct trace_buffer *buffer);
size_t ring_buffer_nr_dirty_pages(struct trace_buffer *buffer, int cpu);
struct buffer_data_read_page;
-struct buffer_data_read_page *
-ring_buffer_alloc_read_page(struct trace_buffer *buffer, int cpu);
+int ring_buffer_alloc_read_page(struct trace_buffer *buffer, int cpu,
+ struct buffer_data_read_page **rpage);
void ring_buffer_free_read_page(struct trace_buffer *buffer, int cpu,
struct buffer_data_read_page *page);
int ring_buffer_read_page(struct trace_buffer *buffer,
struct buffer_data_read_page *data_page,
size_t len, int cpu, int full);
void *ring_buffer_read_page_data(struct buffer_data_read_page *page);
+unsigned int ring_buffer_read_page_size(struct buffer_data_read_page *rpage);
struct trace_seq;

kernel/trace/ftrace.c

View file
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index f9d80c7..53d5db6 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -75,6 +75,8 @@
.func_hash = &opsname.local_hash, \
.local_hash.regex_lock = __MUTEX_INITIALIZER(opsname.local_hash.regex_lock), \
.subop_list = LIST_HEAD_INIT(opsname.subop_list),
+/* Used only to synchronize the initialization of ftrace_ops */
+static DEFINE_MUTEX(ops_mutex);
#else
#define INIT_OPS_HASH(opsname)
#endif
@@ -159,11 +161,18 @@ const struct ftrace_ops ftrace_nop_ops = {
static inline void ftrace_ops_init(struct ftrace_ops *ops)
{
#ifdef CONFIG_DYNAMIC_FTRACE
- if (!(ops->flags & FTRACE_OPS_FL_INITIALIZED)) {
+ unsigned long flags = smp_load_acquire(&ops->flags);
+
+ if (!(flags & FTRACE_OPS_FL_INITIALIZED)) {
+ guard(mutex)(&ops_mutex);
+ /* Could have been initialized before lock taken */
+ if (unlikely(ops->flags & FTRACE_OPS_FL_INITIALIZED))
+ return;
mutex_init(&ops->local_hash.regex_lock);
INIT_LIST_HEAD(&ops->subop_list);
ops->func_hash = &ops->local_hash;
- ops->flags |= FTRACE_OPS_FL_INITIALIZED;
+ flags = ops->flags | FTRACE_OPS_FL_INITIALIZED;
+ smp_store_release(&ops->flags, flags);
}
#endif
}
@@ -4677,7 +4686,8 @@ ftrace_avail_addrs_open(struct inode *inode, struct file *file)
/**
* ftrace_regex_open - initialize function tracer filter files
- * @ops: The ftrace_ops that hold the hash filters
+ * @tr: The trace_array that holds the ftrace_ops [optional]
+ * @ops: The ftrace_ops that hold the hash filters [optional]
* @flag: The type of filter to process
* @inode: The inode, usually passed in to your open routine
* @file: The file, usually passed in to your open routine
@@ -4691,26 +4701,45 @@ ftrace_avail_addrs_open(struct inode *inode, struct file *file)
* tracing_lseek() should be used as the lseek routine, and
* release must call ftrace_regex_release().
*
+ * Note, If @tr is not NULL, its reference has to be taken before
+ * @ops may be referenced.
+ * If @ops is NULL and @tr is not, then @tr->ops is used.
+ * If @tr is NULL and @ops is not then @ops->private is uesd for @tr.
+ * If both @tr and @ops are NULL, then the &global_ops is
+ * to be used, and @tr will be the global_ops.private pointer.
+ *
* Returns: 0 on success or a negative errno value on failure
*/
int
-ftrace_regex_open(struct ftrace_ops *ops, int flag,
+ftrace_regex_open(struct trace_array *tr, struct ftrace_ops *ops, int flag,
struct inode *inode, struct file *file)
{
- struct ftrace_iterator *iter;
+ struct ftrace_iterator *iter = NULL;
struct ftrace_hash *hash;
struct list_head *mod_head;
- struct trace_array *tr = ops->private;
- int ret = -ENOMEM;
-
- ftrace_ops_init(ops);
+ int ret = -ENODEV;
if (unlikely(ftrace_disabled))
return -ENODEV;
+ if (!tr) {
+ if (!ops)
+ ops = &global_ops;
+ tr = ops->private;
+ }
+
if (tracing_check_open_get_tr(tr))
return -ENODEV;
+ if (!ops)
+ ops = tr->ops;
+
+ if (WARN_ON_ONCE(!ops))
+ goto out;
+
+ ftrace_ops_init(ops);
+
+ ret = -ENOMEM;
iter = kzalloc_obj(*iter);
if (!iter)
goto out;
@@ -4788,21 +4817,19 @@ ftrace_regex_open(struct ftrace_ops *ops, int flag,
static int
ftrace_filter_open(struct inode *inode, struct file *file)
{
- struct ftrace_ops *ops = inode->i_private;
+ struct trace_array *tr = inode->i_private;
- /* Checks for tracefs lockdown */
- return ftrace_regex_open(ops,
- FTRACE_ITER_FILTER | FTRACE_ITER_DO_PROBES,
- inode, file);
+ return ftrace_regex_open(tr, NULL,
+ FTRACE_ITER_FILTER | FTRACE_ITER_DO_PROBES,
+ inode, file);
}
static int
ftrace_notrace_open(struct inode *inode, struct file *file)
{
- struct ftrace_ops *ops = inode->i_private;
+ struct trace_array *tr = inode->i_private;
- /* Checks for tracefs lockdown */
- return ftrace_regex_open(ops, FTRACE_ITER_NOTRACE,
+ return ftrace_regex_open(tr, NULL, FTRACE_ITER_NOTRACE,
inode, file);
}
@@ -7492,15 +7519,15 @@ static const struct file_operations ftrace_graph_notrace_fops = {
};
#endif /* CONFIG_FUNCTION_GRAPH_TRACER */
-void ftrace_create_filter_files(struct ftrace_ops *ops,
+void ftrace_create_filter_files(struct trace_array *tr,
struct dentry *parent)
{
trace_create_file("set_ftrace_filter", TRACE_MODE_WRITE, parent,
- ops, &ftrace_filter_fops);
+ tr, &ftrace_filter_fops);
trace_create_file("set_ftrace_notrace", TRACE_MODE_WRITE, parent,
- ops, &ftrace_notrace_fops);
+ tr, &ftrace_notrace_fops);
}
/*
@@ -7525,7 +7552,6 @@ void ftrace_destroy_filter_files(struct ftrace_ops *ops)
static __init int ftrace_init_dyn_tracefs(struct dentry *d_tracer)
{
-
trace_create_file("available_filter_functions", TRACE_MODE_READ,
d_tracer, NULL, &ftrace_avail_fops);
@@ -7538,7 +7564,7 @@ static __init int ftrace_init_dyn_tracefs(struct dentry *d_tracer)
trace_create_file("touched_functions", TRACE_MODE_READ,
d_tracer, NULL, &ftrace_touched_fops);
- ftrace_create_filter_files(&global_ops, d_tracer);
+ ftrace_create_filter_files(NULL, d_tracer);
#ifdef CONFIG_FUNCTION_GRAPH_TRACER
trace_create_file("set_graph_function", TRACE_MODE_WRITE, d_tracer,

kernel/trace/ring_buffer.c

View file
diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c
index 9726413..9c03a55 100644
--- a/kernel/trace/ring_buffer.c
+++ b/kernel/trace/ring_buffer.c
@@ -330,6 +330,14 @@ struct buffer_data_read_page {
struct buffer_data_page *data; /* actual data, stored in this page */
};
+static __always_inline unsigned int rb_read_page_capacity(struct buffer_data_read_page *rpage)
+{
+ return (PAGE_SIZE << rpage->order) - BUF_PAGE_HDR_SIZE;
+}
+
+/* The number of bits for static buffer ids */
+#define RB_STATIC_BITS 30
+
/*
* Note, the buffer_page list must be first. The buffer pages
* are allocated in cache lines, which means that each buffer
@@ -345,7 +353,7 @@ struct buffer_page {
local_t entries; /* entries on this page */
unsigned long real_end; /* real end of data */
unsigned order; /* order of the page */
- u32 id:30; /* ID for external mapping */
+ u32 id:RB_STATIC_BITS; /* ID for external mapping */
u32 range:1; /* Mapped via a range */
struct buffer_data_page *page; /* Actual data page */
};
@@ -652,6 +660,15 @@ static bool rb_is_static(struct ring_buffer_per_cpu *cpu_buffer)
return cpu_buffer->user_mapped || cpu_buffer->remote || cpu_buffer->ring_meta;
}
+static unsigned long rb_static_max_pages(void)
+{
+ /*
+ * Static ring buffers are using bpage::id and must account for the
+ * reader page.
+ */
+ return (1UL << RB_STATIC_BITS) - 1;
+}
+
struct ring_buffer_iter {
struct ring_buffer_per_cpu *cpu_buffer;
unsigned long head;
@@ -1669,7 +1686,7 @@ out_locked:
* This is used to help find the next per cpu subbuffer within a mapped range.
*/
static unsigned long
-rb_range_align_subbuf(unsigned long addr, int subbuf_size, int nr_subbufs)
+rb_range_align_subbuf(unsigned long addr, unsigned int subbuf_size, unsigned long nr_subbufs)
{
addr += sizeof(struct ring_buffer_cpu_meta) +
sizeof(int) * nr_subbufs;
@@ -1679,13 +1696,12 @@ rb_range_align_subbuf(unsigned long addr, int subbuf_size, int nr_subbufs)
/*
* Return the ring_buffer_meta for a given @cpu.
*/
-static void *rb_range_meta(struct trace_buffer *buffer, int nr_pages, int cpu)
+static void *rb_range_meta(struct trace_buffer *buffer, unsigned long nr_pages, int cpu)
{
- int subbuf_size = rb_subbuf_size(buffer);
+ unsigned int subbuf_size = rb_subbuf_size(buffer);
struct ring_buffer_cpu_meta *meta;
struct ring_buffer_meta *bmeta;
- unsigned long ptr;
- int nr_subbufs;
+ unsigned long ptr, nr_subbufs;
bmeta = buffer->meta;
if (!bmeta)
@@ -1731,7 +1747,7 @@ static void *rb_range_meta(struct trace_buffer *buffer, int nr_pages, int cpu)
/* Return the start of subbufs given the meta pointer */
static void *rb_subbufs_from_meta(struct ring_buffer_cpu_meta *meta)
{
- int subbuf_size = meta->subbuf_size;
+ unsigned int subbuf_size = meta->subbuf_size;
unsigned long ptr;
ptr = (unsigned long)meta;
@@ -1743,11 +1759,11 @@ static void *rb_subbufs_from_meta(struct ring_buffer_cpu_meta *meta)
/*
* Return a specific sub-buffer for a given @cpu defined by @idx.
*/
-static void *rb_range_buffer(struct ring_buffer_per_cpu *cpu_buffer, int idx)
+static void *rb_range_buffer(struct ring_buffer_per_cpu *cpu_buffer, unsigned long idx)
{
struct ring_buffer_cpu_meta *meta;
+ unsigned int subbuf_size;
unsigned long ptr;
- int subbuf_size;
meta = rb_range_meta(cpu_buffer->buffer, 0, cpu_buffer->cpu);
if (!meta)
@@ -1763,7 +1779,7 @@ static void *rb_range_buffer(struct ring_buffer_per_cpu *cpu_buffer, int idx)
ptr = (unsigned long)rb_subbufs_from_meta(meta);
- ptr += subbuf_size * idx;
+ ptr += (unsigned long)subbuf_size * idx;
if (ptr + subbuf_size > cpu_buffer->buffer->range_addr_end)
return NULL;
@@ -1840,13 +1856,12 @@ static bool rb_meta_init(struct trace_buffer *buffer, int scratch_size)
* must be the same.
*/
static bool rb_cpu_meta_valid(struct ring_buffer_cpu_meta *meta, int cpu,
- struct trace_buffer *buffer, int nr_pages,
+ struct trace_buffer *buffer, unsigned long nr_pages,
unsigned long *subbuf_mask)
{
- int subbuf_size = PAGE_SIZE;
unsigned long buffers_start;
unsigned long buffers_end;
- int i;
+ unsigned long i;
if (!subbuf_mask)
return false;
@@ -1856,8 +1871,13 @@ static bool rb_cpu_meta_valid(struct ring_buffer_cpu_meta *meta, int cpu,
return false;
}
+ if (meta->nr_subbufs != nr_pages + 1) {
+ pr_info("Ring buffer boot meta [%d] invalid nr_subbufs\n", cpu);
+ return false;
+ }
+
buffers_start = meta->first_buffer;
- buffers_end = meta->first_buffer + (subbuf_size * meta->nr_subbufs);
+ buffers_end = meta->first_buffer + (meta->nr_subbufs * PAGE_SIZE);
/* Is the head and commit buffers within the range of buffers? */
if (meta->head_buffer < buffers_start ||
@@ -2095,8 +2115,8 @@ static void rb_meta_validate_events(struct ring_buffer_per_cpu *cpu_buffer)
struct buffer_page *head_page, *orig_head, *orig_reader;
struct rb_validation_state state = { 0 };
bool skip = false;
+ unsigned long i;
int ret;
- int i;
if (!meta || !meta->head_buffer)
return;
@@ -2147,7 +2167,7 @@ static void rb_meta_validate_events(struct ring_buffer_per_cpu *cpu_buffer)
rb_validate_buffer(head_page, cpu_buffer, meta, &state, 0, state.ts);
}
if (i)
- pr_info("Ring buffer [%d] rewound %d pages\n", cpu_buffer->cpu, i);
+ pr_info("Ring buffer [%d] rewound %lu pages\n", cpu_buffer->cpu, i);
/* The last rewound page must be skipped. */
if (head_page != orig_head)
@@ -2231,7 +2251,8 @@ static void rb_meta_validate_events(struct ring_buffer_per_cpu *cpu_buffer)
}
}
-static void rb_range_meta_init(struct trace_buffer *buffer, int nr_pages, int scratch_size)
+static void rb_range_meta_init(struct trace_buffer *buffer, unsigned long nr_pages,
+ int scratch_size)
{
struct ring_buffer_cpu_meta *meta;
unsigned long *subbuf_mask;
@@ -2331,8 +2352,8 @@ static int rbm_show(struct seq_file *m, void *v)
rb_meta_subbuf_idx(meta, (void *)meta->head_buffer));
seq_printf(m, "commit_buffer: %d\n",
rb_meta_subbuf_idx(meta, (void *)meta->commit_buffer));
- seq_printf(m, "subbuf_size: %d\n", meta->subbuf_size);
- seq_printf(m, "nr_subbufs: %d\n", meta->nr_subbufs);
+ seq_printf(m, "subbuf_size: %u\n", meta->subbuf_size);
+ seq_printf(m, "nr_subbufs: %u\n", meta->nr_subbufs);
return 0;
}
@@ -2417,7 +2438,7 @@ static void *ring_buffer_desc_page(struct ring_buffer_desc *desc, unsigned int p
}
static int __rb_allocate_pages(struct ring_buffer_per_cpu *cpu_buffer,
- long nr_pages, struct list_head *pages)
+ unsigned long nr_pages, struct list_head *pages)
{
struct trace_buffer *buffer = cpu_buffer->buffer;
struct ring_buffer_cpu_meta *meta = NULL;
@@ -2545,7 +2566,7 @@ static int rb_allocate_pages(struct ring_buffer_per_cpu *cpu_buffer,
}
static struct ring_buffer_per_cpu *
-rb_allocate_cpu_buffer(struct trace_buffer *buffer, long nr_pages, int cpu)
+rb_allocate_cpu_buffer(struct trace_buffer *buffer, unsigned long nr_pages, int cpu)
{
struct ring_buffer_per_cpu *cpu_buffer __free(kfree) =
alloc_cpu_buffer(cpu);
@@ -2702,8 +2723,8 @@ static void rb_test_inject_invalid_pages(struct trace_buffer *buffer)
struct ring_buffer_cpu_meta *meta;
struct buffer_data_page *dpage;
unsigned long entry_bytes = 0;
+ unsigned int subbuf_size;
unsigned long ptr;
- int subbuf_size;
int invalid = 0;
int cpu;
int i;
@@ -2773,8 +2794,8 @@ static struct trace_buffer *alloc_buffer(unsigned long size, unsigned flags,
struct ring_buffer_remote *remote)
{
struct trace_buffer *buffer __free(kfree) = NULL;
- long nr_pages;
- int subbuf_size;
+ unsigned int subbuf_size;
+ unsigned long nr_pages;
int bsize;
int cpu;
int ret;
@@ -2828,6 +2849,8 @@ static struct trace_buffer *alloc_buffer(unsigned long size, unsigned flags,
size = end - buffers_start;
size = size / nr_cpu_ids;
+ if (size < sizeof(struct ring_buffer_cpu_meta))
+ goto fail_free_buffers;
/*
* The number of sub-buffers (nr_pages) is determined by the
* total size allocated minus the meta data size.
@@ -2837,6 +2860,10 @@ static struct trace_buffer *alloc_buffer(unsigned long size, unsigned flags,
*/
nr_pages = (size - sizeof(struct ring_buffer_cpu_meta)) /
(subbuf_size + sizeof(int));
+
+ if (nr_pages > rb_static_max_pages())
+ goto fail_free_buffers;
+
/* Need at least two pages plus the reader page */
if (nr_pages < 3)
goto fail_free_buffers;
@@ -2869,6 +2896,10 @@ static struct trace_buffer *alloc_buffer(unsigned long size, unsigned flags,
/* The writer is remote. This ring-buffer is read-only */
atomic_inc(&buffer->record_disabled);
nr_pages = desc->nr_page_va - 1;
+
+ if (nr_pages > rb_static_max_pages())
+ goto fail_free_buffers;
+
if (nr_pages < 2)
goto fail_free_buffers;
} else {
@@ -5853,12 +5884,12 @@ __rb_get_reader_page_from_remote(struct ring_buffer_per_cpu *cpu_buffer)
static struct buffer_page *
__rb_get_reader_page(struct ring_buffer_per_cpu *cpu_buffer)
{
- int max_loops = cpu_buffer->ring_meta ? cpu_buffer->nr_pages : 3;
+ unsigned long max_loops = cpu_buffer->ring_meta ? cpu_buffer->nr_pages : 3;
struct buffer_page *reader = NULL;
+ unsigned long nr_loops = 0;
unsigned long overwrite;
unsigned long flags;
int missed_events = 0;
- int nr_loops = 0;
bool ret;
local_irq_save(flags);
@@ -6176,8 +6207,8 @@ rb_iter_peek(struct ring_buffer_iter *iter, u64 *ts)
struct trace_buffer *buffer;
struct ring_buffer_per_cpu *cpu_buffer;
struct ring_buffer_event *event;
- int nr_loops = 0;
- int max_loops;
+ unsigned long nr_loops = 0;
+ unsigned long max_loops;
if (ts)
*ts = 0;
@@ -6993,56 +7024,78 @@ EXPORT_SYMBOL_GPL(ring_buffer_swap_cpu);
* ring_buffer_alloc_read_page - allocate a page to read from buffer
* @buffer: the buffer to allocate for.
* @cpu: the cpu buffer to allocate.
+ * @rpage: pointer to pass in an already allocated page (can be NULL)
+ * and returns the allocated page.
*
- * This function is used in conjunction with ring_buffer_read_page.
+ * This function is used in conjunction with ring_buffer_read_page().
* When reading a full page from the ring buffer, these functions
* can be used to speed up the process. The calling function should
* allocate a few pages first with this function. Then when it
* needs to get pages from the ring buffer, it passes the result
- * of this function into ring_buffer_read_page, which will swap
+ * of this function into ring_buffer_read_page(), which will swap
* the page that was allocated, with the read page of the buffer.
*
+ * If @rpage is provided, and it has a different order than the current
+ * subbuffer order, its payload will be freed and re-allocated. If it
+ * already matches the order, it is simply returned.
+ *
* Returns:
- * The page allocated, or ERR_PTR
+ * 0 on success, < 0 on error
*/
-struct buffer_data_read_page *
-ring_buffer_alloc_read_page(struct trace_buffer *buffer, int cpu)
+int ring_buffer_alloc_read_page(struct trace_buffer *buffer, int cpu,
+ struct buffer_data_read_page **rpage)
{
struct ring_buffer_per_cpu *cpu_buffer;
- struct buffer_data_read_page *bpage = NULL;
unsigned long flags;
+ unsigned int order;
if (!cpumask_test_cpu(cpu, buffer->cpumask))
- return ERR_PTR(-ENODEV);
+ return -ENODEV;
- bpage = kzalloc_obj(*bpage);
- if (!bpage)
- return ERR_PTR(-ENOMEM);
+ if (!rpage)
+ return -EINVAL;
- bpage->order = buffer->subbuf_order;
+ order = READ_ONCE(buffer->subbuf_order);
+
+ if (*rpage) {
+ if ((*rpage)->order == order)
+ return 0;
+
+ /* We can reuse rpage, but we discard the payload */
+ free_pages((unsigned long)(*rpage)->data, (*rpage)->order);
+ (*rpage)->data = NULL;
+ } else {
+ *rpage = kzalloc_obj(**rpage);
+ if (!*rpage)
+ return -ENOMEM;
+ }
+
+ (*rpage)->order = order;
cpu_buffer = buffer->buffers[cpu];
+
local_irq_save(flags);
arch_spin_lock(&cpu_buffer->lock);
if (cpu_buffer->free_page.data) {
- *bpage = cpu_buffer->free_page;
+ **rpage = cpu_buffer->free_page;
cpu_buffer->free_page.data = NULL;
}
arch_spin_unlock(&cpu_buffer->lock);
local_irq_restore(flags);
- if (bpage->data) {
- rb_init_data_page(bpage->data);
+ if ((*rpage)->data) {
+ rb_init_data_page((*rpage)->data);
} else {
- bpage->data = alloc_cpu_data(cpu, bpage->order);
- if (!bpage->data) {
- kfree(bpage);
- return ERR_PTR(-ENOMEM);
+ (*rpage)->data = alloc_cpu_data(cpu, (*rpage)->order);
+ if (!(*rpage)->data) {
+ kfree(*rpage);
+ *rpage = NULL;
+ return -ENOMEM;
}
}
- return bpage;
+ return 0;
}
EXPORT_SYMBOL_GPL(ring_buffer_alloc_read_page);
@@ -7050,21 +7103,30 @@ EXPORT_SYMBOL_GPL(ring_buffer_alloc_read_page);
* ring_buffer_free_read_page - free an allocated read page
* @buffer: the buffer the page was allocate for
* @cpu: the cpu buffer the page came from
- * @data_page: the page to free
+ * @rpage: the buffer_data_read_page to free
*
* Free a page allocated from ring_buffer_alloc_read_page.
*/
void ring_buffer_free_read_page(struct trace_buffer *buffer, int cpu,
- struct buffer_data_read_page *data_page)
+ struct buffer_data_read_page *rpage)
{
struct ring_buffer_per_cpu *cpu_buffer;
- struct buffer_data_page *dpage = data_page->data;
- struct page *page = virt_to_page(dpage);
+ struct buffer_data_page *dpage;
unsigned long flags;
+ struct page *page;
if (!buffer || !buffer->buffers || !buffer->buffers[cpu])
return;
+ if (!rpage)
+ return;
+
+ dpage = rpage->data;

This diff is incomplete. Download the complete patch.

kernel/trace/ring_buffer_benchmark.c

View file

This diff is incomplete. Download the complete patch.

kernel/trace/trace.c

View file

This diff is incomplete. Download the complete patch.

kernel/trace/trace.h

View file

This diff is incomplete. Download the complete patch.

kernel/trace/trace_events.c

View file

This diff is incomplete. Download the complete patch.

kernel/trace/trace_functions.c

View file

This diff is incomplete. Download the complete patch.

kernel/trace/trace_stack.c

View file

This diff is incomplete. Download the complete patch.