In this post we will learn what is max sampling rate parameters, sample overflow interrupts between kernel ticks and how linux kernel dynamically adjusts the max sample rate when it detects that an interrupt from sample overflow is taking too long.

Understanding Linux perf Sampling Limits

Linux perf can generate samples at very high rates, especially when a hardware event such as INST_RETIRED.ANY is configured with a small sampling period. Each overflow may enter a critical interrupt path (often an NMI on x86), so an excessively high sampling rate can consume a large fraction of CPU time or, in the worst case, prevent the system from making useful progress.

This post walks through the main kernel-side controls that limit perf sampling overhead:

• kernel.perf_event_max_sample_rate
• kernel.perf_cpu_time_max_percent
• the kernel’s HZ/tick-based accounting

The discussion follows the Linux kernel source in kernel/events/core.c and focuses on period-based hardware sampling rather than perf’s frequency mode.

Sources:

• Linux perf core source: https://github.com/torvalds/linux/blob/master/kernel/events/core.c
• Kernel sysctl documentation: https://docs.kernel.org/admin-guide/sysctl/kernel.html

1. Relevant perf sysctls

A familiar perf setting is kernel.perf_event_paranoid, which controls which performance-monitoring facilities are available to unprivileged users.

In kernel/events/core.c, the relevant sysctl table includes:

static const struct ctl_table events_core_sysctl_table[] = {
    {
        .procname      = "perf_event_paranoid",
        .data          = &sysctl_perf_event_paranoid,
        .maxlen        = sizeof(sysctl_perf_event_paranoid),
        .mode          = 0644,
        .proc_handler  = proc_dointvec,
    },
    {
        .procname      = "perf_event_max_sample_rate",
        .data          = &sysctl_perf_event_sample_rate,
        .maxlen        = sizeof(sysctl_perf_event_sample_rate),
        .mode          = 0644,
        .proc_handler  = perf_event_max_sample_rate_handler,
        .extra1        = SYSCTL_ONE,
    },
    {
        .procname      = "perf_cpu_time_max_percent",
        .data          = &sysctl_perf_cpu_time_max_percent,
        .maxlen        = sizeof(sysctl_perf_cpu_time_max_percent),
        .mode          = 0644,
        .proc_handler  = perf_cpu_time_max_percent_handler,
        .extra1        = SYSCTL_ZERO,
        .extra2        = SYSCTL_ONE_HUNDRED,
    },
};

proc_dointvec is a generic sysctl handler for integer values. On reads, it formats the kernel integer as text. On writes, it parses the user-provided text and stores the integer value.

proc_dointvec_minmax is a related helper that also enforces bounds specified by .extra1 and .extra2.

What is more interesting for sampling is that perf_event_max_sample_rate and perf_cpu_time_max_percent use custom handlers. Those handlers update internal perf limits whenever the sysctl values change.

2. Default variables

The relevant defaults are defined as follows:

int sysctl_perf_event_paranoid __read_mostly = 2;

/* Minimum for 512 kiB + 1 user control page. 'free' kiB per user. */
static int sysctl_perf_event_mlock __read_mostly =
    512 + (PAGE_SIZE / 1024);

/* max perf event sample rate */
#define DEFAULT_MAX_SAMPLE_RATE         100000
#define DEFAULT_SAMPLE_PERIOD_NS        \
    (NSEC_PER_SEC / DEFAULT_MAX_SAMPLE_RATE)
#define DEFAULT_CPU_TIME_MAX_PERCENT    25

int sysctl_perf_event_sample_rate __read_mostly =
    DEFAULT_MAX_SAMPLE_RATE;

static int sysctl_perf_cpu_time_max_percent __read_mostly =
    DEFAULT_CPU_TIME_MAX_PERCENT;

static int max_samples_per_tick __read_mostly =
    DIV_ROUND_UP(DEFAULT_MAX_SAMPLE_RATE, HZ);

static int perf_sample_period_ns __read_mostly =
    DEFAULT_SAMPLE_PERIOD_NS;

static int perf_sample_allowed_ns __read_mostly =
    DEFAULT_SAMPLE_PERIOD_NS * DEFAULT_CPU_TIME_MAX_PERCENT / 100;

A few notes:

• perf_event_paranoid = 2 means unprivileged users cannot perform kernel profiling. Values at this level also inherit the restrictions of lower paranoia levels.
• perf_event_mlock_kb is measured in KiB, not pages. Its default is 512 KiB + one page. On a system with 4 KiB pages, this appears as 516.

  PAGE_SIZE = 4096 bytes
  PAGE_SIZE / 1024
  = 4096 bytes / (1024 bytes/KiB)
  = 4 KiB
  512 + 4 = 516
  

• DEFAULT_MAX_SAMPLE_RATE is the initial global maximum sampling rate used by perf’s throttling policy. The value can later be changed manually or reduced dynamically by the kernel.

3. From maximum sample rate to an expected time interval

The default maximum sampling rate is:

Because:

perf computes:

Therefore:

perf_sample_period_ns = 10,000 ns

means:

At a sampling rate of 100,000 samples/s, the corresponding average wall-clock interval is 10 µs per sample.

This variable is easy to confuse with perf_event_attr.sample_period, but they are completely different.

perf_sample_period_ns is a time quantity derived from the kernel’s maximum sampling-rate policy. It does not mean “sample every 10,000 retired instructions.”

4. perf_cpu_time_max_percent

By default:

perf_cpu_time_max_percent = 25

The kernel documentation describes this value as a hint for how much CPU time perf should be allowed to spend processing samples. It is not a strict reservation of 25% of the CPU.

The initial threshold is calculated as:

static int perf_sample_allowed_ns __read_mostly =
    DEFAULT_SAMPLE_PERIOD_NS *
    DEFAULT_CPU_TIME_MAX_PERCENT / 100;

With the defaults:

So initially:

perf_sample_period_ns  = 10,000 ns
perf_sample_allowed_ns =  2,500 ns

Conceptually:

One expected 10 µs sampling interval:

|----------------------------------------|
0                                       10 µs

|----------|
   2.5 µs

^ target sample-processing time implied by the 25% policy

The relationship is approximately:

For example:

A more precise description of perf_sample_allowed_ns is therefore:

It is the threshold against which perf compares its smoothed estimate of sample-processing time when deciding whether the current maximum sample rate is too aggressive.

It is not a hard deadline applied independently to every sample.

5. max_samples_per_tick and HZ

The second quantity derived from the maximum sample rate is:

static int max_samples_per_tick __read_mostly =
    DIV_ROUND_UP(DEFAULT_MAX_SAMPLE_RATE, HZ);

HZ is a kernel build-time configuration that defines the kernel’s nominal tick frequency. Common configurations include 100, 250, and 1000 Hz.

For example, with:

HZ = 250

the nominal tick interval is:

The initial interrupt threshold is:

Therefore, at the defaults:

max sample rate          = 100,000 samples/s
HZ                       = 250 ticks/s
nominal tick interval    = 4 ms
max samples per tick     = 400
sample-rate time period  = 10 µs
CPU-time threshold       = 2.5 µs

max_samples_per_tick limits how many sampling interrupts a perf event may generate before the next perf task-tick accounting boundary. Under normal periodic ticking, this corresponds approximately to a budget of max_samples_per_tick interrupts every 1/HZ seconds.

6. What happens when an event exceeds max_samples_per_tick?

The relevant accounting logic is:

static int
__perf_event_account_interrupt(struct perf_event *event, int throttle)
{
    struct hw_perf_event *hwc = &event->hw;
    int ret = 0;
    u64 seq;

    seq = __this_cpu_read(perf_throttled_seq);
    if (seq != hwc->interrupts_seq) {
        hwc->interrupts_seq = seq;
        hwc->interrupts = 1;
    } else {
        hwc->interrupts++;
    }

    if (unlikely(throttle &&
                 hwc->interrupts >= max_samples_per_tick)) {
        __this_cpu_inc(perf_throttled_count);
        tick_dep_set_cpu(smp_processor_id(), TICK_DEP_BIT_PERF_EVENTS);
        perf_event_throttle_group(event);
        ret = 1;
    }

    ...
}

seq = __this_cpu_read(perf_throttled_seq); reads the current CPU’s perf throttling sequence number. This sequence is incremented by perf_event_task_tick() and therefore marks a new interrupt-accounting epoch.

In contrast, hwc->interrupts_seq records the most recent perf_throttled_seq value observed by this particular perf event. During one accounting epoch, the event may generate multiple sampling-overflow interrupts, which are counted in hwc->interrupts.

When a sampling interrupt occurs, perf compares the current per-CPU sequence number, seq, with the event’s saved sequence number, hwc->interrupts_seq. If they differ, a new accounting epoch has begun. Perf updates hwc->interrupts_seq to the current seq and sets hwc->interrupts = 1, because the interrupt currently being handled is the first interrupt for this event in the new epoch.

If the two sequence numbers are the same, the event is still within the same accounting epoch, so perf simply increments hwc->interrupts for each additional sampling-overflow interrupt.

Therefore, the interrupt count belongs to the individual perf event (event->hw.interrupts). When the per-CPU perf throttling sequence changes, the event begins counting interrupts again for the new accounting epoch.

If hwc->interrupts reaches max_samples_per_tick, perf may throttle the event or event group. When this happens, tick_dep_set_cpu(..., TICK_DEP_BIT_PERF_EVENTS) establishes a tick dependency on that CPU. This is especially important on tickless (NO_HZ) systems: after the sampling event is throttled, it may no longer generate PMIs on its own, so perf needs the CPU tick to remain available for subsequent throttling bookkeeping and unthrottling. In other words, perf requests that the periodic scheduler tick not be suppressed while a throttled perf event still depends on it.

The important point is that perf does not merely discard excess sample records after the threshold is reached.

The throttle path is:

static void perf_event_throttle(struct perf_event *event)
{
    if (event->state != PERF_EVENT_STATE_ACTIVE)
        return;

    event->hw.interrupts = MAX_INTERRUPTS;
    event->pmu->stop(event, 0);
    ...
}

and perf_event_throttle_group() applies this to the leader and its sibling events.

So, once the threshold is reached, perf stops the active event/group at the PMU. The event is later unthrottled and restarted.

This distinction matters:

Ring-buffer overflow:
PMU keeps counting -> sample is produced -> record cannot be stored

Perf interrupt throttling:
too many sampling interrupts -> PMU event/group is stopped -> no sample is produced

Therefore, severe undercounting caused by throttling does not require a PERF_RECORD_LOST record. The missing measurements may never have been generated because the event was stopped.

7. Distinguishing the user-defined PMU sample period

Suppose we configure:

attr.sample_period = 500;

for INST_RETIRED.ANY.

This means approximately:

subject to the usual hardware effects such as interrupt latency/skid.

This is fundamentally different from perf_sample_period_ns:

The user’s PMU sampling period determines how aggressively the hardware tries to generate samples. The kernel-side limits determine whether perf allows that rate to continue.

8. Two related kernel protections

The default values can be summarized as:

DEFAULT_MAX_SAMPLE_RATE = 100000
              |
              +-----------------------------+
              |                             |
              v                             v
         divide by HZ              divide 1 second by rate
              |                             |
              v                             v
 max_samples_per_tick          perf_sample_period_ns
   400 if HZ=250                    10 µs
                                            |
                                            | x 25%
                                            v
                                 perf_sample_allowed_ns
                                         2.5 µs

These support two related mechanisms.

Protection A: interrupt-count throttling

Question:

Is this sampling event generating too many interrupts during the current perf tick-accounting epoch?

The relevant threshold is:

max_samples_per_tick

Conceptually:

perf tick epoch N

PMI
PMI
PMI
...
PMI #400
    |
    v
perf_event_throttle_group()
    |
    v
PMU event/group stopped

next perf tick / scheduling opportunity
    |
    v
unthrottle/restart

Protection B: dynamic sample-cost control

Question:

Is perf spending too much CPU time processing samples?

The relevant comparison is between perf’s smoothed sample-processing time and:

perf_sample_allowed_ns

If processing becomes too expensive, Linux reduces max_samples_per_tick and kernel.perf_event_max_sample_rate.

These mechanisms are related: the time-based controller can lower the global sample-rate ceiling, and that lower ceiling produces a smaller per-tick interrupt threshold.

9. Changing kernel.perf_event_max_sample_rate

Suppose we run:

sudo sysctl -w kernel.perf_event_max_sample_rate=63250

The custom handler is invoked:

static int perf_event_max_sample_rate_handler(
        const struct ctl_table *table, int write,
        void *buffer, size_t *lenp, loff_t *ppos)
{
    int ret;
    int perf_cpu = sysctl_perf_cpu_time_max_percent;

    /* If throttling is disabled don't allow the write: */
    if (write && (perf_cpu == 100 || perf_cpu == 0))
        return -EINVAL;

    ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
    if (ret || !write)
        return ret;

    max_samples_per_tick =
        DIV_ROUND_UP(sysctl_perf_event_sample_rate, HZ);

    perf_sample_period_ns =
        NSEC_PER_SEC / sysctl_perf_event_sample_rate;

    update_perf_cpu_limits();

    return 0;
}

After the sysctl value has been parsed, the handler recomputes all quantities derived from the configured sample-rate ceiling.

9.1 New samples-per-tick threshold

For:

max sample rate = 63,250 samples/s
HZ              = 250

we obtain:

9.2 New time interval implied by the sample-rate ceiling

The handler also computes:

perf_sample_period_ns =
    NSEC_PER_SEC / sysctl_perf_event_sample_rate;

so:

Thus:

perf_sample_period_ns ≈ 15.81 µs

This answers the question:

If sampling were occurring at the configured maximum rate, what wall-clock interval would correspond to one sample?

10. Updating perf_sample_allowed_ns

The handler then calls:

static void update_perf_cpu_limits(void)
{
    u64 tmp = perf_sample_period_ns;

    tmp *= sysctl_perf_cpu_time_max_percent;
    tmp = div_u64(tmp, 100);
    if (!tmp)
        tmp = 1;

    WRITE_ONCE(perf_sample_allowed_ns, tmp);
}

This computes:

For 63,250 samples/s:

With:

perf_cpu_time_max_percent = 25

we obtain:

Therefore:

perf_sample_period_ns  ≈ 15.81 µs
perf_sample_allowed_ns ≈  3.95 µs

Conceptually:

At 63,250 samples/s:

sample                         sample
  |                              |
  v                              v
  +------------------------------+
              15.81 µs

  |-------|
   3.95 µs

  ^ sample-cost threshold implied by a 25% CPU target

Again, perf_sample_allowed_ns is best understood as a threshold used by the dynamic feedback controller, not as a hard deadline for one particular interrupt.

WRITE_ONCE() simply makes this shared kernel variable update explicit and prevents undesirable compiler transformations; it is not what performs the throttling.

11. How perf_cpu_time_max_percent enables dynamic throttling

The sysctl handler is:

static int perf_cpu_time_max_percent_handler(
        const struct ctl_table *table, int write,
        void *buffer, size_t *lenp, loff_t *ppos)
{
    int ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);

    if (ret || !write)
        return ret;

    if (sysctl_perf_cpu_time_max_percent == 100 ||
        sysctl_perf_cpu_time_max_percent == 0) {
        printk(KERN_WARNING
               "perf: Dynamic interrupt throttling disabled, can hang your system!\n");
        WRITE_ONCE(perf_sample_allowed_ns, 0);
    } else {
        update_perf_cpu_limits();
    }

    return 0;
}

In the source version shown above, setting the value to 0 or 100 sets:

perf_sample_allowed_ns = 0

and disables the duration-based feedback mechanism. This is important because the current kernel documentation describes the 100 case differently. When documenting or reproducing behavior, use the source corresponding to the exact kernel version being tested.

Disabling this dynamic CPU-cost controller does not mean that every other perf throttling mechanism disappears. The separate interrupt-count threshold (max_samples_per_tick) is a distinct part of the overflow path.

12. Where the kernel dynamically lowers the sampling rate

The actual feedback controller is perf_sample_event_took().

At a high level it does the following:

1. Maintain a per-CPU smoothed estimate of sample-processing duration.
2. Compare that estimate with perf_sample_allowed_ns.
3. If the average is too high, add a 25% safety margin.
4. Compute how many such samples fit within the allowed CPU-time fraction of one tick.
5. Lower max_samples_per_tick and kernel.perf_event_max_sample_rate accordingly.

The smoothing code is:

#define NR_ACCUMULATED_SAMPLES 128
static DEFINE_PER_CPU(u64, running_sample_length);

running_len = __this_cpu_read(running_sample_length);
running_len -= running_len / NR_ACCUMULATED_SAMPLES;
running_len += sample_len_ns;
__this_cpu_write(running_sample_length, running_len);

avg_len = running_len / NR_ACCUMULATED_SAMPLES;

This is approximately an exponentially weighted moving average:

where $x$ is the duration of the newest sample-handling operation.

The estimate is intentionally biased low during startup because the code divides by 128 before 128 observations have accumulated.

If:

avg_len <= perf_sample_allowed_ns

nothing changes.

Otherwise, Linux increases the measured average by 25%:

avg_len += avg_len / 4;

and computes a new samples-per-tick threshold:

max = (TICK_NSEC / 100) * sysctl_perf_cpu_time_max_percent;

if (avg_len < max)
    max /= (u32)avg_len;
else
    max = 1;

Conceptually:

The kernel then updates:

WRITE_ONCE(perf_sample_allowed_ns, avg_len);
WRITE_ONCE(max_samples_per_tick, max);

sysctl_perf_event_sample_rate = max * HZ;
perf_sample_period_ns =
    NSEC_PER_SEC / sysctl_perf_event_sample_rate;

For example, suppose:

HZ                       = 250
nominal tick             = 4 ms
perf_cpu_time_max_percent = 25%
measured average          = 5 µs/sample

After the 25% safety margin:

The CPU-time budget corresponding to 25% of a 4 ms tick is:

Therefore:

With HZ = 250, this corresponds to:

So the feedback loop can look like:

initial maximum rate
100,000 samples/s
        |
        v
samples are more expensive than expected
        |
        v
estimate safe samples/tick from measured cost
        |
        v
160 samples/tick
        |
        v
40,000 samples/s

This is the mechanism behind kernel messages such as:

perf: interrupt took too long (... > ...), lowering kernel.perf_event_max_sample_rate to ...

13. Final mental model

The easiest way to remember the relationship is:

User-selected PMU period
(attr.sample_period)
        |
        v
How frequently the hardware tries to overflow
        |
        v
PMU sampling interrupts
        |
        +-----------------------------+
        |                             |
        v                             v
interrupt count                 processing cost
        |                             |
        v                             v
max_samples_per_tick      perf_sample_allowed_ns
        |                             |
        +-------------+---------------+
                      |
                      v
              perf throttling policy
                      |
                      v
       stop/unthrottle PMU event/group

And the kernel parameters play different roles:

The most important practical distinction is that perf throttling is not the same as ring-buffer loss. When the interrupt threshold is exceeded, perf can stop the PMU event/group itself. In that case, the measurements are never generated, so userspace may see no PERF_RECORD_LOST indication even though the sampled count is far below the true workload count.