History

mm/page_alloc: delete vm.percpu_pagelist_fraction

Patch series "Calculate pcp->high based on zone sizes and active CPUs", v2. The per-cpu page allocator (PCP) is meant to reduce contention on the zone lock but the sizing of batch and high is archaic and neither takes the zone size into account or the number of CPUs local to a zone. With larger zones and more CPUs per node, the contention is getting worse. Furthermore, the fact that vm.percpu_pagelist_fraction adjusts both batch and high values means that the sysctl can reduce zone lock contention but also increase allocation latencies. This series disassociates pcp->high from pcp->batch and then scales pcp->high based on the size of the local zone with limited impact to reclaim and accounting for active CPUs but leaves pcp->batch static. It also adapts the number of pages that can be on the pcp list based on recent freeing patterns. The motivation is partially to adjust to larger memory sizes but is also driven by the fact that large batches of page freeing via release_pages() often shows zone contention as a major part of the problem. Another is a bug report based on an older kernel where a multi-terabyte process can takes several minutes to exit. A workaround was to use vm.percpu_pagelist_fraction to increase the pcp->high value but testing indicated that a production workload could not use the same values because of an increase in allocation latencies. Unfortunately, I cannot reproduce this test case myself as the multi-terabyte machines are in active use but it should alleviate the problem. The series aims to address both and partially acts as a pre-requisite. pcp only works with order-0 which is useless for SLUB (when using high orders) and THP (unconditionally). To store high-order pages on PCP, the pcp->high values need to be increased first. This patch (of 6): The vm.percpu_pagelist_fraction is used to increase the batch and high limits for the per-cpu page allocator (PCP). The intent behind the sysctl is to reduce zone lock acquisition when allocating/freeing pages but it has a problem. While it can decrease contention, it can also increase latency on the allocation side due to unreasonably large batch sizes. This leads to games where an administrator adjusts percpu_pagelist_fraction on the fly to work around contention and allocation latency problems. This series aims to alleviate the problems with zone lock contention while avoiding the allocation-side latency problems. For the purposes of review, it's easier to remove this sysctl now and reintroduce a similar sysctl later in the series that deals only with pcp->high.

Mel Gorman committed 5 years ago

Commit bbbecb31 parent 151e0844 files · +4 −81Browse files

Unified · Split

4 files
  1. Documentation/admin-guide/sysctl/vm.rst +0 −19
  2. include/linux/mmzone.h +0 −3
  3. kernel/sysctl.c +0 −8
  4. mm/page_alloc.c +4 −51
Documentation/admin-guide/sysctl/vm.rst+0−19
@@ -64,7 +64,6 @@ Currently, these files are in /proc/sys/vm: · 63 unmodified lines
64- overcommit_ratio64- overcommit_ratio
65- page-cluster65- page-cluster
66- panic_on_oom66- panic_on_oom
67- percpu_pagelist_fraction
68- stat_interval67- stat_interval
69- stat_refresh68- stat_refresh
70- numa_stat69- numa_stat
@@ -790,24 +789,6 @@ panic_on_oom=2+kdump gives you very strong tool to investigate · 719 unmodified lines
790why oom happens. You can get snapshot.789why oom happens. You can get snapshot.
791 790
792 791
793percpu_pagelist_fraction
794========================
795
796This is the fraction of pages at most (high mark pcp->high) in each zone that
797are allocated for each per cpu page list. The min value for this is 8. It
798means that we don't allow more than 1/8th of pages in each zone to be
799allocated in any single per_cpu_pagelist. This entry only changes the value
800of hot per cpu pagelists. User can specify a number like 100 to allocate
8011/100th of each zone to each per cpu page list.
802
803The batch value of each per cpu pagelist is also updated as a result. It is
804set to pcp->high/4. The upper limit of batch is (PAGE_SHIFT * 8)
805
806The initial value is zero. Kernel does not use this value at boot time to set
807the high water marks for each per cpu page list. If the user writes '0' to this
808sysctl, it will revert to this default behavior.
809
810
811stat_interval792stat_interval
812=============793=============
813 794
include/linux/mmzone.h+0−3
@@ -1027,15 +1027,12 @@ int watermark_scale_factor_sysctl_handler(struct ctl_table *, int, void *, · 1026 unmodified lines
1027extern int sysctl_lowmem_reserve_ratio[MAX_NR_ZONES];1027extern int sysctl_lowmem_reserve_ratio[MAX_NR_ZONES];
1028int lowmem_reserve_ratio_sysctl_handler(struct ctl_table *, int, void *,1028int lowmem_reserve_ratio_sysctl_handler(struct ctl_table *, int, void *,
1029 size_t *, loff_t *);1029 size_t *, loff_t *);
1030int percpu_pagelist_fraction_sysctl_handler(struct ctl_table *, int,
1031 void *, size_t *, loff_t *);
1032int sysctl_min_unmapped_ratio_sysctl_handler(struct ctl_table *, int,1030int sysctl_min_unmapped_ratio_sysctl_handler(struct ctl_table *, int,
1033 void *, size_t *, loff_t *);1031 void *, size_t *, loff_t *);
1034int sysctl_min_slab_ratio_sysctl_handler(struct ctl_table *, int,1032int sysctl_min_slab_ratio_sysctl_handler(struct ctl_table *, int,
1035 void *, size_t *, loff_t *);1033 void *, size_t *, loff_t *);
1036int numa_zonelist_order_handler(struct ctl_table *, int,1034int numa_zonelist_order_handler(struct ctl_table *, int,
1037 void *, size_t *, loff_t *);1035 void *, size_t *, loff_t *);
1038extern int percpu_pagelist_fraction;
1039extern char numa_zonelist_order[];1036extern char numa_zonelist_order[];
1040#define NUMA_ZONELIST_ORDER_LEN 161037#define NUMA_ZONELIST_ORDER_LEN 16
1041 1038
kernel/sysctl.c+0−8
@@ -2908,14 +2908,6 @@ static struct ctl_table vm_table[] = { · 2907 unmodified lines
2908 .extra1 = SYSCTL_ONE,2908 .extra1 = SYSCTL_ONE,
2909 .extra2 = &one_thousand,2909 .extra2 = &one_thousand,
2910 },2910 },
2911 {
2912 .procname = "percpu_pagelist_fraction",
2913 .data = &percpu_pagelist_fraction,
2914 .maxlen = sizeof(percpu_pagelist_fraction),
2915 .mode = 0644,
2916 .proc_handler = percpu_pagelist_fraction_sysctl_handler,
2917 .extra1 = SYSCTL_ZERO,
2918 },
2919 {2911 {
2920 .procname = "page_lock_unfairness",2912 .procname = "page_lock_unfairness",
2921 .data = &sysctl_page_lock_unfairness,2913 .data = &sysctl_page_lock_unfairness,
mm/page_alloc.c+4−51
@@ -120,7 +120,6 @@ typedef int __bitwise fpi_t; · 119 unmodified lines
120 120
121/* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */121/* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */
122static DEFINE_MUTEX(pcp_batch_high_lock);122static DEFINE_MUTEX(pcp_batch_high_lock);
123#define MIN_PERCPU_PAGELIST_FRACTION (8)
124 123
125struct pagesets {124struct pagesets {
126 local_lock_t lock;125 local_lock_t lock;
@@ -193,7 +192,6 @@ EXPORT_SYMBOL(_totalram_pages); · 66 unmodified lines
193unsigned long totalreserve_pages __read_mostly;192unsigned long totalreserve_pages __read_mostly;
194unsigned long totalcma_pages __read_mostly;193unsigned long totalcma_pages __read_mostly;
195 194
196int percpu_pagelist_fraction;
197gfp_t gfp_allowed_mask __read_mostly = GFP_BOOT_MASK;195gfp_t gfp_allowed_mask __read_mostly = GFP_BOOT_MASK;
198DEFINE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_ALLOC_DEFAULT_ON, init_on_alloc);196DEFINE_STATIC_KEY_MAYBE(CONFIG_INIT_ON_ALLOC_DEFAULT_ON, init_on_alloc);
199EXPORT_SYMBOL(init_on_alloc);197EXPORT_SYMBOL(init_on_alloc);
@@ -6735,22 +6733,15 @@ static void __zone_set_pageset_high_and_batch(struct zone *zone, unsigned long h · 6535 unmodified lines
6735 6733
6736/*6734/*
6737 * Calculate and set new high and batch values for all per-cpu pagesets of a6735 * Calculate and set new high and batch values for all per-cpu pagesets of a
6738 * zone, based on the zone's size and the percpu_pagelist_fraction sysctl.6736 * zone based on the zone's size.
6739 */6737 */
6740static void zone_set_pageset_high_and_batch(struct zone *zone)6738static void zone_set_pageset_high_and_batch(struct zone *zone)
6741{6739{
6742 unsigned long new_high, new_batch;6740 unsigned long new_high, new_batch;
6743 6741
6744 if (percpu_pagelist_fraction) {6742 new_batch = zone_batchsize(zone);
6745 new_high = zone_managed_pages(zone) / percpu_pagelist_fraction;6743 new_high = 6 * new_batch;
6746 new_batch = max(1UL, new_high / 4);6744 new_batch = max(1UL, 1 * new_batch);
6747 if ((new_high / 4) > (PAGE_SHIFT * 8))
6748 new_batch = PAGE_SHIFT * 8;
6749 } else {
6750 new_batch = zone_batchsize(zone);
6751 new_high = 6 * new_batch;
6752 new_batch = max(1UL, 1 * new_batch);
6753 }
6754 6745
6755 if (zone->pageset_high == new_high &&6746 if (zone->pageset_high == new_high &&
6756 zone->pageset_batch == new_batch)6747 zone->pageset_batch == new_batch)
@@ -8413,44 +8404,6 @@ int lowmem_reserve_ratio_sysctl_handler(struct ctl_table *table, int write, · 1656 unmodified lines
8413 return 0;8404 return 0;
8414}8405}
8415 8406
8416/*
8417 * percpu_pagelist_fraction - changes the pcp->high for each zone on each
8418 * cpu. It is the fraction of total pages in each zone that a hot per cpu
8419 * pagelist can have before it gets flushed back to buddy allocator.
8420 */
8421int percpu_pagelist_fraction_sysctl_handler(struct ctl_table *table, int write,
8422 void *buffer, size_t *length, loff_t *ppos)
8423{
8424 struct zone *zone;
8425 int old_percpu_pagelist_fraction;
8426 int ret;
8427
8428 mutex_lock(&pcp_batch_high_lock);
8429 old_percpu_pagelist_fraction = percpu_pagelist_fraction;
8430
8431 ret = proc_dointvec_minmax(table, write, buffer, length, ppos);
8432 if (!write || ret < 0)
8433 goto out;
8434
8435 /* Sanity checking to avoid pcp imbalance */
8436 if (percpu_pagelist_fraction &&
8437 percpu_pagelist_fraction < MIN_PERCPU_PAGELIST_FRACTION) {
8438 percpu_pagelist_fraction = old_percpu_pagelist_fraction;
8439 ret = -EINVAL;
8440 goto out;
8441 }
8442
8443 /* No change? */
8444 if (percpu_pagelist_fraction == old_percpu_pagelist_fraction)
8445 goto out;
8446
8447 for_each_populated_zone(zone)
8448 zone_set_pageset_high_and_batch(zone);
8449out:
8450 mutex_unlock(&pcp_batch_high_lock);
8451 return ret;
8452}
8453
8454#ifndef __HAVE_ARCH_RESERVED_KERNEL_PAGES8407#ifndef __HAVE_ARCH_RESERVED_KERNEL_PAGES
8455/*8408/*
8456 * Returns the number of pages that arch has reserved but8409 * Returns the number of pages that arch has reserved but