History

mm/mempolicy: introduce MPOL_WEIGHTED_INTERLEAVE for weighted interleaving

When a system has multiple NUMA nodes and it becomes bandwidth hungry, using the current MPOL_INTERLEAVE could be an wise option. However, if those NUMA nodes consist of different types of memory such as socket-attached DRAM and CXL/PCIe attached DRAM, the round-robin based interleave policy does not optimally distribute data to make use of their different bandwidth characteristics. Instead, interleave is more effective when the allocation policy follows each NUMA nodes' bandwidth weight rather than a simple 1:1 distribution. This patch introduces a new memory policy, MPOL_WEIGHTED_INTERLEAVE, enabling weighted interleave between NUMA nodes. Weighted interleave allows for proportional distribution of memory across multiple numa nodes, preferably apportioned to match the bandwidth of each node. For example, if a system has 1 CPU node (0), and 2 memory nodes (0,1), with bandwidth of (100GB/s, 50GB/s) respectively, the appropriate weight distribution is (2:1). Weights for each node can be assigned via the new sysfs extension: /sys/kernel/mm/mempolicy/weighted_interleave/ For now, the default value of all nodes will be `1`, which matches the behavior of standard 1:1 round-robin interleave. An extension will be added in the future to allow default values to be registered at kernel and device bringup time. The policy allocates a number of pages equal to the set weights. For example, if the weights are (2,1), then 2 pages will be allocated on node0 for every 1 page allocated on node1. The new flag MPOL_WEIGHTED_INTERLEAVE can be used in set_mempolicy(2) and mbind(2). Some high level notes about the pieces of weighted interleave: current->il_prev: Tracks the node previously allocated from. current->il_weight: The active weight of the current node (current->il_prev) When this reaches 0, current->il_prev is set to the next node and current->il_weight is set to the next weight. weighted_interleave_nodes: Counts the number of allocations as they occur, and applies the weight for the current node. When the weight reaches 0, switch to the next node. Operates only on task->mempolicy. weighted_interleave_nid: Gets the total weight of the nodemask as well as each individual node weight, then calculates the node based on the given index. Operates on VMA policies. bulk_array_weighted_interleave: Gets the total weight of the nodemask as well as each individual node weight, then calculates the number of "interleave rounds" as well as any delta ("partial round"). Calculates the number of pages for each node and allocates them. If a node was scheduled for interleave via interleave_nodes, the current weight will be allocated first. Operates only on the task->mempolicy. One piece of complexity is the interaction between a recent refactor which split the logic to acquire the "ilx" (interleave index) of an allocation and the actually application of the interleave. If a call to alloc_pages_mpol() were made with a weighted-interleave policy and ilx set to NO_INTERLEAVE_INDEX, weighted_interleave_nodes() would operate on a VMA policy - violating the description above. An inspection of all callers of alloc_pages_mpol() shows that all external callers set ilx to `0`, an index value, or will call get_vma_policy() to acquire the ilx. For example, mm/shmem.c may call into alloc_pages_mpol. The call stacks all set (pgoff_t ilx) or end up in `get_vma_policy()`. This enforces the `weighted_interleave_nodes()` and `weighted_interleave_nid()` policy requirements (task/vma respectively).

Gregory Price committed 2 years ago

Commit fa3bea41 parent 9685e6e4 files · +225 −4Browse files

Unified · Split

4 files
  1. Documentation/admin-guide/mm/numa_memory_policy.rst +9 −0
  2. include/linux/sched.h +1 −0
  3. include/uapi/linux/mempolicy.h +1 −0
  4. mm/mempolicy.c +214 −4
Documentation/admin-guide/mm/numa_memory_policy.rst+9−0
@@ -250,6 +250,15 @@ MPOL_PREFERRED_MANY · 249 unmodified lines
250250 can fall back to all existing numa nodes. This is effectively
251251 MPOL_PREFERRED allowed for a mask rather than a single node.
252252
253+MPOL_WEIGHTED_INTERLEAVE
254+ This mode operates the same as MPOL_INTERLEAVE, except that
255+ interleaving behavior is executed based on weights set in
256+ /sys/kernel/mm/mempolicy/weighted_interleave/
257+
258+ Weighted interleave allocates pages on nodes according to a
259+ weight. For example if nodes [0,1] are weighted [5,2], 5 pages
260+ will be allocated on node0 for every 2 pages allocated on node1.
261+
253262 NUMA memory policy supports the following optional mode flags:
254263
255264 MPOL_F_STATIC_NODES
include/linux/sched.h+1−0
@@ -1259,6 +1259,7 @@ struct task_struct { · 1258 unmodified lines
12591259 /* Protected by alloc_lock: */
12601260 struct mempolicy *mempolicy;
12611261 short il_prev;
1262+ u8 il_weight;
12621263 short pref_node_fork;
12631264 #endif
12641265 #ifdef CONFIG_NUMA_BALANCING
include/uapi/linux/mempolicy.h+1−0
@@ -23,6 +23,7 @@ enum { · 22 unmodified lines
2323 MPOL_INTERLEAVE,
2424 MPOL_LOCAL,
2525 MPOL_PREFERRED_MANY,
26+ MPOL_WEIGHTED_INTERLEAVE,
2627 MPOL_MAX, /* always last member of enum */
2728 };
2829
mm/mempolicy.c+214−4
@@ -19,6 +19,13 @@ · 18 unmodified lines
1919 * for anonymous memory. For process policy an process counter
2020 * is used.
2121 *
22+ * weighted interleave
23+ * Allocate memory interleaved over a set of nodes based on
24+ * a set of weights (per-node), with normal fallback if it
25+ * fails. Otherwise operates the same as interleave.
26+ * Example: nodeset(0,1) & weights (2,1) - 2 pages allocated
27+ * on node 0 for every 1 page allocated on node 1.
28+ *
2229 * bind Only allocate memory on a specific set of nodes,
2330 * no fallback.
2431 * FIXME: memory is allocated starting with the first node
@@ -441,6 +448,10 @@ static const struct mempolicy_operations mpol_ops[MPOL_MAX] = { · 416 unmodified lines
441448 .create = mpol_new_nodemask,
442449 .rebind = mpol_rebind_preferred,
443450 },
451+ [MPOL_WEIGHTED_INTERLEAVE] = {
452+ .create = mpol_new_nodemask,
453+ .rebind = mpol_rebind_nodemask,
454+ },
444455 };
445456
446457 static bool migrate_folio_add(struct folio *folio, struct list_head *foliolist,
@@ -858,8 +869,11 @@ static long do_set_mempolicy(unsigned short mode, unsigned short flags, · 411 unmodified lines
858869
859870 old = current->mempolicy;
860871 current->mempolicy = new;
861− if (new && new->mode == MPOL_INTERLEAVE)
872+ if (new && (new->mode == MPOL_INTERLEAVE ||
873+ new->mode == MPOL_WEIGHTED_INTERLEAVE)) {
862874 current->il_prev = MAX_NUMNODES-1;
875+ current->il_weight = 0;
876+ }
863877 task_unlock(current);
864878 mpol_put(old);
865879 ret = 0;
@@ -884,6 +898,7 @@ static void get_policy_nodemask(struct mempolicy *pol, nodemask_t *nodes) · 18 unmodified lines
884898 case MPOL_INTERLEAVE:
885899 case MPOL_PREFERRED:
886900 case MPOL_PREFERRED_MANY:
901+ case MPOL_WEIGHTED_INTERLEAVE:
887902 *nodes = pol->nodes;
888903 break;
889904 case MPOL_LOCAL:
@@ -968,6 +983,13 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask, · 78 unmodified lines
968983 } else if (pol == current->mempolicy &&
969984 pol->mode == MPOL_INTERLEAVE) {
970985 *policy = next_node_in(current->il_prev, pol->nodes);
986+ } else if (pol == current->mempolicy &&
987+ pol->mode == MPOL_WEIGHTED_INTERLEAVE) {
988+ if (current->il_weight)
989+ *policy = current->il_prev;
990+ else
991+ *policy = next_node_in(current->il_prev,
992+ pol->nodes);
971993 } else {
972994 err = -EINVAL;
973995 goto out;
@@ -1332,7 +1354,8 @@ static long do_mbind(unsigned long start, unsigned long len, · 358 unmodified lines
13321354 * VMAs, the nodes will still be interleaved from the targeted
13331355 * nodemask, but one by one may be selected differently.
13341356 */
1335− if (new->mode == MPOL_INTERLEAVE) {
1357+ if (new->mode == MPOL_INTERLEAVE ||
1358+ new->mode == MPOL_WEIGHTED_INTERLEAVE) {
13361359 struct page *page;
13371360 unsigned int order;
13381361 unsigned long addr = -EFAULT;
@@ -1780,7 +1803,8 @@ struct mempolicy *__get_vma_policy(struct vm_area_struct *vma, · 441 unmodified lines
17801803 * @vma: virtual memory area whose policy is sought
17811804 * @addr: address in @vma for shared policy lookup
17821805 * @order: 0, or appropriate huge_page_order for interleaving
1783− * @ilx: interleave index (output), for use only when MPOL_INTERLEAVE
1806+ * @ilx: interleave index (output), for use only when MPOL_INTERLEAVE or
1807+ * MPOL_WEIGHTED_INTERLEAVE
17841808 *
17851809 * Returns effective policy for a VMA at specified address.
17861810 * Falls back to current->mempolicy or system default policy, as necessary.
@@ -1797,7 +1821,8 @@ struct mempolicy *get_vma_policy(struct vm_area_struct *vma, · 10 unmodified lines
17971821 pol = __get_vma_policy(vma, addr, ilx);
17981822 if (!pol)
17991823 pol = get_task_policy(current);
1800− if (pol->mode == MPOL_INTERLEAVE) {
1824+ if (pol->mode == MPOL_INTERLEAVE ||
1825+ pol->mode == MPOL_WEIGHTED_INTERLEAVE) {
18011826 *ilx += vma->vm_pgoff >> order;
18021827 *ilx += (addr - vma->vm_start) >> (PAGE_SHIFT + order);
18031828 }
@@ -1847,6 +1872,22 @@ bool apply_policy_zone(struct mempolicy *policy, enum zone_type zone) · 43 unmodified lines
18471872 return zone >= dynamic_policy_zone;
18481873 }
18491874
1875+static unsigned int weighted_interleave_nodes(struct mempolicy *policy)
1876+{
1877+ unsigned int node = current->il_prev;
1878+
1879+ if (!current->il_weight || !node_isset(node, policy->nodes)) {
1880+ node = next_node_in(node, policy->nodes);
1881+ /* can only happen if nodemask is being rebound */
1882+ if (node == MAX_NUMNODES)
1883+ return node;
1884+ current->il_prev = node;
1885+ current->il_weight = get_il_weight(node);
1886+ }
1887+ current->il_weight--;
1888+ return node;
1889+}
1890+
18501891 /* Do dynamic interleaving for a process */
18511892 static unsigned int interleave_nodes(struct mempolicy *policy)
18521893 {
@@ -1881,6 +1922,9 @@ unsigned int mempolicy_slab_node(void) · 28 unmodified lines
18811922 case MPOL_INTERLEAVE:
18821923 return interleave_nodes(policy);
18831924
1925+ case MPOL_WEIGHTED_INTERLEAVE:
1926+ return weighted_interleave_nodes(policy);
1927+
18841928 case MPOL_BIND:
18851929 case MPOL_PREFERRED_MANY:
18861930 {
@@ -1919,6 +1963,45 @@ static unsigned int read_once_policy_nodemask(struct mempolicy *pol, · 32 unmodified lines
19191963 return nodes_weight(*mask);
19201964 }
19211965
1966+static unsigned int weighted_interleave_nid(struct mempolicy *pol, pgoff_t ilx)
1967+{
1968+ nodemask_t nodemask;
1969+ unsigned int target, nr_nodes;
1970+ u8 *table;
1971+ unsigned int weight_total = 0;
1972+ u8 weight;
1973+ int nid;
1974+
1975+ nr_nodes = read_once_policy_nodemask(pol, &nodemask);
1976+ if (!nr_nodes)
1977+ return numa_node_id();
1978+
1979+ rcu_read_lock();
1980+ table = rcu_dereference(iw_table);
1981+ /* calculate the total weight */
1982+ for_each_node_mask(nid, nodemask) {
1983+ /* detect system default usage */
1984+ weight = table ? table[nid] : 1;
1985+ weight = weight ? weight : 1;
1986+ weight_total += weight;
1987+ }
1988+
1989+ /* Calculate the node offset based on totals */
1990+ target = ilx % weight_total;
1991+ nid = first_node(nodemask);
1992+ while (target) {
1993+ /* detect system default usage */
1994+ weight = table ? table[nid] : 1;
1995+ weight = weight ? weight : 1;
1996+ if (target < weight)
1997+ break;
1998+ target -= weight;
1999+ nid = next_node_in(nid, nodemask);
2000+ }
2001+ rcu_read_unlock();
2002+ return nid;
2003+}
2004+
19222005 /*
19232006 * Do static interleaving for interleave index @ilx. Returns the ilx'th
19242007 * node in pol->nodes (starting from ilx=0), wrapping around if ilx
@@ -1979,6 +2062,11 @@ static nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *pol, · 54 unmodified lines
19792062 *nid = (ilx == NO_INTERLEAVE_INDEX) ?
19802063 interleave_nodes(pol) : interleave_nid(pol, ilx);
19812064 break;
2065+ case MPOL_WEIGHTED_INTERLEAVE:
2066+ *nid = (ilx == NO_INTERLEAVE_INDEX) ?
2067+ weighted_interleave_nodes(pol) :
2068+ weighted_interleave_nid(pol, ilx);
2069+ break;
19822070 }
19832071
19842072 return nodemask;
@@ -2040,6 +2128,7 @@ bool init_nodemask_of_mempolicy(nodemask_t *mask) · 55 unmodified lines
20402128 case MPOL_PREFERRED_MANY:
20412129 case MPOL_BIND:
20422130 case MPOL_INTERLEAVE:
2131+ case MPOL_WEIGHTED_INTERLEAVE:
20432132 *mask = mempolicy->nodes;
20442133 break;
20452134
@@ -2140,6 +2229,7 @@ struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, · 94 unmodified lines
21402229 * node in its nodemask, we allocate the standard way.
21412230 */
21422231 if (pol->mode != MPOL_INTERLEAVE &&
2232+ pol->mode != MPOL_WEIGHTED_INTERLEAVE &&
21432233 (!nodemask || node_isset(nid, *nodemask))) {
21442234 /*
21452235 * First, try to allocate THP only on local node, but
@@ -2275,6 +2365,114 @@ static unsigned long alloc_pages_bulk_array_interleave(gfp_t gfp, · 129 unmodified lines
22752365 return total_allocated;
22762366 }
22772367
2368+static unsigned long alloc_pages_bulk_array_weighted_interleave(gfp_t gfp,
2369+ struct mempolicy *pol, unsigned long nr_pages,
2370+ struct page **page_array)
2371+{
2372+ struct task_struct *me = current;
2373+ unsigned long total_allocated = 0;
2374+ unsigned long nr_allocated = 0;
2375+ unsigned long rounds;
2376+ unsigned long node_pages, delta;
2377+ u8 *table, *weights, weight;
2378+ unsigned int weight_total = 0;
2379+ unsigned long rem_pages = nr_pages;
2380+ nodemask_t nodes;
2381+ int nnodes, node;
2382+ int resume_node = MAX_NUMNODES - 1;
2383+ u8 resume_weight = 0;
2384+ int prev_node;
2385+ int i;
2386+
2387+ if (!nr_pages)
2388+ return 0;
2389+
2390+ nnodes = read_once_policy_nodemask(pol, &nodes);
2391+ if (!nnodes)
2392+ return 0;
2393+
2394+ /* Continue allocating from most recent node and adjust the nr_pages */
2395+ node = me->il_prev;
2396+ weight = me->il_weight;
2397+ if (weight && node_isset(node, nodes)) {
2398+ node_pages = min(rem_pages, weight);
2399+ nr_allocated = __alloc_pages_bulk(gfp, node, NULL, node_pages,
2400+ NULL, page_array);
2401+ page_array += nr_allocated;
2402+ total_allocated += nr_allocated;
2403+ /* if that's all the pages, no need to interleave */
2404+ if (rem_pages <= weight) {
2405+ me->il_weight -= rem_pages;
2406+ return total_allocated;
2407+ }
2408+ /* Otherwise we adjust remaining pages, continue from there */
2409+ rem_pages -= weight;
2410+ }
2411+ /* clear active weight in case of an allocation failure */
2412+ me->il_weight = 0;
2413+ prev_node = node;
2414+
2415+ /* create a local copy of node weights to operate on outside rcu */
2416+ weights = kzalloc(nr_node_ids, GFP_KERNEL);
2417+ if (!weights)
2418+ return total_allocated;
2419+
2420+ rcu_read_lock();
2421+ table = rcu_dereference(iw_table);
2422+ if (table)
2423+ memcpy(weights, table, nr_node_ids);
2424+ rcu_read_unlock();
2425+
2426+ /* calculate total, detect system default usage */
2427+ for_each_node_mask(node, nodes) {
2428+ if (!weights[node])
2429+ weights[node] = 1;
2430+ weight_total += weights[node];
2431+ }
2432+
2433+ /*
2434+ * Calculate rounds/partial rounds to minimize __alloc_pages_bulk calls.
2435+ * Track which node weighted interleave should resume from.
2436+ *
2437+ * if (rounds > 0) and (delta == 0), resume_node will always be
2438+ * the node following prev_node and its weight.
2439+ */
2440+ rounds = rem_pages / weight_total;
2441+ delta = rem_pages % weight_total;
2442+ resume_node = next_node_in(prev_node, nodes);
2443+ resume_weight = weights[resume_node];
2444+ for (i = 0; i < nnodes; i++) {
2445+ node = next_node_in(prev_node, nodes);
2446+ weight = weights[node];
2447+ node_pages = weight * rounds;
2448+ /* If a delta exists, add this node's portion of the delta */
2449+ if (delta > weight) {
2450+ node_pages += weight;
2451+ delta -= weight;
2452+ } else if (delta) {
2453+ /* when delta is depleted, resume from that node */
2454+ node_pages += delta;
2455+ resume_node = node;
2456+ resume_weight = weight - delta;
2457+ delta = 0;
2458+ }
2459+ /* node_pages can be 0 if an allocation fails and rounds == 0 */
2460+ if (!node_pages)
2461+ break;
2462+ nr_allocated = __alloc_pages_bulk(gfp, node, NULL, node_pages,
2463+ NULL, page_array);
2464+ page_array += nr_allocated;
2465+ total_allocated += nr_allocated;
2466+ if (total_allocated == nr_pages)
2467+ break;
2468+ prev_node = node;
2469+ }
2470+ me->il_prev = resume_node;
2471+ me->il_weight = resume_weight;
2472+ kfree(weights);
2473+ return total_allocated;
2474+}
2475+
22782476 static unsigned long alloc_pages_bulk_array_preferred_many(gfp_t gfp, int nid,
22792477 struct mempolicy *pol, unsigned long nr_pages,
22802478 struct page **page_array)
@@ -2315,6 +2513,10 @@ unsigned long alloc_pages_bulk_array_mempolicy(gfp_t gfp, · 34 unmodified lines
23152513 return alloc_pages_bulk_array_interleave(gfp, pol,
23162514 nr_pages, page_array);
23172515
2516+ if (pol->mode == MPOL_WEIGHTED_INTERLEAVE)
2517+ return alloc_pages_bulk_array_weighted_interleave(
2518+ gfp, pol, nr_pages, page_array);
2519+
23182520 if (pol->mode == MPOL_PREFERRED_MANY)
23192521 return alloc_pages_bulk_array_preferred_many(gfp,
23202522 numa_node_id(), pol, nr_pages, page_array);
@@ -2390,6 +2592,7 @@ bool __mpol_equal(struct mempolicy *a, struct mempolicy *b) · 69 unmodified lines
23902592 case MPOL_INTERLEAVE:
23912593 case MPOL_PREFERRED:
23922594 case MPOL_PREFERRED_MANY:
2595+ case MPOL_WEIGHTED_INTERLEAVE:
23932596 return !!nodes_equal(a->nodes, b->nodes);
23942597 case MPOL_LOCAL:
23952598 return true;
@@ -2526,6 +2729,10 @@ int mpol_misplaced(struct folio *folio, struct vm_area_struct *vma, · 130 unmodified lines
25262729 polnid = interleave_nid(pol, ilx);
25272730 break;
25282731
2732+ case MPOL_WEIGHTED_INTERLEAVE:
2733+ polnid = weighted_interleave_nid(pol, ilx);
2734+ break;
2735+
25292736 case MPOL_PREFERRED:
25302737 if (node_isset(curnid, pol->nodes))
25312738 goto out;
@@ -2900,6 +3107,7 @@ static const char * const policy_modes[] = · 368 unmodified lines
29003107 [MPOL_PREFERRED] = "prefer",
29013108 [MPOL_BIND] = "bind",
29023109 [MPOL_INTERLEAVE] = "interleave",
3110+ [MPOL_WEIGHTED_INTERLEAVE] = "weighted interleave",
29033111 [MPOL_LOCAL] = "local",
29043112 [MPOL_PREFERRED_MANY] = "prefer (many)",
29053113 };
@@ -2959,6 +3167,7 @@ int mpol_parse_str(char *str, struct mempolicy **mpol) · 53 unmodified lines
29593167 }
29603168 break;
29613169 case MPOL_INTERLEAVE:
3170+ case MPOL_WEIGHTED_INTERLEAVE:
29623171 /*
29633172 * Default to online nodes with memory if no nodelist
29643173 */
@@ -3069,6 +3278,7 @@ void mpol_to_str(char *buffer, int maxlen, struct mempolicy *pol) · 104 unmodified lines
30693278 case MPOL_PREFERRED_MANY:
30703279 case MPOL_BIND:
30713280 case MPOL_INTERLEAVE:
3281+ case MPOL_WEIGHTED_INTERLEAVE:
30723282 nodes = pol->nodes;
30733283 break;
30743284 default:

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t