For testing mainline, the kvm.sh rcutorture script is the preferred approach to testing. This commit therefore adds it to the torture.txt documentation.
Unified · Split
| @@ -124,9 +124,14 @@ using a dynamically allocated srcu_struct (hence "srcud-" rather than · 123 unmodified lines | |||
| 124 | debugging. The final "T" entry contains the totals of the counters. | 124 | debugging. The final "T" entry contains the totals of the counters. |
| 125 | 125 | ||
| 126 | 126 | ||
| 127 | USAGE | 127 | USAGE ON SPECIFIC KERNEL BUILDS |
| 128 | 128 | ||
| 129 | The following script may be used to torture RCU: | 129 | It is sometimes desirable to torture RCU on a specific kernel build, |
| 130 | for example, when preparing to put that kernel build into production. | ||
| 131 | In that case, the kernel should be built with CONFIG_RCU_TORTURE_TEST=m | ||
| 132 | so that the test can be started using modprobe and terminated using rmmod. | ||
| 133 | |||
| 134 | For example, the following script may be used to torture RCU: | ||
| 130 | 135 | ||
| 131 | #!/bin/sh | 136 | #!/bin/sh |
| 132 | 137 | ||
| @@ -142,8 +147,136 @@ checked for such errors. The "rmmod" command forces a "SUCCESS", · 9 unmodified lines | |||
| 142 | two are self-explanatory, while the last indicates that while there | 147 | two are self-explanatory, while the last indicates that while there |
| 143 | were no RCU failures, CPU-hotplug problems were detected. | 148 | were no RCU failures, CPU-hotplug problems were detected. |
| 144 | 149 | ||
| 145 | However, the tools/testing/selftests/rcutorture/bin/kvm.sh script | 150 | |
| 146 | provides better automation, including automatic failure analysis. | 151 | USAGE ON MAINLINE KERNELS |
| 147 | It assumes a qemu/kvm-enabled platform, and runs guest OSes out of initrd. | 152 | |
| 148 | See tools/testing/selftests/rcutorture/doc/initrd.txt for instructions | 153 | When using rcutorture to test changes to RCU itself, it is often |
| 149 | on setting up such an initrd. | 154 | necessary to build a number of kernels in order to test that change |
| 155 | across a broad range of combinations of the relevant Kconfig options | ||
| 156 | and of the relevant kernel boot parameters. In this situation, use | ||
| 157 | of modprobe and rmmod can be quite time-consuming and error-prone. | ||
| 158 | |||
| 159 | Therefore, the tools/testing/selftests/rcutorture/bin/kvm.sh | ||
| 160 | script is available for mainline testing for x86, arm64, and | ||
| 161 | powerpc. By default, it will run the series of tests specified by | ||
| 162 | tools/testing/selftests/rcutorture/configs/rcu/CFLIST, with each test | ||
| 163 | running for 30 minutes within a guest OS using a minimal userspace | ||
| 164 | supplied by an automatically generated initrd. After the tests are | ||
| 165 | complete, the resulting build products and console output are analyzed | ||
| 166 | for errors and the results of the runs are summarized. | ||
| 167 | |||
| 168 | On larger systems, rcutorture testing can be accelerated by passing the | ||
| 169 | --cpus argument to kvm.sh. For example, on a 64-CPU system, "--cpus 43" | ||
| 170 | would use up to 43 CPUs to run tests concurrently, which as of v5.4 would | ||
| 171 | complete all the scenarios in two batches, reducing the time to complete | ||
| 172 | from about eight hours to about one hour (not counting the time to build | ||
| 173 | the sixteen kernels). The "--dryrun sched" argument will not run tests, | ||
| 174 | but rather tell you how the tests would be scheduled into batches. This | ||
| 175 | can be useful when working out how many CPUs to specify in the --cpus | ||
| 176 | argument. | ||
| 177 | |||
| 178 | Not all changes require that all scenarios be run. For example, a change | ||
| 179 | to Tree SRCU might run only the SRCU-N and SRCU-P scenarios using the | ||
| 180 | --configs argument to kvm.sh as follows: "--configs 'SRCU-N SRCU-P'". | ||
| 181 | Large systems can run multiple copies of of the full set of scenarios, | ||
| 182 | for example, a system with 448 hardware threads can run five instances | ||
| 183 | of the full set concurrently. To make this happen: | ||
| 184 | |||
| 185 | kvm.sh --cpus 448 --configs '5*CFLIST' | ||
| 186 | |||
| 187 | Alternatively, such a system can run 56 concurrent instances of a single | ||
| 188 | eight-CPU scenario: | ||
| 189 | |||
| 190 | kvm.sh --cpus 448 --configs '56*TREE04' | ||
| 191 | |||
| 192 | Or 28 concurrent instances of each of two eight-CPU scenarios: | ||
| 193 | |||
| 194 | kvm.sh --cpus 448 --configs '28*TREE03 28*TREE04' | ||
| 195 | |||
| 196 | Of course, each concurrent instance will use memory, which can be | ||
| 197 | limited using the --memory argument, which defaults to 512M. Small | ||
| 198 | values for memory may require disabling the callback-flooding tests | ||
| 199 | using the --bootargs parameter discussed below. | ||
| 200 | |||
| 201 | Sometimes additional debugging is useful, and in such cases the --kconfig | ||
| 202 | parameter to kvm.sh may be used, for example, "--kconfig 'CONFIG_KASAN=y'". | ||
| 203 | |||
| 204 | Kernel boot arguments can also be supplied, for example, to control | ||
| 205 | rcutorture's module parameters. For example, to test a change to RCU's | ||
| 206 | CPU stall-warning code, use "--bootargs 'rcutorture.stall_cpu=30'". | ||
| 207 | This will of course result in the scripting reporting a failure, namely | ||
| 208 | the resuling RCU CPU stall warning. As noted above, reducing memory may | ||
| 209 | require disabling rcutorture's callback-flooding tests: | ||
| 210 | |||
| 211 | kvm.sh --cpus 448 --configs '56*TREE04' --memory 128M \ | ||
| 212 | --bootargs 'rcutorture.fwd_progress=0' | ||
| 213 | |||
| 214 | Sometimes all that is needed is a full set of kernel builds. This is | ||
| 215 | what the --buildonly argument does. | ||
| 216 | |||
| 217 | Finally, the --trust-make argument allows each kernel build to reuse what | ||
| 218 | it can from the previous kernel build. | ||
| 219 | |||
| 220 | There are additional more arcane arguments that are documented in the | ||
| 221 | source code of the kvm.sh script. | ||
| 222 | |||
| 223 | If a run contains failures, the number of buildtime and runtime failures | ||
| 224 | is listed at the end of the kvm.sh output, which you really should redirect | ||
| 225 | to a file. The build products and console output of each run is kept in | ||
| 226 | tools/testing/selftests/rcutorture/res in timestamped directories. A | ||
| 227 | given directory can be supplied to kvm-find-errors.sh in order to have | ||
| 228 | it cycle you through summaries of errors and full error logs. For example: | ||
| 229 | |||
| 230 | tools/testing/selftests/rcutorture/bin/kvm-find-errors.sh \ | ||
| 231 | tools/testing/selftests/rcutorture/res/2020.01.20-15.54.23 | ||
| 232 | |||
| 233 | However, it is often more convenient to access the files directly. | ||
| 234 | Files pertaining to all scenarios in a run reside in the top-level | ||
| 235 | directory (2020.01.20-15.54.23 in the example above), while per-scenario | ||
| 236 | files reside in a subdirectory named after the scenario (for example, | ||
| 237 | "TREE04"). If a given scenario ran more than once (as in "--configs | ||
| 238 | '56*TREE04'" above), the directories corresponding to the second and | ||
| 239 | subsequent runs of that scenario include a sequence number, for example, | ||
| 240 | "TREE04.2", "TREE04.3", and so on. | ||
| 241 | |||
| 242 | The most frequently used file in the top-level directory is testid.txt. | ||
| 243 | If the test ran in a git repository, then this file contains the commit | ||
| 244 | that was tested and any uncommitted changes in diff format. | ||
| 245 | |||
| 246 | The most frequently used files in each per-scenario-run directory are: | ||
| 247 | |||
| 248 | .config: This file contains the Kconfig options. | ||
| 249 | |||
| 250 | Make.out: This contains build output for a specific scenario. | ||
| 251 | |||
| 252 | console.log: This contains the console output for a specific scenario. | ||
| 253 | This file may be examined once the kernel has booted, but | ||
| 254 | it might not exist if the build failed. | ||
| 255 | |||
| 256 | vmlinux: This contains the kernel, which can be useful with tools like | ||
| 257 | objdump and gdb. | ||
| 258 | |||
| 259 | A number of additional files are available, but are less frequently used. | ||
| 260 | Many are intended for debugging of rcutorture itself or of its scripting. | ||
| 261 | |||
| 262 | As of v5.4, a successful run with the default set of scenarios produces | ||
| 263 | the following summary at the end of the run on a 12-CPU system: | ||
| 264 | |||
| 265 | SRCU-N ------- 804233 GPs (148.932/s) [srcu: g10008272 f0x0 ] | ||
| 266 | SRCU-P ------- 202320 GPs (37.4667/s) [srcud: g1809476 f0x0 ] | ||
| 267 | SRCU-t ------- 1122086 GPs (207.794/s) [srcu: g0 f0x0 ] | ||
| 268 | SRCU-u ------- 1111285 GPs (205.794/s) [srcud: g1 f0x0 ] | ||
| 269 | TASKS01 ------- 19666 GPs (3.64185/s) [tasks: g0 f0x0 ] | ||
| 270 | TASKS02 ------- 20541 GPs (3.80389/s) [tasks: g0 f0x0 ] | ||
| 271 | TASKS03 ------- 19416 GPs (3.59556/s) [tasks: g0 f0x0 ] | ||
| 272 | TINY01 ------- 836134 GPs (154.84/s) [rcu: g0 f0x0 ] n_max_cbs: 34198 | ||
| 273 | TINY02 ------- 850371 GPs (157.476/s) [rcu: g0 f0x0 ] n_max_cbs: 2631 | ||
| 274 | TREE01 ------- 162625 GPs (30.1157/s) [rcu: g1124169 f0x0 ] | ||
| 275 | TREE02 ------- 333003 GPs (61.6672/s) [rcu: g2647753 f0x0 ] n_max_cbs: 35844 | ||
| 276 | TREE03 ------- 306623 GPs (56.782/s) [rcu: g2975325 f0x0 ] n_max_cbs: 1496497 | ||
| 277 | CPU count limited from 16 to 12 | ||
| 278 | TREE04 ------- 246149 GPs (45.5831/s) [rcu: g1695737 f0x0 ] n_max_cbs: 434961 | ||
| 279 | TREE05 ------- 314603 GPs (58.2598/s) [rcu: g2257741 f0x2 ] n_max_cbs: 193997 | ||
| 280 | TREE07 ------- 167347 GPs (30.9902/s) [rcu: g1079021 f0x0 ] n_max_cbs: 478732 | ||
| 281 | CPU count limited from 16 to 12 | ||
| 282 | TREE09 ------- 752238 GPs (139.303/s) [rcu: g13075057 f0x0 ] n_max_cbs: 99011 | ||