For testing mainline, the kvm.sh rcutorture script is the preferred approach to testing. This commit therefore adds it to the torture.txt documentation.
Unified · Split
| @@ -124,9 +124,14 @@ using a dynamically allocated srcu_struct (hence "srcud-" rather than · 123 unmodified lines | ||
| 124 | 124 | debugging. The final "T" entry contains the totals of the counters. |
| 125 | 125 | |
| 126 | 126 | |
| 127 | −USAGE | |
| 127 | +USAGE ON SPECIFIC KERNEL BUILDS | |
| 128 | 128 | |
| 129 | −The following script may be used to torture RCU: | |
| 129 | +It is sometimes desirable to torture RCU on a specific kernel build, | |
| 130 | +for example, when preparing to put that kernel build into production. | |
| 131 | +In that case, the kernel should be built with CONFIG_RCU_TORTURE_TEST=m | |
| 132 | +so that the test can be started using modprobe and terminated using rmmod. | |
| 133 | + | |
| 134 | +For example, the following script may be used to torture RCU: | |
| 130 | 135 | |
| 131 | 136 | #!/bin/sh |
| 132 | 137 | |
| @@ -142,8 +147,136 @@ checked for such errors. The "rmmod" command forces a "SUCCESS", · 9 unmodified lines | ||
| 142 | 147 | two are self-explanatory, while the last indicates that while there |
| 143 | 148 | were no RCU failures, CPU-hotplug problems were detected. |
| 144 | 149 | |
| 145 | −However, the tools/testing/selftests/rcutorture/bin/kvm.sh script | |
| 146 | −provides better automation, including automatic failure analysis. | |
| 147 | −It assumes a qemu/kvm-enabled platform, and runs guest OSes out of initrd. | |
| 148 | −See tools/testing/selftests/rcutorture/doc/initrd.txt for instructions | |
| 149 | −on setting up such an initrd. | |
| 150 | + | |
| 151 | +USAGE ON MAINLINE KERNELS | |
| 152 | + | |
| 153 | +When using rcutorture to test changes to RCU itself, it is often | |
| 154 | +necessary to build a number of kernels in order to test that change | |
| 155 | +across a broad range of combinations of the relevant Kconfig options | |
| 156 | +and of the relevant kernel boot parameters. In this situation, use | |
| 157 | +of modprobe and rmmod can be quite time-consuming and error-prone. | |
| 158 | + | |
| 159 | +Therefore, the tools/testing/selftests/rcutorture/bin/kvm.sh | |
| 160 | +script is available for mainline testing for x86, arm64, and | |
| 161 | +powerpc. By default, it will run the series of tests specified by | |
| 162 | +tools/testing/selftests/rcutorture/configs/rcu/CFLIST, with each test | |
| 163 | +running for 30 minutes within a guest OS using a minimal userspace | |
| 164 | +supplied by an automatically generated initrd. After the tests are | |
| 165 | +complete, the resulting build products and console output are analyzed | |
| 166 | +for errors and the results of the runs are summarized. | |
| 167 | + | |
| 168 | +On larger systems, rcutorture testing can be accelerated by passing the | |
| 169 | +--cpus argument to kvm.sh. For example, on a 64-CPU system, "--cpus 43" | |
| 170 | +would use up to 43 CPUs to run tests concurrently, which as of v5.4 would | |
| 171 | +complete all the scenarios in two batches, reducing the time to complete | |
| 172 | +from about eight hours to about one hour (not counting the time to build | |
| 173 | +the sixteen kernels). The "--dryrun sched" argument will not run tests, | |
| 174 | +but rather tell you how the tests would be scheduled into batches. This | |
| 175 | +can be useful when working out how many CPUs to specify in the --cpus | |
| 176 | +argument. | |
| 177 | + | |
| 178 | +Not all changes require that all scenarios be run. For example, a change | |
| 179 | +to Tree SRCU might run only the SRCU-N and SRCU-P scenarios using the | |
| 180 | +--configs argument to kvm.sh as follows: "--configs 'SRCU-N SRCU-P'". | |
| 181 | +Large systems can run multiple copies of of the full set of scenarios, | |
| 182 | +for example, a system with 448 hardware threads can run five instances | |
| 183 | +of the full set concurrently. To make this happen: | |
| 184 | + | |
| 185 | + kvm.sh --cpus 448 --configs '5*CFLIST' | |
| 186 | + | |
| 187 | +Alternatively, such a system can run 56 concurrent instances of a single | |
| 188 | +eight-CPU scenario: | |
| 189 | + | |
| 190 | + kvm.sh --cpus 448 --configs '56*TREE04' | |
| 191 | + | |
| 192 | +Or 28 concurrent instances of each of two eight-CPU scenarios: | |
| 193 | + | |
| 194 | + kvm.sh --cpus 448 --configs '28*TREE03 28*TREE04' | |
| 195 | + | |
| 196 | +Of course, each concurrent instance will use memory, which can be | |
| 197 | +limited using the --memory argument, which defaults to 512M. Small | |
| 198 | +values for memory may require disabling the callback-flooding tests | |
| 199 | +using the --bootargs parameter discussed below. | |
| 200 | + | |
| 201 | +Sometimes additional debugging is useful, and in such cases the --kconfig | |
| 202 | +parameter to kvm.sh may be used, for example, "--kconfig 'CONFIG_KASAN=y'". | |
| 203 | + | |
| 204 | +Kernel boot arguments can also be supplied, for example, to control | |
| 205 | +rcutorture's module parameters. For example, to test a change to RCU's | |
| 206 | +CPU stall-warning code, use "--bootargs 'rcutorture.stall_cpu=30'". | |
| 207 | +This will of course result in the scripting reporting a failure, namely | |
| 208 | +the resuling RCU CPU stall warning. As noted above, reducing memory may | |
| 209 | +require disabling rcutorture's callback-flooding tests: | |
| 210 | + | |
| 211 | + kvm.sh --cpus 448 --configs '56*TREE04' --memory 128M \ | |
| 212 | + --bootargs 'rcutorture.fwd_progress=0' | |
| 213 | + | |
| 214 | +Sometimes all that is needed is a full set of kernel builds. This is | |
| 215 | +what the --buildonly argument does. | |
| 216 | + | |
| 217 | +Finally, the --trust-make argument allows each kernel build to reuse what | |
| 218 | +it can from the previous kernel build. | |
| 219 | + | |
| 220 | +There are additional more arcane arguments that are documented in the | |
| 221 | +source code of the kvm.sh script. | |
| 222 | + | |
| 223 | +If a run contains failures, the number of buildtime and runtime failures | |
| 224 | +is listed at the end of the kvm.sh output, which you really should redirect | |
| 225 | +to a file. The build products and console output of each run is kept in | |
| 226 | +tools/testing/selftests/rcutorture/res in timestamped directories. A | |
| 227 | +given directory can be supplied to kvm-find-errors.sh in order to have | |
| 228 | +it cycle you through summaries of errors and full error logs. For example: | |
| 229 | + | |
| 230 | + tools/testing/selftests/rcutorture/bin/kvm-find-errors.sh \ | |
| 231 | + tools/testing/selftests/rcutorture/res/2020.01.20-15.54.23 | |
| 232 | + | |
| 233 | +However, it is often more convenient to access the files directly. | |
| 234 | +Files pertaining to all scenarios in a run reside in the top-level | |
| 235 | +directory (2020.01.20-15.54.23 in the example above), while per-scenario | |
| 236 | +files reside in a subdirectory named after the scenario (for example, | |
| 237 | +"TREE04"). If a given scenario ran more than once (as in "--configs | |
| 238 | +'56*TREE04'" above), the directories corresponding to the second and | |
| 239 | +subsequent runs of that scenario include a sequence number, for example, | |
| 240 | +"TREE04.2", "TREE04.3", and so on. | |
| 241 | + | |
| 242 | +The most frequently used file in the top-level directory is testid.txt. | |
| 243 | +If the test ran in a git repository, then this file contains the commit | |
| 244 | +that was tested and any uncommitted changes in diff format. | |
| 245 | + | |
| 246 | +The most frequently used files in each per-scenario-run directory are: | |
| 247 | + | |
| 248 | +.config: This file contains the Kconfig options. | |
| 249 | + | |
| 250 | +Make.out: This contains build output for a specific scenario. | |
| 251 | + | |
| 252 | +console.log: This contains the console output for a specific scenario. | |
| 253 | + This file may be examined once the kernel has booted, but | |
| 254 | + it might not exist if the build failed. | |
| 255 | + | |
| 256 | +vmlinux: This contains the kernel, which can be useful with tools like | |
| 257 | + objdump and gdb. | |
| 258 | + | |
| 259 | +A number of additional files are available, but are less frequently used. | |
| 260 | +Many are intended for debugging of rcutorture itself or of its scripting. | |
| 261 | + | |
| 262 | +As of v5.4, a successful run with the default set of scenarios produces | |
| 263 | +the following summary at the end of the run on a 12-CPU system: | |
| 264 | + | |
| 265 | +SRCU-N ------- 804233 GPs (148.932/s) [srcu: g10008272 f0x0 ] | |
| 266 | +SRCU-P ------- 202320 GPs (37.4667/s) [srcud: g1809476 f0x0 ] | |
| 267 | +SRCU-t ------- 1122086 GPs (207.794/s) [srcu: g0 f0x0 ] | |
| 268 | +SRCU-u ------- 1111285 GPs (205.794/s) [srcud: g1 f0x0 ] | |
| 269 | +TASKS01 ------- 19666 GPs (3.64185/s) [tasks: g0 f0x0 ] | |
| 270 | +TASKS02 ------- 20541 GPs (3.80389/s) [tasks: g0 f0x0 ] | |
| 271 | +TASKS03 ------- 19416 GPs (3.59556/s) [tasks: g0 f0x0 ] | |
| 272 | +TINY01 ------- 836134 GPs (154.84/s) [rcu: g0 f0x0 ] n_max_cbs: 34198 | |
| 273 | +TINY02 ------- 850371 GPs (157.476/s) [rcu: g0 f0x0 ] n_max_cbs: 2631 | |
| 274 | +TREE01 ------- 162625 GPs (30.1157/s) [rcu: g1124169 f0x0 ] | |
| 275 | +TREE02 ------- 333003 GPs (61.6672/s) [rcu: g2647753 f0x0 ] n_max_cbs: 35844 | |
| 276 | +TREE03 ------- 306623 GPs (56.782/s) [rcu: g2975325 f0x0 ] n_max_cbs: 1496497 | |
| 277 | +CPU count limited from 16 to 12 | |
| 278 | +TREE04 ------- 246149 GPs (45.5831/s) [rcu: g1695737 f0x0 ] n_max_cbs: 434961 | |
| 279 | +TREE05 ------- 314603 GPs (58.2598/s) [rcu: g2257741 f0x2 ] n_max_cbs: 193997 | |
| 280 | +TREE07 ------- 167347 GPs (30.9902/s) [rcu: g1079021 f0x0 ] n_max_cbs: 478732 | |
| 281 | +CPU count limited from 16 to 12 | |
| 282 | +TREE09 ------- 752238 GPs (139.303/s) [rcu: g13075057 f0x0 ] n_max_cbs: 99011 | |