History

Merge tag 'linux_kselftest-kunit-7.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest

Pull kunit updates from Shuah Khan: "Fixes and new kunit and tools, enable new configs: - configs: enable GPIO kunit test cases in all_tests.config - string-stream: Replace strlcat() with strscpy() and seq_buf - configs: enable GPIO kunit test cases in all_tests.config Documentation: - Test config entries shouldn't select other configs - Fix outdated FAQ entries Add the ability to skip entire test suites and an example test suite that can be skipped at runtime: - Add ability to skip entire test suites - Add example of test suite that can be skipped at runtime" * tag 'linux_kselftest-kunit-7.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest: kunit: tool: fix _list_tests filtering wrong variable when list has TAP prefix kunit: configs: enable GPIO kunit test cases in all_tests.config kunit: string-stream: Replace strlcat() with strscpy() and seq_buf Documentation: kunit: Fix outdated FAQ entries Documentation: kunit: Test Kconfig entries shouldn't select other configs kunit: Add example of test suite that can be skipped at runtime kunit,rust: Add ability to skip entire test suites

Linus Torvalds committed 2 months ago

Commit fd89b0b2 parents fc8c78b dea754d11 files · +117 −35Browse files

Unified · Split

11 files
  1. Documentation/dev-tools/kunit/faq.rst +24 −19
  2. Documentation/dev-tools/kunit/style.rst +4 −3
  3. include/kunit/test.h +1 −0
  4. lib/kunit/debugfs.c +21 −9
  5. lib/kunit/kunit-example-test.c +29 −0
  6. lib/kunit/string-stream.c +7 −2
  7. lib/kunit/test.c +16 −1
  8. rust/kernel/kunit.rs +1 −0
  9. tools/testing/kunit/configs/all_tests.config +1 −0
  10. tools/testing/kunit/kunit.py +1 −1
  11. tools/testing/kunit/kunit_tool_test.py +12 −0
Documentation/dev-tools/kunit/faq.rst+24−19
@@ -25,19 +25,21 @@ disqualifying any of them from being considered unit testing frameworks. · 24 unmodified lines
25Does KUnit support running on architectures other than UML?25Does KUnit support running on architectures other than UML?
26===========================================================26===========================================================
27 27
28Yes, mostly.28Yes. KUnit can run on any architecture, though the kunit.py tool can only
29build and run kernels for some architectures (of which UML is the default).
29 30
30For the most part, the KUnit core framework (what we use to write the tests)31You can build and run tests without kunit.py at all on any architecture by
31can compile to any architecture. It compiles like just another part of the32enabling ``CONFIG_KUNIT=y`` and booting the kernel.
32kernel and runs when the kernel boots, or when built as a module, when the33See Documentation/dev-tools/kunit/run_manual.rst for more details.
33module is loaded. However, there is infrastructure, like the KUnit Wrapper
34(``tools/testing/kunit/kunit.py``) that might not support some architectures
35(see :ref:`kunit-on-qemu`).
36 34
37In short, yes, you can run KUnit on other architectures, but it might require35Alternatively, kunit.py supports many common architectures using
38more work than using KUnit on UML.36cross-compilers and the qemu emulator. This can be done using the ``--arch``
37parameter when running the tests, and the ``--cross_compile`` parameter
38when building (if the architecture is not supported by the host compiler).
39See :ref:`kunit-on-qemu` for more details.
39 40
40For more information, see :ref:`kunit-on-non-uml`.41When writing tests targeting other architectures, it's worth keeping the tips
42on the :ref:`kunit-on-non-uml` page in mind.
41 43
42.. _kinds-of-tests:44.. _kinds-of-tests:
43 45
@@ -78,27 +80,30 @@ things to try. · 34 unmodified lines
78 down where an issue is occurring. (If you think the parser is at fault, you80 down where an issue is occurring. (If you think the parser is at fault, you
79 can run it manually against ``stdin`` or a file with ``kunit.py parse``.)81 can run it manually against ``stdin`` or a file with ``kunit.py parse``.)
803. Running the UML kernel directly can often reveal issues or error messages,823. Running the UML kernel directly can often reveal issues or error messages,
81 ``kunit_tool`` ignores. This should be as simple as running ``./vmlinux``83 ``kunit_tool`` ignores. This should be as simple as runningi the ``vmlinux``
82 after building the UML kernel (for example, by using ``kunit.py build``).84 binary in the output directory (by default ``./.kunit/vmlinux``) after
85 building the UML kernel (for example, by using ``kunit.py build``).
83 Note that UML has some unusual requirements (such as the host having a tmpfs86 Note that UML has some unusual requirements (such as the host having a tmpfs
84 filesystem mounted), and has had issues in the past when built statically and87 filesystem mounted), and has had issues in the past when built statically and
85 the host has KASLR enabled. (On older host kernels, you may need to run88 the host has KASLR enabled. (On older host kernels, you may need to run
86 ``setarch `uname -m` -R ./vmlinux`` to disable KASLR.)89 ``setarch `uname -m` -R ./vmlinux`` to disable KASLR.)
874. Make sure the kernel .config has ``CONFIG_KUNIT=y`` and at least one test904. Try running KUnit on a different architecture by using the ``--arch``
91 option. On an x86_64 host, using ``--arch=x86_64`` is a good first step.
925. Make sure the kernel .config has ``CONFIG_KUNIT=y`` and at least one test
88 (e.g. ``CONFIG_KUNIT_EXAMPLE_TEST=y``). kunit_tool will keep its .config93 (e.g. ``CONFIG_KUNIT_EXAMPLE_TEST=y``). kunit_tool will keep its .config
89 around, so you can see what config was used after running ``kunit.py run``.94 around, so you can see what config was used after running ``kunit.py run``.
90 It also preserves any config changes you might make, so you can95 It also preserves any config changes you might make, so you can
91 enable/disable things with ``make ARCH=um menuconfig`` or similar, and then96 enable/disable things with ``make ARCH=um menuconfig`` or similar, and then
92 re-run kunit_tool.97 re-run kunit_tool.
935. Try to run ``make ARCH=um defconfig`` before running ``kunit.py run``. This986. Try to run ``make ARCH=um defconfig`` before running ``kunit.py run``. This
94 may help clean up any residual config items which could be causing problems.99 may help clean up any residual config items which could be causing problems.
956. Finally, try running KUnit outside UML. KUnit and KUnit tests can be1007. Finally, try running KUnit manually, instead of via ``kunit.py``. KUnit can
96 built into any kernel, or can be built as a module and loaded at runtime.101 be built into any kernel, or can be built as a module and loaded at runtime.
97 Doing so should allow you to determine if UML is causing the issue you're102 When tests are built-in, they will execute when the kernel boots, and
98 seeing. When tests are built-in, they will execute when the kernel boots, and
99 modules will automatically execute associated tests when loaded. Test results103 modules will automatically execute associated tests when loaded. Test results
100 can be collected from ``/sys/kernel/debug/kunit/<test suite>/results``, and104 can be collected from ``/sys/kernel/debug/kunit/<test suite>/results``, and
101 can be parsed with ``kunit.py parse``. For more details, see :ref:`kunit-on-qemu`.105 can be parsed with ``kunit.py parse``. For more details, see
106 Documentation/dev-tools/kunit/run_manual.rst
102 107
103If none of the above tricks help, you are always welcome to email any issues to108If none of the above tricks help, you are always welcome to email any issues to
104kunit-dev@googlegroups.com.109kunit-dev@googlegroups.com.
Documentation/dev-tools/kunit/style.rst+4−3
@@ -164,9 +164,10 @@ This Kconfig entry must: · 163 unmodified lines
164* be visible only if ``CONFIG_KUNIT_ALL_TESTS`` is not enabled.164* be visible only if ``CONFIG_KUNIT_ALL_TESTS`` is not enabled.
165* have a default value of ``CONFIG_KUNIT_ALL_TESTS``.165* have a default value of ``CONFIG_KUNIT_ALL_TESTS``.
166* have a brief description of KUnit in the help text.166* have a brief description of KUnit in the help text.
167 167* depend on the feature being tested, rather than selecting it (so that
168If we are not able to meet above conditions (for example, the test is unable to168 enabling ``CONFIG_KUNIT_ALL_TESTS`` does not enable unrelated functionality).
169be built as a module), Kconfig entries for tests should be tristate.169* be ``tristate``, unless there is a specific reason that the test cannot be
170 built as a module.
170 171
171For example, a Kconfig entry might look like:172For example, a Kconfig entry might look like:
172 173
include/kunit/test.h+1−0
@@ -285,6 +285,7 @@ struct kunit_suite { · 284 unmodified lines
285 struct string_stream *log;285 struct string_stream *log;
286 int suite_init_err;286 int suite_init_err;
287 bool is_init;287 bool is_init;
288 enum kunit_status status;
288};289};
289 290
290/* Stores an array of suites, end points one past the end */291/* Stores an array of suites, end points one past the end */
lib/kunit/debugfs.c+21−9
@@ -76,18 +76,30 @@ static int debugfs_print_results(struct seq_file *seq, void *v) · 75 unmodified lines
76 seq_puts(seq, "KTAP version 1\n");76 seq_puts(seq, "KTAP version 1\n");
77 seq_puts(seq, "1..1\n");77 seq_puts(seq, "1..1\n");
78 78
79 /* Print suite header because it is not stored in the test logs. */79 if (suite->status != KUNIT_SKIPPED) {
80 seq_puts(seq, KUNIT_SUBTEST_INDENT "KTAP version 1\n");80 /* Print suite header because it is not stored in the test logs. */
81 seq_printf(seq, KUNIT_SUBTEST_INDENT "# Subtest: %s\n", suite->name);81 seq_puts(seq,
82 seq_printf(seq, KUNIT_SUBTEST_INDENT "1..%zd\n", kunit_suite_num_test_cases(suite));82 KUNIT_SUBTEST_INDENT "KTAP version 1\n");
83 83 seq_printf(seq,
84 kunit_suite_for_each_test_case(suite, test_case)84 KUNIT_SUBTEST_INDENT "# Subtest: %s\n",
85 debugfs_print_result(seq, test_case->log);85 suite->name);
86 seq_printf(seq,
87 KUNIT_SUBTEST_INDENT "1..%zd\n",
88 kunit_suite_num_test_cases(suite));
89
90 kunit_suite_for_each_test_case(suite, test_case)
91 debugfs_print_result(seq, test_case->log);
92 }
86 93
87 debugfs_print_result(seq, suite->log);94 debugfs_print_result(seq, suite->log);
88 95
89 seq_printf(seq, "%s %d %s\n",96 if (suite->status != KUNIT_SKIPPED)
90 kunit_status_to_ok_not_ok(success), 1, suite->name);97 seq_printf(seq, "%s %d %s\n",
98 kunit_status_to_ok_not_ok(success), 1, suite->name);
99 else
100 seq_printf(seq, "%s %d %s # SKIP %s\n",
101 kunit_status_to_ok_not_ok(success), 1, suite->name,
102 suite->status_comment);
91 return 0;103 return 0;
92}104}
93 105
lib/kunit/kunit-example-test.c+29−0
@@ -591,5 +591,34 @@ static struct kunit_suite example_init_test_suite = { · 590 unmodified lines
591 */591 */
592kunit_test_init_section_suites(&example_init_test_suite);592kunit_test_init_section_suites(&example_init_test_suite);
593 593
594/*
595 * This test should always be skipped.
596 */
597static void example_skip_suite_test(struct kunit *test)
598{
599 /* This line should never be seen */
600 KUNIT_FAIL(test, "You should not see a this.");
601}
602
603static struct kunit_case example_skip_suite_test_cases[] = {
604 KUNIT_CASE(example_skip_suite_test),
605 {}
606};
607
608static int example_skip_suite_init(struct kunit_suite *suite)
609{
610 kunit_mark_skipped(suite, "Test suite expected to be skipped");
611 return 0;
612}
613
614static struct kunit_suite example_test_skip_suite = {
615 .name = "example_skip_suite",
616 .suite_init = example_skip_suite_init,
617 .test_cases = example_skip_suite_test_cases,
618};
619
620/* This registers a test suite that will be skipped */
621kunit_test_suite(example_test_skip_suite);
622
594MODULE_DESCRIPTION("Example KUnit test suite");623MODULE_DESCRIPTION("Example KUnit test suite");
595MODULE_LICENSE("GPL v2");624MODULE_LICENSE("GPL v2");
lib/kunit/string-stream.c+7−2
@@ -9,6 +9,7 @@ · 8 unmodified lines
9#include <kunit/static_stub.h>9#include <kunit/static_stub.h>
10#include <kunit/test.h>10#include <kunit/test.h>
11#include <linux/list.h>11#include <linux/list.h>
12#include <linux/seq_buf.h>
12#include <linux/slab.h>13#include <linux/slab.h>
13 14
14#include "string-stream.h"15#include "string-stream.h"
@@ -74,7 +75,8 @@ int string_stream_vadd(struct string_stream *stream, · 59 unmodified lines
74 75
75 /* Append newline if necessary. */76 /* Append newline if necessary. */
76 if (frag_container->fragment[result_len - 1] != '\n')77 if (frag_container->fragment[result_len - 1] != '\n')
77 result_len = strlcat(frag_container->fragment, "\n", buf_len);78 result_len += strscpy(frag_container->fragment + result_len,
79 "\n", buf_len - result_len);
78 } else {80 } else {
79 result_len = vsnprintf(frag_container->fragment, buf_len, fmt, args);81 result_len = vsnprintf(frag_container->fragment, buf_len, fmt, args);
80 }82 }
@@ -118,15 +120,18 @@ char *string_stream_get_string(struct string_stream *stream) · 37 unmodified lines
118{120{
119 struct string_stream_fragment *frag_container;121 struct string_stream_fragment *frag_container;
120 size_t buf_len = stream->length + 1; /* +1 for null byte. */122 size_t buf_len = stream->length + 1; /* +1 for null byte. */
123 struct seq_buf sb;
121 char *buf;124 char *buf;
122 125
123 buf = kzalloc(buf_len, stream->gfp);126 buf = kzalloc(buf_len, stream->gfp);
124 if (!buf)127 if (!buf)
125 return NULL;128 return NULL;
126 129
130 seq_buf_init(&sb, buf, buf_len);
131
127 spin_lock(&stream->lock);132 spin_lock(&stream->lock);
128 list_for_each_entry(frag_container, &stream->fragments, node)133 list_for_each_entry(frag_container, &stream->fragments, node)
129 strlcat(buf, frag_container->fragment, buf_len);134 seq_buf_puts(&sb, frag_container->fragment);
130 spin_unlock(&stream->lock);135 spin_unlock(&stream->lock);
131 136
132 return buf;137 return buf;
lib/kunit/test.c+16−1
@@ -214,12 +214,18 @@ enum kunit_status kunit_suite_has_succeeded(struct kunit_suite *suite) · 213 unmodified lines
214 const struct kunit_case *test_case;214 const struct kunit_case *test_case;
215 enum kunit_status status = KUNIT_SKIPPED;215 enum kunit_status status = KUNIT_SKIPPED;
216 216
217 if (suite->status == KUNIT_SKIPPED)
218 return KUNIT_SKIPPED;
219
217 if (suite->suite_init_err)220 if (suite->suite_init_err)
218 return KUNIT_FAILURE;221 return KUNIT_FAILURE;
219 222
220 kunit_suite_for_each_test_case(suite, test_case) {223 kunit_suite_for_each_test_case(suite, test_case) {
221 if (test_case->status == KUNIT_FAILURE)224 if (test_case->status == KUNIT_FAILURE) {
225 /* Update the kunit_suite status also */
226 suite->status = KUNIT_FAILURE;
222 return KUNIT_FAILURE;227 return KUNIT_FAILURE;
228 }
223 else if (test_case->status == KUNIT_SUCCESS)229 else if (test_case->status == KUNIT_SUCCESS)
224 status = KUNIT_SUCCESS;230 status = KUNIT_SUCCESS;
225 }231 }
@@ -795,12 +801,20 @@ int kunit_run_tests(struct kunit_suite *suite) · 569 unmodified lines
795 /* Taint the kernel so we know we've run tests. */801 /* Taint the kernel so we know we've run tests. */
796 add_taint(TAINT_TEST, LOCKDEP_STILL_OK);802 add_taint(TAINT_TEST, LOCKDEP_STILL_OK);
797 803
804 if (suite->status == KUNIT_SKIPPED)
805 goto suite_end;
806
798 if (suite->suite_init) {807 if (suite->suite_init) {
799 suite->suite_init_err = suite->suite_init(suite);808 suite->suite_init_err = suite->suite_init(suite);
800 if (suite->suite_init_err) {809 if (suite->suite_init_err) {
810 suite->status = KUNIT_FAILURE;
801 kunit_err(suite, KUNIT_SUBTEST_INDENT811 kunit_err(suite, KUNIT_SUBTEST_INDENT
802 "# failed to initialize (%d)", suite->suite_init_err);812 "# failed to initialize (%d)", suite->suite_init_err);
803 goto suite_end;813 goto suite_end;
814
815 } else if (suite->status == KUNIT_SKIPPED) {
816 /* Skip this kunit suite */
817 goto suite_end;
804 }818 }
805 }819 }
806 820
@@ -825,6 +839,7 @@ static void kunit_init_suite(struct kunit_suite *suite) · 18 unmodified lines
825 kunit_debugfs_create_suite(suite);839 kunit_debugfs_create_suite(suite);
826 suite->status_comment[0] = '\0';840 suite->status_comment[0] = '\0';
827 suite->suite_init_err = 0;841 suite->suite_init_err = 0;
842 suite->status = KUNIT_SUCCESS;
828 843
829 if (suite->log)844 if (suite->log)
830 string_stream_clear(suite->log);845 string_stream_clear(suite->log);
rust/kernel/kunit.rs+1−0
@@ -288,6 +288,7 @@ macro_rules! kunit_unsafe_test_suite { · 287 unmodified lines
288 log: ::core::ptr::null_mut(),288 log: ::core::ptr::null_mut(),
289 suite_init_err: 0,289 suite_init_err: 0,
290 is_init: false,290 is_init: false,
291 status: kernel::bindings::kunit_status_KUNIT_SUCCESS,
291 };292 };
292 293
293 #[used(compiler)]294 #[used(compiler)]
tools/testing/kunit/configs/all_tests.config+1−0
@@ -21,6 +21,7 @@ CONFIG_VFAT_FS=y · 20 unmodified lines
21CONFIG_PCI=y21CONFIG_PCI=y
22CONFIG_USB4=y22CONFIG_USB4=y
23CONFIG_I2C=y23CONFIG_I2C=y
24CONFIG_GPIOLIB=y
24 25
25CONFIG_NET=y26CONFIG_NET=y
26CONFIG_MCTP=y27CONFIG_MCTP=y
tools/testing/kunit/kunit.py+1−1
@@ -126,7 +126,7 @@ def _list_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) · 125 unmodified lines
126 lines.pop()126 lines.pop()
127 127
128 # Filter out any extraneous non-test output that might have gotten mixed in.128 # Filter out any extraneous non-test output that might have gotten mixed in.
129 return [l for l in output if re.match(r'^[^\s.]+\.[^\s.]+$', l)]129 return [l for l in lines if re.match(r'^[^\s.]+\.[^\s.]+$', l)]
130 130
131def _list_tests_attr(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) -> Iterable[str]:131def _list_tests_attr(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) -> Iterable[str]:
132 args = ['kunit.action=list_attr']132 args = ['kunit.action=list_attr']
tools/testing/kunit/kunit_tool_test.py+12−0
@@ -979,6 +979,18 @@ class KUnitMainTest(unittest.TestCase): · 978 unmodified lines
979 self.linux_source_mock.run_kernel.assert_called_once_with(979 self.linux_source_mock.run_kernel.assert_called_once_with(
980 args=['kunit.action=list'], build_dir='.kunit', filter_glob='suite*', filter='', filter_action=None, timeout=300)980 args=['kunit.action=list'], build_dir='.kunit', filter_glob='suite*', filter='', filter_action=None, timeout=300)
981 981
982 def test_list_tests_with_prefix(self):
983 want = ['suite.test1', 'suite.test2', 'suite2.test1']
984 self.linux_source_mock.run_kernel.return_value = [
985 '[ 0.100000] TAP version 14',
986 '[ 0.200000] suite.test1',
987 '[ 0.200000] suite.test2',
988 '[ 0.300000] suite2.test1']
989
990 got = kunit._list_tests(self.linux_source_mock,
991 kunit.KunitExecRequest(None, None, None, False, False, '.kunit', 300, 'suite*', '', None, None, 'suite', False, False, False))
992 self.assertEqual(got, want)
993
982 @mock.patch.object(kunit, '_list_tests')994 @mock.patch.object(kunit, '_list_tests')
983 def test_run_isolated_by_suite(self, mock_tests):995 def test_run_isolated_by_suite(self, mock_tests):
984 mock_tests.return_value = ['suite.test1', 'suite.test2', 'suite2.test1']996 mock_tests.return_value = ['suite.test1', 'suite.test2', 'suite2.test1']