Merge tag 'linux_kselftest-kunit-7.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest

Pull kunit updates from Shuah Khan: "Fixes and new kunit and tools, enable new configs: - configs: enable GPIO kunit test cases in all_tests.config - string-stream: Replace strlcat() with strscpy() and seq_buf - configs: enable GPIO kunit test cases in all_tests.config Documentation: - Test config entries shouldn't select other configs - Fix outdated FAQ entries Add the ability to skip entire test suites and an example test suite that can be skipped at runtime: - Add ability to skip entire test suites - Add example of test suite that can be skipped at runtime" * tag 'linux_kselftest-kunit-7.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest: kunit: tool: fix _list_tests filtering wrong variable when list has TAP prefix kunit: configs: enable GPIO kunit test cases in all_tests.config kunit: string-stream: Replace strlcat() with strscpy() and seq_buf Documentation: kunit: Fix outdated FAQ entries Documentation: kunit: Test Kconfig entries shouldn't select other configs kunit: Add example of test suite that can be skipped at runtime kunit,rust: Add ability to skip entire test suites

Linus Torvalds committed 2 months ago

Commit fd89b0b2 parents fc8c78b dea754d11 files · +117 −35Browse files

Unified · Split

11 files

Documentation/dev-tools/kunit/faq.rst +24 −19
Documentation/dev-tools/kunit/style.rst +4 −3
include/kunit/test.h +1 −0
lib/kunit/debugfs.c +21 −9
lib/kunit/kunit-example-test.c +29 −0
lib/kunit/string-stream.c +7 −2
lib/kunit/test.c +16 −1
rust/kernel/kunit.rs +1 −0
tools/testing/kunit/configs/all_tests.config +1 −0
tools/testing/kunit/kunit.py +1 −1
tools/testing/kunit/kunit_tool_test.py +12 −0

Documentation/dev-tools/kunit/faq.rst+24−19

@@ -25,19 +25,21 @@ disqualifying any of them from being considered unit testing frameworks. · 24 unmodified lines
25	25	Does KUnit support running on architectures other than UML?
26	26	===========================================================
27	27
28		−Yes, mostly.
	28	+Yes. KUnit can run on any architecture, though the kunit.py tool can only
	29	+build and run kernels for some architectures (of which UML is the default).
29	30
30		−For the most part, the KUnit core framework (what we use to write the tests)
31		−can compile to any architecture. It compiles like just another part of the
32		−kernel and runs when the kernel boots, or when built as a module, when the
33		−module is loaded. However, there is infrastructure, like the KUnit Wrapper
34		−(``tools/testing/kunit/kunit.py``) that might not support some architectures
35		−(see :ref:`kunit-on-qemu`).
	31	+You can build and run tests without kunit.py at all on any architecture by
	32	+enabling ``CONFIG_KUNIT=y`` and booting the kernel.
	33	+See Documentation/dev-tools/kunit/run_manual.rst for more details.
36	34
37		−In short, yes, you can run KUnit on other architectures, but it might require
38		−more work than using KUnit on UML.
	35	+Alternatively, kunit.py supports many common architectures using
	36	+cross-compilers and the qemu emulator. This can be done using the ``--arch``
	37	+parameter when running the tests, and the ``--cross_compile`` parameter
	38	+when building (if the architecture is not supported by the host compiler).
	39	+See :ref:`kunit-on-qemu` for more details.
39	40
40		−For more information, see :ref:`kunit-on-non-uml`.
	41	+When writing tests targeting other architectures, it's worth keeping the tips
	42	+on the :ref:`kunit-on-non-uml` page in mind.
41	43
42	44	.. _kinds-of-tests:
43	45
@@ -78,27 +80,30 @@ things to try. · 34 unmodified lines
78	80	down where an issue is occurring. (If you think the parser is at fault, you
79	81	can run it manually against ``stdin`` or a file with ``kunit.py parse``.)
80	82	3. Running the UML kernel directly can often reveal issues or error messages,
81		− ``kunit_tool`` ignores. This should be as simple as running ``./vmlinux``
82		− after building the UML kernel (for example, by using ``kunit.py build``).
	83	+ ``kunit_tool`` ignores. This should be as simple as runningi the ``vmlinux``
	84	+ binary in the output directory (by default ``./.kunit/vmlinux``) after
	85	+ building the UML kernel (for example, by using ``kunit.py build``).
83	86	Note that UML has some unusual requirements (such as the host having a tmpfs
84	87	filesystem mounted), and has had issues in the past when built statically and
85	88	the host has KASLR enabled. (On older host kernels, you may need to run
86	89	``setarch `uname -m` -R ./vmlinux`` to disable KASLR.)
87		−4. Make sure the kernel .config has ``CONFIG_KUNIT=y`` and at least one test
	90	+4. Try running KUnit on a different architecture by using the ``--arch``
	91	+ option. On an x86_64 host, using ``--arch=x86_64`` is a good first step.
	92	+5. Make sure the kernel .config has ``CONFIG_KUNIT=y`` and at least one test
88	93	(e.g. ``CONFIG_KUNIT_EXAMPLE_TEST=y``). kunit_tool will keep its .config
89	94	around, so you can see what config was used after running ``kunit.py run``.
90	95	It also preserves any config changes you might make, so you can
91	96	enable/disable things with ``make ARCH=um menuconfig`` or similar, and then
92	97	re-run kunit_tool.
93		−5. Try to run ``make ARCH=um defconfig`` before running ``kunit.py run``. This
	98	+6. Try to run ``make ARCH=um defconfig`` before running ``kunit.py run``. This
94	99	may help clean up any residual config items which could be causing problems.
95		−6. Finally, try running KUnit outside UML. KUnit and KUnit tests can be
96		− built into any kernel, or can be built as a module and loaded at runtime.
97		− Doing so should allow you to determine if UML is causing the issue you're
98		− seeing. When tests are built-in, they will execute when the kernel boots, and
	100	+7. Finally, try running KUnit manually, instead of via ``kunit.py``. KUnit can
	101	+ be built into any kernel, or can be built as a module and loaded at runtime.
	102	+ When tests are built-in, they will execute when the kernel boots, and
99	103	modules will automatically execute associated tests when loaded. Test results
100	104	can be collected from ``/sys/kernel/debug/kunit/<test suite>/results``, and
101		− can be parsed with ``kunit.py parse``. For more details, see :ref:`kunit-on-qemu`.
	105	+ can be parsed with ``kunit.py parse``. For more details, see
	106	+ Documentation/dev-tools/kunit/run_manual.rst
102	107
103	108	If none of the above tricks help, you are always welcome to email any issues to
104	109	kunit-dev@googlegroups.com.

Documentation/dev-tools/kunit/style.rst+4−3

@@ -164,9 +164,10 @@ This Kconfig entry must: · 163 unmodified lines
164	164	* be visible only if ``CONFIG_KUNIT_ALL_TESTS`` is not enabled.
165	165	* have a default value of ``CONFIG_KUNIT_ALL_TESTS``.
166	166	* have a brief description of KUnit in the help text.
167		−
168		−If we are not able to meet above conditions (for example, the test is unable to
169		−be built as a module), Kconfig entries for tests should be tristate.
	167	+* depend on the feature being tested, rather than selecting it (so that
	168	+ enabling ``CONFIG_KUNIT_ALL_TESTS`` does not enable unrelated functionality).
	169	+* be ``tristate``, unless there is a specific reason that the test cannot be
	170	+ built as a module.
170	171
171	172	For example, a Kconfig entry might look like:
172	173

include/kunit/test.h+1−0

@@ -285,6 +285,7 @@ struct kunit_suite { · 284 unmodified lines
285	285	struct string_stream *log;
286	286	int suite_init_err;
287	287	bool is_init;
	288	+ enum kunit_status status;
288	289	};
289	290
290	291	/* Stores an array of suites, end points one past the end */

lib/kunit/debugfs.c+21−9

@@ -76,18 +76,30 @@ static int debugfs_print_results(struct seq_file seq, void v) · 75 unmodified lines
76	76	seq_puts(seq, "KTAP version 1\n");
77	77	seq_puts(seq, "1..1\n");
78	78
79		− /* Print suite header because it is not stored in the test logs. */
80		− seq_puts(seq, KUNIT_SUBTEST_INDENT "KTAP version 1\n");
81		− seq_printf(seq, KUNIT_SUBTEST_INDENT "# Subtest: %s\n", suite->name);
82		− seq_printf(seq, KUNIT_SUBTEST_INDENT "1..%zd\n", kunit_suite_num_test_cases(suite));
83		−
84		− kunit_suite_for_each_test_case(suite, test_case)
85		− debugfs_print_result(seq, test_case->log);
	79	+ if (suite->status != KUNIT_SKIPPED) {
	80	+ /* Print suite header because it is not stored in the test logs. */
	81	+ seq_puts(seq,
	82	+ KUNIT_SUBTEST_INDENT "KTAP version 1\n");
	83	+ seq_printf(seq,
	84	+ KUNIT_SUBTEST_INDENT "# Subtest: %s\n",
	85	+ suite->name);
	86	+ seq_printf(seq,
	87	+ KUNIT_SUBTEST_INDENT "1..%zd\n",
	88	+ kunit_suite_num_test_cases(suite));
	89	+
	90	+ kunit_suite_for_each_test_case(suite, test_case)
	91	+ debugfs_print_result(seq, test_case->log);
	92	+ }
86	93
87	94	debugfs_print_result(seq, suite->log);
88	95
89		− seq_printf(seq, "%s %d %s\n",
90		− kunit_status_to_ok_not_ok(success), 1, suite->name);
	96	+ if (suite->status != KUNIT_SKIPPED)
	97	+ seq_printf(seq, "%s %d %s\n",
	98	+ kunit_status_to_ok_not_ok(success), 1, suite->name);
	99	+ else
	100	+ seq_printf(seq, "%s %d %s # SKIP %s\n",
	101	+ kunit_status_to_ok_not_ok(success), 1, suite->name,
	102	+ suite->status_comment);
91	103	return 0;
92	104	}
93	105

lib/kunit/kunit-example-test.c+29−0

@@ -591,5 +591,34 @@ static struct kunit_suite example_init_test_suite = { · 590 unmodified lines
591	591	*/
592	592	kunit_test_init_section_suites(&example_init_test_suite);
593	593
	594	+/*
	595	+ * This test should always be skipped.
	596	+ */
	597	+static void example_skip_suite_test(struct kunit *test)
	598	+{
	599	+ /* This line should never be seen */
	600	+ KUNIT_FAIL(test, "You should not see a this.");
	601	+}
	602	+
	603	+static struct kunit_case example_skip_suite_test_cases[] = {
	604	+ KUNIT_CASE(example_skip_suite_test),
	605	+ {}
	606	+};
	607	+
	608	+static int example_skip_suite_init(struct kunit_suite *suite)
	609	+{
	610	+ kunit_mark_skipped(suite, "Test suite expected to be skipped");
	611	+ return 0;
	612	+}
	613	+
	614	+static struct kunit_suite example_test_skip_suite = {
	615	+ .name = "example_skip_suite",
	616	+ .suite_init = example_skip_suite_init,
	617	+ .test_cases = example_skip_suite_test_cases,
	618	+};
	619	+
	620	+/* This registers a test suite that will be skipped */
	621	+kunit_test_suite(example_test_skip_suite);
	622	+
594	623	MODULE_DESCRIPTION("Example KUnit test suite");
595	624	MODULE_LICENSE("GPL v2");

lib/kunit/string-stream.c+7−2

@@ -9,6 +9,7 @@ · 8 unmodified lines
9	9	#include <kunit/static_stub.h>
10	10	#include <kunit/test.h>
11	11	#include <linux/list.h>
	12	+#include <linux/seq_buf.h>
12	13	#include <linux/slab.h>
13	14
14	15	#include "string-stream.h"
@@ -74,7 +75,8 @@ int string_stream_vadd(struct string_stream *stream, · 59 unmodified lines
74	75
75	76	/* Append newline if necessary. */
76	77	if (frag_container->fragment[result_len - 1] != '\n')
77		− result_len = strlcat(frag_container->fragment, "\n", buf_len);
	78	+ result_len += strscpy(frag_container->fragment + result_len,
	79	+ "\n", buf_len - result_len);
78	80	} else {
79	81	result_len = vsnprintf(frag_container->fragment, buf_len, fmt, args);
80	82	}
@@ -118,15 +120,18 @@ char string_stream_get_string(struct string_stream stream) · 37 unmodified lines
118	120	{
119	121	struct string_stream_fragment *frag_container;
120	122	size_t buf_len = stream->length + 1; /* +1 for null byte. */
	123	+ struct seq_buf sb;
121	124	char *buf;
122	125
123	126	buf = kzalloc(buf_len, stream->gfp);
124	127	if (!buf)
125	128	return NULL;
126	129
	130	+ seq_buf_init(&sb, buf, buf_len);
	131	+
127	132	spin_lock(&stream->lock);
128	133	list_for_each_entry(frag_container, &stream->fragments, node)
129		− strlcat(buf, frag_container->fragment, buf_len);
	134	+ seq_buf_puts(&sb, frag_container->fragment);
130	135	spin_unlock(&stream->lock);
131	136
132	137	return buf;

lib/kunit/test.c+16−1

@@ -214,12 +214,18 @@ enum kunit_status kunit_suite_has_succeeded(struct kunit_suite *suite) · 213 unmodified lines
214	214	const struct kunit_case *test_case;
215	215	enum kunit_status status = KUNIT_SKIPPED;
216	216
	217	+ if (suite->status == KUNIT_SKIPPED)
	218	+ return KUNIT_SKIPPED;
	219	+
217	220	if (suite->suite_init_err)
218	221	return KUNIT_FAILURE;
219	222
220	223	kunit_suite_for_each_test_case(suite, test_case) {
221		− if (test_case->status == KUNIT_FAILURE)
	224	+ if (test_case->status == KUNIT_FAILURE) {
	225	+ /* Update the kunit_suite status also */
	226	+ suite->status = KUNIT_FAILURE;
222	227	return KUNIT_FAILURE;
	228	+ }
223	229	else if (test_case->status == KUNIT_SUCCESS)
224	230	status = KUNIT_SUCCESS;
225	231	}
@@ -795,12 +801,20 @@ int kunit_run_tests(struct kunit_suite *suite) · 569 unmodified lines
795	801	/* Taint the kernel so we know we've run tests. */
796	802	add_taint(TAINT_TEST, LOCKDEP_STILL_OK);
797	803
	804	+ if (suite->status == KUNIT_SKIPPED)
	805	+ goto suite_end;
	806	+
798	807	if (suite->suite_init) {
799	808	suite->suite_init_err = suite->suite_init(suite);
800	809	if (suite->suite_init_err) {
	810	+ suite->status = KUNIT_FAILURE;
801	811	kunit_err(suite, KUNIT_SUBTEST_INDENT
802	812	"# failed to initialize (%d)", suite->suite_init_err);
803	813	goto suite_end;
	814	+
	815	+ } else if (suite->status == KUNIT_SKIPPED) {
	816	+ /* Skip this kunit suite */
	817	+ goto suite_end;
804	818	}
805	819	}
806	820
@@ -825,6 +839,7 @@ static void kunit_init_suite(struct kunit_suite *suite) · 18 unmodified lines
825	839	kunit_debugfs_create_suite(suite);
826	840	suite->status_comment[0] = '\0';
827	841	suite->suite_init_err = 0;
	842	+ suite->status = KUNIT_SUCCESS;
828	843
829	844	if (suite->log)
830	845	string_stream_clear(suite->log);

rust/kernel/kunit.rs+1−0

@@ -288,6 +288,7 @@ macro_rules! kunit_unsafe_test_suite { · 287 unmodified lines
288	288	log: ::core::ptr::null_mut(),
289	289	suite_init_err: 0,
290	290	is_init: false,
	291	+ status: kernel::bindings::kunit_status_KUNIT_SUCCESS,
291	292	};
292	293
293	294	#[used(compiler)]

tools/testing/kunit/configs/all_tests.config+1−0

@@ -21,6 +21,7 @@ CONFIG_VFAT_FS=y · 20 unmodified lines
21	21	CONFIG_PCI=y
22	22	CONFIG_USB4=y
23	23	CONFIG_I2C=y
	24	+CONFIG_GPIOLIB=y
24	25
25	26	CONFIG_NET=y
26	27	CONFIG_MCTP=y

tools/testing/kunit/kunit.py+1−1

@@ -126,7 +126,7 @@ def _list_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) · 125 unmodified lines
126	126	lines.pop()
127	127
128	128	# Filter out any extraneous non-test output that might have gotten mixed in.
129		− return [l for l in output if re.match(r'^[^\s.]+\.[^\s.]+$', l)]
	129	+ return [l for l in lines if re.match(r'^[^\s.]+\.[^\s.]+$', l)]
130	130
131	131	def _list_tests_attr(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) -> Iterable[str]:
132	132	args = ['kunit.action=list_attr']

tools/testing/kunit/kunit_tool_test.py+12−0

@@ -979,6 +979,18 @@ class KUnitMainTest(unittest.TestCase): · 978 unmodified lines
979	979	self.linux_source_mock.run_kernel.assert_called_once_with(
980	980	args=['kunit.action=list'], build_dir='.kunit', filter_glob='suite*', filter='', filter_action=None, timeout=300)
981	981
	982	+ def test_list_tests_with_prefix(self):
	983	+ want = ['suite.test1', 'suite.test2', 'suite2.test1']
	984	+ self.linux_source_mock.run_kernel.return_value = [
	985	+ '[ 0.100000] TAP version 14',
	986	+ '[ 0.200000] suite.test1',
	987	+ '[ 0.200000] suite.test2',
	988	+ '[ 0.300000] suite2.test1']
	989	+
	990	+ got = kunit._list_tests(self.linux_source_mock,
	991	+ kunit.KunitExecRequest(None, None, None, False, False, '.kunit', 300, 'suite*', '', None, None, 'suite', False, False, False))
	992	+ self.assertEqual(got, want)
	993	+
982	994	@mock.patch.object(kunit, '_list_tests')
983	995	def test_run_isolated_by_suite(self, mock_tests):
984	996	mock_tests.return_value = ['suite.test1', 'suite.test2', 'suite2.test1']