Pull non-MM updates from Andrew Morton: - "panic: sys_info: Refactor and fix a potential issue" (Andy Shevchenko) fixes a build issue and does some cleanup in ib/sys_info.c - "Implement mul_u64_u64_div_u64_roundup()" (David Laight) enhances the 64-bit math code on behalf of a PWM driver and beefs up the test module for these library functions - "scripts/gdb/symbols: make BPF debug info available to GDB" (Ilya Leoshkevich) makes BPF symbol names, sizes, and line numbers available to the GDB debugger - "Enable hung_task and lockup cases to dump system info on demand" (Feng Tang) adds a sysctl which can be used to cause additional info dumping when the hung-task and lockup detectors fire - "lib/base64: add generic encoder/decoder, migrate users" (Kuan-Wei Chiu) adds a general base64 encoder/decoder to lib/ and migrates several users away from their private implementations - "rbree: inline rb_first() and rb_last()" (Eric Dumazet) makes TCP a little faster - "liveupdate: Rework KHO for in-kernel users" (Pasha Tatashin) reworks the KEXEC Handover interfaces in preparation for Live Update Orchestrator (LUO), and possibly for other future clients - "kho: simplify state machine and enable dynamic updates" (Pasha Tatashin) increases the flexibility of KEXEC Handover. Also preparation for LUO - "Live Update Orchestrator" (Pasha Tatashin) is a major new feature targeted at cloud environments. Quoting the cover letter: This series introduces the Live Update Orchestrator, a kernel subsystem designed to facilitate live kernel updates using a kexec-based reboot. This capability is critical for cloud environments, allowing hypervisors to be updated with minimal downtime for running virtual machines. LUO achieves this by preserving the state of selected resources, such as memory, devices and their dependencies, across the kernel transition. As a key feature, this series includes support for preserving memfd file descriptors, which allows critical in-memory data, such as guest RAM or any other large memory region, to be maintained in RAM across the kexec reboot. Mike Rappaport merits a mention here, for his extensive review and testing work. - "kexec: reorganize kexec and kdump sysfs" (Sourabh Jain) moves the kexec and kdump sysfs entries from /sys/kernel/ to /sys/kernel/kexec/ and adds back-compatibility symlinks which can hopefully be removed one day - "kho: fixes for vmalloc restoration" (Mike Rapoport) fixes a BUG which was being hit during KHO restoration of vmalloc() regions * tag 'mm-nonmm-stable-2025-12-06-11-14' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (139 commits) calibrate: update header inclusion Reinstate "resource: avoid unnecessary lookups in find_next_iomem_res()" vmcoreinfo: track and log recoverable hardware errors kho: fix restoring of contiguous ranges of order-0 pages kho: kho_restore_vmalloc: fix initialization of pages array MAINTAINERS: TPM DEVICE DRIVER: update the W-tag init: replace simple_strtoul with kstrtoul to improve lpj_setup KHO: fix boot failure due to kmemleak access to non-PRESENT pages Documentation/ABI: new kexec and kdump sysfs interface Documentation/ABI: mark old kexec sysfs deprecated kexec: move sysfs entries to /sys/kernel/kexec test_kho: always print restore status kho: free chunks using free_page() instead of kfree() selftests/liveupdate: add kexec test for multiple and empty sessions selftests/liveupdate: add simple kexec-based selftest for LUO selftests/liveupdate: add userspace API selftests docs: add documentation for memfd preservation via LUO mm: memfd_luo: allow preserving memfd liveupdate: luo_file: add private argument to store runtime state mm: shmem: export some functions to internal.h ...
Unified · Split
| @@ -303,6 +303,7 @@ Hans de Goede <hansg@kernel.org> <hdegoede@redhat.com> · 302 unmodified lines | ||
| 303 | 303 | Hans Verkuil <hverkuil@kernel.org> <hverkuil@xs4all.nl> |
| 304 | 304 | Hans Verkuil <hverkuil@kernel.org> <hverkuil-cisco@xs4all.nl> |
| 305 | 305 | Hans Verkuil <hverkuil@kernel.org> <hansverk@cisco.com> |
| 306 | +Hao Ge <hao.ge@linux.dev> <gehao@kylinos.cn> | |
| 306 | 307 | Harry Yoo <harry.yoo@oracle.com> <42.hyeyoo@gmail.com> |
| 307 | 308 | Heiko Carstens <hca@linux.ibm.com> <h.carstens@de.ibm.com> |
| 308 | 309 | Heiko Carstens <hca@linux.ibm.com> <heiko.carstens@de.ibm.com> |
| @@ -503,9 +504,7 @@ Mark Brown <broonie@sirena.org.uk> · 194 unmodified lines | ||
| 503 | 504 | Mark Starovoytov <mstarovo@pm.me> <mstarovoitov@marvell.com> |
| 504 | 505 | Markus Schneider-Pargmann <msp@baylibre.com> <mpa@pengutronix.de> |
| 505 | 506 | Mark Yao <markyao0591@gmail.com> <mark.yao@rock-chips.com> |
| 506 | −Martin Kepplinger <martink@posteo.de> <martin.kepplinger@ginzinger.com> | |
| 507 | −Martin Kepplinger <martink@posteo.de> <martin.kepplinger@puri.sm> | |
| 508 | −Martin Kepplinger <martink@posteo.de> <martin.kepplinger@theobroma-systems.com> | |
| 507 | +Martin Kepplinger-Novakovic <martink@posteo.de> <martin.kepplinger-novakovic@ginzinger.com> | |
| 509 | 508 | Martyna Szapar-Mudlaw <martyna.szapar-mudlaw@linux.intel.com> <martyna.szapar-mudlaw@intel.com> |
| 510 | 509 | Mathieu Othacehe <othacehe@gnu.org> <m.othacehe@gmail.com> |
| 511 | 510 | Mat Martineau <martineau@kernel.org> <mathew.j.martineau@linux.intel.com> |
| @@ -856,6 +855,9 @@ Vivien Didelot <vivien.didelot@gmail.com> <vivien.didelot@savoirfairelinux.com> · 344 unmodified lines | ||
| 856 | 855 | Vlad Dogaru <ddvlad@gmail.com> <vlad.dogaru@intel.com> |
| 857 | 856 | Vladimir Davydov <vdavydov.dev@gmail.com> <vdavydov@parallels.com> |
| 858 | 857 | Vladimir Davydov <vdavydov.dev@gmail.com> <vdavydov@virtuozzo.com> |
| 858 | +WangYuli <wangyuli@aosc.io> <wangyl5933@chinaunicom.cn> | |
| 859 | +WangYuli <wangyuli@aosc.io> <wangyuli@deepin.org> | |
| 860 | +WangYuli <wangyuli@aosc.io> <wangyuli@uniontech.com> | |
| 859 | 861 | Weiwen Hu <huweiwen@linux.alibaba.com> <sehuww@mail.scut.edu.cn> |
| 860 | 862 | WeiXiong Liao <gmpy.liaowx@gmail.com> <liaoweixiong@allwinnertech.com> |
| 861 | 863 | Wen Gong <quic_wgong@quicinc.com> <wgong@codeaurora.org> |
| @@ -867,6 +869,7 @@ Yakir Yang <kuankuan.y@gmail.com> <ykk@rock-chips.com> · 5 unmodified lines | ||
| 867 | 869 | Yanteng Si <si.yanteng@linux.dev> <siyanteng@loongson.cn> |
| 868 | 870 | Ying Huang <huang.ying.caritas@gmail.com> <ying.huang@intel.com> |
| 869 | 871 | Yosry Ahmed <yosry.ahmed@linux.dev> <yosryahmed@google.com> |
| 872 | +Yu-Chun Lin <eleanor.lin@realtek.com> <eleanor15x@gmail.com> | |
| 870 | 873 | Yusuke Goda <goda.yusuke@renesas.com> |
| 871 | 874 | Zack Rusin <zack.rusin@broadcom.com> <zackr@vmware.com> |
| 872 | 875 | Zhu Yanjun <zyjzyj2000@gmail.com> <yanjunz@nvidia.com> |
| @@ -2056,16 +2056,15 @@ S: Korte Heul 95 · 2055 unmodified lines | ||
| 2056 | 2056 | S: 1403 ND BUSSUM |
| 2057 | 2057 | S: The Netherlands |
| 2058 | 2058 | |
| 2059 | −N: Martin Kepplinger | |
| 2059 | +N: Martin Kepplinger-Novakovic | |
| 2060 | 2060 | E: martink@posteo.de |
| 2061 | −E: martin.kepplinger@puri.sm | |
| 2062 | −W: http://www.martinkepplinger.com | |
| 2063 | 2061 | P: 4096R/5AB387D3 F208 2B88 0F9E 4239 3468 6E3F 5003 98DF 5AB3 87D3 |
| 2064 | 2062 | D: mma8452 accelerators iio driver |
| 2065 | 2063 | D: pegasus_notetaker input driver |
| 2064 | +D: imx8m media and hi846 sensor driver | |
| 2066 | 2065 | D: Kernel fixes and cleanups |
| 2067 | −S: Garnisonstraße 26 | |
| 2068 | −S: 4020 Linz | |
| 2066 | +S: Keplerstr. 6 | |
| 2067 | +S: 4050 Traun | |
| 2069 | 2068 | S: Austria |
| 2070 | 2069 | |
| 2071 | 2070 | N: Karl Keyte |
| @@ -0,0 +1,71 @@ | ||
| 1 | +NOTE: all the ABIs listed in this file are deprecated and will be removed after 2028. | |
| 2 | + | |
| 3 | +Here are the alternative ABIs: | |
| 4 | ++------------------------------------+-----------------------------------------+ | |
| 5 | +| Deprecated | Alternative | | |
| 6 | ++------------------------------------+-----------------------------------------+ | |
| 7 | +| /sys/kernel/kexec_loaded | /sys/kernel/kexec/loaded | | |
| 8 | ++------------------------------------+-----------------------------------------+ | |
| 9 | +| /sys/kernel/kexec_crash_loaded | /sys/kernel/kexec/crash_loaded | | |
| 10 | ++------------------------------------+-----------------------------------------+ | |
| 11 | +| /sys/kernel/kexec_crash_size | /sys/kernel/kexec/crash_size | | |
| 12 | ++------------------------------------+-----------------------------------------+ | |
| 13 | +| /sys/kernel/crash_elfcorehdr_size | /sys/kernel/kexec/crash_elfcorehdr_size | | |
| 14 | ++------------------------------------+-----------------------------------------+ | |
| 15 | +| /sys/kernel/kexec_crash_cma_ranges | /sys/kernel/kexec/crash_cma_ranges | | |
| 16 | ++------------------------------------+-----------------------------------------+ | |
| 17 | + | |
| 18 | + | |
| 19 | +What: /sys/kernel/kexec_loaded | |
| 20 | +Date: Jun 2006 | |
| 21 | +Contact: kexec@lists.infradead.org | |
| 22 | +Description: read only | |
| 23 | + Indicates whether a new kernel image has been loaded | |
| 24 | + into memory using the kexec system call. It shows 1 if | |
| 25 | + a kexec image is present and ready to boot, or 0 if none | |
| 26 | + is loaded. | |
| 27 | +User: kexec tools, kdump service | |
| 28 | + | |
| 29 | +What: /sys/kernel/kexec_crash_loaded | |
| 30 | +Date: Jun 2006 | |
| 31 | +Contact: kexec@lists.infradead.org | |
| 32 | +Description: read only | |
| 33 | + Indicates whether a crash (kdump) kernel is currently | |
| 34 | + loaded into memory. It shows 1 if a crash kernel has been | |
| 35 | + successfully loaded for panic handling, or 0 if no crash | |
| 36 | + kernel is present. | |
| 37 | +User: Kexec tools, Kdump service | |
| 38 | + | |
| 39 | +What: /sys/kernel/kexec_crash_size | |
| 40 | +Date: Dec 2009 | |
| 41 | +Contact: kexec@lists.infradead.org | |
| 42 | +Description: read/write | |
| 43 | + Shows the amount of memory reserved for loading the crash | |
| 44 | + (kdump) kernel. It reports the size, in bytes, of the | |
| 45 | + crash kernel area defined by the crashkernel= parameter. | |
| 46 | + This interface also allows reducing the crashkernel | |
| 47 | + reservation by writing a smaller value, and the reclaimed | |
| 48 | + space is added back to the system RAM. | |
| 49 | +User: Kdump service | |
| 50 | + | |
| 51 | +What: /sys/kernel/crash_elfcorehdr_size | |
| 52 | +Date: Aug 2023 | |
| 53 | +Contact: kexec@lists.infradead.org | |
| 54 | +Description: read only | |
| 55 | + Indicates the preferred size of the memory buffer for the | |
| 56 | + ELF core header used by the crash (kdump) kernel. It defines | |
| 57 | + how much space is needed to hold metadata about the crashed | |
| 58 | + system, including CPU and memory information. This information | |
| 59 | + is used by the user space utility kexec to support updating the | |
| 60 | + in-kernel kdump image during hotplug operations. | |
| 61 | +User: Kexec tools | |
| 62 | + | |
| 63 | +What: /sys/kernel/kexec_crash_cma_ranges | |
| 64 | +Date: Nov 2025 | |
| 65 | +Contact: kexec@lists.infradead.org | |
| 66 | +Description: read only | |
| 67 | + Provides information about the memory ranges reserved from | |
| 68 | + the Contiguous Memory Allocator (CMA) area that are allocated | |
| 69 | + to the crash (kdump) kernel. It lists the start and end physical | |
| 70 | + addresses of CMA regions assigned for crashkernel use. | |
| 71 | +User: kdump service | |
| @@ -0,0 +1,61 @@ | ||
| 1 | +What: /sys/kernel/kexec/* | |
| 2 | +Date: Nov 2025 | |
| 3 | +Contact: kexec@lists.infradead.org | |
| 4 | +Description: | |
| 5 | + The /sys/kernel/kexec/* directory contains sysfs files | |
| 6 | + that provide information about the configuration status | |
| 7 | + of kexec and kdump. | |
| 8 | + | |
| 9 | +What: /sys/kernel/kexec/loaded | |
| 10 | +Date: Nov 2025 | |
| 11 | +Contact: kexec@lists.infradead.org | |
| 12 | +Description: read only | |
| 13 | + Indicates whether a new kernel image has been loaded | |
| 14 | + into memory using the kexec system call. It shows 1 if | |
| 15 | + a kexec image is present and ready to boot, or 0 if none | |
| 16 | + is loaded. | |
| 17 | +User: kexec tools, kdump service | |
| 18 | + | |
| 19 | +What: /sys/kernel/kexec/crash_loaded | |
| 20 | +Date: Nov 2025 | |
| 21 | +Contact: kexec@lists.infradead.org | |
| 22 | +Description: read only | |
| 23 | + Indicates whether a crash (kdump) kernel is currently | |
| 24 | + loaded into memory. It shows 1 if a crash kernel has been | |
| 25 | + successfully loaded for panic handling, or 0 if no crash | |
| 26 | + kernel is present. | |
| 27 | +User: Kexec tools, Kdump service | |
| 28 | + | |
| 29 | +What: /sys/kernel/kexec/crash_size | |
| 30 | +Date: Nov 2025 | |
| 31 | +Contact: kexec@lists.infradead.org | |
| 32 | +Description: read/write | |
| 33 | + Shows the amount of memory reserved for loading the crash | |
| 34 | + (kdump) kernel. It reports the size, in bytes, of the | |
| 35 | + crash kernel area defined by the crashkernel= parameter. | |
| 36 | + This interface also allows reducing the crashkernel | |
| 37 | + reservation by writing a smaller value, and the reclaimed | |
| 38 | + space is added back to the system RAM. | |
| 39 | +User: Kdump service | |
| 40 | + | |
| 41 | +What: /sys/kernel/kexec/crash_elfcorehdr_size | |
| 42 | +Date: Nov 2025 | |
| 43 | +Contact: kexec@lists.infradead.org | |
| 44 | +Description: read only | |
| 45 | + Indicates the preferred size of the memory buffer for the | |
| 46 | + ELF core header used by the crash (kdump) kernel. It defines | |
| 47 | + how much space is needed to hold metadata about the crashed | |
| 48 | + system, including CPU and memory information. This information | |
| 49 | + is used by the user space utility kexec to support updating the | |
| 50 | + in-kernel kdump image during hotplug operations. | |
| 51 | +User: Kexec tools | |
| 52 | + | |
| 53 | +What: /sys/kernel/kexec/crash_cma_ranges | |
| 54 | +Date: Nov 2025 | |
| 55 | +Contact: kexec@lists.infradead.org | |
| 56 | +Description: read only | |
| 57 | + Provides information about the memory ranges reserved from | |
| 58 | + the Contiguous Memory Allocator (CMA) area that are allocated | |
| 59 | + to the crash (kdump) kernel. It lists the start and end physical | |
| 60 | + addresses of CMA regions assigned for crashkernel use. | |
| 61 | +User: kdump service | |
| @@ -223,12 +223,13 @@ The flags are:: · 222 unmodified lines | ||
| 223 | 223 | f Include the function name |
| 224 | 224 | s Include the source file name |
| 225 | 225 | l Include line number |
| 226 | + d Include call trace | |
| 226 | 227 | |
| 227 | 228 | For ``print_hex_dump_debug()`` and ``print_hex_dump_bytes()``, only |
| 228 | 229 | the ``p`` flag has meaning, other flags are ignored. |
| 229 | 230 | |
| 230 | −Note the regexp ``^[-+=][fslmpt_]+$`` matches a flags specification. | |
| 231 | −To clear all flags at once, use ``=_`` or ``-fslmpt``. | |
| 231 | +Note the regexp ``^[-+=][fslmptd_]+$`` matches a flags specification. | |
| 232 | +To clear all flags at once, use ``=_`` or ``-fslmptd``. | |
| 232 | 233 | |
| 233 | 234 | |
| 234 | 235 | Debug messages during Boot Process |
| @@ -2114,14 +2114,20 @@ Kernel parameters · 2113 unmodified lines | ||
| 2114 | 2114 | the added memory block itself do not be affected. |
| 2115 | 2115 | |
| 2116 | 2116 | hung_task_panic= |
| 2117 | − [KNL] Should the hung task detector generate panics. | |
| 2118 | − Format: 0 | 1 | |
| 2117 | + [KNL] Number of hung tasks to trigger kernel panic. | |
| 2118 | + Format: <int> | |
| 2119 | + | |
| 2120 | + When set to a non-zero value, a kernel panic will be triggered if | |
| 2121 | + the number of detected hung tasks reaches this value. | |
| 2122 | + | |
| 2123 | + 0: don't panic | |
| 2124 | + 1: panic immediately on first hung task | |
| 2125 | + N: panic after N hung tasks are detected in a single scan | |
| 2119 | 2126 | |
| 2120 | − A value of 1 instructs the kernel to panic when a | |
| 2121 | − hung task is detected. The default value is controlled | |
| 2122 | − by the CONFIG_BOOTPARAM_HUNG_TASK_PANIC build-time | |
| 2123 | − option. The value selected by this boot parameter can | |
| 2124 | − be changed later by the kernel.hung_task_panic sysctl. | |
| 2127 | + The default value is controlled by the | |
| 2128 | + CONFIG_BOOTPARAM_HUNG_TASK_PANIC build-time option. The value | |
| 2129 | + selected by this boot parameter can be changed later by the | |
| 2130 | + kernel.hung_task_panic sysctl. | |
| 2125 | 2131 | |
| 2126 | 2132 | hvc_iucv= [S390] Number of z/VM IUCV hypervisor console (HVC) |
| 2127 | 2133 | terminal devices. Valid values: 0..8 |
| @@ -397,13 +397,14 @@ a hung task is detected. · 396 unmodified lines | ||
| 397 | 397 | hung_task_panic |
| 398 | 398 | =============== |
| 399 | 399 | |
| 400 | −Controls the kernel's behavior when a hung task is detected. | |
| 400 | +When set to a non-zero value, a kernel panic will be triggered if the | |
| 401 | +number of hung tasks found during a single scan reaches this value. | |
| 401 | 402 | This file shows up if ``CONFIG_DETECT_HUNG_TASK`` is enabled. |
| 402 | 403 | |
| 403 | −= ================================================= | |
| 404 | += ======================================================= | |
| 404 | 405 | 0 Continue operation. This is the default behavior. |
| 405 | −1 Panic immediately. | |
| 406 | −= ================================================= | |
| 406 | +N Panic when N hung tasks are found during a single scan. | |
| 407 | += ======================================================= | |
| 407 | 408 | |
| 408 | 409 | |
| 409 | 410 | hung_task_check_count |
| @@ -421,6 +422,11 @@ the system boot. · 11 unmodified lines | ||
| 421 | 422 | |
| 422 | 423 | This file shows up if ``CONFIG_DETECT_HUNG_TASK`` is enabled. |
| 423 | 424 | |
| 425 | +hung_task_sys_info | |
| 426 | +================== | |
| 427 | +A comma separated list of extra system information to be dumped when | |
| 428 | +hung task is detected, for example, "tasks,mem,timers,locks,...". | |
| 429 | +Refer 'panic_sys_info' section below for more details. | |
| 424 | 430 | |
| 425 | 431 | hung_task_timeout_secs |
| 426 | 432 | ====================== |
| @@ -515,6 +521,15 @@ default), only processes with the CAP_SYS_ADMIN capability may create · 88 unmodified lines | ||
| 515 | 521 | io_uring instances. |
| 516 | 522 | |
| 517 | 523 | |
| 524 | +kernel_sys_info | |
| 525 | +=============== | |
| 526 | +A comma separated list of extra system information to be dumped when | |
| 527 | +soft/hard lockup is detected, for example, "tasks,mem,timers,locks,...". | |
| 528 | +Refer 'panic_sys_info' section below for more details. | |
| 529 | + | |
| 530 | +It serves as the default kernel control knob, which will take effect | |
| 531 | +when a kernel module calls sys_info() with parameter==0. | |
| 532 | + | |
| 518 | 533 | kexec_load_disabled |
| 519 | 534 | =================== |
| 520 | 535 | |
| @@ -576,6 +591,11 @@ if leaking kernel pointer values to unprivileged users is a concern. · 55 unmodified lines | ||
| 576 | 591 | When ``kptr_restrict`` is set to 2, kernel pointers printed using |
| 577 | 592 | %pK will be replaced with 0s regardless of privileges. |
| 578 | 593 | |
| 594 | +softlockup_sys_info & hardlockup_sys_info | |
| 595 | +========================================= | |
| 596 | +A comma separated list of extra system information to be dumped when | |
| 597 | +soft/hard lockup is detected, for example, "tasks,mem,timers,locks,...". | |
| 598 | +Refer 'panic_sys_info' section below for more details. | |
| 579 | 599 | |
| 580 | 600 | modprobe |
| 581 | 601 | ======== |
| @@ -910,8 +930,8 @@ to 'panic_print'. Possible values are: · 328 unmodified lines | ||
| 910 | 930 | ============= =================================================== |
| 911 | 931 | tasks print all tasks info |
| 912 | 932 | mem print system memory info |
| 913 | −timer print timers info | |
| 914 | −lock print locks info if CONFIG_LOCKDEP is on | |
| 933 | +timers print timers info | |
| 934 | +locks print locks info if CONFIG_LOCKDEP is on | |
| 915 | 935 | ftrace print ftrace buffer |
| 916 | 936 | all_bt print all CPUs backtrace (if available in the arch) |
| 917 | 937 | blocked_tasks print only tasks in uninterruptible (blocked) state |
| @@ -138,6 +138,7 @@ Documents that don't fit elsewhere or which have yet to be categorized. · 137 unmodified lines | ||
| 138 | 138 | :maxdepth: 1 |
| 139 | 139 | |
| 140 | 140 | librs |
| 141 | + liveupdate | |
| 141 | 142 | netlink |
| 142 | 143 | |
| 143 | 144 | .. only:: subproject and html |
| @@ -70,5 +70,5 @@ in the FDT. That state is called the KHO finalization phase. · 69 unmodified lines | ||
| 70 | 70 | |
| 71 | 71 | Public API |
| 72 | 72 | ========== |
| 73 | −.. kernel-doc:: kernel/kexec_handover.c | |
| 73 | +.. kernel-doc:: kernel/liveupdate/kexec_handover.c | |
| 74 | 74 | :export: |
| @@ -0,0 +1,61 @@ | ||
| 1 | +.. SPDX-License-Identifier: GPL-2.0 | |
| 2 | + | |
| 3 | +======================== | |
| 4 | +Live Update Orchestrator | |
| 5 | +======================== | |
| 6 | +:Author: Pasha Tatashin <pasha.tatashin@soleen.com> | |
| 7 | + | |
| 8 | +.. kernel-doc:: kernel/liveupdate/luo_core.c | |
| 9 | + :doc: Live Update Orchestrator (LUO) | |
| 10 | + | |
| 11 | +LUO Sessions | |
| 12 | +============ | |
| 13 | +.. kernel-doc:: kernel/liveupdate/luo_session.c | |
| 14 | + :doc: LUO Sessions | |
| 15 | + | |
| 16 | +LUO Preserving File Descriptors | |
| 17 | +=============================== | |
| 18 | +.. kernel-doc:: kernel/liveupdate/luo_file.c | |
| 19 | + :doc: LUO File Descriptors | |
| 20 | + | |
| 21 | +Live Update Orchestrator ABI | |
| 22 | +============================ | |
| 23 | +.. kernel-doc:: include/linux/kho/abi/luo.h | |
| 24 | + :doc: Live Update Orchestrator ABI | |
| 25 | + | |
| 26 | +The following types of file descriptors can be preserved | |
| 27 | + | |
| 28 | +.. toctree:: | |
| 29 | + :maxdepth: 1 | |
| 30 | + | |
| 31 | + ../mm/memfd_preservation | |
| 32 | + | |
| 33 | +Public API | |
| 34 | +========== | |
| 35 | +.. kernel-doc:: include/linux/liveupdate.h | |
| 36 | + | |
| 37 | +.. kernel-doc:: include/linux/kho/abi/luo.h | |
| 38 | + :functions: | |
| 39 | + | |
| 40 | +.. kernel-doc:: kernel/liveupdate/luo_core.c | |
| 41 | + :export: | |
| 42 | + | |
| 43 | +.. kernel-doc:: kernel/liveupdate/luo_file.c | |
| 44 | + :export: | |
| 45 | + | |
| 46 | +Internal API | |
| 47 | +============ | |
| 48 | +.. kernel-doc:: kernel/liveupdate/luo_core.c | |
| 49 | + :internal: | |
| 50 | + | |
| 51 | +.. kernel-doc:: kernel/liveupdate/luo_session.c | |
| 52 | + :internal: | |
| 53 | + | |
| 54 | +.. kernel-doc:: kernel/liveupdate/luo_file.c | |
| 55 | + :internal: | |
| 56 | + | |
| 57 | +See Also | |
| 58 | +======== | |
| 59 | + | |
| 60 | +- :doc:`Live Update uAPI </userspace-api/liveupdate>` | |
| 61 | +- :doc:`/core-api/kho/concepts` | |
| @@ -1238,6 +1238,16 @@ Others · 1237 unmodified lines | ||
| 1238 | 1238 | The patch file does not appear to be in unified-diff format. Please |
| 1239 | 1239 | regenerate the patch file before sending it to the maintainer. |
| 1240 | 1240 | |
| 1241 | + **PLACEHOLDER_USE** | |
| 1242 | + Detects unhandled placeholder text left in cover letters or commit headers/logs. | |
| 1243 | + Common placeholders include lines like:: | |
| 1244 | + | |
| 1245 | + *** SUBJECT HERE *** | |
| 1246 | + *** BLURB HERE *** | |
| 1247 | + | |
| 1248 | + These typically come from autogenerated templates. Replace them with a proper | |
| 1249 | + subject and description before sending. | |
| 1250 | + | |
| 1241 | 1251 | **PRINTF_0XDECIMAL** |
| 1242 | 1252 | Prefixing 0x with decimal output is defective and should be corrected. |
| 1243 | 1253 | |
| @@ -0,0 +1,60 @@ | ||
| 1 | +.. SPDX-License-Identifier: GPL-2.0 | |
| 2 | + | |
| 3 | +================================================= | |
| 4 | +Recoverable Hardware Error Tracking in vmcoreinfo | |
| 5 | +================================================= | |
| 6 | + | |
| 7 | +Overview | |
| 8 | +-------- | |
| 9 | + | |
| 10 | +This feature provides a generic infrastructure within the Linux kernel to track | |
| 11 | +and log recoverable hardware errors. These are hardware recoverable errors | |
| 12 | +visible that might not cause immediate panics but may influence health, mainly | |
| 13 | +because new code path will be executed in the kernel. | |
| 14 | + | |
| 15 | +By recording counts and timestamps of recoverable errors into the vmcoreinfo | |
| 16 | +crash dump notes, this infrastructure aids post-mortem crash analysis tools in | |
| 17 | +correlating hardware events with kernel failures. This enables faster triage | |
| 18 | +and better understanding of root causes, especially in large-scale cloud | |
| 19 | +environments where hardware issues are common. | |
| 20 | + | |
| 21 | +Benefits | |
| 22 | +-------- | |
| 23 | + | |
| 24 | +- Facilitates correlation of hardware recoverable errors with kernel panics or | |
| 25 | + unusual code paths that lead to system crashes. | |
| 26 | +- Provides operators and cloud providers quick insights, improving reliability | |
| 27 | + and reducing troubleshooting time. | |
| 28 | +- Complements existing full hardware diagnostics without replacing them. | |
| 29 | + | |
| 30 | +Data Exposure and Consumption | |
| 31 | +----------------------------- | |
| 32 | + | |
| 33 | +- The tracked error data consists of per-error-type counts and timestamps of | |
| 34 | + last occurrence. | |
| 35 | +- This data is stored in the `hwerror_data` array, categorized by error source | |
| 36 | + types like CPU, memory, PCI, CXL, and others. | |
| 37 | +- It is exposed via vmcoreinfo crash dump notes and can be read using tools | |
| 38 | + like `crash`, `drgn`, or other kernel crash analysis utilities. | |
| 39 | +- There is no other way to read these data other than from crash dumps. | |
| 40 | +- These errors are divided by area, which includes CPU, Memory, PCI, CXL and | |
| 41 | + others. | |
| 42 | + | |
| 43 | +Typical usage example (in drgn REPL): | |
| 44 | + | |
| 45 | +.. code-block:: python | |
| 46 | + | |
| 47 | + >>> prog['hwerror_data'] | |
| 48 | + (struct hwerror_info[HWERR_RECOV_MAX]){ | |
| 49 | + { | |
| 50 | + .count = (int)844, | |
| 51 | + .timestamp = (time64_t)1752852018, | |
| 52 | + }, | |
| 53 | + ... | |
| 54 | + } | |
| 55 | + | |
| 56 | +Enabling | |
| 57 | +-------- | |
| 58 | + | |
| 59 | +- This feature is enabled when CONFIG_VMCORE_INFO is set. | |
| 60 | + | |
| @@ -97,6 +97,7 @@ Subsystem-specific APIs · 96 unmodified lines | ||
| 97 | 97 | gpio/index |
| 98 | 98 | hsi |
| 99 | 99 | hte/index |
| 100 | + hw-recoverable-errors | |
| 100 | 101 | i2c |
| 101 | 102 | iio/index |
| 102 | 103 | infiniband |
| @@ -48,6 +48,7 @@ documentation, or deleted if it has served its purpose. · 47 unmodified lines | ||
| 48 | 48 | hugetlbfs_reserv |
| 49 | 49 | ksm |
| 50 | 50 | memory-model |
| 51 | + memfd_preservation | |
| 51 | 52 | mmu_notifier |
| 52 | 53 | multigen_lru |
| 53 | 54 | numa |
| @@ -0,0 +1,23 @@ | ||
| 1 | +.. SPDX-License-Identifier: GPL-2.0-or-later | |
| 2 | + | |
| 3 | +========================== | |
| 4 | +Memfd Preservation via LUO | |
| 5 | +========================== | |
| 6 | + | |
| 7 | +.. kernel-doc:: mm/memfd_luo.c | |
| 8 | + :doc: Memfd Preservation via LUO | |
| 9 | + | |
| 10 | +Memfd Preservation ABI | |
| 11 | +====================== | |
| 12 | + | |
| 13 | +.. kernel-doc:: include/linux/kho/abi/memfd.h | |
| 14 | + :doc: DOC: memfd Live Update ABI | |
| 15 | + | |
| 16 | +.. kernel-doc:: include/linux/kho/abi/memfd.h | |
| 17 | + :internal: | |
| 18 | + | |
| 19 | +See Also | |
| 20 | +======== | |
| 21 | + | |
| 22 | +- :doc:`/core-api/liveupdate` | |
| 23 | +- :doc:`/core-api/kho/concepts` | |
| @@ -61,6 +61,7 @@ Everything else · 60 unmodified lines | ||
| 61 | 61 | :maxdepth: 1 |
| 62 | 62 | |
| 63 | 63 | ELF |
| 64 | + liveupdate | |
| 64 | 65 | netlink/index |
| 65 | 66 | sysfs-platform_profile |
| 66 | 67 | vduse |
| @@ -385,6 +385,8 @@ Code Seq# Include File Comments · 384 unmodified lines | ||
| 385 | 385 | 0xB8 01-02 uapi/misc/mrvl_cn10k_dpi.h Marvell CN10K DPI driver |
| 386 | 386 | 0xB8 all uapi/linux/mshv.h Microsoft Hyper-V /dev/mshv driver |
| 387 | 387 | <mailto:linux-hyperv@vger.kernel.org> |
| 388 | +0xBA 00-0F uapi/linux/liveupdate.h Pasha Tatashin | |
| 389 | + <mailto:pasha.tatashin@soleen.com> | |
| 388 | 390 | 0xC0 00-0F linux/usb/iowarrior.h |
| 389 | 391 | 0xCA 00-0F uapi/misc/cxl.h Dead since 6.15 |
| 390 | 392 | 0xCA 10-2F uapi/misc/ocxl.h |
| @@ -0,0 +1,20 @@ | ||
| 1 | +.. SPDX-License-Identifier: GPL-2.0 | |
| 2 | + | |
| 3 | +================ | |
| 4 | +Live Update uAPI | |
| 5 | +================ | |
| 6 | +:Author: Pasha Tatashin <pasha.tatashin@soleen.com> | |
| 7 | + | |
| 8 | +ioctl interface | |
| 9 | +=============== | |
| 10 | +.. kernel-doc:: kernel/liveupdate/luo_core.c | |
| 11 | + :doc: LUO ioctl Interface | |
| 12 | + | |
| 13 | +ioctl uAPI | |
| 14 | +=========== | |
| 15 | +.. kernel-doc:: include/uapi/linux/liveupdate.h | |
| 16 | + | |
| 17 | +See Also | |
| 18 | +======== | |
| 19 | + | |
| 20 | +- :doc:`Live Update Orchestrator </core-api/liveupdate>` | |
| @@ -11659,7 +11659,7 @@ T: git git://linuxtv.org/media.git · 11658 unmodified lines | ||
| 11659 | 11659 | F: drivers/media/i2c/hi556.c |
| 11660 | 11660 | |
| 11661 | 11661 | HYNIX HI846 SENSOR DRIVER |
| 11662 | −M: Martin Kepplinger <martin.kepplinger@puri.sm> | |
| 11662 | +M: Martin Kepplinger-Novakovic <martink@posteo.de> | |
| 11663 | 11663 | L: linux-media@vger.kernel.org |
| 11664 | 11664 | S: Maintained |
| 11665 | 11665 | F: drivers/media/i2c/hi846.c |
| @@ -11744,6 +11744,7 @@ HUNG TASK DETECTOR · 78 unmodified lines | ||
| 11744 | 11744 | M: Andrew Morton <akpm@linux-foundation.org> |
| 11745 | 11745 | R: Lance Yang <lance.yang@linux.dev> |
| 11746 | 11746 | R: Masami Hiramatsu <mhiramat@kernel.org> |
| 11747 | +R: Petr Mladek <pmladek@suse.com> | |
| 11747 | 11748 | L: linux-kernel@vger.kernel.org |
| 11748 | 11749 | S: Maintained |
| 11749 | 11750 | F: include/linux/hung_task.h |
| @@ -13891,14 +13892,15 @@ F: kernel/kexec* · 2141 unmodified lines | ||
| 13891 | 13892 | KEXEC HANDOVER (KHO) |
| 13892 | 13893 | M: Alexander Graf <graf@amazon.com> |
| 13893 | 13894 | M: Mike Rapoport <rppt@kernel.org> |
| 13894 | −M: Changyuan Lyu <changyuanl@google.com> | |
| 13895 | +M: Pasha Tatashin <pasha.tatashin@soleen.com> | |
| 13896 | +R: Pratyush Yadav <pratyush@kernel.org> | |
| 13895 | 13897 | L: kexec@lists.infradead.org |
| 13896 | 13898 | L: linux-mm@kvack.org |
| 13897 | 13899 | S: Maintained |
| 13898 | 13900 | F: Documentation/admin-guide/mm/kho.rst |
| 13899 | 13901 | F: Documentation/core-api/kho/* |
| 13900 | 13902 | F: include/linux/kexec_handover.h |
| 13901 | −F: kernel/kexec_handover.c | |
| 13903 | +F: kernel/liveupdate/kexec_handover* | |
| 13902 | 13904 | F: lib/test_kho.c |
| 13903 | 13905 | F: tools/testing/selftests/kho/ |
| 13904 | 13906 | |
| @@ -14567,6 +14569,22 @@ F: samples/livepatch/ · 662 unmodified lines | ||
| 14567 | 14569 | F: scripts/livepatch/ |
| 14568 | 14570 | F: tools/testing/selftests/livepatch/ |
| 14569 | 14571 | |
| 14572 | +LIVE UPDATE | |
| 14573 | +M: Pasha Tatashin <pasha.tatashin@soleen.com> | |
| 14574 | +M: Mike Rapoport <rppt@kernel.org> | |
| 14575 | +R: Pratyush Yadav <pratyush@kernel.org> | |
| 14576 | +L: linux-kernel@vger.kernel.org | |
| 14577 | +S: Maintained | |
| 14578 | +F: Documentation/core-api/liveupdate.rst | |
| 14579 | +F: Documentation/mm/memfd_preservation.rst | |
| 14580 | +F: Documentation/userspace-api/liveupdate.rst | |
| 14581 | +F: include/linux/liveupdate.h | |
| 14582 | +F: include/linux/liveupdate/ | |
| 14583 | +F: include/uapi/linux/liveupdate.h | |
| 14584 | +F: kernel/liveupdate/ | |
| 14585 | +F: mm/memfd_luo.c | |
| 14586 | +F: tools/testing/selftests/liveupdate/ | |
| 14587 | + | |
| 14570 | 14588 | LLC (802.2) |
| 14571 | 14589 | L: netdev@vger.kernel.org |
| 14572 | 14590 | S: Odd fixes |
| @@ -15668,7 +15686,7 @@ F: include/media/imx.h · 1095 unmodified lines | ||
| 15668 | 15686 | MEDIA DRIVERS FOR FREESCALE IMX7/8 |
| 15669 | 15687 | M: Rui Miguel Silva <rmfrfs@gmail.com> |
| 15670 | 15688 | M: Laurent Pinchart <laurent.pinchart@ideasonboard.com> |
| 15671 | −M: Martin Kepplinger <martin.kepplinger@puri.sm> | |
| 15689 | +M: Martin Kepplinger-Novakovic <martink@posteo.de> | |
| 15672 | 15690 | R: Purism Kernel Team <kernel@puri.sm> |
| 15673 | 15691 | R: Frank Li <Frank.Li@nxp.com> |
| 15674 | 15692 | L: imx@lists.linux.dev |
| @@ -18420,10 +18438,11 @@ F: net/sunrpc/ · 2745 unmodified lines | ||
| 18420 | 18438 | |
| 18421 | 18439 | NILFS2 FILESYSTEM |
| 18422 | 18440 | M: Ryusuke Konishi <konishi.ryusuke@gmail.com> |
| 18441 | +M: Viacheslav Dubeyko <slava@dubeyko.com> | |
| 18423 | 18442 | L: linux-nilfs@vger.kernel.org |
| 18424 | −S: Supported | |
| 18443 | +S: Maintained | |
| 18425 | 18444 | W: https://nilfs.sourceforge.io/ |
| 18426 | −T: git https://github.com/konis/nilfs2.git | |
| 18445 | +T: git git://git.kernel.org/pub/scm/linux/kernel/git/vdubeyko/nilfs2.git | |
| 18427 | 18446 | F: Documentation/filesystems/nilfs2.rst |
| 18428 | 18447 | F: fs/nilfs2/ |
| 18429 | 18448 | F: include/trace/events/nilfs2.h |
| @@ -25103,7 +25122,6 @@ F: drivers/regulator/sy8106a-regulator.c · 6673 unmodified lines | ||
| 25103 | 25122 | |
| 25104 | 25123 | SYNC FILE FRAMEWORK |
| 25105 | 25124 | M: Sumit Semwal <sumit.semwal@linaro.org> |
| 25106 | −R: Gustavo Padovan <gustavo@padovan.org> | |
| 25107 | 25125 | L: linux-media@vger.kernel.org |
| 25108 | 25126 | L: dri-devel@lists.freedesktop.org |
| 25109 | 25127 | S: Maintained |
| @@ -26308,7 +26326,7 @@ M: Jarkko Sakkinen <jarkko@kernel.org> · 1198 unmodified lines | ||
| 26308 | 26326 | R: Jason Gunthorpe <jgg@ziepe.ca> |
| 26309 | 26327 | L: linux-integrity@vger.kernel.org |
| 26310 | 26328 | S: Maintained |
| 26311 | −W: https://codeberg.org/jarkko/linux-tpmdd-test | |
| 26329 | +W: https://git.kernel.org/pub/scm/linux/kernel/git/jarkko/linux-tpmdd-test.git/about/ | |
| 26312 | 26330 | Q: https://patchwork.kernel.org/project/linux-integrity/list/ |
| 26313 | 26331 | T: git git://git.kernel.org/pub/scm/linux/kernel/git/jarkko/linux-tpmdd.git |
| 26314 | 26332 | F: Documentation/devicetree/bindings/tpm/ |
| @@ -232,17 +232,14 @@ config HAVE_EFFICIENT_UNALIGNED_ACCESS · 231 unmodified lines | ||
| 232 | 232 | config ARCH_USE_BUILTIN_BSWAP |
| 233 | 233 | bool |
| 234 | 234 | help |
| 235 | − Modern versions of GCC (since 4.4) have builtin functions | |
| 236 | − for handling byte-swapping. Using these, instead of the old | |
| 237 | − inline assembler that the architecture code provides in the | |
| 238 | − __arch_bswapXX() macros, allows the compiler to see what's | |
| 239 | − happening and offers more opportunity for optimisation. In | |
| 240 | − particular, the compiler will be able to combine the byteswap | |
| 241 | − with a nearby load or store and use load-and-swap or | |
| 242 | − store-and-swap instructions if the architecture has them. It | |
| 243 | − should almost *never* result in code which is worse than the | |
| 244 | − hand-coded assembler in <asm/swab.h>. But just in case it | |
| 245 | − does, the use of the builtins is optional. | |
| 235 | + GCC and Clang have builtin functions for handling byte-swapping. | |
| 236 | + Using these allows the compiler to see what's happening and | |
| 237 | + offers more opportunity for optimisation. In particular, the | |
| 238 | + compiler will be able to combine the byteswap with a nearby load | |
| 239 | + or store and use load-and-swap or store-and-swap instructions if | |
| 240 | + the architecture has them. It should almost *never* result in code | |
| 241 | + which is worse than the hand-coded assembler in <asm/swab.h>. | |
| 242 | + But just in case it does, the use of the builtins is optional. | |
| 246 | 243 | |
| 247 | 244 | Any architecture with load-and-swap or store-and-swap |
| 248 | 245 | instructions should set this. And it shouldn't hurt to set it |
| @@ -1161,8 +1161,6 @@ config AEABI · 1160 unmodified lines | ||
| 1161 | 1161 | disambiguate both ABIs and allow for backward compatibility support |
| 1162 | 1162 | (selected with CONFIG_OABI_COMPAT). |
| 1163 | 1163 | |
| 1164 | − To use this you need GCC version 4.0.0 or later. | |
| 1165 | − | |
| 1166 | 1164 | config OABI_COMPAT |
| 1167 | 1165 | bool "Allow old ABI binaries to run with this kernel (EXPERIMENTAL)" |
| 1168 | 1166 | depends on AEABI && !THUMB2_KERNEL |
| @@ -308,7 +308,7 @@ CONFIG_PANIC_ON_OOPS=y · 307 unmodified lines | ||
| 308 | 308 | CONFIG_PANIC_TIMEOUT=-1 |
| 309 | 309 | CONFIG_SOFTLOCKUP_DETECTOR=y |
| 310 | 310 | CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=y |
| 311 | −CONFIG_BOOTPARAM_HUNG_TASK_PANIC=y | |
| 311 | +CONFIG_BOOTPARAM_HUNG_TASK_PANIC=1 | |
| 312 | 312 | CONFIG_WQ_WATCHDOG=y |
| 313 | 313 | # CONFIG_SCHED_DEBUG is not set |
| 314 | 314 | CONFIG_FUNCTION_TRACER=y |
| @@ -5,4 +5,12 @@ · 4 unmodified lines | ||
| 5 | 5 | /* crash kernel regions are Page size agliged */ |
| 6 | 6 | #define CRASH_ALIGN PAGE_SIZE |
| 7 | 7 | |
| 8 | +#ifdef CONFIG_ARCH_HAS_GENERIC_CRASHKERNEL_RESERVATION | |
| 9 | +static inline bool arch_add_crash_res_to_iomem(void) | |
| 10 | +{ | |
| 11 | + return false; | |
| 12 | +} | |
| 13 | +#define arch_add_crash_res_to_iomem arch_add_crash_res_to_iomem | |
| 14 | +#endif | |
| 15 | + | |
| 8 | 16 | #endif /* _ASM_POWERPC_CRASH_RESERVE_H */ |
| @@ -60,6 +60,12 @@ static inline u64 div_u64_rem(u64 dividend, u32 divisor, u32 *remainder) · 59 unmodified lines | ||
| 60 | 60 | } |
| 61 | 61 | #define div_u64_rem div_u64_rem |
| 62 | 62 | |
| 63 | +/* | |
| 64 | + * gcc tends to zero extend 32bit values and do full 64bit maths. | |
| 65 | + * Define asm functions that avoid this. | |
| 66 | + * (clang generates better code for the C versions.) | |
| 67 | + */ | |
| 68 | +#ifndef __clang__ | |
| 63 | 69 | static inline u64 mul_u32_u32(u32 a, u32 b) |
| 64 | 70 | { |
| 65 | 71 | u32 high, low; |
| @@ -71,6 +77,19 @@ static inline u64 mul_u32_u32(u32 a, u32 b) · 5 unmodified lines | ||
| 71 | 77 | } |
| 72 | 78 | #define mul_u32_u32 mul_u32_u32 |
| 73 | 79 | |
| 80 | +static inline u64 add_u64_u32(u64 a, u32 b) | |
| 81 | +{ | |
| 82 | + u32 high = a >> 32, low = a; | |
| 83 | + | |
| 84 | + asm ("addl %[b], %[low]; adcl $0, %[high]" | |
| 85 | + : [low] "+r" (low), [high] "+r" (high) | |
| 86 | + : [b] "rm" (b) ); | |
| 87 | + | |
| 88 | + return low | (u64)high << 32; | |
| 89 | +} | |
| 90 | +#define add_u64_u32 add_u64_u32 | |
| 91 | +#endif | |
| 92 | + | |
| 74 | 93 | /* |
| 75 | 94 | * __div64_32() is never called on x86, so prevent the |
| 76 | 95 | * generic definition from getting built. |
| @@ -84,21 +103,25 @@ static inline u64 mul_u32_u32(u32 a, u32 b) · 7 unmodified lines | ||
| 84 | 103 | * Will generate an #DE when the result doesn't fit u64, could fix with an |
| 85 | 104 | * __ex_table[] entry when it becomes an issue. |
| 86 | 105 | */ |
| 87 | −static inline u64 mul_u64_u64_div_u64(u64 a, u64 mul, u64 div) | |
| 106 | +static inline u64 mul_u64_add_u64_div_u64(u64 rax, u64 mul, u64 add, u64 div) | |
| 88 | 107 | { |
| 89 | − u64 q; | |
| 108 | + u64 rdx; | |
| 109 | + | |
| 110 | + asm ("mulq %[mul]" : "+a" (rax), "=d" (rdx) : [mul] "rm" (mul)); | |
| 111 | + | |
| 112 | + if (!statically_true(!add)) | |
| 113 | + asm ("addq %[add], %[lo]; adcq $0, %[hi]" : | |
| 114 | + [lo] "+r" (rax), [hi] "+r" (rdx) : [add] "irm" (add)); | |
| 90 | 115 | |
| 91 | − asm ("mulq %2; divq %3" : "=a" (q) | |
| 92 | − : "a" (a), "rm" (mul), "rm" (div) | |
| 93 | − : "rdx"); | |
| 116 | + asm ("divq %[div]" : "+a" (rax), "+d" (rdx) : [div] "rm" (div)); | |
| 94 | 117 | |
| 95 | − return q; | |
| 118 | + return rax; | |
| 96 | 119 | } |
| 97 | −#define mul_u64_u64_div_u64 mul_u64_u64_div_u64 | |
| 120 | +#define mul_u64_add_u64_div_u64 mul_u64_add_u64_div_u64 | |
| 98 | 121 | |
| 99 | 122 | static inline u64 mul_u64_u32_div(u64 a, u32 mul, u32 div) |
| 100 | 123 | { |
| 101 | − return mul_u64_u64_div_u64(a, mul, div); | |
| 124 | + return mul_u64_add_u64_div_u64(a, mul, 0, div); | |
| 102 | 125 | } |
| 103 | 126 | #define mul_u64_u32_div mul_u64_u32_div |
| 104 | 127 | |
| @@ -45,6 +45,7 @@ · 44 unmodified lines | ||
| 45 | 45 | #include <linux/task_work.h> |
| 46 | 46 | #include <linux/hardirq.h> |
| 47 | 47 | #include <linux/kexec.h> |
| 48 | +#include <linux/vmcore_info.h> | |
| 48 | 49 | |
| 49 | 50 | #include <asm/fred.h> |
| 50 | 51 | #include <asm/cpu_device_id.h> |
| @@ -1729,6 +1730,9 @@ noinstr void do_machine_check(struct pt_regs *regs) · 1678 unmodified lines | ||
| 1729 | 1730 | } |
| 1730 | 1731 | |
| 1731 | 1732 | out: |
| 1733 | + /* Given it didn't panic, mark it as recoverable */ | |
| 1734 | + hwerr_log_error_type(HWERR_RECOV_OTHERS); | |
| 1735 | + | |
| 1732 | 1736 | instrumentation_end(); |
| 1733 | 1737 | |
| 1734 | 1738 | clear: |
| @@ -44,6 +44,7 @@ · 43 unmodified lines | ||
| 44 | 44 | #include <linux/uuid.h> |
| 45 | 45 | #include <linux/ras.h> |
| 46 | 46 | #include <linux/task_work.h> |
| 47 | +#include <linux/vmcore_info.h> | |
| 47 | 48 | |
| 48 | 49 | #include <acpi/actbl1.h> |
| 49 | 50 | #include <acpi/ghes.h> |
| @@ -864,6 +865,40 @@ int cxl_cper_kfifo_get(struct cxl_cper_work_data *wd) · 814 unmodified lines | ||
| 864 | 865 | } |
| 865 | 866 | EXPORT_SYMBOL_NS_GPL(cxl_cper_kfifo_get, "CXL"); |
| 866 | 867 | |
| 868 | +static void ghes_log_hwerr(int sev, guid_t *sec_type) | |
| 869 | +{ | |
| 870 | + if (sev != CPER_SEV_RECOVERABLE) | |
| 871 | + return; | |
| 872 | + | |
| 873 | + if (guid_equal(sec_type, &CPER_SEC_PROC_ARM) || | |
| 874 | + guid_equal(sec_type, &CPER_SEC_PROC_GENERIC) || | |
| 875 | + guid_equal(sec_type, &CPER_SEC_PROC_IA)) { | |
| 876 | + hwerr_log_error_type(HWERR_RECOV_CPU); | |
| 877 | + return; | |
| 878 | + } | |
| 879 | + | |
| 880 | + if (guid_equal(sec_type, &CPER_SEC_CXL_PROT_ERR) || | |
| 881 | + guid_equal(sec_type, &CPER_SEC_CXL_GEN_MEDIA_GUID) || | |
| 882 | + guid_equal(sec_type, &CPER_SEC_CXL_DRAM_GUID) || | |
| 883 | + guid_equal(sec_type, &CPER_SEC_CXL_MEM_MODULE_GUID)) { | |
| 884 | + hwerr_log_error_type(HWERR_RECOV_CXL); | |
| 885 | + return; | |
| 886 | + } | |
| 887 | + | |
| 888 | + if (guid_equal(sec_type, &CPER_SEC_PCIE) || | |
| 889 | + guid_equal(sec_type, &CPER_SEC_PCI_X_BUS)) { | |
| 890 | + hwerr_log_error_type(HWERR_RECOV_PCI); | |
| 891 | + return; | |
| 892 | + } | |
| 893 | + | |
| 894 | + if (guid_equal(sec_type, &CPER_SEC_PLATFORM_MEM)) { | |
| 895 | + hwerr_log_error_type(HWERR_RECOV_MEMORY); | |
| 896 | + return; | |
| 897 | + } | |
| 898 | + | |
| 899 | + hwerr_log_error_type(HWERR_RECOV_OTHERS); | |
| 900 | +} | |
| 901 | + | |
| 867 | 902 | static void ghes_do_proc(struct ghes *ghes, |
| 868 | 903 | const struct acpi_hest_generic_status *estatus) |
| 869 | 904 | { |
| @@ -885,6 +920,7 @@ static void ghes_do_proc(struct ghes *ghes, · 15 unmodified lines | ||
| 885 | 920 | if (gdata->validation_bits & CPER_SEC_VALID_FRU_TEXT) |
| 886 | 921 | fru_text = gdata->fru_text; |
| 887 | 922 | |
| 923 | + ghes_log_hwerr(sev, sec_type); | |
| 888 | 924 | if (guid_equal(sec_type, &CPER_SEC_PLATFORM_MEM)) { |
| 889 | 925 | struct cper_sec_mem_err *mem_err = acpi_hest_get_payload(gdata); |
| 890 | 926 | |
| @@ -178,7 +178,7 @@ struct nvme_dhchap_key *nvme_auth_extract_key(unsigned char *secret, · 177 unmodified lines | ||
| 178 | 178 | if (!key) |
| 179 | 179 | return ERR_PTR(-ENOMEM); |
| 180 | 180 | |
| 181 | − key_len = base64_decode(secret, allocated_len, key->key); | |
| 181 | + key_len = base64_decode(secret, allocated_len, key->key, true, BASE64_STD); | |
| 182 | 182 | if (key_len < 0) { |
| 183 | 183 | pr_debug("base64 key decoding error %d\n", |
| 184 | 184 | key_len); |
| @@ -663,7 +663,7 @@ int nvme_auth_generate_digest(u8 hmac_id, u8 *psk, size_t psk_len, · 478 unmodified lines | ||
| 663 | 663 | if (ret) |
| 664 | 664 | goto out_free_digest; |
| 665 | 665 | |
| 666 | − ret = base64_encode(digest, digest_len, enc); | |
| 666 | + ret = base64_encode(digest, digest_len, enc, true, BASE64_STD); | |
| 667 | 667 | if (ret < hmac_len) { |
| 668 | 668 | ret = -ENOKEY; |
| 669 | 669 | goto out_free_digest; |
| @@ -30,6 +30,7 @@ · 29 unmodified lines | ||
| 30 | 30 | #include <linux/kfifo.h> |
| 31 | 31 | #include <linux/ratelimit.h> |
| 32 | 32 | #include <linux/slab.h> |
| 33 | +#include <linux/vmcore_info.h> | |
| 33 | 34 | #include <acpi/apei.h> |
| 34 | 35 | #include <acpi/ghes.h> |
| 35 | 36 | #include <ras/ras_event.h> |
| @@ -765,6 +766,7 @@ static void pci_dev_aer_stats_incr(struct pci_dev *pdev, · 729 unmodified lines | ||
| 765 | 766 | break; |
| 766 | 767 | case AER_NONFATAL: |
| 767 | 768 | aer_info->dev_total_nonfatal_errs++; |
| 769 | + hwerr_log_error_type(HWERR_RECOV_PCI); | |
| 768 | 770 | counter = &aer_info->dev_nonfatal_errs[0]; |
| 769 | 771 | max = AER_MAX_TYPEOF_UNCOR_ERRS; |
| 770 | 772 | break; |
| @@ -15,59 +15,6 @@ · 14 unmodified lines | ||
| 15 | 15 | #include "mds_client.h" |
| 16 | 16 | #include "crypto.h" |
| 17 | 17 | |
| 18 | −/* | |
| 19 | − * The base64url encoding used by fscrypt includes the '_' character, which may | |
| 20 | − * cause problems in snapshot names (which can not start with '_'). Thus, we | |
| 21 | − * used the base64 encoding defined for IMAP mailbox names (RFC 3501) instead, | |
| 22 | − * which replaces '-' and '_' by '+' and ','. | |
| 23 | − */ | |
| 24 | −static const char base64_table[65] = | |
| 25 | − "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+,"; | |
| 26 | − | |
| 27 | −int ceph_base64_encode(const u8 *src, int srclen, char *dst) | |
| 28 | −{ | |
| 29 | − u32 ac = 0; | |
| 30 | − int bits = 0; | |
| 31 | − int i; | |
| 32 | − char *cp = dst; | |
| 33 | − | |
| 34 | − for (i = 0; i < srclen; i++) { | |
| 35 | − ac = (ac << 8) | src[i]; | |
| 36 | − bits += 8; | |
| 37 | − do { | |
| 38 | − bits -= 6; | |
| 39 | − *cp++ = base64_table[(ac >> bits) & 0x3f]; | |
| 40 | − } while (bits >= 6); | |
| 41 | − } | |
| 42 | − if (bits) | |
| 43 | − *cp++ = base64_table[(ac << (6 - bits)) & 0x3f]; | |
| 44 | − return cp - dst; | |
| 45 | −} | |
| 46 | − | |
| 47 | −int ceph_base64_decode(const char *src, int srclen, u8 *dst) | |
| 48 | −{ | |
| 49 | − u32 ac = 0; | |
| 50 | − int bits = 0; | |
| 51 | − int i; | |
| 52 | − u8 *bp = dst; | |
| 53 | − | |
| 54 | − for (i = 0; i < srclen; i++) { | |
| 55 | − const char *p = strchr(base64_table, src[i]); | |
| 56 | − | |
| 57 | − if (p == NULL || src[i] == 0) | |
| 58 | − return -1; | |
| 59 | − ac = (ac << 6) | (p - base64_table); | |
| 60 | − bits += 6; | |
| 61 | − if (bits >= 8) { | |
| 62 | − bits -= 8; | |
| 63 | − *bp++ = (u8)(ac >> bits); | |
| 64 | − } | |
| 65 | − } | |
| 66 | − if (ac & ((1 << bits) - 1)) | |
| 67 | − return -1; | |
| 68 | − return bp - dst; | |
| 69 | −} | |
| 70 | − | |
| 71 | 18 | static int ceph_crypt_get_context(struct inode *inode, void *ctx, size_t len) |
| 72 | 19 | { |
| 73 | 20 | struct ceph_inode_info *ci = ceph_inode(inode); |
| @@ -318,7 +265,7 @@ int ceph_encode_encrypted_dname(struct inode *parent, char *buf, int elen) · 244 unmodified lines | ||
| 318 | 265 | } |
| 319 | 266 | |
| 320 | 267 | /* base64 encode the encrypted name */ |
| 321 | − elen = ceph_base64_encode(cryptbuf, len, p); | |
| 268 | + elen = base64_encode(cryptbuf, len, p, false, BASE64_IMAP); | |
| 322 | 269 | doutc(cl, "base64-encoded ciphertext name = %.*s\n", elen, p); |
| 323 | 270 | |
| 324 | 271 | /* To understand the 240 limit, see CEPH_NOHASH_NAME_MAX comments */ |
| @@ -412,7 +359,8 @@ int ceph_fname_to_usr(const struct ceph_fname *fname, struct fscrypt_str *tname, · 87 unmodified lines | ||
| 412 | 359 | tname = &_tname; |
| 413 | 360 | } |
| 414 | 361 | |
| 415 | − declen = ceph_base64_decode(name, name_len, tname->name); | |
| 362 | + declen = base64_decode(name, name_len, | |
| 363 | + tname->name, false, BASE64_IMAP); | |
| 416 | 364 | if (declen <= 0) { |
| 417 | 365 | ret = -EIO; |
| 418 | 366 | goto out; |
| @@ -426,7 +374,7 @@ int ceph_fname_to_usr(const struct ceph_fname *fname, struct fscrypt_str *tname, · 7 unmodified lines | ||
| 426 | 374 | |
| 427 | 375 | ret = fscrypt_fname_disk_to_usr(dir, 0, 0, &iname, oname); |
| 428 | 376 | if (!ret && (dir != fname->dir)) { |
| 429 | − char tmp_buf[CEPH_BASE64_CHARS(NAME_MAX)]; | |
| 377 | + char tmp_buf[BASE64_CHARS(NAME_MAX)]; | |
| 430 | 378 | |
| 431 | 379 | name_len = snprintf(tmp_buf, sizeof(tmp_buf), "_%.*s_%ld", |
| 432 | 380 | oname->len, oname->name, dir->i_ino); |
| @@ -8,6 +8,7 @@ · 7 unmodified lines | ||
| 8 | 8 | |
| 9 | 9 | #include <crypto/sha2.h> |
| 10 | 10 | #include <linux/fscrypt.h> |
| 11 | +#include <linux/base64.h> | |
| 11 | 12 | |
| 12 | 13 | #define CEPH_FSCRYPT_BLOCK_SHIFT 12 |
| 13 | 14 | #define CEPH_FSCRYPT_BLOCK_SIZE (_AC(1, UL) << CEPH_FSCRYPT_BLOCK_SHIFT) |
| @@ -89,11 +90,6 @@ static inline u32 ceph_fscrypt_auth_len(struct ceph_fscrypt_auth *fa) · 75 unmodified lines | ||
| 89 | 90 | */ |
| 90 | 91 | #define CEPH_NOHASH_NAME_MAX (180 - SHA256_DIGEST_SIZE) |
| 91 | 92 | |
| 92 | −#define CEPH_BASE64_CHARS(nbytes) DIV_ROUND_UP((nbytes) * 4, 3) | |
| 93 | − | |
| 94 | −int ceph_base64_encode(const u8 *src, int srclen, char *dst); | |
| 95 | −int ceph_base64_decode(const char *src, int srclen, u8 *dst); | |
| 96 | − | |
| 97 | 93 | void ceph_fscrypt_set_ops(struct super_block *sb); |
| 98 | 94 | |
| 99 | 95 | void ceph_fscrypt_free_dummy_policy(struct ceph_fs_client *fsc); |
| @@ -998,13 +998,14 @@ static int prep_encrypted_symlink_target(struct ceph_mds_request *req, · 997 unmodified lines | ||
| 998 | 998 | if (err) |
| 999 | 999 | goto out; |
| 1000 | 1000 | |
| 1001 | − req->r_path2 = kmalloc(CEPH_BASE64_CHARS(osd_link.len) + 1, GFP_KERNEL); | |
| 1001 | + req->r_path2 = kmalloc(BASE64_CHARS(osd_link.len) + 1, GFP_KERNEL); | |
| 1002 | 1002 | if (!req->r_path2) { |
| 1003 | 1003 | err = -ENOMEM; |
| 1004 | 1004 | goto out; |
| 1005 | 1005 | } |
| 1006 | 1006 | |
| 1007 | − len = ceph_base64_encode(osd_link.name, osd_link.len, req->r_path2); | |
| 1007 | + len = base64_encode(osd_link.name, osd_link.len, | |
| 1008 | + req->r_path2, false, BASE64_IMAP); | |
| 1008 | 1009 | req->r_path2[len] = '\0'; |
| 1009 | 1010 | out: |
| 1010 | 1011 | fscrypt_fname_free_buffer(&osd_link); |
| @@ -947,7 +947,7 @@ static int decode_encrypted_symlink(struct ceph_mds_client *mdsc, · 946 unmodified lines | ||
| 947 | 947 | if (!sym) |
| 948 | 948 | return -ENOMEM; |
| 949 | 949 | |
| 950 | − declen = ceph_base64_decode(encsym, enclen, sym); | |
| 950 | + declen = base64_decode(encsym, enclen, sym, false, BASE64_IMAP); | |
| 951 | 951 | if (declen < 0) { |
| 952 | 952 | pr_err_client(cl, |
| 953 | 953 | "can't decode symlink (%d). Content: %.*s\n", |
| @@ -16,6 +16,7 @@ · 15 unmodified lines | ||
| 16 | 16 | #include <linux/export.h> |
| 17 | 17 | #include <linux/namei.h> |
| 18 | 18 | #include <linux/scatterlist.h> |
| 19 | +#include <linux/base64.h> | |
| 19 | 20 | |
| 20 | 21 | #include "fscrypt_private.h" |
| 21 | 22 | |
| @@ -71,7 +72,7 @@ struct fscrypt_nokey_name { · 49 unmodified lines | ||
| 71 | 72 | |
| 72 | 73 | /* Encoded size of max-size no-key name */ |
| 73 | 74 | #define FSCRYPT_NOKEY_NAME_MAX_ENCODED \ |
| 74 | − FSCRYPT_BASE64URL_CHARS(FSCRYPT_NOKEY_NAME_MAX) | |
| 75 | + BASE64_CHARS(FSCRYPT_NOKEY_NAME_MAX) | |
| 75 | 76 | |
| 76 | 77 | static inline bool fscrypt_is_dot_dotdot(const struct qstr *str) |
| 77 | 78 | { |
| @@ -162,84 +163,6 @@ static int fname_decrypt(const struct inode *inode, · 84 unmodified lines | ||
| 162 | 163 | return 0; |
| 163 | 164 | } |
| 164 | 165 | |
| 165 | −static const char base64url_table[65] = | |
| 166 | − "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_"; | |
| 167 | − | |
| 168 | −#define FSCRYPT_BASE64URL_CHARS(nbytes) DIV_ROUND_UP((nbytes) * 4, 3) | |
| 169 | − | |
| 170 | −/** | |
| 171 | − * fscrypt_base64url_encode() - base64url-encode some binary data | |
| 172 | − * @src: the binary data to encode | |
| 173 | − * @srclen: the length of @src in bytes | |
| 174 | − * @dst: (output) the base64url-encoded string. Not NUL-terminated. | |
| 175 | − * | |
| 176 | − * Encodes data using base64url encoding, i.e. the "Base 64 Encoding with URL | |
| 177 | − * and Filename Safe Alphabet" specified by RFC 4648. '='-padding isn't used, | |
| 178 | − * as it's unneeded and not required by the RFC. base64url is used instead of | |
| 179 | − * base64 to avoid the '/' character, which isn't allowed in filenames. | |
| 180 | − * | |
| 181 | − * Return: the length of the resulting base64url-encoded string in bytes. | |
| 182 | − * This will be equal to FSCRYPT_BASE64URL_CHARS(srclen). | |
| 183 | − */ | |
| 184 | −static int fscrypt_base64url_encode(const u8 *src, int srclen, char *dst) | |
| 185 | −{ | |
| 186 | − u32 ac = 0; | |
| 187 | − int bits = 0; | |
| 188 | − int i; | |
| 189 | − char *cp = dst; | |
| 190 | − | |
| 191 | − for (i = 0; i < srclen; i++) { | |
| 192 | − ac = (ac << 8) | src[i]; | |
| 193 | − bits += 8; | |
| 194 | − do { | |
| 195 | − bits -= 6; | |
| 196 | − *cp++ = base64url_table[(ac >> bits) & 0x3f]; | |
| 197 | − } while (bits >= 6); | |
| 198 | − } | |
| 199 | − if (bits) | |
| 200 | − *cp++ = base64url_table[(ac << (6 - bits)) & 0x3f]; | |
| 201 | − return cp - dst; | |
| 202 | −} | |
| 203 | − | |
| 204 | −/** | |
| 205 | − * fscrypt_base64url_decode() - base64url-decode a string | |
| 206 | − * @src: the string to decode. Doesn't need to be NUL-terminated. | |
| 207 | − * @srclen: the length of @src in bytes | |
| 208 | − * @dst: (output) the decoded binary data | |
| 209 | − * | |
| 210 | − * Decodes a string using base64url encoding, i.e. the "Base 64 Encoding with | |
| 211 | − * URL and Filename Safe Alphabet" specified by RFC 4648. '='-padding isn't | |
| 212 | − * accepted, nor are non-encoding characters such as whitespace. | |
| 213 | − * | |
| 214 | − * This implementation hasn't been optimized for performance. | |
| 215 | − * | |
| 216 | − * Return: the length of the resulting decoded binary data in bytes, | |
| 217 | − * or -1 if the string isn't a valid base64url string. | |
| 218 | − */ | |
| 219 | −static int fscrypt_base64url_decode(const char *src, int srclen, u8 *dst) | |
| 220 | −{ | |
| 221 | − u32 ac = 0; | |
| 222 | − int bits = 0; | |
| 223 | − int i; | |
| 224 | − u8 *bp = dst; | |
| 225 | − | |
| 226 | − for (i = 0; i < srclen; i++) { | |
| 227 | − const char *p = strchr(base64url_table, src[i]); | |
| 228 | − | |
| 229 | − if (p == NULL || src[i] == 0) | |
| 230 | − return -1; | |
| 231 | − ac = (ac << 6) | (p - base64url_table); | |
| 232 | − bits += 6; | |
| 233 | − if (bits >= 8) { | |
| 234 | − bits -= 8; | |
| 235 | − *bp++ = (u8)(ac >> bits); | |
| 236 | − } | |
| 237 | − } | |
| 238 | − if (ac & ((1 << bits) - 1)) | |
| 239 | − return -1; | |
| 240 | − return bp - dst; | |
| 241 | −} | |
| 242 | − | |
| 243 | 166 | bool __fscrypt_fname_encrypted_size(const union fscrypt_policy *policy, |
| 244 | 167 | u32 orig_len, u32 max_len, |
| 245 | 168 | u32 *encrypted_len_ret) |
| @@ -387,8 +310,8 @@ int fscrypt_fname_disk_to_usr(const struct inode *inode, · 141 unmodified lines | ||
| 387 | 310 | nokey_name.sha256); |
| 388 | 311 | size = FSCRYPT_NOKEY_NAME_MAX; |
| 389 | 312 | } |
| 390 | − oname->len = fscrypt_base64url_encode((const u8 *)&nokey_name, size, | |
| 391 | − oname->name); | |
| 313 | + oname->len = base64_encode((const u8 *)&nokey_name, size, | |
| 314 | + oname->name, false, BASE64_URLSAFE); | |
| 392 | 315 | return 0; |
| 393 | 316 | } |
| 394 | 317 | EXPORT_SYMBOL(fscrypt_fname_disk_to_usr); |
| @@ -467,8 +390,8 @@ int fscrypt_setup_filename(struct inode *dir, const struct qstr *iname, · 72 unmodified lines | ||
| 467 | 390 | if (fname->crypto_buf.name == NULL) |
| 468 | 391 | return -ENOMEM; |
| 469 | 392 | |
| 470 | − ret = fscrypt_base64url_decode(iname->name, iname->len, | |
| 471 | − fname->crypto_buf.name); | |
| 393 | + ret = base64_decode(iname->name, iname->len, | |
| 394 | + fname->crypto_buf.name, false, BASE64_URLSAFE); | |
| 472 | 395 | if (ret < (int)offsetof(struct fscrypt_nokey_name, bytes[1]) || |
| 473 | 396 | (ret > offsetof(struct fscrypt_nokey_name, sha256) && |
| 474 | 397 | ret != FSCRYPT_NOKEY_NAME_MAX)) { |
| @@ -49,7 +49,7 @@ static int nilfs_ioctl_wrap_copy(struct the_nilfs *nilfs, · 48 unmodified lines | ||
| 49 | 49 | void *, size_t, size_t)) |
| 50 | 50 | { |
| 51 | 51 | void *buf; |
| 52 | − void __user *base = (void __user *)(unsigned long)argv->v_base; | |
| 52 | + void __user *base = u64_to_user_ptr(argv->v_base); | |
| 53 | 53 | size_t maxmembs, total, n; |
| 54 | 54 | ssize_t nr; |
| 55 | 55 | int ret, i; |
| @@ -836,7 +836,6 @@ static int nilfs_ioctl_clean_segments(struct inode *inode, struct file *filp, · 780 unmodified lines | ||
| 836 | 836 | sizeof(struct nilfs_bdesc), |
| 837 | 837 | sizeof(__u64), |
| 838 | 838 | }; |
| 839 | − void __user *base; | |
| 840 | 839 | void *kbufs[5]; |
| 841 | 840 | struct the_nilfs *nilfs; |
| 842 | 841 | size_t len, nsegs; |
| @@ -863,7 +862,7 @@ static int nilfs_ioctl_clean_segments(struct inode *inode, struct file *filp, · 20 unmodified lines | ||
| 863 | 862 | * use kmalloc() for its buffer because the memory used for the |
| 864 | 863 | * segment numbers is small enough. |
| 865 | 864 | */ |
| 866 | − kbufs[4] = memdup_array_user((void __user *)(unsigned long)argv[4].v_base, | |
| 865 | + kbufs[4] = memdup_array_user(u64_to_user_ptr(argv[4].v_base), | |
| 867 | 866 | nsegs, sizeof(__u64)); |
| 868 | 867 | if (IS_ERR(kbufs[4])) { |
| 869 | 868 | ret = PTR_ERR(kbufs[4]); |
| @@ -883,20 +882,14 @@ static int nilfs_ioctl_clean_segments(struct inode *inode, struct file *filp, · 13 unmodified lines | ||
| 883 | 882 | goto out_free; |
| 884 | 883 | |
| 885 | 884 | len = argv[n].v_size * argv[n].v_nmembs; |
| 886 | − base = (void __user *)(unsigned long)argv[n].v_base; | |
| 887 | 885 | if (len == 0) { |
| 888 | 886 | kbufs[n] = NULL; |
| 889 | 887 | continue; |
| 890 | 888 | } |
| 891 | 889 | |
| 892 | − kbufs[n] = vmalloc(len); | |
| 893 | − if (!kbufs[n]) { | |
| 894 | − ret = -ENOMEM; | |
| 895 | − goto out_free; | |
| 896 | − } | |
| 897 | − if (copy_from_user(kbufs[n], base, len)) { | |
| 898 | − ret = -EFAULT; | |
| 899 | − vfree(kbufs[n]); | |
| 890 | + kbufs[n] = vmemdup_user(u64_to_user_ptr(argv[n].v_base), len); | |
| 891 | + if (IS_ERR(kbufs[n])) { | |
| 892 | + ret = PTR_ERR(kbufs[n]); | |
| 900 | 893 | goto out_free; |
| 901 | 894 | } |
| 902 | 895 | } |
| @@ -928,7 +921,7 @@ static int nilfs_ioctl_clean_segments(struct inode *inode, struct file *filp, · 25 unmodified lines | ||
| 928 | 921 | |
| 929 | 922 | out_free: |
| 930 | 923 | while (--n >= 0) |
| 931 | − vfree(kbufs[n]); | |
| 924 | + kvfree(kbufs[n]); | |
| 932 | 925 | kfree(kbufs[4]); |
| 933 | 926 | out: |
| 934 | 927 | mnt_drop_write_file(filp); |
| @@ -1181,7 +1174,6 @@ static int nilfs_ioctl_set_suinfo(struct inode *inode, struct file *filp, · 246 unmodified lines | ||
| 1181 | 1174 | struct nilfs_transaction_info ti; |
| 1182 | 1175 | struct nilfs_argv argv; |
| 1183 | 1176 | size_t len; |
| 1184 | − void __user *base; | |
| 1185 | 1177 | void *kbuf; |
| 1186 | 1178 | int ret; |
| 1187 | 1179 | |
| @@ -1212,18 +1204,12 @@ static int nilfs_ioctl_set_suinfo(struct inode *inode, struct file *filp, · 24 unmodified lines | ||
| 1212 | 1204 | goto out; |
| 1213 | 1205 | } |
| 1214 | 1206 | |
| 1215 | − base = (void __user *)(unsigned long)argv.v_base; | |
| 1216 | − kbuf = vmalloc(len); | |
| 1217 | − if (!kbuf) { | |
| 1218 | − ret = -ENOMEM; | |
| 1207 | + kbuf = vmemdup_user(u64_to_user_ptr(argv.v_base), len); | |
| 1208 | + if (IS_ERR(kbuf)) { | |
| 1209 | + ret = PTR_ERR(kbuf); | |
| 1219 | 1210 | goto out; |
| 1220 | 1211 | } |
| 1221 | 1212 | |
| 1222 | − if (copy_from_user(kbuf, base, len)) { | |
| 1223 | − ret = -EFAULT; | |
| 1224 | − goto out_free; | |
| 1225 | − } | |
| 1226 | − | |
| 1227 | 1213 | nilfs_transaction_begin(inode->i_sb, &ti, 0); |
| 1228 | 1214 | ret = nilfs_sufile_set_suinfo(nilfs->ns_sufile, kbuf, argv.v_size, |
| 1229 | 1215 | argv.v_nmembs); |
| @@ -1232,8 +1218,7 @@ static int nilfs_ioctl_set_suinfo(struct inode *inode, struct file *filp, · 2 unmodified lines | ||
| 1232 | 1218 | else |
| 1233 | 1219 | nilfs_transaction_commit(inode->i_sb); /* never fails */ |
| 1234 | 1220 | |
| 1235 | −out_free: | |
| 1236 | − vfree(kbuf); | |
| 1221 | + kvfree(kbuf); | |
| 1237 | 1222 | out: |
| 1238 | 1223 | mnt_drop_write_file(filp); |
| 1239 | 1224 | return ret; |
| @@ -302,8 +302,21 @@ static int ocfs2_check_dir_entry(struct inode *dir, · 301 unmodified lines | ||
| 302 | 302 | unsigned long offset) |
| 303 | 303 | { |
| 304 | 304 | const char *error_msg = NULL; |
| 305 | − const int rlen = le16_to_cpu(de->rec_len); | |
| 306 | − const unsigned long next_offset = ((char *) de - buf) + rlen; | |
| 305 | + unsigned long next_offset; | |
| 306 | + int rlen; | |
| 307 | + | |
| 308 | + if (offset > size - OCFS2_DIR_REC_LEN(1)) { | |
| 309 | + /* Dirent is (maybe partially) beyond the buffer | |
| 310 | + * boundaries so touching 'de' members is unsafe. | |
| 311 | + */ | |
| 312 | + mlog(ML_ERROR, "directory entry (#%llu: offset=%lu) " | |
| 313 | + "too close to end or out-of-bounds", | |
| 314 | + (unsigned long long)OCFS2_I(dir)->ip_blkno, offset); | |
| 315 | + return 0; | |
| 316 | + } | |
| 317 | + | |
| 318 | + rlen = le16_to_cpu(de->rec_len); | |
| 319 | + next_offset = ((char *) de - buf) + rlen; | |
| 307 | 320 | |
| 308 | 321 | if (unlikely(rlen < OCFS2_DIR_REC_LEN(1))) |
| 309 | 322 | error_msg = "rec_len is smaller than minimal"; |
| @@ -778,6 +791,14 @@ static int ocfs2_dx_dir_lookup_rec(struct inode *inode, · 468 unmodified lines | ||
| 778 | 791 | struct ocfs2_extent_block *eb; |
| 779 | 792 | struct ocfs2_extent_rec *rec = NULL; |
| 780 | 793 | |
| 794 | + if (le16_to_cpu(el->l_count) != | |
| 795 | + ocfs2_extent_recs_per_dx_root(inode->i_sb)) { | |
| 796 | + ret = ocfs2_error(inode->i_sb, | |
| 797 | + "Inode %lu has invalid extent list length %u\n", | |
| 798 | + inode->i_ino, le16_to_cpu(el->l_count)); | |
| 799 | + goto out; | |
| 800 | + } | |
| 801 | + | |
| 781 | 802 | if (el->l_tree_depth) { |
| 782 | 803 | ret = ocfs2_find_leaf(INODE_CACHE(inode), el, major_hash, |
| 783 | 804 | &eb_bh); |
| @@ -3423,6 +3444,14 @@ static int ocfs2_find_dir_space_id(struct inode *dir, struct buffer_head *di_bh, · 2639 unmodified lines | ||
| 3423 | 3444 | offset += le16_to_cpu(de->rec_len); |
| 3424 | 3445 | } |
| 3425 | 3446 | |
| 3447 | + if (!last_de) { | |
| 3448 | + ret = ocfs2_error(sb, "Directory entry (#%llu: size=%lld) " | |
| 3449 | + "is unexpectedly short", | |
| 3450 | + (unsigned long long)OCFS2_I(dir)->ip_blkno, | |
| 3451 | + i_size_read(dir)); | |
| 3452 | + goto out; | |
| 3453 | + } | |
| 3454 | + | |
| 3426 | 3455 | /* |
| 3427 | 3456 | * We're going to require expansion of the directory - figure |
| 3428 | 3457 | * out how many blocks we'll need so that a place for the |
| @@ -4104,10 +4133,15 @@ static int ocfs2_expand_inline_dx_root(struct inode *dir, · 675 unmodified lines | ||
| 4104 | 4133 | } |
| 4105 | 4134 | |
| 4106 | 4135 | dx_root->dr_flags &= ~OCFS2_DX_FLAG_INLINE; |
| 4107 | − memset(&dx_root->dr_list, 0, osb->sb->s_blocksize - | |
| 4108 | − offsetof(struct ocfs2_dx_root_block, dr_list)); | |
| 4136 | + | |
| 4137 | + dx_root->dr_list.l_tree_depth = 0; | |
| 4109 | 4138 | dx_root->dr_list.l_count = |
| 4110 | 4139 | cpu_to_le16(ocfs2_extent_recs_per_dx_root(osb->sb)); |
| 4140 | + dx_root->dr_list.l_next_free_rec = 0; | |
| 4141 | + memset(&dx_root->dr_list.l_recs, 0, | |
| 4142 | + osb->sb->s_blocksize - | |
| 4143 | + (offsetof(struct ocfs2_dx_root_block, dr_list) + | |
| 4144 | + offsetof(struct ocfs2_extent_list, l_recs))); | |
| 4111 | 4145 | |
| 4112 | 4146 | /* This should never fail considering we start with an empty |
| 4113 | 4147 | * dx_root. */ |
| @@ -201,13 +201,15 @@ bail: · 200 unmodified lines | ||
| 201 | 201 | static int ocfs2_dinode_has_extents(struct ocfs2_dinode *di) |
| 202 | 202 | { |
| 203 | 203 | /* inodes flagged with other stuff in id2 */ |
| 204 | − if (di->i_flags & (OCFS2_SUPER_BLOCK_FL | OCFS2_LOCAL_ALLOC_FL | | |
| 205 | − OCFS2_CHAIN_FL | OCFS2_DEALLOC_FL)) | |
| 204 | + if (le32_to_cpu(di->i_flags) & | |
| 205 | + (OCFS2_SUPER_BLOCK_FL | OCFS2_LOCAL_ALLOC_FL | OCFS2_CHAIN_FL | | |
| 206 | + OCFS2_DEALLOC_FL)) | |
| 206 | 207 | return 0; |
| 207 | 208 | /* i_flags doesn't indicate when id2 is a fast symlink */ |
| 208 | − if (S_ISLNK(di->i_mode) && di->i_size && di->i_clusters == 0) | |
| 209 | + if (S_ISLNK(le16_to_cpu(di->i_mode)) && le64_to_cpu(di->i_size) && | |
| 210 | + !le32_to_cpu(di->i_clusters)) | |
| 209 | 211 | return 0; |
| 210 | − if (di->i_dyn_features & OCFS2_INLINE_DATA_FL) | |
| 212 | + if (le16_to_cpu(di->i_dyn_features) & OCFS2_INLINE_DATA_FL) | |
| 211 | 213 | return 0; |
| 212 | 214 | |
| 213 | 215 | return 1; |
| @@ -1460,7 +1462,7 @@ int ocfs2_validate_inode_block(struct super_block *sb, · 1246 unmodified lines | ||
| 1460 | 1462 | goto bail; |
| 1461 | 1463 | } |
| 1462 | 1464 | |
| 1463 | − if (!(di->i_flags & cpu_to_le32(OCFS2_VALID_FL))) { | |
| 1465 | + if (!(le32_to_cpu(di->i_flags) & OCFS2_VALID_FL)) { | |
| 1464 | 1466 | rc = ocfs2_error(sb, |
| 1465 | 1467 | "Invalid dinode #%llu: OCFS2_VALID_FL not set\n", |
| 1466 | 1468 | (unsigned long long)bh->b_blocknr); |
| @@ -1484,6 +1486,41 @@ int ocfs2_validate_inode_block(struct super_block *sb, · 17 unmodified lines | ||
| 1484 | 1486 | goto bail; |
| 1485 | 1487 | } |
| 1486 | 1488 | |
| 1489 | + if ((le16_to_cpu(di->i_dyn_features) & OCFS2_INLINE_DATA_FL) && | |
| 1490 | + le32_to_cpu(di->i_clusters)) { | |
| 1491 | + rc = ocfs2_error(sb, "Invalid dinode %llu: %u clusters\n", | |
| 1492 | + (unsigned long long)bh->b_blocknr, | |
| 1493 | + le32_to_cpu(di->i_clusters)); | |
| 1494 | + goto bail; | |
| 1495 | + } | |
| 1496 | + | |
| 1497 | + if (le32_to_cpu(di->i_flags) & OCFS2_CHAIN_FL) { | |
| 1498 | + struct ocfs2_chain_list *cl = &di->id2.i_chain; | |
| 1499 | + u16 bpc = 1 << (OCFS2_SB(sb)->s_clustersize_bits - | |
| 1500 | + sb->s_blocksize_bits); | |
| 1501 | + | |
| 1502 | + if (le16_to_cpu(cl->cl_count) != ocfs2_chain_recs_per_inode(sb)) { | |
| 1503 | + rc = ocfs2_error(sb, "Invalid dinode %llu: chain list count %u\n", | |
| 1504 | + (unsigned long long)bh->b_blocknr, | |
| 1505 | + le16_to_cpu(cl->cl_count)); | |
| 1506 | + goto bail; | |
| 1507 | + } | |
| 1508 | + if (le16_to_cpu(cl->cl_next_free_rec) > le16_to_cpu(cl->cl_count)) { | |
| 1509 | + rc = ocfs2_error(sb, "Invalid dinode %llu: chain list index %u\n", | |
| 1510 | + (unsigned long long)bh->b_blocknr, | |
| 1511 | + le16_to_cpu(cl->cl_next_free_rec)); | |
| 1512 | + goto bail; | |
| 1513 | + } | |
| 1514 | + if (OCFS2_SB(sb)->bitmap_blkno && | |
| 1515 | + OCFS2_SB(sb)->bitmap_blkno != le64_to_cpu(di->i_blkno) && | |
| 1516 | + le16_to_cpu(cl->cl_bpc) != bpc) { | |
| 1517 | + rc = ocfs2_error(sb, "Invalid dinode %llu: bits per cluster %u\n", | |
| 1518 | + (unsigned long long)bh->b_blocknr, | |
| 1519 | + le16_to_cpu(cl->cl_bpc)); | |
| 1520 | + goto bail; | |
| 1521 | + } | |
| 1522 | + } | |
| 1523 | + | |
| 1487 | 1524 | rc = 0; |
| 1488 | 1525 | |
| 1489 | 1526 | bail: |
| @@ -1671,6 +1708,8 @@ int ocfs2_read_inode_block_full(struct inode *inode, struct buffer_head **bh, · 181 unmodified lines | ||
| 1671 | 1708 | rc = ocfs2_read_blocks(INODE_CACHE(inode), OCFS2_I(inode)->ip_blkno, |
| 1672 | 1709 | 1, &tmp, flags, ocfs2_validate_inode_block); |
| 1673 | 1710 | |
| 1711 | + if (rc < 0) | |
| 1712 | + make_bad_inode(inode); | |
| 1674 | 1713 | /* If ocfs2_read_blocks() got us a new bh, pass it up. */ |
| 1675 | 1714 | if (!rc && !*bh) |
| 1676 | 1715 | *bh = tmp; |
| @@ -98,7 +98,13 @@ static int __ocfs2_move_extent(handle_t *handle, · 97 unmodified lines | ||
| 98 | 98 | |
| 99 | 99 | rec = &el->l_recs[index]; |
| 100 | 100 | |
| 101 | − BUG_ON(ext_flags != rec->e_flags); | |
| 101 | + if (ext_flags != rec->e_flags) { | |
| 102 | + ret = ocfs2_error(inode->i_sb, | |
| 103 | + "Inode %llu has corrupted extent %d with flags 0x%x at cpos %u\n", | |
| 104 | + (unsigned long long)ino, index, rec->e_flags, cpos); | |
| 105 | + goto out; | |
| 106 | + } | |
| 107 | + | |
| 102 | 108 | /* |
| 103 | 109 | * after moving/defraging to new location, the extent is not going |
| 104 | 110 | * to be refcounted anymore. |
| @@ -1036,6 +1042,12 @@ int ocfs2_ioctl_move_extents(struct file *filp, void __user *argp) · 931 unmodified lines | ||
| 1036 | 1042 | if (range.me_threshold > i_size_read(inode)) |
| 1037 | 1043 | range.me_threshold = i_size_read(inode); |
| 1038 | 1044 | |
| 1045 | + if (range.me_flags & ~(OCFS2_MOVE_EXT_FL_AUTO_DEFRAG | | |
| 1046 | + OCFS2_MOVE_EXT_FL_PART_DEFRAG)) { | |
| 1047 | + status = -EINVAL; | |
| 1048 | + goto out_free; | |
| 1049 | + } | |
| 1050 | + | |
| 1039 | 1051 | if (range.me_flags & OCFS2_MOVE_EXT_FL_AUTO_DEFRAG) { |
| 1040 | 1052 | context->auto_defrag = 1; |
| 1041 | 1053 | |
| @@ -468,7 +468,8 @@ struct ocfs2_extent_list { · 467 unmodified lines | ||
| 468 | 468 | __le16 l_reserved1; |
| 469 | 469 | __le64 l_reserved2; /* Pad to |
| 470 | 470 | sizeof(ocfs2_extent_rec) */ |
| 471 | −/*10*/ struct ocfs2_extent_rec l_recs[]; /* Extent records */ | |
| 471 | + /* Extent records */ | |
| 472 | +/*10*/ struct ocfs2_extent_rec l_recs[] __counted_by_le(l_count); | |
| 472 | 473 | }; |
| 473 | 474 | |
| 474 | 475 | /* |
| @@ -482,7 +483,8 @@ struct ocfs2_chain_list { · 7 unmodified lines | ||
| 482 | 483 | __le16 cl_count; /* Total chains in this list */ |
| 483 | 484 | __le16 cl_next_free_rec; /* Next unused chain slot */ |
| 484 | 485 | __le64 cl_reserved1; |
| 485 | −/*10*/ struct ocfs2_chain_rec cl_recs[]; /* Chain records */ | |
| 486 | + /* Chain records */ | |
| 487 | +/*10*/ struct ocfs2_chain_rec cl_recs[] __counted_by_le(cl_count); | |
| 486 | 488 | }; |
| 487 | 489 | |
| 488 | 490 | /* |
| @@ -494,7 +496,8 @@ struct ocfs2_truncate_log { · 5 unmodified lines | ||
| 494 | 496 | /*00*/ __le16 tl_count; /* Total records in this log */ |
| 495 | 497 | __le16 tl_used; /* Number of records in use */ |
| 496 | 498 | __le32 tl_reserved1; |
| 497 | −/*08*/ struct ocfs2_truncate_rec tl_recs[]; /* Truncate records */ | |
| 499 | + /* Truncate records */ | |
| 500 | +/*08*/ struct ocfs2_truncate_rec tl_recs[] __counted_by_le(tl_count); | |
| 498 | 501 | }; |
| 499 | 502 | |
| 500 | 503 | /* |
| @@ -796,9 +799,10 @@ struct ocfs2_dx_entry_list { · 295 unmodified lines | ||
| 796 | 799 | * possible in de_entries */ |
| 797 | 800 | __le16 de_num_used; /* Current number of |
| 798 | 801 | * de_entries entries */ |
| 799 | − struct ocfs2_dx_entry de_entries[]; /* Indexed dir entries | |
| 800 | − * in a packed array of | |
| 801 | − * length de_num_used */ | |
| 802 | + /* Indexed dir entries in a packed | |
| 803 | + * array of length de_num_used. | |
| 804 | + */ | |
| 805 | + struct ocfs2_dx_entry de_entries[] __counted_by_le(de_count); | |
| 802 | 806 | }; |
| 803 | 807 | |
| 804 | 808 | #define OCFS2_DX_FLAG_INLINE 0x01 |
| @@ -934,7 +938,8 @@ struct ocfs2_refcount_list { · 129 unmodified lines | ||
| 934 | 938 | __le16 rl_used; /* Current number of used records */ |
| 935 | 939 | __le32 rl_reserved2; |
| 936 | 940 | __le64 rl_reserved1; /* Pad to sizeof(ocfs2_refcount_record) */ |
| 937 | −/*10*/ struct ocfs2_refcount_rec rl_recs[]; /* Refcount records */ | |
| 941 | + /* Refcount records */ | |
| 942 | +/*10*/ struct ocfs2_refcount_rec rl_recs[] __counted_by_le(rl_count); | |
| 938 | 943 | }; |
| 939 | 944 | |
| 940 | 945 | |
| @@ -1020,7 +1025,8 @@ struct ocfs2_xattr_header { · 79 unmodified lines | ||
| 1020 | 1025 | buckets. A block uses |
| 1021 | 1026 | xb_check and sets |
| 1022 | 1027 | this field to zero.) */ |
| 1023 | − struct ocfs2_xattr_entry xh_entries[]; /* xattr entry list. */ | |
| 1028 | + /* xattr entry list. */ | |
| 1029 | + struct ocfs2_xattr_entry xh_entries[] __counted_by_le(xh_count); | |
| 1024 | 1030 | }; |
| 1025 | 1031 | |
| 1026 | 1032 | /* |
| @@ -34,6 +34,7 @@ · 33 unmodified lines | ||
| 34 | 34 | #include <linux/pagevec.h> |
| 35 | 35 | #include <linux/swap.h> |
| 36 | 36 | #include <linux/security.h> |
| 37 | +#include <linux/string.h> | |
| 37 | 38 | #include <linux/fsnotify.h> |
| 38 | 39 | #include <linux/quotaops.h> |
| 39 | 40 | #include <linux/namei.h> |
| @@ -621,7 +622,7 @@ static int ocfs2_create_refcount_tree(struct inode *inode, · 581 unmodified lines | ||
| 621 | 622 | /* Initialize ocfs2_refcount_block. */ |
| 622 | 623 | rb = (struct ocfs2_refcount_block *)new_bh->b_data; |
| 623 | 624 | memset(rb, 0, inode->i_sb->s_blocksize); |
| 624 | − strcpy((void *)rb, OCFS2_REFCOUNT_BLOCK_SIGNATURE); | |
| 625 | + strscpy(rb->rf_signature, OCFS2_REFCOUNT_BLOCK_SIGNATURE); | |
| 625 | 626 | rb->rf_suballoc_slot = cpu_to_le16(meta_ac->ac_alloc_slot); |
| 626 | 627 | rb->rf_suballoc_loc = cpu_to_le64(suballoc_loc); |
| 627 | 628 | rb->rf_suballoc_bit = cpu_to_le16(suballoc_bit_start); |
| @@ -1562,7 +1563,7 @@ static int ocfs2_new_leaf_refcount_block(handle_t *handle, · 934 unmodified lines | ||
| 1562 | 1563 | /* Initialize ocfs2_refcount_block. */ |
| 1563 | 1564 | new_rb = (struct ocfs2_refcount_block *)new_bh->b_data; |
| 1564 | 1565 | memset(new_rb, 0, sb->s_blocksize); |
| 1565 | − strcpy((void *)new_rb, OCFS2_REFCOUNT_BLOCK_SIGNATURE); | |
| 1566 | + strscpy(new_rb->rf_signature, OCFS2_REFCOUNT_BLOCK_SIGNATURE); | |
| 1566 | 1567 | new_rb->rf_suballoc_slot = cpu_to_le16(meta_ac->ac_alloc_slot); |
| 1567 | 1568 | new_rb->rf_suballoc_loc = cpu_to_le64(suballoc_loc); |
| 1568 | 1569 | new_rb->rf_suballoc_bit = cpu_to_le16(suballoc_bit_start); |
| @@ -2908,7 +2908,7 @@ static int ocfs2_create_xattr_block(struct inode *inode, · 2907 unmodified lines | ||
| 2908 | 2908 | /* Initialize ocfs2_xattr_block */ |
| 2909 | 2909 | xblk = (struct ocfs2_xattr_block *)new_bh->b_data; |
| 2910 | 2910 | memset(xblk, 0, inode->i_sb->s_blocksize); |
| 2911 | − strcpy((void *)xblk, OCFS2_XATTR_BLOCK_SIGNATURE); | |
| 2911 | + strscpy(xblk->xb_signature, OCFS2_XATTR_BLOCK_SIGNATURE); | |
| 2912 | 2912 | xblk->xb_suballoc_slot = cpu_to_le16(ctxt->meta_ac->ac_alloc_slot); |
| 2913 | 2913 | xblk->xb_suballoc_loc = cpu_to_le64(suballoc_loc); |
| 2914 | 2914 | xblk->xb_suballoc_bit = cpu_to_le16(suballoc_bit_start); |
| @@ -20,7 +20,6 @@ · 19 unmodified lines | ||
| 20 | 20 | |
| 21 | 21 | #define KPMSIZE sizeof(u64) |
| 22 | 22 | #define KPMMASK (KPMSIZE - 1) |
| 23 | −#define KPMBITS (KPMSIZE * BITS_PER_BYTE) | |
| 24 | 23 | |
| 25 | 24 | enum kpage_operation { |
| 26 | 25 | KPAGE_FLAGS, |
| @@ -8,9 +8,15 @@ · 7 unmodified lines | ||
| 8 | 8 | |
| 9 | 9 | #include <linux/types.h> |
| 10 | 10 | |
| 11 | +enum base64_variant { | |
| 12 | + BASE64_STD, /* RFC 4648 (standard) */ | |
| 13 | + BASE64_URLSAFE, /* RFC 4648 (base64url) */ | |
| 14 | + BASE64_IMAP, /* RFC 3501 */ | |
| 15 | +}; | |
| 16 | + | |
| 11 | 17 | #define BASE64_CHARS(nbytes) DIV_ROUND_UP((nbytes) * 4, 3) |
| 12 | 18 | |
| 13 | −int base64_encode(const u8 *src, int len, char *dst); | |
| 14 | −int base64_decode(const char *src, int len, u8 *dst); | |
| 19 | +int base64_encode(const u8 *src, int len, char *dst, bool padding, enum base64_variant variant); | |
| 20 | +int base64_decode(const char *src, int len, u8 *dst, bool padding, enum base64_variant variant); | |
| 15 | 21 | |
| 16 | 22 | #endif /* _LINUX_BASE64_H */ |
| @@ -273,12 +273,6 @@ static inline void *offset_to_ptr(const int *off) · 272 unmodified lines | ||
| 273 | 273 | |
| 274 | 274 | #endif /* __ASSEMBLY__ */ |
| 275 | 275 | |
| 276 | −#ifdef CONFIG_64BIT | |
| 277 | −#define ARCH_SEL(a,b) a | |
| 278 | −#else | |
| 279 | −#define ARCH_SEL(a,b) b | |
| 280 | −#endif | |
| 281 | − | |
| 282 | 276 | /* |
| 283 | 277 | * Force the compiler to emit 'sym' as a symbol, so that we can reference |
| 284 | 278 | * it from inline assembler. Necessary in case 'sym' could be inlined |
| @@ -32,6 +32,12 @@ int __init parse_crashkernel(char *cmdline, unsigned long long system_ram, · 31 unmodified lines | ||
| 32 | 32 | void __init reserve_crashkernel_cma(unsigned long long cma_size); |
| 33 | 33 | |
| 34 | 34 | #ifdef CONFIG_ARCH_HAS_GENERIC_CRASHKERNEL_RESERVATION |
| 35 | +#ifndef arch_add_crash_res_to_iomem | |
| 36 | +static inline bool arch_add_crash_res_to_iomem(void) | |
| 37 | +{ | |
| 38 | + return true; | |
| 39 | +} | |
| 40 | +#endif | |
| 35 | 41 | #ifndef DEFAULT_CRASH_KERNEL_LOW_SIZE |
| 36 | 42 | #define DEFAULT_CRASH_KERNEL_LOW_SIZE (128UL << 20) |
| 37 | 43 | #endif |
| @@ -38,11 +38,12 @@ struct _ddebug { · 37 unmodified lines | ||
| 38 | 38 | #define _DPRINTK_FLAGS_INCL_LINENO (1<<3) |
| 39 | 39 | #define _DPRINTK_FLAGS_INCL_TID (1<<4) |
| 40 | 40 | #define _DPRINTK_FLAGS_INCL_SOURCENAME (1<<5) |
| 41 | +#define _DPRINTK_FLAGS_INCL_STACK (1<<6) | |
| 41 | 42 | |
| 42 | 43 | #define _DPRINTK_FLAGS_INCL_ANY \ |
| 43 | 44 | (_DPRINTK_FLAGS_INCL_MODNAME | _DPRINTK_FLAGS_INCL_FUNCNAME |\ |
| 44 | 45 | _DPRINTK_FLAGS_INCL_LINENO | _DPRINTK_FLAGS_INCL_TID |\ |
| 45 | − _DPRINTK_FLAGS_INCL_SOURCENAME) | |
| 46 | + _DPRINTK_FLAGS_INCL_SOURCENAME | _DPRINTK_FLAGS_INCL_STACK) | |
| 46 | 47 | |
| 47 | 48 | #if defined DEBUG |
| 48 | 49 | #define _DPRINTK_FLAGS_DEFAULT _DPRINTK_FLAGS_PRINT |
| @@ -160,6 +161,12 @@ void __dynamic_ibdev_dbg(struct _ddebug *descriptor, · 111 unmodified lines | ||
| 160 | 161 | const struct ib_device *ibdev, |
| 161 | 162 | const char *fmt, ...); |
| 162 | 163 | |
| 164 | +#define __dynamic_dump_stack(desc) \ | |
| 165 | +{ \ | |
| 166 | + if (desc.flags & _DPRINTK_FLAGS_INCL_STACK) \ | |
| 167 | + dump_stack(); \ | |
| 168 | +} | |
| 169 | + | |
| 163 | 170 | #define DEFINE_DYNAMIC_DEBUG_METADATA_CLS(name, cls, fmt) \ |
| 164 | 171 | static struct _ddebug __aligned(8) \ |
| 165 | 172 | __section("__dyndbg") name = { \ |
| @@ -220,8 +227,10 @@ void __dynamic_ibdev_dbg(struct _ddebug *descriptor, · 54 unmodified lines | ||
| 220 | 227 | */ |
| 221 | 228 | #define __dynamic_func_call_cls(id, cls, fmt, func, ...) do { \ |
| 222 | 229 | DEFINE_DYNAMIC_DEBUG_METADATA_CLS(id, cls, fmt); \ |
| 223 | − if (DYNAMIC_DEBUG_BRANCH(id)) \ | |
| 230 | + if (DYNAMIC_DEBUG_BRANCH(id)) { \ | |
| 224 | 231 | func(&id, ##__VA_ARGS__); \ |
| 232 | + __dynamic_dump_stack(id); \ | |
| 233 | + } \ | |
| 225 | 234 | } while (0) |
| 226 | 235 | #define __dynamic_func_call(id, fmt, func, ...) \ |
| 227 | 236 | __dynamic_func_call_cls(id, _DPRINTK_CLASS_DFLT, fmt, \ |
| @@ -229,8 +238,10 @@ void __dynamic_ibdev_dbg(struct _ddebug *descriptor, · 1 unmodified line | ||
| 229 | 238 | |
| 230 | 239 | #define __dynamic_func_call_cls_no_desc(id, cls, fmt, func, ...) do { \ |
| 231 | 240 | DEFINE_DYNAMIC_DEBUG_METADATA_CLS(id, cls, fmt); \ |
| 232 | − if (DYNAMIC_DEBUG_BRANCH(id)) \ | |
| 241 | + if (DYNAMIC_DEBUG_BRANCH(id)) { \ | |
| 233 | 242 | func(__VA_ARGS__); \ |
| 243 | + __dynamic_dump_stack(id); \ | |
| 244 | + } \ | |
| 234 | 245 | } while (0) |
| 235 | 246 | #define __dynamic_func_call_no_desc(id, fmt, func, ...) \ |
| 236 | 247 | __dynamic_func_call_cls_no_desc(id, _DPRINTK_CLASS_DFLT, \ |
| @@ -2,22 +2,16 @@ · 1 unmodified line | ||
| 2 | 2 | #ifndef LINUX_KEXEC_HANDOVER_H |
| 3 | 3 | #define LINUX_KEXEC_HANDOVER_H |
| 4 | 4 | |
| 5 | −#include <linux/types.h> | |
| 5 | +#include <linux/err.h> | |
| 6 | 6 | #include <linux/errno.h> |
| 7 | +#include <linux/types.h> | |
| 7 | 8 | |
| 8 | 9 | struct kho_scratch { |
| 9 | 10 | phys_addr_t addr; |
| 10 | 11 | phys_addr_t size; |
| 11 | 12 | }; |
| 12 | 13 | |
| 13 | −/* KHO Notifier index */ | |
| 14 | −enum kho_event { | |
| 15 | − KEXEC_KHO_FINALIZE = 0, | |
| 16 | − KEXEC_KHO_ABORT = 1, | |
| 17 | −}; | |
| 18 | − | |
| 19 | 14 | struct folio; |
| 20 | −struct notifier_block; | |
| 21 | 15 | struct page; |
| 22 | 16 | |
| 23 | 17 | #define DECLARE_KHOSER_PTR(name, type) \ |
| @@ -37,8 +31,6 @@ struct page; · 13 unmodified lines | ||
| 37 | 31 | (typeof((s).ptr))((s).phys ? phys_to_virt((s).phys) : NULL); \ |
| 38 | 32 | }) |
| 39 | 33 | |
| 40 | −struct kho_serialization; | |
| 41 | − | |
| 42 | 34 | struct kho_vmalloc_chunk; |
| 43 | 35 | struct kho_vmalloc { |
| 44 | 36 | DECLARE_KHOSER_PTR(first, struct kho_vmalloc_chunk *); |
| @@ -52,17 +44,21 @@ bool kho_is_enabled(void); · 7 unmodified lines | ||
| 52 | 44 | bool is_kho_boot(void); |
| 53 | 45 | |
| 54 | 46 | int kho_preserve_folio(struct folio *folio); |
| 47 | +void kho_unpreserve_folio(struct folio *folio); | |
| 55 | 48 | int kho_preserve_pages(struct page *page, unsigned int nr_pages); |
| 49 | +void kho_unpreserve_pages(struct page *page, unsigned int nr_pages); | |
| 56 | 50 | int kho_preserve_vmalloc(void *ptr, struct kho_vmalloc *preservation); |
| 51 | +void kho_unpreserve_vmalloc(struct kho_vmalloc *preservation); | |
| 52 | +void *kho_alloc_preserve(size_t size); | |
| 53 | +void kho_unpreserve_free(void *mem); | |
| 54 | +void kho_restore_free(void *mem); | |
| 57 | 55 | struct folio *kho_restore_folio(phys_addr_t phys); |
| 58 | 56 | struct page *kho_restore_pages(phys_addr_t phys, unsigned int nr_pages); |
| 59 | 57 | void *kho_restore_vmalloc(const struct kho_vmalloc *preservation); |
| 60 | −int kho_add_subtree(struct kho_serialization *ser, const char *name, void *fdt); | |
| 58 | +int kho_add_subtree(const char *name, void *fdt); | |
| 59 | +void kho_remove_subtree(void *fdt); | |
| 61 | 60 | int kho_retrieve_subtree(const char *name, phys_addr_t *phys); |
| 62 | 61 | |
| 63 | −int register_kho_notifier(struct notifier_block *nb); | |
| 64 | −int unregister_kho_notifier(struct notifier_block *nb); | |
| 65 | − | |
| 66 | 62 | void kho_memory_init(void); |
| 67 | 63 | |
| 68 | 64 | void kho_populate(phys_addr_t fdt_phys, u64 fdt_len, phys_addr_t scratch_phys, |
| @@ -83,17 +79,31 @@ static inline int kho_preserve_folio(struct folio *folio) · 14 unmodified lines | ||
| 83 | 79 | return -EOPNOTSUPP; |
| 84 | 80 | } |
| 85 | 81 | |
| 82 | +static inline void kho_unpreserve_folio(struct folio *folio) { } | |
| 83 | + | |
| 86 | 84 | static inline int kho_preserve_pages(struct page *page, unsigned int nr_pages) |
| 87 | 85 | { |
| 88 | 86 | return -EOPNOTSUPP; |
| 89 | 87 | } |
| 90 | 88 | |
| 89 | +static inline void kho_unpreserve_pages(struct page *page, unsigned int nr_pages) { } | |
| 90 | + | |
| 91 | 91 | static inline int kho_preserve_vmalloc(void *ptr, |
| 92 | 92 | struct kho_vmalloc *preservation) |
| 93 | 93 | { |
| 94 | 94 | return -EOPNOTSUPP; |
| 95 | 95 | } |
| 96 | 96 | |
| 97 | +static inline void kho_unpreserve_vmalloc(struct kho_vmalloc *preservation) { } | |
| 98 | + | |
| 99 | +static inline void *kho_alloc_preserve(size_t size) | |
| 100 | +{ | |
| 101 | + return ERR_PTR(-EOPNOTSUPP); | |
| 102 | +} | |
| 103 | + | |
| 104 | +static inline void kho_unpreserve_free(void *mem) { } | |
| 105 | +static inline void kho_restore_free(void *mem) { } | |
| 106 | + | |
| 97 | 107 | static inline struct folio *kho_restore_folio(phys_addr_t phys) |
| 98 | 108 | { |
| 99 | 109 | return NULL; |
| @@ -110,30 +120,19 @@ static inline void *kho_restore_vmalloc(const struct kho_vmalloc *preservation) · 10 unmodified lines | ||
| 110 | 120 | return NULL; |
| 111 | 121 | } |
| 112 | 122 | |
| 113 | −static inline int kho_add_subtree(struct kho_serialization *ser, | |
| 114 | − const char *name, void *fdt) | |
| 115 | −{ | |
| 116 | − return -EOPNOTSUPP; | |
| 117 | −} | |
| 118 | − | |
| 119 | −static inline int kho_retrieve_subtree(const char *name, phys_addr_t *phys) | |
| 123 | +static inline int kho_add_subtree(const char *name, void *fdt) | |
| 120 | 124 | { |
| 121 | 125 | return -EOPNOTSUPP; |
| 122 | 126 | } |
| 123 | 127 | |
| 124 | −static inline int register_kho_notifier(struct notifier_block *nb) | |
| 125 | −{ | |
| 126 | − return -EOPNOTSUPP; | |
| 127 | −} | |
| 128 | +static inline void kho_remove_subtree(void *fdt) { } | |
| 128 | 129 | |
| 129 | −static inline int unregister_kho_notifier(struct notifier_block *nb) | |
| 130 | +static inline int kho_retrieve_subtree(const char *name, phys_addr_t *phys) | |
| 130 | 131 | { |
| 131 | 132 | return -EOPNOTSUPP; |
| 132 | 133 | } |
| 133 | 134 | |
| 134 | −static inline void kho_memory_init(void) | |
| 135 | −{ | |
| 136 | −} | |
| 135 | +static inline void kho_memory_init(void) { } | |
| 137 | 136 | |
| 138 | 137 | static inline void kho_populate(phys_addr_t fdt_phys, u64 fdt_len, |
| 139 | 138 | phys_addr_t scratch_phys, u64 scratch_len) |
| @@ -0,0 +1,166 @@ | ||
| 1 | +/* SPDX-License-Identifier: GPL-2.0 */ | |
| 2 | + | |
| 3 | +/* | |
| 4 | + * Copyright (c) 2025, Google LLC. | |
| 5 | + * Pasha Tatashin <pasha.tatashin@soleen.com> | |
| 6 | + */ | |
| 7 | + | |
| 8 | +/** | |
| 9 | + * DOC: Live Update Orchestrator ABI | |
| 10 | + * | |
| 11 | + * This header defines the stable Application Binary Interface used by the | |
| 12 | + * Live Update Orchestrator to pass state from a pre-update kernel to a | |
| 13 | + * post-update kernel. The ABI is built upon the Kexec HandOver framework | |
| 14 | + * and uses a Flattened Device Tree to describe the preserved data. | |
| 15 | + * | |
| 16 | + * This interface is a contract. Any modification to the FDT structure, node | |
| 17 | + * properties, compatible strings, or the layout of the `__packed` serialization | |
| 18 | + * structures defined here constitutes a breaking change. Such changes require | |
| 19 | + * incrementing the version number in the relevant `_COMPATIBLE` string to | |
| 20 | + * prevent a new kernel from misinterpreting data from an old kernel. | |
| 21 | + * | |
| 22 | + * Changes are allowed provided the compatibility version is incremented; | |
| 23 | + * however, backward/forward compatibility is only guaranteed for kernels | |
| 24 | + * supporting the same ABI version. | |
| 25 | + * | |
| 26 | + * FDT Structure Overview: | |
| 27 | + * The entire LUO state is encapsulated within a single KHO entry named "LUO". | |
| 28 | + * This entry contains an FDT with the following layout: | |
| 29 | + * | |
| 30 | + * .. code-block:: none | |
| 31 | + * | |
| 32 | + * / { | |
| 33 | + * compatible = "luo-v1"; | |
| 34 | + * liveupdate-number = <...>; | |
| 35 | + * | |
| 36 | + * luo-session { | |
| 37 | + * compatible = "luo-session-v1"; | |
| 38 | + * luo-session-header = <phys_addr_of_session_header_ser>; | |
| 39 | + * }; | |
| 40 | + * }; | |
| 41 | + * | |
| 42 | + * Main LUO Node (/): | |
| 43 | + * | |
| 44 | + * - compatible: "luo-v1" | |
| 45 | + * Identifies the overall LUO ABI version. | |
| 46 | + * - liveupdate-number: u64 | |
| 47 | + * A counter tracking the number of successful live updates performed. | |
| 48 | + * | |
| 49 | + * Session Node (luo-session): | |
| 50 | + * This node describes all preserved user-space sessions. | |
| 51 | + * | |
| 52 | + * - compatible: "luo-session-v1" | |
| 53 | + * Identifies the session ABI version. | |
| 54 | + * - luo-session-header: u64 | |
| 55 | + * The physical address of a `struct luo_session_header_ser`. This structure | |
| 56 | + * is the header for a contiguous block of memory containing an array of | |
| 57 | + * `struct luo_session_ser`, one for each preserved session. | |
| 58 | + * | |
| 59 | + * Serialization Structures: | |
| 60 | + * The FDT properties point to memory regions containing arrays of simple, | |
| 61 | + * `__packed` structures. These structures contain the actual preserved state. | |
| 62 | + * | |
| 63 | + * - struct luo_session_header_ser: | |
| 64 | + * Header for the session array. Contains the total page count of the | |
| 65 | + * preserved memory block and the number of `struct luo_session_ser` | |
| 66 | + * entries that follow. | |
| 67 | + * | |
| 68 | + * - struct luo_session_ser: | |
| 69 | + * Metadata for a single session, including its name and a physical pointer | |
| 70 | + * to another preserved memory block containing an array of | |
| 71 | + * `struct luo_file_ser` for all files in that session. | |
| 72 | + * | |
| 73 | + * - struct luo_file_ser: | |
| 74 | + * Metadata for a single preserved file. Contains the `compatible` string to | |
| 75 | + * find the correct handler in the new kernel, a user-provided `token` for | |
| 76 | + * identification, and an opaque `data` handle for the handler to use. | |
| 77 | + */ | |
| 78 | + | |
| 79 | +#ifndef _LINUX_KHO_ABI_LUO_H | |
| 80 | +#define _LINUX_KHO_ABI_LUO_H | |
| 81 | + | |
| 82 | +#include <uapi/linux/liveupdate.h> | |
| 83 | + | |
| 84 | +/* | |
| 85 | + * The LUO FDT hooks all LUO state for sessions, fds, etc. | |
| 86 | + * In the root it also carries "liveupdate-number" 64-bit property that | |
| 87 | + * corresponds to the number of live-updates performed on this machine. | |
| 88 | + */ | |
| 89 | +#define LUO_FDT_SIZE PAGE_SIZE | |
| 90 | +#define LUO_FDT_KHO_ENTRY_NAME "LUO" | |
| 91 | +#define LUO_FDT_COMPATIBLE "luo-v1" | |
| 92 | +#define LUO_FDT_LIVEUPDATE_NUM "liveupdate-number" | |
| 93 | + | |
| 94 | +#define LIVEUPDATE_HNDL_COMPAT_LENGTH 48 | |
| 95 | + | |
| 96 | +/** | |
| 97 | + * struct luo_file_ser - Represents the serialized preserves files. | |
| 98 | + * @compatible: File handler compatible string. | |
| 99 | + * @data: Private data | |
| 100 | + * @token: User provided token for this file | |
| 101 | + * | |
| 102 | + * If this structure is modified, LUO_SESSION_COMPATIBLE must be updated. | |
| 103 | + */ | |
| 104 | +struct luo_file_ser { | |
| 105 | + char compatible[LIVEUPDATE_HNDL_COMPAT_LENGTH]; | |
| 106 | + u64 data; | |
| 107 | + u64 token; | |
| 108 | +} __packed; | |
| 109 | + | |
| 110 | +/** | |
| 111 | + * struct luo_file_set_ser - Represents the serialized metadata for file set | |
| 112 | + * @files: The physical address of a contiguous memory block that holds | |
| 113 | + * the serialized state of files (array of luo_file_ser) in this file | |
| 114 | + * set. | |
| 115 | + * @count: The total number of files that were part of this session during | |
| 116 | + * serialization. Used for iteration and validation during | |
| 117 | + * restoration. | |
| 118 | + */ | |
| 119 | +struct luo_file_set_ser { | |
| 120 | + u64 files; | |
| 121 | + u64 count; | |
| 122 | +} __packed; | |
| 123 | + | |
| 124 | +/* | |
| 125 | + * LUO FDT session node | |
| 126 | + * LUO_FDT_SESSION_HEADER: is a u64 physical address of struct | |
| 127 | + * luo_session_header_ser | |
| 128 | + */ | |
| 129 | +#define LUO_FDT_SESSION_NODE_NAME "luo-session" | |
| 130 | +#define LUO_FDT_SESSION_COMPATIBLE "luo-session-v2" | |
| 131 | +#define LUO_FDT_SESSION_HEADER "luo-session-header" | |
| 132 | + | |
| 133 | +/** | |
| 134 | + * struct luo_session_header_ser - Header for the serialized session data block. | |
| 135 | + * @count: The number of `struct luo_session_ser` entries that immediately | |
| 136 | + * follow this header in the memory block. | |
| 137 | + * | |
| 138 | + * This structure is located at the beginning of a contiguous block of | |
| 139 | + * physical memory preserved across the kexec. It provides the necessary | |
| 140 | + * metadata to interpret the array of session entries that follow. | |
| 141 | + * | |
| 142 | + * If this structure is modified, `LUO_FDT_SESSION_COMPATIBLE` must be updated. | |
| 143 | + */ | |
| 144 | +struct luo_session_header_ser { | |
| 145 | + u64 count; | |
| 146 | +} __packed; | |
| 147 | + | |
| 148 | +/** | |
| 149 | + * struct luo_session_ser - Represents the serialized metadata for a LUO session. | |
| 150 | + * @name: The unique name of the session, provided by the userspace at | |
| 151 | + * the time of session creation. | |
| 152 | + * @file_set_ser: Serialized files belonging to this session, | |
| 153 | + * | |
| 154 | + * This structure is used to package session-specific metadata for transfer | |
| 155 | + * between kernels via Kexec Handover. An array of these structures (one per | |
| 156 | + * session) is created and passed to the new kernel, allowing it to reconstruct | |
| 157 | + * the session context. | |
| 158 | + * | |
| 159 | + * If this structure is modified, `LUO_FDT_SESSION_COMPATIBLE` must be updated. | |
| 160 | + */ | |
| 161 | +struct luo_session_ser { | |
| 162 | + char name[LIVEUPDATE_SESSION_NAME_LENGTH]; | |
| 163 | + struct luo_file_set_ser file_set_ser; | |
| 164 | +} __packed; | |
| 165 | + | |
| 166 | +#endif /* _LINUX_KHO_ABI_LUO_H */ | |
| @@ -0,0 +1,77 @@ | ||
| 1 | +/* SPDX-License-Identifier: GPL-2.0 */ | |
| 2 | + | |
| 3 | +/* | |
| 4 | + * Copyright (c) 2025, Google LLC. | |
| 5 | + * Pasha Tatashin <pasha.tatashin@soleen.com> | |
| 6 | + * | |
| 7 | + * Copyright (C) 2025 Amazon.com Inc. or its affiliates. | |
| 8 | + * Pratyush Yadav <ptyadav@amazon.de> | |
| 9 | + */ | |
| 10 | + | |
| 11 | +#ifndef _LINUX_KHO_ABI_MEMFD_H | |
| 12 | +#define _LINUX_KHO_ABI_MEMFD_H | |
| 13 | + | |
| 14 | +#include <linux/types.h> | |
| 15 | +#include <linux/kexec_handover.h> | |
| 16 | + | |
| 17 | +/** | |
| 18 | + * DOC: memfd Live Update ABI | |
| 19 | + * | |
| 20 | + * This header defines the ABI for preserving the state of a memfd across a | |
| 21 | + * kexec reboot using the LUO. | |
| 22 | + * | |
| 23 | + * The state is serialized into a packed structure `struct memfd_luo_ser` | |
| 24 | + * which is handed over to the next kernel via the KHO mechanism. | |
| 25 | + * | |
| 26 | + * This interface is a contract. Any modification to the structure layout | |
| 27 | + * constitutes a breaking change. Such changes require incrementing the | |
| 28 | + * version number in the MEMFD_LUO_FH_COMPATIBLE string. | |
| 29 | + */ | |
| 30 | + | |
| 31 | +/** | |
| 32 | + * MEMFD_LUO_FOLIO_DIRTY - The folio is dirty. | |
| 33 | + * | |
| 34 | + * This flag indicates the folio contains data from user. A non-dirty folio is | |
| 35 | + * one that was allocated (say using fallocate(2)) but not written to. | |
| 36 | + */ | |
| 37 | +#define MEMFD_LUO_FOLIO_DIRTY BIT(0) | |
| 38 | + | |
| 39 | +/** | |
| 40 | + * MEMFD_LUO_FOLIO_UPTODATE - The folio is up-to-date. | |
| 41 | + * | |
| 42 | + * An up-to-date folio has been zeroed out. shmem zeroes out folios on first | |
| 43 | + * use. This flag tracks which folios need zeroing. | |
| 44 | + */ | |
| 45 | +#define MEMFD_LUO_FOLIO_UPTODATE BIT(1) | |
| 46 | + | |
| 47 | +/** | |
| 48 | + * struct memfd_luo_folio_ser - Serialized state of a single folio. | |
| 49 | + * @pfn: The page frame number of the folio. | |
| 50 | + * @flags: Flags to describe the state of the folio. | |
| 51 | + * @index: The page offset (pgoff_t) of the folio within the original file. | |
| 52 | + */ | |
| 53 | +struct memfd_luo_folio_ser { | |
| 54 | + u64 pfn:52; | |
| 55 | + u64 flags:12; | |
| 56 | + u64 index; | |
| 57 | +} __packed; | |
| 58 | + | |
| 59 | +/** | |
| 60 | + * struct memfd_luo_ser - Main serialization structure for a memfd. | |
| 61 | + * @pos: The file's current position (f_pos). | |
| 62 | + * @size: The total size of the file in bytes (i_size). | |
| 63 | + * @nr_folios: Number of folios in the folios array. | |
| 64 | + * @folios: KHO vmalloc descriptor pointing to the array of | |
| 65 | + * struct memfd_luo_folio_ser. | |
| 66 | + */ | |
| 67 | +struct memfd_luo_ser { | |
| 68 | + u64 pos; | |
| 69 | + u64 size; | |
| 70 | + u64 nr_folios; | |
| 71 | + struct kho_vmalloc folios; | |
| 72 | +} __packed; | |
| 73 | + | |
| 74 | +/* The compatibility string for memfd file handler */ | |
| 75 | +#define MEMFD_LUO_FH_COMPATIBLE "memfd-v1" | |
| 76 | + | |
| 77 | +#endif /* _LINUX_KHO_ABI_MEMFD_H */ | |
| @@ -0,0 +1,138 @@ | ||
| 1 | +/* SPDX-License-Identifier: GPL-2.0 */ | |
| 2 | + | |
| 3 | +/* | |
| 4 | + * Copyright (c) 2025, Google LLC. | |
| 5 | + * Pasha Tatashin <pasha.tatashin@soleen.com> | |
| 6 | + */ | |
| 7 | +#ifndef _LINUX_LIVEUPDATE_H | |
| 8 | +#define _LINUX_LIVEUPDATE_H | |
| 9 | + | |
| 10 | +#include <linux/bug.h> | |
| 11 | +#include <linux/compiler.h> | |
| 12 | +#include <linux/kho/abi/luo.h> | |
| 13 | +#include <linux/list.h> | |
| 14 | +#include <linux/types.h> | |
| 15 | +#include <uapi/linux/liveupdate.h> | |
| 16 | + | |
| 17 | +struct liveupdate_file_handler; | |
| 18 | +struct file; | |
| 19 | + | |
| 20 | +/** | |
| 21 | + * struct liveupdate_file_op_args - Arguments for file operation callbacks. | |
| 22 | + * @handler: The file handler being called. | |
| 23 | + * @retrieved: The retrieve status for the 'can_finish / finish' | |
| 24 | + * operation. | |
| 25 | + * @file: The file object. For retrieve: [OUT] The callback sets | |
| 26 | + * this to the new file. For other ops: [IN] The caller sets | |
| 27 | + * this to the file being operated on. | |
| 28 | + * @serialized_data: The opaque u64 handle, preserve/prepare/freeze may update | |
| 29 | + * this field. | |
| 30 | + * @private_data: Private data for the file used to hold runtime state that | |
| 31 | + * is not preserved. Set by the handler's .preserve() | |
| 32 | + * callback, and must be freed in the handler's | |
| 33 | + * .unpreserve() callback. | |
| 34 | + * | |
| 35 | + * This structure bundles all parameters for the file operation callbacks. | |
| 36 | + * The 'data' and 'file' fields are used for both input and output. | |
| 37 | + */ | |
| 38 | +struct liveupdate_file_op_args { | |
| 39 | + struct liveupdate_file_handler *handler; | |
| 40 | + bool retrieved; | |
| 41 | + struct file *file; | |
| 42 | + u64 serialized_data; | |
| 43 | + void *private_data; | |
| 44 | +}; | |
| 45 | + | |
| 46 | +/** | |
| 47 | + * struct liveupdate_file_ops - Callbacks for live-updatable files. | |
| 48 | + * @can_preserve: Required. Lightweight check to see if this handler is | |
| 49 | + * compatible with the given file. | |
| 50 | + * @preserve: Required. Performs state-saving for the file. | |
| 51 | + * @unpreserve: Required. Cleans up any resources allocated by @preserve. | |
| 52 | + * @freeze: Optional. Final actions just before kernel transition. | |
| 53 | + * @unfreeze: Optional. Undo freeze operations. | |
| 54 | + * @retrieve: Required. Restores the file in the new kernel. | |
| 55 | + * @can_finish: Optional. Check if this FD can finish, i.e. all restoration | |
| 56 | + * pre-requirements for this FD are satisfied. Called prior to | |
| 57 | + * finish, in order to do successful finish calls for all | |
| 58 | + * resources in the session. | |
| 59 | + * @finish: Required. Final cleanup in the new kernel. | |
| 60 | + * @owner: Module reference | |
| 61 | + * | |
| 62 | + * All operations (except can_preserve) receive a pointer to a | |
| 63 | + * 'struct liveupdate_file_op_args' containing the necessary context. | |
| 64 | + */ | |
| 65 | +struct liveupdate_file_ops { | |
| 66 | + bool (*can_preserve)(struct liveupdate_file_handler *handler, | |
| 67 | + struct file *file); | |
| 68 | + int (*preserve)(struct liveupdate_file_op_args *args); | |
| 69 | + void (*unpreserve)(struct liveupdate_file_op_args *args); | |
| 70 | + int (*freeze)(struct liveupdate_file_op_args *args); | |
| 71 | + void (*unfreeze)(struct liveupdate_file_op_args *args); | |
| 72 | + int (*retrieve)(struct liveupdate_file_op_args *args); | |
| 73 | + bool (*can_finish)(struct liveupdate_file_op_args *args); | |
| 74 | + void (*finish)(struct liveupdate_file_op_args *args); | |
| 75 | + struct module *owner; | |
| 76 | +}; | |
| 77 | + | |
| 78 | +/** | |
| 79 | + * struct liveupdate_file_handler - Represents a handler for a live-updatable file type. | |
| 80 | + * @ops: Callback functions | |
| 81 | + * @compatible: The compatibility string (e.g., "memfd-v1", "vfiofd-v1") | |
| 82 | + * that uniquely identifies the file type this handler | |
| 83 | + * supports. This is matched against the compatible string | |
| 84 | + * associated with individual &struct file instances. | |
| 85 | + * | |
| 86 | + * Modules that want to support live update for specific file types should | |
| 87 | + * register an instance of this structure. LUO uses this registration to | |
| 88 | + * determine if a given file can be preserved and to find the appropriate | |
| 89 | + * operations to manage its state across the update. | |
| 90 | + */ | |
| 91 | +struct liveupdate_file_handler { | |
| 92 | + const struct liveupdate_file_ops *ops; | |
| 93 | + const char compatible[LIVEUPDATE_HNDL_COMPAT_LENGTH]; | |
| 94 | + | |
| 95 | + /* private: */ | |
| 96 | + | |
| 97 | + /* | |
| 98 | + * Used for linking this handler instance into a global list of | |
| 99 | + * registered file handlers. | |
| 100 | + */ | |
| 101 | + struct list_head __private list; | |
| 102 | +}; | |
| 103 | + | |
| 104 | +#ifdef CONFIG_LIVEUPDATE | |
| 105 | + | |
| 106 | +/* Return true if live update orchestrator is enabled */ | |
| 107 | +bool liveupdate_enabled(void); | |
| 108 | + | |
| 109 | +/* Called during kexec to tell LUO that entered into reboot */ | |
| 110 | +int liveupdate_reboot(void); | |
| 111 | + | |
| 112 | +int liveupdate_register_file_handler(struct liveupdate_file_handler *fh); | |
| 113 | +int liveupdate_unregister_file_handler(struct liveupdate_file_handler *fh); | |
| 114 | + | |
| 115 | +#else /* CONFIG_LIVEUPDATE */ | |
| 116 | + | |
| 117 | +static inline bool liveupdate_enabled(void) | |
| 118 | +{ | |
| 119 | + return false; | |
| 120 | +} | |
| 121 | + | |
| 122 | +static inline int liveupdate_reboot(void) | |
| 123 | +{ | |
| 124 | + return 0; | |
| 125 | +} | |
| 126 | + | |
| 127 | +static inline int liveupdate_register_file_handler(struct liveupdate_file_handler *fh) | |
| 128 | +{ | |
| 129 | + return -EOPNOTSUPP; | |
| 130 | +} | |
| 131 | + | |
| 132 | +static inline int liveupdate_unregister_file_handler(struct liveupdate_file_handler *fh) | |
| 133 | +{ | |
| 134 | + return -EOPNOTSUPP; | |
| 135 | +} | |
| 136 | + | |
| 137 | +#endif /* CONFIG_LIVEUPDATE */ | |
| 138 | +#endif /* _LINUX_LIVEUPDATE_H */ | |
| @@ -148,11 +148,16 @@ __STRUCT_FRACT(u32) · 147 unmodified lines | ||
| 148 | 148 | |
| 149 | 149 | /** |
| 150 | 150 | * abs - return absolute value of an argument |
| 151 | − * @x: the value. If it is unsigned type, it is converted to signed type first. | |
| 152 | − * char is treated as if it was signed (regardless of whether it really is) | |
| 153 | − * but the macro's return type is preserved as char. | |
| 151 | + * @x: the value. | |
| 154 | 152 | * |
| 155 | − * Return: an absolute value of x. | |
| 153 | + * If it is unsigned type, @x is converted to signed type first. | |
| 154 | + * char is treated as if it was signed (regardless of whether it really is) | |
| 155 | + * but the macro's return type is preserved as char. | |
| 156 | + * | |
| 157 | + * NOTE, for signed type if @x is the minimum, the returned result is undefined | |
| 158 | + * as there is not enough bits to represent it as a positive number. | |
| 159 | + * | |
| 160 | + * Return: an absolute value of @x. | |
| 156 | 161 | */ |
| 157 | 162 | #define abs(x) __abs_choose_expr(x, long long, \ |
| 158 | 163 | __abs_choose_expr(x, long, \ |
| @@ -158,6 +158,17 @@ static inline u64 mul_u32_u32(u32 a, u32 b) · 157 unmodified lines | ||
| 158 | 158 | } |
| 159 | 159 | #endif |
| 160 | 160 | |
| 161 | +#ifndef add_u64_u32 | |
| 162 | +/* | |
| 163 | + * Many a GCC version also messes this up. | |
| 164 | + * Zero extending b and then spilling everything to stack. | |
| 165 | + */ | |
| 166 | +static inline u64 add_u64_u32(u64 a, u32 b) | |
| 167 | +{ | |
| 168 | + return a + b; | |
| 169 | +} | |
| 170 | +#endif | |
| 171 | + | |
| 161 | 172 | #if defined(CONFIG_ARCH_SUPPORTS_INT128) && defined(__SIZEOF_INT128__) |
| 162 | 173 | |
| 163 | 174 | #ifndef mul_u64_u32_shr |
| @@ -282,7 +293,53 @@ static inline u64 mul_u64_u32_div(u64 a, u32 mul, u32 divisor) · 118 unmodified lines | ||
| 282 | 293 | } |
| 283 | 294 | #endif /* mul_u64_u32_div */ |
| 284 | 295 | |
| 285 | −u64 mul_u64_u64_div_u64(u64 a, u64 mul, u64 div); | |
| 296 | +/** | |
| 297 | + * mul_u64_add_u64_div_u64 - unsigned 64bit multiply, add, and divide | |
| 298 | + * @a: first unsigned 64bit multiplicand | |
| 299 | + * @b: second unsigned 64bit multiplicand | |
| 300 | + * @c: unsigned 64bit addend | |
| 301 | + * @d: unsigned 64bit divisor | |
| 302 | + * | |
| 303 | + * Multiply two 64bit values together to generate a 128bit product | |
| 304 | + * add a third value and then divide by a fourth. | |
| 305 | + * The Generic code divides by 0 if @d is zero and returns ~0 on overflow. | |
| 306 | + * Architecture specific code may trap on zero or overflow. | |
| 307 | + * | |
| 308 | + * Return: (@a * @b + @c) / @d | |
| 309 | + */ | |
| 310 | +u64 mul_u64_add_u64_div_u64(u64 a, u64 b, u64 c, u64 d); | |
| 311 | + | |
| 312 | +/** | |
| 313 | + * mul_u64_u64_div_u64 - unsigned 64bit multiply and divide | |
| 314 | + * @a: first unsigned 64bit multiplicand | |
| 315 | + * @b: second unsigned 64bit multiplicand | |
| 316 | + * @d: unsigned 64bit divisor | |
| 317 | + * | |
| 318 | + * Multiply two 64bit values together to generate a 128bit product | |
| 319 | + * and then divide by a third value. | |
| 320 | + * The Generic code divides by 0 if @d is zero and returns ~0 on overflow. | |
| 321 | + * Architecture specific code may trap on zero or overflow. | |
| 322 | + * | |
| 323 | + * Return: @a * @b / @d | |
| 324 | + */ | |
| 325 | +#define mul_u64_u64_div_u64(a, b, d) mul_u64_add_u64_div_u64(a, b, 0, d) | |
| 326 | + | |
| 327 | +/** | |
| 328 | + * mul_u64_u64_div_u64_roundup - unsigned 64bit multiply and divide rounded up | |
| 329 | + * @a: first unsigned 64bit multiplicand | |
| 330 | + * @b: second unsigned 64bit multiplicand | |
| 331 | + * @d: unsigned 64bit divisor | |
| 332 | + * | |
| 333 | + * Multiply two 64bit values together to generate a 128bit product | |
| 334 | + * and then divide and round up. | |
| 335 | + * The Generic code divides by 0 if @d is zero and returns ~0 on overflow. | |
| 336 | + * Architecture specific code may trap on zero or overflow. | |
| 337 | + * | |
| 338 | + * Return: (@a * @b + @d - 1) / @d | |
| 339 | + */ | |
| 340 | +#define mul_u64_u64_div_u64_roundup(a, b, d) \ | |
| 341 | + ({ u64 _tmp = (d); mul_u64_add_u64_div_u64(a, b, _tmp - 1, _tmp); }) | |
| 342 | + | |
| 286 | 343 | |
| 287 | 344 | /** |
| 288 | 345 | * DIV64_U64_ROUND_UP - unsigned 64bit divide with 64bit divisor rounded up |
| @@ -16,7 +16,7 @@ · 15 unmodified lines | ||
| 16 | 16 | bool __ret_cond = !!(condition); \ |
| 17 | 17 | bool __ret_once = false; \ |
| 18 | 18 | \ |
| 19 | − if (unlikely(__ret_cond && !__already_done)) { \ | |
| 19 | + if (unlikely(__ret_cond) && unlikely(!__already_done)) {\ | |
| 20 | 20 | __already_done = true; \ |
| 21 | 21 | __ret_once = true; \ |
| 22 | 22 | } \ |
| @@ -86,7 +86,6 @@ static inline void set_arch_panic_timeout(int timeout, int arch_default_timeout) · 85 unmodified lines | ||
| 86 | 86 | struct taint_flag { |
| 87 | 87 | char c_true; /* character printed when tainted */ |
| 88 | 88 | char c_false; /* character printed when not tainted */ |
| 89 | − bool module; /* also show as a per-module taint flag */ | |
| 90 | 89 | const char *desc; /* verbose description of the set taint flag */ |
| 91 | 90 | }; |
| 92 | 91 | |
| @@ -43,8 +43,36 @@ extern void rb_erase(struct rb_node *, struct rb_root *); · 42 unmodified lines | ||
| 43 | 43 | /* Find logical next and previous nodes in a tree */ |
| 44 | 44 | extern struct rb_node *rb_next(const struct rb_node *); |
| 45 | 45 | extern struct rb_node *rb_prev(const struct rb_node *); |
| 46 | −extern struct rb_node *rb_first(const struct rb_root *); | |
| 47 | −extern struct rb_node *rb_last(const struct rb_root *); | |
| 46 | + | |
| 47 | +/* | |
| 48 | + * This function returns the first node (in sort order) of the tree. | |
| 49 | + */ | |
| 50 | +static inline struct rb_node *rb_first(const struct rb_root *root) | |
| 51 | +{ | |
| 52 | + struct rb_node *n; | |
| 53 | + | |
| 54 | + n = root->rb_node; | |
| 55 | + if (!n) | |
| 56 | + return NULL; | |
| 57 | + while (n->rb_left) | |
| 58 | + n = n->rb_left; | |
| 59 | + return n; | |
| 60 | +} | |
| 61 | + | |
| 62 | +/* | |
| 63 | + * This function returns the last node (in sort order) of the tree. | |
| 64 | + */ | |
| 65 | +static inline struct rb_node *rb_last(const struct rb_root *root) | |
| 66 | +{ | |
| 67 | + struct rb_node *n; | |
| 68 | + | |
| 69 | + n = root->rb_node; | |
| 70 | + if (!n) | |
| 71 | + return NULL; | |
| 72 | + while (n->rb_right) | |
| 73 | + n = n->rb_right; | |
| 74 | + return n; | |
| 75 | +} | |
| 48 | 76 | |
| 49 | 77 | /* Postorder iteration - always visit the parent after its children */ |
| 50 | 78 | extern struct rb_node *rb_first_postorder(const struct rb_root *); |
| @@ -10,6 +10,7 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <linux/xattr.h> |
| 11 | 11 | #include <linux/fs_parser.h> |
| 12 | 12 | #include <linux/userfaultfd_k.h> |
| 13 | +#include <linux/bits.h> | |
| 13 | 14 | |
| 14 | 15 | struct swap_iocb; |
| 15 | 16 | |
| @@ -19,6 +20,19 @@ struct swap_iocb; · 3 unmodified lines | ||
| 19 | 20 | #define SHMEM_MAXQUOTAS 2 |
| 20 | 21 | #endif |
| 21 | 22 | |
| 23 | +/* Suppress pre-accounting of the entire object size. */ | |
| 24 | +#define SHMEM_F_NORESERVE BIT(0) | |
| 25 | +/* Disallow swapping. */ | |
| 26 | +#define SHMEM_F_LOCKED BIT(1) | |
| 27 | +/* | |
| 28 | + * Disallow growing, shrinking, or hole punching in the inode. Combined with | |
| 29 | + * folio pinning, makes sure the inode's mapping stays fixed. | |
| 30 | + * | |
| 31 | + * In some ways similar to F_SEAL_GROW | F_SEAL_SHRINK, but can be removed and | |
| 32 | + * isn't directly visible to userspace. | |
| 33 | + */ | |
| 34 | +#define SHMEM_F_MAPPING_FROZEN BIT(2) | |
| 35 | + | |
| 22 | 36 | struct shmem_inode_info { |
| 23 | 37 | spinlock_t lock; |
| 24 | 38 | unsigned int seals; /* shmem seals */ |
| @@ -186,6 +200,15 @@ static inline bool shmem_file(struct file *file) · 161 unmodified lines | ||
| 186 | 200 | return shmem_mapping(file->f_mapping); |
| 187 | 201 | } |
| 188 | 202 | |
| 203 | +/* Must be called with inode lock taken exclusive. */ | |
| 204 | +static inline void shmem_freeze(struct inode *inode, bool freeze) | |
| 205 | +{ | |
| 206 | + if (freeze) | |
| 207 | + SHMEM_I(inode)->flags |= SHMEM_F_MAPPING_FROZEN; | |
| 208 | + else | |
| 209 | + SHMEM_I(inode)->flags &= ~SHMEM_F_MAPPING_FROZEN; | |
| 210 | +} | |
| 211 | + | |
| 189 | 212 | /* |
| 190 | 213 | * If fallocate(FALLOC_FL_KEEP_SIZE) has been used, there may be pages |
| 191 | 214 | * beyond i_size's notion of EOF, which fallocate has committed to reserving: |
| @@ -14,7 +14,7 @@ · 13 unmodified lines | ||
| 14 | 14 | #define SYS_INFO_LOCKS 0x00000008 |
| 15 | 15 | #define SYS_INFO_FTRACE 0x00000010 |
| 16 | 16 | #define SYS_INFO_PANIC_CONSOLE_REPLAY 0x00000020 |
| 17 | −#define SYS_INFO_ALL_CPU_BT 0x00000040 | |
| 17 | +#define SYS_INFO_ALL_BT 0x00000040 | |
| 18 | 18 | #define SYS_INFO_BLOCKED_TASKS 0x00000080 |
| 19 | 19 | |
| 20 | 20 | void sys_info(unsigned long si_mask); |
| @@ -161,8 +161,6 @@ __copy_to_user(void __user *to, const void *from, unsigned long n) · 160 unmodified lines | ||
| 161 | 161 | * directly in the normal copy_to/from_user(), the other ones go |
| 162 | 162 | * through an extern _copy_to/from_user(), which expands the same code |
| 163 | 163 | * here. |
| 164 | − * | |
| 165 | − * Rust code always uses the extern definition. | |
| 166 | 164 | */ |
| 167 | 165 | static inline __must_check unsigned long |
| 168 | 166 | _inline_copy_from_user(void *to, const void __user *from, unsigned long n) |
| @@ -192,8 +190,10 @@ fail: · 23 unmodified lines | ||
| 192 | 190 | memset(to + (n - res), 0, res); |
| 193 | 191 | return res; |
| 194 | 192 | } |
| 193 | +#ifndef INLINE_COPY_FROM_USER | |
| 195 | 194 | extern __must_check unsigned long |
| 196 | 195 | _copy_from_user(void *, const void __user *, unsigned long); |
| 196 | +#endif | |
| 197 | 197 | |
| 198 | 198 | static inline __must_check unsigned long |
| 199 | 199 | _inline_copy_to_user(void __user *to, const void *from, unsigned long n) |
| @@ -207,8 +207,10 @@ _inline_copy_to_user(void __user *to, const void *from, unsigned long n) · 7 unmodified lines | ||
| 207 | 207 | } |
| 208 | 208 | return n; |
| 209 | 209 | } |
| 210 | +#ifndef INLINE_COPY_TO_USER | |
| 210 | 211 | extern __must_check unsigned long |
| 211 | 212 | _copy_to_user(void __user *, const void *, unsigned long); |
| 213 | +#endif | |
| 212 | 214 | |
| 213 | 215 | static __always_inline unsigned long __must_check |
| 214 | 216 | copy_from_user(void *to, const void __user *from, unsigned long n) |
| @@ -136,10 +136,10 @@ · 135 unmodified lines | ||
| 136 | 136 | #define PTR_IF(cond, ptr) ((cond) ? (ptr) : NULL) |
| 137 | 137 | |
| 138 | 138 | /** |
| 139 | − * to_user_ptr - cast a pointer passed as u64 from user space to void __user * | |
| 139 | + * u64_to_user_ptr - cast a pointer passed as u64 from user space to void __user * | |
| 140 | 140 | * @x: The u64 value from user space, usually via IOCTL |
| 141 | 141 | * |
| 142 | − * to_user_ptr() simply casts a pointer passed as u64 from user space to void | |
| 142 | + * u64_to_user_ptr() simply casts a pointer passed as u64 from user space to void | |
| 143 | 143 | * __user * correctly. Using this lets us get rid of all the tiresome casts. |
| 144 | 144 | */ |
| 145 | 145 | #define u64_to_user_ptr(x) \ |
| @@ -5,6 +5,7 @@ · 4 unmodified lines | ||
| 5 | 5 | #include <linux/linkage.h> |
| 6 | 6 | #include <linux/elfcore.h> |
| 7 | 7 | #include <linux/elf.h> |
| 8 | +#include <uapi/linux/vmcore.h> | |
| 8 | 9 | |
| 9 | 10 | #define CRASH_CORE_NOTE_HEAD_BYTES ALIGN(sizeof(struct elf_note), 4) |
| 10 | 11 | #define CRASH_CORE_NOTE_NAME_BYTES ALIGN(sizeof(NN_PRSTATUS), 4) |
| @@ -77,4 +78,11 @@ extern u32 *vmcoreinfo_note; · 66 unmodified lines | ||
| 77 | 78 | Elf_Word *append_elf_note(Elf_Word *buf, char *name, unsigned int type, |
| 78 | 79 | void *data, size_t data_len); |
| 79 | 80 | void final_note(Elf_Word *buf); |
| 81 | + | |
| 82 | +#ifdef CONFIG_VMCORE_INFO | |
| 83 | +void hwerr_log_error_type(enum hwerr_error_type src); | |
| 84 | +#else | |
| 85 | +static inline void hwerr_log_error_type(enum hwerr_error_type src) {}; | |
| 86 | +#endif | |
| 87 | + | |
| 80 | 88 | #endif /* LINUX_VMCORE_INFO_H */ |
| @@ -141,21 +141,7 @@ static inline unsigned long xxhash(const void *input, size_t length, · 140 unmodified lines | ||
| 141 | 141 | */ |
| 142 | 142 | |
| 143 | 143 | /** |
| 144 | − * struct xxh32_state - private xxh32 state, do not use members directly | |
| 145 | − */ | |
| 146 | −struct xxh32_state { | |
| 147 | − uint32_t total_len_32; | |
| 148 | − uint32_t large_len; | |
| 149 | − uint32_t v1; | |
| 150 | − uint32_t v2; | |
| 151 | − uint32_t v3; | |
| 152 | − uint32_t v4; | |
| 153 | − uint32_t mem32[4]; | |
| 154 | − uint32_t memsize; | |
| 155 | −}; | |
| 156 | − | |
| 157 | −/** | |
| 158 | − * struct xxh32_state - private xxh64 state, do not use members directly | |
| 144 | + * struct xxh64_state - private xxh64 state, do not use members directly | |
| 159 | 145 | */ |
| 160 | 146 | struct xxh64_state { |
| 161 | 147 | uint64_t total_len; |
| @@ -167,16 +153,6 @@ struct xxh64_state { · 5 unmodified lines | ||
| 167 | 153 | uint32_t memsize; |
| 168 | 154 | }; |
| 169 | 155 | |
| 170 | −/** | |
| 171 | − * xxh32_reset() - reset the xxh32 state to start a new hashing operation | |
| 172 | − * | |
| 173 | − * @state: The xxh32 state to reset. | |
| 174 | − * @seed: Initialize the hash state with this seed. | |
| 175 | − * | |
| 176 | − * Call this function on any xxh32_state to prepare for a new hashing operation. | |
| 177 | − */ | |
| 178 | −void xxh32_reset(struct xxh32_state *state, uint32_t seed); | |
| 179 | − | |
| 180 | 156 | /** |
| 181 | 157 | * xxh64_reset() - reset the xxh64 state to start a new hashing operation |
| 182 | 158 | * |
| @@ -210,24 +186,4 @@ int xxh64_update(struct xxh64_state *state, const void *input, size_t length); · 27 unmodified lines | ||
| 210 | 186 | */ |
| 211 | 187 | uint64_t xxh64_digest(const struct xxh64_state *state); |
| 212 | 188 | |
| 213 | −/*-************************** | |
| 214 | − * Utils | |
| 215 | − ***************************/ | |
| 216 | − | |
| 217 | −/** | |
| 218 | − * xxh32_copy_state() - copy the source state into the destination state | |
| 219 | − * | |
| 220 | − * @src: The source xxh32 state. | |
| 221 | − * @dst: The destination xxh32 state. | |
| 222 | − */ | |
| 223 | −void xxh32_copy_state(struct xxh32_state *dst, const struct xxh32_state *src); | |
| 224 | − | |
| 225 | −/** | |
| 226 | − * xxh64_copy_state() - copy the source state into the destination state | |
| 227 | − * | |
| 228 | − * @src: The source xxh64 state. | |
| 229 | − * @dst: The destination xxh64 state. | |
| 230 | − */ | |
| 231 | −void xxh64_copy_state(struct xxh64_state *dst, const struct xxh64_state *src); | |
| 232 | − | |
| 233 | 189 | #endif /* XXHASH_H */ |
| @@ -0,0 +1,216 @@ | ||
| 1 | +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ | |
| 2 | + | |
| 3 | +/* | |
| 4 | + * Userspace interface for /dev/liveupdate | |
| 5 | + * Live Update Orchestrator | |
| 6 | + * | |
| 7 | + * Copyright (c) 2025, Google LLC. | |
| 8 | + * Pasha Tatashin <pasha.tatashin@soleen.com> | |
| 9 | + */ | |
| 10 | + | |
| 11 | +#ifndef _UAPI_LIVEUPDATE_H | |
| 12 | +#define _UAPI_LIVEUPDATE_H | |
| 13 | + | |
| 14 | +#include <linux/ioctl.h> | |
| 15 | +#include <linux/types.h> | |
| 16 | + | |
| 17 | +/** | |
| 18 | + * DOC: General ioctl format | |
| 19 | + * | |
| 20 | + * The ioctl interface follows a general format to allow for extensibility. Each | |
| 21 | + * ioctl is passed in a structure pointer as the argument providing the size of | |
| 22 | + * the structure in the first u32. The kernel checks that any structure space | |
| 23 | + * beyond what it understands is 0. This allows userspace to use the backward | |
| 24 | + * compatible portion while consistently using the newer, larger, structures. | |
| 25 | + * | |
| 26 | + * ioctls use a standard meaning for common errnos: | |
| 27 | + * | |
| 28 | + * - ENOTTY: The IOCTL number itself is not supported at all | |
| 29 | + * - E2BIG: The IOCTL number is supported, but the provided structure has | |
| 30 | + * non-zero in a part the kernel does not understand. | |
| 31 | + * - EOPNOTSUPP: The IOCTL number is supported, and the structure is | |
| 32 | + * understood, however a known field has a value the kernel does not | |
| 33 | + * understand or support. | |
| 34 | + * - EINVAL: Everything about the IOCTL was understood, but a field is not | |
| 35 | + * correct. | |
| 36 | + * - ENOENT: A provided token does not exist. | |
| 37 | + * - ENOMEM: Out of memory. | |
| 38 | + * - EOVERFLOW: Mathematics overflowed. | |
| 39 | + * | |
| 40 | + * As well as additional errnos, within specific ioctls. | |
| 41 | + */ | |
| 42 | + | |
| 43 | +/* The ioctl type, documented in ioctl-number.rst */ | |
| 44 | +#define LIVEUPDATE_IOCTL_TYPE 0xBA | |
| 45 | + | |
| 46 | +/* The maximum length of session name including null termination */ | |
| 47 | +#define LIVEUPDATE_SESSION_NAME_LENGTH 64 | |
| 48 | + | |
| 49 | +/* The /dev/liveupdate ioctl commands */ | |
| 50 | +enum { | |
| 51 | + LIVEUPDATE_CMD_BASE = 0x00, | |
| 52 | + LIVEUPDATE_CMD_CREATE_SESSION = LIVEUPDATE_CMD_BASE, | |
| 53 | + LIVEUPDATE_CMD_RETRIEVE_SESSION = 0x01, | |
| 54 | +}; | |
| 55 | + | |
| 56 | +/* ioctl commands for session file descriptors */ | |
| 57 | +enum { | |
| 58 | + LIVEUPDATE_CMD_SESSION_BASE = 0x40, | |
| 59 | + LIVEUPDATE_CMD_SESSION_PRESERVE_FD = LIVEUPDATE_CMD_SESSION_BASE, | |
| 60 | + LIVEUPDATE_CMD_SESSION_RETRIEVE_FD = 0x41, | |
| 61 | + LIVEUPDATE_CMD_SESSION_FINISH = 0x42, | |
| 62 | +}; | |
| 63 | + | |
| 64 | +/** | |
| 65 | + * struct liveupdate_ioctl_create_session - ioctl(LIVEUPDATE_IOCTL_CREATE_SESSION) | |
| 66 | + * @size: Input; sizeof(struct liveupdate_ioctl_create_session) | |
| 67 | + * @fd: Output; The new file descriptor for the created session. | |
| 68 | + * @name: Input; A null-terminated string for the session name, max | |
| 69 | + * length %LIVEUPDATE_SESSION_NAME_LENGTH including termination | |
| 70 | + * character. | |
| 71 | + * | |
| 72 | + * Creates a new live update session for managing preserved resources. | |
| 73 | + * This ioctl can only be called on the main /dev/liveupdate device. | |
| 74 | + * | |
| 75 | + * Return: 0 on success, negative error code on failure. | |
| 76 | + */ | |
| 77 | +struct liveupdate_ioctl_create_session { | |
| 78 | + __u32 size; | |
| 79 | + __s32 fd; | |
| 80 | + __u8 name[LIVEUPDATE_SESSION_NAME_LENGTH]; | |
| 81 | +}; | |
| 82 | + | |
| 83 | +#define LIVEUPDATE_IOCTL_CREATE_SESSION \ | |
| 84 | + _IO(LIVEUPDATE_IOCTL_TYPE, LIVEUPDATE_CMD_CREATE_SESSION) | |
| 85 | + | |
| 86 | +/** | |
| 87 | + * struct liveupdate_ioctl_retrieve_session - ioctl(LIVEUPDATE_IOCTL_RETRIEVE_SESSION) | |
| 88 | + * @size: Input; sizeof(struct liveupdate_ioctl_retrieve_session) | |
| 89 | + * @fd: Output; The new file descriptor for the retrieved session. | |
| 90 | + * @name: Input; A null-terminated string identifying the session to retrieve. | |
| 91 | + * The name must exactly match the name used when the session was | |
| 92 | + * created in the previous kernel. | |
| 93 | + * | |
| 94 | + * Retrieves a handle (a new file descriptor) for a preserved session by its | |
| 95 | + * name. This is the primary mechanism for a userspace agent to regain control | |
| 96 | + * of its preserved resources after a live update. | |
| 97 | + * | |
| 98 | + * The userspace application provides the null-terminated `name` of a session | |
| 99 | + * it created before the live update. If a preserved session with a matching | |
| 100 | + * name is found, the kernel instantiates it and returns a new file descriptor | |
| 101 | + * in the `fd` field. This new session FD can then be used for all file-specific | |
| 102 | + * operations, such as restoring individual file descriptors with | |
| 103 | + * LIVEUPDATE_SESSION_RETRIEVE_FD. | |
| 104 | + * | |
| 105 | + * It is the responsibility of the userspace application to know the names of | |
| 106 | + * the sessions it needs to retrieve. If no session with the given name is | |
| 107 | + * found, the ioctl will fail with -ENOENT. | |
| 108 | + * | |
| 109 | + * This ioctl can only be called on the main /dev/liveupdate device when the | |
| 110 | + * system is in the LIVEUPDATE_STATE_UPDATED state. | |
| 111 | + */ | |
| 112 | +struct liveupdate_ioctl_retrieve_session { | |
| 113 | + __u32 size; | |
| 114 | + __s32 fd; | |
| 115 | + __u8 name[LIVEUPDATE_SESSION_NAME_LENGTH]; | |
| 116 | +}; | |
| 117 | + | |
| 118 | +#define LIVEUPDATE_IOCTL_RETRIEVE_SESSION \ | |
| 119 | + _IO(LIVEUPDATE_IOCTL_TYPE, LIVEUPDATE_CMD_RETRIEVE_SESSION) | |
| 120 | + | |
| 121 | +/* Session specific IOCTLs */ | |
| 122 | + | |
| 123 | +/** | |
| 124 | + * struct liveupdate_session_preserve_fd - ioctl(LIVEUPDATE_SESSION_PRESERVE_FD) | |
| 125 | + * @size: Input; sizeof(struct liveupdate_session_preserve_fd) | |
| 126 | + * @fd: Input; The user-space file descriptor to be preserved. | |
| 127 | + * @token: Input; An opaque, unique token for preserved resource. | |
| 128 | + * | |
| 129 | + * Holds parameters for preserving a file descriptor. | |
| 130 | + * | |
| 131 | + * User sets the @fd field identifying the file descriptor to preserve | |
| 132 | + * (e.g., memfd, kvm, iommufd, VFIO). The kernel validates if this FD type | |
| 133 | + * and its dependencies are supported for preservation. If validation passes, | |
| 134 | + * the kernel marks the FD internally and *initiates the process* of preparing | |
| 135 | + * its state for saving. The actual snapshotting of the state typically occurs | |
| 136 | + * during the subsequent %LIVEUPDATE_IOCTL_PREPARE execution phase, though | |
| 137 | + * some finalization might occur during freeze. | |
| 138 | + * On successful validation and initiation, the kernel uses the @token | |
| 139 | + * field with an opaque identifier representing the resource being preserved. | |
| 140 | + * This token confirms the FD is targeted for preservation and is required for | |
| 141 | + * the subsequent %LIVEUPDATE_SESSION_RETRIEVE_FD call after the live update. | |
| 142 | + * | |
| 143 | + * Return: 0 on success (validation passed, preservation initiated), negative | |
| 144 | + * error code on failure (e.g., unsupported FD type, dependency issue, | |
| 145 | + * validation failed). | |
| 146 | + */ | |
| 147 | +struct liveupdate_session_preserve_fd { | |
| 148 | + __u32 size; | |
| 149 | + __s32 fd; | |
| 150 | + __aligned_u64 token; | |
| 151 | +}; | |
| 152 | + | |
| 153 | +#define LIVEUPDATE_SESSION_PRESERVE_FD \ | |
| 154 | + _IO(LIVEUPDATE_IOCTL_TYPE, LIVEUPDATE_CMD_SESSION_PRESERVE_FD) | |
| 155 | + | |
| 156 | +/** | |
| 157 | + * struct liveupdate_session_retrieve_fd - ioctl(LIVEUPDATE_SESSION_RETRIEVE_FD) | |
| 158 | + * @size: Input; sizeof(struct liveupdate_session_retrieve_fd) | |
| 159 | + * @fd: Output; The new file descriptor representing the fully restored | |
| 160 | + * kernel resource. | |
| 161 | + * @token: Input; An opaque, token that was used to preserve the resource. | |
| 162 | + * | |
| 163 | + * Retrieve a previously preserved file descriptor. | |
| 164 | + * | |
| 165 | + * User sets the @token field to the value obtained from a successful | |
| 166 | + * %LIVEUPDATE_IOCTL_FD_PRESERVE call before the live update. On success, | |
| 167 | + * the kernel restores the state (saved during the PREPARE/FREEZE phases) | |
| 168 | + * associated with the token and populates the @fd field with a new file | |
| 169 | + * descriptor referencing the restored resource in the current (new) kernel. | |
| 170 | + * This operation must be performed *before* signaling completion via | |
| 171 | + * %LIVEUPDATE_IOCTL_FINISH. | |
| 172 | + * | |
| 173 | + * Return: 0 on success, negative error code on failure (e.g., invalid token). | |
| 174 | + */ | |
| 175 | +struct liveupdate_session_retrieve_fd { | |
| 176 | + __u32 size; | |
| 177 | + __s32 fd; | |
| 178 | + __aligned_u64 token; | |
| 179 | +}; | |
| 180 | + | |
| 181 | +#define LIVEUPDATE_SESSION_RETRIEVE_FD \ | |
| 182 | + _IO(LIVEUPDATE_IOCTL_TYPE, LIVEUPDATE_CMD_SESSION_RETRIEVE_FD) | |
| 183 | + | |
| 184 | +/** | |
| 185 | + * struct liveupdate_session_finish - ioctl(LIVEUPDATE_SESSION_FINISH) | |
| 186 | + * @size: Input; sizeof(struct liveupdate_session_finish) | |
| 187 | + * @reserved: Input; Must be zero. Reserved for future use. | |
| 188 | + * | |
| 189 | + * Signals the completion of the restoration process for a retrieved session. | |
| 190 | + * This is the final operation that should be performed on a session file | |
| 191 | + * descriptor after a live update. | |
| 192 | + * | |
| 193 | + * This ioctl must be called once all required file descriptors for the session | |
| 194 | + * have been successfully retrieved (using %LIVEUPDATE_SESSION_RETRIEVE_FD) and | |
| 195 | + * are fully restored from the userspace and kernel perspective. | |
| 196 | + * | |
| 197 | + * Upon success, the kernel releases its ownership of the preserved resources | |
| 198 | + * associated with this session. This allows internal resources to be freed, | |
| 199 | + * typically by decrementing reference counts on the underlying preserved | |
| 200 | + * objects. | |
| 201 | + * | |
| 202 | + * If this operation fails, the resources remain preserved in memory. Userspace | |
| 203 | + * may attempt to call finish again. The resources will otherwise be reset | |
| 204 | + * during the next live update cycle. | |
| 205 | + * | |
| 206 | + * Return: 0 on success, negative error code on failure. | |
| 207 | + */ | |
| 208 | +struct liveupdate_session_finish { | |
| 209 | + __u32 size; | |
| 210 | + __u32 reserved; | |
| 211 | +}; | |
| 212 | + | |
| 213 | +#define LIVEUPDATE_SESSION_FINISH \ | |
| 214 | + _IO(LIVEUPDATE_IOCTL_TYPE, LIVEUPDATE_CMD_SESSION_FINISH) | |
| 215 | + | |
| 216 | +#endif /* _UAPI_LIVEUPDATE_H */ | |
| @@ -15,4 +15,13 @@ struct vmcoredd_header { · 14 unmodified lines | ||
| 15 | 15 | __u8 dump_name[VMCOREDD_MAX_NAME_BYTES]; /* Device dump's name */ |
| 16 | 16 | }; |
| 17 | 17 | |
| 18 | +enum hwerr_error_type { | |
| 19 | + HWERR_RECOV_CPU, | |
| 20 | + HWERR_RECOV_MEMORY, | |
| 21 | + HWERR_RECOV_PCI, | |
| 22 | + HWERR_RECOV_CXL, | |
| 23 | + HWERR_RECOV_OTHERS, | |
| 24 | + HWERR_RECOV_MAX, | |
| 25 | +}; | |
| 26 | + | |
| 18 | 27 | #endif /* _UAPI_VMCORE_H */ |
| @@ -1519,6 +1519,24 @@ config BOOT_CONFIG_EMBED_FILE · 1518 unmodified lines | ||
| 1519 | 1519 | This bootconfig will be used if there is no initrd or no other |
| 1520 | 1520 | bootconfig in the initrd. |
| 1521 | 1521 | |
| 1522 | +config CMDLINE_LOG_WRAP_IDEAL_LEN | |
| 1523 | + int "Length to try to wrap the cmdline when logged at boot" | |
| 1524 | + default 1021 | |
| 1525 | + range 0 1021 | |
| 1526 | + help | |
| 1527 | + At boot time, the kernel command line is logged to the console. | |
| 1528 | + The log message will start with the prefix "Kernel command line: ". | |
| 1529 | + The log message will attempt to be wrapped (split into multiple log | |
| 1530 | + messages) at spaces based on CMDLINE_LOG_WRAP_IDEAL_LEN characters. | |
| 1531 | + If wrapping happens, each log message will start with the prefix and | |
| 1532 | + all but the last message will end with " \". Messages may exceed the | |
| 1533 | + ideal length if a place to wrap isn't found before the specified | |
| 1534 | + number of characters. | |
| 1535 | + | |
| 1536 | + A value of 0 disables wrapping, though be warned that the maximum | |
| 1537 | + length of a log message (1021 characters) may cause the cmdline to | |
| 1538 | + be truncated. | |
| 1539 | + | |
| 1522 | 1540 | config INITRAMFS_PRESERVE_MTIME |
| 1523 | 1541 | bool "Preserve cpio archive mtimes in initramfs" |
| 1524 | 1542 | depends on BLK_DEV_INITRD |
| @@ -2171,6 +2189,8 @@ config TRACEPOINTS · 646 unmodified lines | ||
| 2171 | 2189 | |
| 2172 | 2190 | source "kernel/Kconfig.kexec" |
| 2173 | 2191 | |
| 2192 | +source "kernel/liveupdate/Kconfig" | |
| 2193 | + | |
| 2174 | 2194 | endmenu # General setup |
| 2175 | 2195 | |
| 2176 | 2196 | source "arch/Kconfig" |
| @@ -5,19 +5,22 @@ · 4 unmodified lines | ||
| 5 | 5 | * Copyright (C) 1991, 1992 Linus Torvalds |
| 6 | 6 | */ |
| 7 | 7 | |
| 8 | −#include <linux/jiffies.h> | |
| 9 | 8 | #include <linux/delay.h> |
| 10 | 9 | #include <linux/init.h> |
| 11 | −#include <linux/timex.h> | |
| 12 | −#include <linux/smp.h> | |
| 10 | +#include <linux/jiffies.h> | |
| 11 | +#include <linux/kstrtox.h> | |
| 13 | 12 | #include <linux/percpu.h> |
| 13 | +#include <linux/printk.h> | |
| 14 | +#include <linux/smp.h> | |
| 15 | +#include <linux/stddef.h> | |
| 16 | +#include <linux/timex.h> | |
| 14 | 17 | |
| 15 | 18 | unsigned long lpj_fine; |
| 16 | 19 | unsigned long preset_lpj; |
| 20 | + | |
| 17 | 21 | static int __init lpj_setup(char *str) |
| 18 | 22 | { |
| 19 | − preset_lpj = simple_strtoul(str,NULL,0); | |
| 20 | − return 1; | |
| 23 | + return kstrtoul(str, 0, &preset_lpj) == 0; | |
| 21 | 24 | } |
| 22 | 25 | |
| 23 | 26 | __setup("lpj=", lpj_setup); |
| @@ -906,6 +906,101 @@ static void __init early_numa_node_init(void) · 905 unmodified lines | ||
| 906 | 906 | #endif |
| 907 | 907 | } |
| 908 | 908 | |
| 909 | +#define KERNEL_CMDLINE_PREFIX "Kernel command line: " | |
| 910 | +#define KERNEL_CMDLINE_PREFIX_LEN (sizeof(KERNEL_CMDLINE_PREFIX) - 1) | |
| 911 | +#define KERNEL_CMDLINE_CONTINUATION " \\" | |
| 912 | +#define KERNEL_CMDLINE_CONTINUATION_LEN (sizeof(KERNEL_CMDLINE_CONTINUATION) - 1) | |
| 913 | + | |
| 914 | +#define MIN_CMDLINE_LOG_WRAP_IDEAL_LEN (KERNEL_CMDLINE_PREFIX_LEN + \ | |
| 915 | + KERNEL_CMDLINE_CONTINUATION_LEN) | |
| 916 | +#define CMDLINE_LOG_WRAP_IDEAL_LEN (CONFIG_CMDLINE_LOG_WRAP_IDEAL_LEN > \ | |
| 917 | + MIN_CMDLINE_LOG_WRAP_IDEAL_LEN ? \ | |
| 918 | + CONFIG_CMDLINE_LOG_WRAP_IDEAL_LEN : \ | |
| 919 | + MIN_CMDLINE_LOG_WRAP_IDEAL_LEN) | |
| 920 | + | |
| 921 | +#define IDEAL_CMDLINE_LEN (CMDLINE_LOG_WRAP_IDEAL_LEN - KERNEL_CMDLINE_PREFIX_LEN) | |
| 922 | +#define IDEAL_CMDLINE_SPLIT_LEN (IDEAL_CMDLINE_LEN - KERNEL_CMDLINE_CONTINUATION_LEN) | |
| 923 | + | |
| 924 | +/** | |
| 925 | + * print_kernel_cmdline() - Print the kernel cmdline with wrapping. | |
| 926 | + * @cmdline: The cmdline to print. | |
| 927 | + * | |
| 928 | + * Print the kernel command line, trying to wrap based on the Kconfig knob | |
| 929 | + * CONFIG_CMDLINE_LOG_WRAP_IDEAL_LEN. | |
| 930 | + * | |
| 931 | + * Wrapping is based on spaces, ignoring quotes. All lines are prefixed | |
| 932 | + * with "Kernel command line: " and lines that are not the last line have | |
| 933 | + * a " \" suffix added to them. The prefix and suffix count towards the | |
| 934 | + * line length for wrapping purposes. The ideal length will be exceeded | |
| 935 | + * if no appropriate place to wrap is found. | |
| 936 | + * | |
| 937 | + * Example output if CONFIG_CMDLINE_LOG_WRAP_IDEAL_LEN is 40: | |
| 938 | + * Kernel command line: loglevel=7 \ | |
| 939 | + * Kernel command line: init=/sbin/init \ | |
| 940 | + * Kernel command line: root=PARTUUID=8c3efc1a-768b-6642-8d0c-89eb782f19f0/PARTNROFF=1 \ | |
| 941 | + * Kernel command line: rootwait ro \ | |
| 942 | + * Kernel command line: my_quoted_arg="The \ | |
| 943 | + * Kernel command line: quick brown fox \ | |
| 944 | + * Kernel command line: jumps over the \ | |
| 945 | + * Kernel command line: lazy dog." | |
| 946 | + */ | |
| 947 | +static void __init print_kernel_cmdline(const char *cmdline) | |
| 948 | +{ | |
| 949 | + size_t len; | |
| 950 | + | |
| 951 | + /* Config option of 0 or anything longer than the max disables wrapping */ | |
| 952 | + if (CONFIG_CMDLINE_LOG_WRAP_IDEAL_LEN == 0 || | |
| 953 | + IDEAL_CMDLINE_LEN >= COMMAND_LINE_SIZE - 1) { | |
| 954 | + pr_notice("%s%s\n", KERNEL_CMDLINE_PREFIX, cmdline); | |
| 955 | + return; | |
| 956 | + } | |
| 957 | + | |
| 958 | + len = strlen(cmdline); | |
| 959 | + while (len > IDEAL_CMDLINE_LEN) { | |
| 960 | + const char *first_space; | |
| 961 | + const char *prev_cutoff; | |
| 962 | + const char *cutoff; | |
| 963 | + int to_print; | |
| 964 | + size_t used; | |
| 965 | + | |
| 966 | + /* Find the last ' ' that wouldn't make the line too long */ | |
| 967 | + prev_cutoff = NULL; | |
| 968 | + cutoff = cmdline; | |
| 969 | + while (true) { | |
| 970 | + cutoff = strchr(cutoff + 1, ' '); | |
| 971 | + if (!cutoff || cutoff - cmdline > IDEAL_CMDLINE_SPLIT_LEN) | |
| 972 | + break; | |
| 973 | + prev_cutoff = cutoff; | |
| 974 | + } | |
| 975 | + if (prev_cutoff) | |
| 976 | + cutoff = prev_cutoff; | |
| 977 | + else if (!cutoff) | |
| 978 | + break; | |
| 979 | + | |
| 980 | + /* Find the beginning and end of the string of spaces */ | |
| 981 | + first_space = cutoff; | |
| 982 | + while (first_space > cmdline && first_space[-1] == ' ') | |
| 983 | + first_space--; | |
| 984 | + to_print = first_space - cmdline; | |
| 985 | + while (*cutoff == ' ') | |
| 986 | + cutoff++; | |
| 987 | + used = cutoff - cmdline; | |
| 988 | + | |
| 989 | + /* If the whole string is used, break and do the final printout */ | |
| 990 | + if (len == used) | |
| 991 | + break; | |
| 992 | + | |
| 993 | + if (to_print) | |
| 994 | + pr_notice("%s%.*s%s\n", KERNEL_CMDLINE_PREFIX, | |
| 995 | + to_print, cmdline, KERNEL_CMDLINE_CONTINUATION); | |
| 996 | + | |
| 997 | + len -= used; | |
| 998 | + cmdline += used; | |
| 999 | + } | |
| 1000 | + if (len) | |
| 1001 | + pr_notice("%s%s\n", KERNEL_CMDLINE_PREFIX, cmdline); | |
| 1002 | +} | |
| 1003 | + | |
| 909 | 1004 | asmlinkage __visible __init __no_sanitize_address __noreturn __no_stack_protector |
| 910 | 1005 | void start_kernel(void) |
| 911 | 1006 | { |
| @@ -942,7 +1037,7 @@ void start_kernel(void) · 30 unmodified lines | ||
| 942 | 1037 | early_numa_node_init(); |
| 943 | 1038 | boot_cpu_hotplug_init(); |
| 944 | 1039 | |
| 945 | − pr_notice("Kernel command line: %s\n", saved_command_line); | |
| 1040 | + print_kernel_cmdline(saved_command_line); | |
| 946 | 1041 | /* parameters may set static keys */ |
| 947 | 1042 | parse_early_param(); |
| 948 | 1043 | after_dashes = parse_args("Booting kernel", |
| @@ -76,10 +76,10 @@ static struct ipc_namespace *create_ipc_ns(struct user_namespace *user_ns, · 75 unmodified lines | ||
| 76 | 76 | |
| 77 | 77 | err = -ENOMEM; |
| 78 | 78 | if (!setup_mq_sysctls(ns)) |
| 79 | − goto fail_put; | |
| 79 | + goto fail_mq_mount; | |
| 80 | 80 | |
| 81 | 81 | if (!setup_ipc_sysctls(ns)) |
| 82 | − goto fail_mq; | |
| 82 | + goto fail_mq_sysctls; | |
| 83 | 83 | |
| 84 | 84 | err = msg_init_ns(ns); |
| 85 | 85 | if (err) |
| @@ -93,9 +93,10 @@ static struct ipc_namespace *create_ipc_ns(struct user_namespace *user_ns, · 7 unmodified lines | ||
| 93 | 93 | |
| 94 | 94 | fail_ipc: |
| 95 | 95 | retire_ipc_sysctls(ns); |
| 96 | −fail_mq: | |
| 96 | +fail_mq_sysctls: | |
| 97 | 97 | retire_mq_sysctls(ns); |
| 98 | − | |
| 98 | +fail_mq_mount: | |
| 99 | + mntput(ns->mq_mnt); | |
| 99 | 100 | fail_put: |
| 100 | 101 | put_user_ns(ns->user_ns); |
| 101 | 102 | ns_common_free(ns); |
| @@ -94,30 +94,6 @@ config KEXEC_JUMP · 93 unmodified lines | ||
| 94 | 94 | Jump between original kernel and kexeced kernel and invoke |
| 95 | 95 | code in physical address mode via KEXEC |
| 96 | 96 | |
| 97 | −config KEXEC_HANDOVER | |
| 98 | − bool "kexec handover" | |
| 99 | − depends on ARCH_SUPPORTS_KEXEC_HANDOVER && ARCH_SUPPORTS_KEXEC_FILE | |
| 100 | − depends on !DEFERRED_STRUCT_PAGE_INIT | |
| 101 | − select MEMBLOCK_KHO_SCRATCH | |
| 102 | − select KEXEC_FILE | |
| 103 | − select DEBUG_FS | |
| 104 | − select LIBFDT | |
| 105 | − select CMA | |
| 106 | − help | |
| 107 | − Allow kexec to hand over state across kernels by generating and | |
| 108 | − passing additional metadata to the target kernel. This is useful | |
| 109 | − to keep data or state alive across the kexec. For this to work, | |
| 110 | − both source and target kernels need to have this option enabled. | |
| 111 | − | |
| 112 | −config KEXEC_HANDOVER_DEBUG | |
| 113 | − bool "Enable Kexec Handover debug checks" | |
| 114 | − depends on KEXEC_HANDOVER | |
| 115 | − help | |
| 116 | − This option enables extra sanity checks for the Kexec Handover | |
| 117 | − subsystem. Since, KHO performance is crucial in live update | |
| 118 | − scenarios and the extra code might be adding overhead it is | |
| 119 | − only optionally enabled. | |
| 120 | − | |
| 121 | 97 | config CRASH_DUMP |
| 122 | 98 | bool "kernel crash dumps" |
| 123 | 99 | default ARCH_DEFAULT_CRASH_DUMP |
| @@ -52,6 +52,7 @@ obj-y += printk/ · 51 unmodified lines | ||
| 52 | 52 | obj-y += irq/ |
| 53 | 53 | obj-y += rcu/ |
| 54 | 54 | obj-y += livepatch/ |
| 55 | +obj-y += liveupdate/ | |
| 55 | 56 | obj-y += dma/ |
| 56 | 57 | obj-y += entry/ |
| 57 | 58 | obj-y += unwind/ |
| @@ -82,8 +83,6 @@ obj-$(CONFIG_CRASH_DUMP_KUNIT_TEST) += crash_core_test.o · 24 unmodified lines | ||
| 82 | 83 | obj-$(CONFIG_KEXEC) += kexec.o |
| 83 | 84 | obj-$(CONFIG_KEXEC_FILE) += kexec_file.o |
| 84 | 85 | obj-$(CONFIG_KEXEC_ELF) += kexec_elf.o |
| 85 | −obj-$(CONFIG_KEXEC_HANDOVER) += kexec_handover.o | |
| 86 | −obj-$(CONFIG_KEXEC_HANDOVER_DEBUG) += kexec_handover_debug.o | |
| 87 | 86 | obj-$(CONFIG_BACKTRACE_SELF_TEST) += backtracetest.o |
| 88 | 87 | obj-$(CONFIG_COMPAT) += compat.o |
| 89 | 88 | obj-$(CONFIG_CGROUPS) += cgroup/ |
| @@ -83,7 +83,7 @@ CONFIG_SLUB_DEBUG_ON=y · 82 unmodified lines | ||
| 83 | 83 | # |
| 84 | 84 | # Debug Oops, Lockups and Hangs |
| 85 | 85 | # |
| 86 | −# CONFIG_BOOTPARAM_HUNG_TASK_PANIC is not set | |
| 86 | +CONFIG_BOOTPARAM_HUNG_TASK_PANIC=0 | |
| 87 | 87 | # CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC is not set |
| 88 | 88 | CONFIG_DEBUG_ATOMIC_SLEEP=y |
| 89 | 89 | CONFIG_DETECT_HUNG_TASK=y |
| @@ -524,6 +524,9 @@ void __init reserve_crashkernel_cma(unsigned long long cma_size) · 523 unmodified lines | ||
| 524 | 524 | #ifndef HAVE_ARCH_ADD_CRASH_RES_TO_IOMEM_EARLY |
| 525 | 525 | static __init int insert_crashkernel_resources(void) |
| 526 | 526 | { |
| 527 | + if (!arch_add_crash_res_to_iomem()) | |
| 528 | + return 0; | |
| 529 | + | |
| 527 | 530 | if (crashk_res.start < crashk_res.end) |
| 528 | 531 | insert_resource(&iomem_resource, &crashk_res); |
| 529 | 532 | |
| @@ -251,10 +251,8 @@ repeat: · 250 unmodified lines | ||
| 251 | 251 | memset(&post, 0, sizeof(post)); |
| 252 | 252 | |
| 253 | 253 | /* don't need to get the RCU readlock here - the process is dead and |
| 254 | − * can't be modifying its own credentials. But shut RCU-lockdep up */ | |
| 255 | − rcu_read_lock(); | |
| 254 | + * can't be modifying its own credentials. */ | |
| 256 | 255 | dec_rlimit_ucounts(task_ucounts(p), UCOUNT_RLIMIT_NPROC, 1); |
| 257 | − rcu_read_unlock(); | |
| 258 | 256 | |
| 259 | 257 | pidfs_exit(p); |
| 260 | 258 | cgroup_task_release(p); |
| @@ -208,15 +208,62 @@ struct vm_stack { · 207 unmodified lines | ||
| 208 | 208 | struct vm_struct *stack_vm_area; |
| 209 | 209 | }; |
| 210 | 210 | |
| 211 | +static struct vm_struct *alloc_thread_stack_node_from_cache(struct task_struct *tsk, int node) | |
| 212 | +{ | |
| 213 | + struct vm_struct *vm_area; | |
| 214 | + unsigned int i; | |
| 215 | + | |
| 216 | + /* | |
| 217 | + * If the node has memory, we are guaranteed the stacks are backed by local pages. | |
| 218 | + * Otherwise the pages are arbitrary. | |
| 219 | + * | |
| 220 | + * Note that depending on cpuset it is possible we will get migrated to a different | |
| 221 | + * node immediately after allocating here, so this does *not* guarantee locality for | |
| 222 | + * arbitrary callers. | |
| 223 | + */ | |
| 224 | + scoped_guard(preempt) { | |
| 225 | + if (node != NUMA_NO_NODE && numa_node_id() != node) | |
| 226 | + return NULL; | |
| 227 | + | |
| 228 | + for (i = 0; i < NR_CACHED_STACKS; i++) { | |
| 229 | + vm_area = this_cpu_xchg(cached_stacks[i], NULL); | |
| 230 | + if (vm_area) | |
| 231 | + return vm_area; | |
| 232 | + } | |
| 233 | + } | |
| 234 | + | |
| 235 | + return NULL; | |
| 236 | +} | |
| 237 | + | |
| 211 | 238 | static bool try_release_thread_stack_to_cache(struct vm_struct *vm_area) |
| 212 | 239 | { |
| 213 | 240 | unsigned int i; |
| 241 | + int nid; | |
| 214 | 242 | |
| 215 | − for (i = 0; i < NR_CACHED_STACKS; i++) { | |
| 216 | − struct vm_struct *tmp = NULL; | |
| 243 | + /* | |
| 244 | + * Don't cache stacks if any of the pages don't match the local domain, unless | |
| 245 | + * there is no local memory to begin with. | |
| 246 | + * | |
| 247 | + * Note that lack of local memory does not automatically mean it makes no difference | |
| 248 | + * performance-wise which other domain backs the stack. In this case we are merely | |
| 249 | + * trying to avoid constantly going to vmalloc. | |
| 250 | + */ | |
| 251 | + scoped_guard(preempt) { | |
| 252 | + nid = numa_node_id(); | |
| 253 | + if (node_state(nid, N_MEMORY)) { | |
| 254 | + for (i = 0; i < vm_area->nr_pages; i++) { | |
| 255 | + struct page *page = vm_area->pages[i]; | |
| 256 | + if (page_to_nid(page) != nid) | |
| 257 | + return false; | |
| 258 | + } | |
| 259 | + } | |
| 260 | + | |
| 261 | + for (i = 0; i < NR_CACHED_STACKS; i++) { | |
| 262 | + struct vm_struct *tmp = NULL; | |
| 217 | 263 | |
| 218 | − if (this_cpu_try_cmpxchg(cached_stacks[i], &tmp, vm_area)) | |
| 219 | − return true; | |
| 264 | + if (this_cpu_try_cmpxchg(cached_stacks[i], &tmp, vm_area)) | |
| 265 | + return true; | |
| 266 | + } | |
| 220 | 267 | } |
| 221 | 268 | return false; |
| 222 | 269 | } |
| @@ -283,13 +330,9 @@ static int alloc_thread_stack_node(struct task_struct *tsk, int node) · 60 unmodified lines | ||
| 283 | 330 | { |
| 284 | 331 | struct vm_struct *vm_area; |
| 285 | 332 | void *stack; |
| 286 | − int i; | |
| 287 | − | |
| 288 | − for (i = 0; i < NR_CACHED_STACKS; i++) { | |
| 289 | − vm_area = this_cpu_xchg(cached_stacks[i], NULL); | |
| 290 | − if (!vm_area) | |
| 291 | − continue; | |
| 292 | 333 | |
| 334 | + vm_area = alloc_thread_stack_node_from_cache(tsk, node); | |
| 335 | + if (vm_area) { | |
| 293 | 336 | if (memcg_charge_kernel_stack(vm_area)) { |
| 294 | 337 | vfree(vm_area->addr); |
| 295 | 338 | return -ENOMEM; |
| @@ -24,6 +24,7 @@ · 23 unmodified lines | ||
| 24 | 24 | #include <linux/sched/sysctl.h> |
| 25 | 25 | #include <linux/hung_task.h> |
| 26 | 26 | #include <linux/rwsem.h> |
| 27 | +#include <linux/sys_info.h> | |
| 27 | 28 | |
| 28 | 29 | #include <trace/events/sched.h> |
| 29 | 30 | |
| @@ -50,7 +51,6 @@ static unsigned long __read_mostly sysctl_hung_task_detect_count; · 20 unmodified lines | ||
| 50 | 51 | * Zero means infinite timeout - no checking done: |
| 51 | 52 | */ |
| 52 | 53 | unsigned long __read_mostly sysctl_hung_task_timeout_secs = CONFIG_DEFAULT_HUNG_TASK_TIMEOUT; |
| 53 | −EXPORT_SYMBOL_GPL(sysctl_hung_task_timeout_secs); | |
| 54 | 54 | |
| 55 | 55 | /* |
| 56 | 56 | * Zero (default value) means use sysctl_hung_task_timeout_secs: |
| @@ -60,12 +60,17 @@ static unsigned long __read_mostly sysctl_hung_task_check_interval_secs; · 3 unmodified lines | ||
| 60 | 60 | static int __read_mostly sysctl_hung_task_warnings = 10; |
| 61 | 61 | |
| 62 | 62 | static int __read_mostly did_panic; |
| 63 | −static bool hung_task_show_lock; | |
| 64 | 63 | static bool hung_task_call_panic; |
| 65 | −static bool hung_task_show_all_bt; | |
| 66 | 64 | |
| 67 | 65 | static struct task_struct *watchdog_task; |
| 68 | 66 | |
| 67 | +/* | |
| 68 | + * A bitmask to control what kinds of system info to be printed when | |
| 69 | + * a hung task is detected, it could be task, memory, lock etc. Refer | |
| 70 | + * include/linux/sys_info.h for detailed bit definition. | |
| 71 | + */ | |
| 72 | +static unsigned long hung_task_si_mask; | |
| 73 | + | |
| 69 | 74 | #ifdef CONFIG_SMP |
| 70 | 75 | /* |
| 71 | 76 | * Should we dump all CPUs backtraces in a hung task event? |
| @@ -81,7 +86,7 @@ static unsigned int __read_mostly sysctl_hung_task_all_cpu_backtrace; · 9 unmodified lines | ||
| 81 | 86 | * hung task is detected: |
| 82 | 87 | */ |
| 83 | 88 | static unsigned int __read_mostly sysctl_hung_task_panic = |
| 84 | − IS_ENABLED(CONFIG_BOOTPARAM_HUNG_TASK_PANIC); | |
| 89 | + CONFIG_BOOTPARAM_HUNG_TASK_PANIC; | |
| 85 | 90 | |
| 86 | 91 | static int |
| 87 | 92 | hung_task_panic(struct notifier_block *this, unsigned long event, void *ptr) |
| @@ -218,8 +223,11 @@ static inline void debug_show_blocker(struct task_struct *task, unsigned long ti · 130 unmodified lines | ||
| 218 | 223 | } |
| 219 | 224 | #endif |
| 220 | 225 | |
| 221 | −static void check_hung_task(struct task_struct *t, unsigned long timeout) | |
| 226 | +static void check_hung_task(struct task_struct *t, unsigned long timeout, | |
| 227 | + unsigned long prev_detect_count) | |
| 222 | 228 | { |
| 229 | + unsigned long total_hung_task; | |
| 230 | + | |
| 223 | 231 | if (!task_is_hung(t, timeout)) |
| 224 | 232 | return; |
| 225 | 233 | |
| @@ -229,11 +237,11 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout) · 3 unmodified lines | ||
| 229 | 237 | */ |
| 230 | 238 | sysctl_hung_task_detect_count++; |
| 231 | 239 | |
| 240 | + total_hung_task = sysctl_hung_task_detect_count - prev_detect_count; | |
| 232 | 241 | trace_sched_process_hang(t); |
| 233 | 242 | |
| 234 | − if (sysctl_hung_task_panic) { | |
| 243 | + if (sysctl_hung_task_panic && total_hung_task >= sysctl_hung_task_panic) { | |
| 235 | 244 | console_verbose(); |
| 236 | − hung_task_show_lock = true; | |
| 237 | 245 | hung_task_call_panic = true; |
| 238 | 246 | } |
| 239 | 247 | |
| @@ -256,10 +264,7 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout) · 16 unmodified lines | ||
| 256 | 264 | " disables this message.\n"); |
| 257 | 265 | sched_show_task(t); |
| 258 | 266 | debug_show_blocker(t, timeout); |
| 259 | − hung_task_show_lock = true; | |
| 260 | 267 | |
| 261 | − if (sysctl_hung_task_all_cpu_backtrace) | |
| 262 | − hung_task_show_all_bt = true; | |
| 263 | 268 | if (!sysctl_hung_task_warnings) |
| 264 | 269 | pr_info("Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings\n"); |
| 265 | 270 | } |
| @@ -300,6 +305,9 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) · 34 unmodified lines | ||
| 300 | 305 | int max_count = sysctl_hung_task_check_count; |
| 301 | 306 | unsigned long last_break = jiffies; |
| 302 | 307 | struct task_struct *g, *t; |
| 308 | + unsigned long prev_detect_count = sysctl_hung_task_detect_count; | |
| 309 | + int need_warning = sysctl_hung_task_warnings; | |
| 310 | + unsigned long si_mask = hung_task_si_mask; | |
| 303 | 311 | |
| 304 | 312 | /* |
| 305 | 313 | * If the system crashed already then all bets are off, |
| @@ -308,7 +316,7 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) · 2 unmodified lines | ||
| 308 | 316 | if (test_taint(TAINT_DIE) || did_panic) |
| 309 | 317 | return; |
| 310 | 318 | |
| 311 | − hung_task_show_lock = false; | |
| 319 | + | |
| 312 | 320 | rcu_read_lock(); |
| 313 | 321 | for_each_process_thread(g, t) { |
| 314 | 322 | |
| @@ -320,18 +328,23 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout) · 5 unmodified lines | ||
| 320 | 328 | last_break = jiffies; |
| 321 | 329 | } |
| 322 | 330 | |
| 323 | − check_hung_task(t, timeout); | |
| 331 | + check_hung_task(t, timeout, prev_detect_count); | |
| 324 | 332 | } |
| 325 | 333 | unlock: |
| 326 | 334 | rcu_read_unlock(); |
| 327 | − if (hung_task_show_lock) | |
| 328 | − debug_show_all_locks(); | |
| 329 | 335 | |
| 330 | − if (hung_task_show_all_bt) { | |
| 331 | − hung_task_show_all_bt = false; | |
| 332 | − trigger_all_cpu_backtrace(); | |
| 336 | + if (!(sysctl_hung_task_detect_count - prev_detect_count)) | |
| 337 | + return; | |
| 338 | + | |
| 339 | + if (need_warning || hung_task_call_panic) { | |
| 340 | + si_mask |= SYS_INFO_LOCKS; | |
| 341 | + | |
| 342 | + if (sysctl_hung_task_all_cpu_backtrace) | |
| 343 | + si_mask |= SYS_INFO_ALL_BT; | |
| 333 | 344 | } |
| 334 | 345 | |
| 346 | + sys_info(si_mask); | |
| 347 | + | |
| 335 | 348 | if (hung_task_call_panic) |
| 336 | 349 | panic("hung_task: blocked tasks"); |
| 337 | 350 | } |
| @@ -389,7 +402,7 @@ static const struct ctl_table hung_task_sysctls[] = { · 51 unmodified lines | ||
| 389 | 402 | .mode = 0644, |
| 390 | 403 | .proc_handler = proc_dointvec_minmax, |
| 391 | 404 | .extra1 = SYSCTL_ZERO, |
| 392 | − .extra2 = SYSCTL_ONE, | |
| 405 | + .extra2 = SYSCTL_INT_MAX, | |
| 393 | 406 | }, |
| 394 | 407 | { |
| 395 | 408 | .procname = "hung_task_check_count", |
| @@ -430,6 +443,13 @@ static const struct ctl_table hung_task_sysctls[] = { · 34 unmodified lines | ||
| 430 | 443 | .mode = 0444, |
| 431 | 444 | .proc_handler = proc_doulongvec_minmax, |
| 432 | 445 | }, |
| 446 | + { | |
| 447 | + .procname = "hung_task_sys_info", | |
| 448 | + .data = &hung_task_si_mask, | |
| 449 | + .maxlen = sizeof(hung_task_si_mask), | |
| 450 | + .mode = 0644, | |
| 451 | + .proc_handler = sysctl_sys_info_handler, | |
| 452 | + }, | |
| 433 | 453 | }; |
| 434 | 454 | |
| 435 | 455 | static void __init hung_task_sysctl_init(void) |
| @@ -15,6 +15,7 @@ · 14 unmodified lines | ||
| 15 | 15 | #include <linux/kexec.h> |
| 16 | 16 | #include <linux/mutex.h> |
| 17 | 17 | #include <linux/list.h> |
| 18 | +#include <linux/liveupdate.h> | |
| 18 | 19 | #include <linux/highmem.h> |
| 19 | 20 | #include <linux/syscalls.h> |
| 20 | 21 | #include <linux/reboot.h> |
| @@ -41,6 +42,7 @@ · 20 unmodified lines | ||
| 41 | 42 | #include <linux/objtool.h> |
| 42 | 43 | #include <linux/kmsg_dump.h> |
| 43 | 44 | #include <linux/dma-map-ops.h> |
| 45 | +#include <linux/sysfs.h> | |
| 44 | 46 | |
| 45 | 47 | #include <asm/page.h> |
| 46 | 48 | #include <asm/sections.h> |
| @@ -742,7 +744,6 @@ static int kimage_load_cma_segment(struct kimage *image, int idx) · 695 unmodified lines | ||
| 742 | 744 | struct kexec_segment *segment = &image->segment[idx]; |
| 743 | 745 | struct page *cma = image->segment_cma[idx]; |
| 744 | 746 | char *ptr = page_address(cma); |
| 745 | − unsigned long maddr; | |
| 746 | 747 | size_t ubytes, mbytes; |
| 747 | 748 | int result = 0; |
| 748 | 749 | unsigned char __user *buf = NULL; |
| @@ -754,15 +755,12 @@ static int kimage_load_cma_segment(struct kimage *image, int idx) · 5 unmodified lines | ||
| 754 | 755 | buf = segment->buf; |
| 755 | 756 | ubytes = segment->bufsz; |
| 756 | 757 | mbytes = segment->memsz; |
| 757 | − maddr = segment->mem; | |
| 758 | 758 | |
| 759 | 759 | /* Then copy from source buffer to the CMA one */ |
| 760 | 760 | while (mbytes) { |
| 761 | 761 | size_t uchunk, mchunk; |
| 762 | 762 | |
| 763 | − ptr += maddr & ~PAGE_MASK; | |
| 764 | − mchunk = min_t(size_t, mbytes, | |
| 765 | − PAGE_SIZE - (maddr & ~PAGE_MASK)); | |
| 763 | + mchunk = min_t(size_t, mbytes, PAGE_SIZE); | |
| 766 | 764 | uchunk = min(ubytes, mchunk); |
| 767 | 765 | |
| 768 | 766 | if (uchunk) { |
| @@ -784,7 +782,6 @@ static int kimage_load_cma_segment(struct kimage *image, int idx) · 15 unmodified lines | ||
| 784 | 782 | } |
| 785 | 783 | |
| 786 | 784 | ptr += mchunk; |
| 787 | − maddr += mchunk; | |
| 788 | 785 | mbytes -= mchunk; |
| 789 | 786 | |
| 790 | 787 | cond_resched(); |
| @@ -839,9 +836,7 @@ static int kimage_load_normal_segment(struct kimage *image, int idx) · 48 unmodified lines | ||
| 839 | 836 | ptr = kmap_local_page(page); |
| 840 | 837 | /* Start with a clear page */ |
| 841 | 838 | clear_page(ptr); |
| 842 | − ptr += maddr & ~PAGE_MASK; | |
| 843 | − mchunk = min_t(size_t, mbytes, | |
| 844 | − PAGE_SIZE - (maddr & ~PAGE_MASK)); | |
| 839 | + mchunk = min_t(size_t, mbytes, PAGE_SIZE); | |
| 845 | 840 | uchunk = min(ubytes, mchunk); |
| 846 | 841 | |
| 847 | 842 | if (uchunk) { |
| @@ -904,9 +899,7 @@ static int kimage_load_crash_segment(struct kimage *image, int idx) · 56 unmodified lines | ||
| 904 | 899 | } |
| 905 | 900 | arch_kexec_post_alloc_pages(page_address(page), 1, 0); |
| 906 | 901 | ptr = kmap_local_page(page); |
| 907 | − ptr += maddr & ~PAGE_MASK; | |
| 908 | − mchunk = min_t(size_t, mbytes, | |
| 909 | − PAGE_SIZE - (maddr & ~PAGE_MASK)); | |
| 902 | + mchunk = min_t(size_t, mbytes, PAGE_SIZE); | |
| 910 | 903 | uchunk = min(ubytes, mchunk); |
| 911 | 904 | if (mchunk > uchunk) { |
| 912 | 905 | /* Zero the trailing part of the page */ |
| @@ -1146,6 +1139,10 @@ int kernel_kexec(void) · 233 unmodified lines | ||
| 1146 | 1139 | goto Unlock; |
| 1147 | 1140 | } |
| 1148 | 1141 | |
| 1142 | + error = liveupdate_reboot(); | |
| 1143 | + if (error) | |
| 1144 | + goto Unlock; | |
| 1145 | + | |
| 1149 | 1146 | #ifdef CONFIG_KEXEC_JUMP |
| 1150 | 1147 | if (kexec_image->preserve_context) { |
| 1151 | 1148 | /* |
| @@ -1229,3 +1226,143 @@ int kernel_kexec(void) · 77 unmodified lines | ||
| 1229 | 1226 | kexec_unlock(); |
| 1230 | 1227 | return error; |
| 1231 | 1228 | } |
| 1229 | + | |
| 1230 | +static ssize_t loaded_show(struct kobject *kobj, | |
| 1231 | + struct kobj_attribute *attr, char *buf) | |
| 1232 | +{ | |
| 1233 | + return sysfs_emit(buf, "%d\n", !!kexec_image); | |
| 1234 | +} | |
| 1235 | +static struct kobj_attribute loaded_attr = __ATTR_RO(loaded); | |
| 1236 | + | |
| 1237 | +#ifdef CONFIG_CRASH_DUMP | |
| 1238 | +static ssize_t crash_loaded_show(struct kobject *kobj, | |
| 1239 | + struct kobj_attribute *attr, char *buf) | |
| 1240 | +{ | |
| 1241 | + return sysfs_emit(buf, "%d\n", kexec_crash_loaded()); | |
| 1242 | +} | |
| 1243 | +static struct kobj_attribute crash_loaded_attr = __ATTR_RO(crash_loaded); | |
| 1244 | + | |
| 1245 | +#ifdef CONFIG_CRASH_RESERVE | |
| 1246 | +static ssize_t crash_cma_ranges_show(struct kobject *kobj, | |
| 1247 | + struct kobj_attribute *attr, char *buf) | |
| 1248 | +{ | |
| 1249 | + | |
| 1250 | + ssize_t len = 0; | |
| 1251 | + int i; | |
| 1252 | + | |
| 1253 | + for (i = 0; i < crashk_cma_cnt; ++i) { | |
| 1254 | + len += sysfs_emit_at(buf, len, "%08llx-%08llx\n", | |
| 1255 | + crashk_cma_ranges[i].start, | |
| 1256 | + crashk_cma_ranges[i].end); | |
| 1257 | + } | |
| 1258 | + return len; | |
| 1259 | +} | |
| 1260 | +static struct kobj_attribute crash_cma_ranges_attr = __ATTR_RO(crash_cma_ranges); | |
| 1261 | +#endif | |
| 1262 | + | |
| 1263 | +static ssize_t crash_size_show(struct kobject *kobj, | |
| 1264 | + struct kobj_attribute *attr, char *buf) | |
| 1265 | +{ | |
| 1266 | + ssize_t size = crash_get_memory_size(); | |
| 1267 | + | |
| 1268 | + if (size < 0) | |
| 1269 | + return size; | |
| 1270 | + | |
| 1271 | + return sysfs_emit(buf, "%zd\n", size); | |
| 1272 | +} | |
| 1273 | +static ssize_t crash_size_store(struct kobject *kobj, | |
| 1274 | + struct kobj_attribute *attr, | |
| 1275 | + const char *buf, size_t count) | |
| 1276 | +{ | |
| 1277 | + unsigned long cnt; | |
| 1278 | + int ret; | |
| 1279 | + | |
| 1280 | + if (kstrtoul(buf, 0, &cnt)) | |
| 1281 | + return -EINVAL; | |
| 1282 | + | |
| 1283 | + ret = crash_shrink_memory(cnt); | |
| 1284 | + return ret < 0 ? ret : count; | |
| 1285 | +} | |
| 1286 | +static struct kobj_attribute crash_size_attr = __ATTR_RW(crash_size); | |
| 1287 | + | |
| 1288 | +#ifdef CONFIG_CRASH_HOTPLUG | |
| 1289 | +static ssize_t crash_elfcorehdr_size_show(struct kobject *kobj, | |
| 1290 | + struct kobj_attribute *attr, char *buf) | |
| 1291 | +{ | |
| 1292 | + unsigned int sz = crash_get_elfcorehdr_size(); | |
| 1293 | + | |
| 1294 | + return sysfs_emit(buf, "%u\n", sz); | |
| 1295 | +} | |
| 1296 | +static struct kobj_attribute crash_elfcorehdr_size_attr = __ATTR_RO(crash_elfcorehdr_size); | |
| 1297 | + | |
| 1298 | +#endif /* CONFIG_CRASH_HOTPLUG */ | |
| 1299 | +#endif /* CONFIG_CRASH_DUMP */ | |
| 1300 | + | |
| 1301 | +static struct attribute *kexec_attrs[] = { | |
| 1302 | + &loaded_attr.attr, | |
| 1303 | +#ifdef CONFIG_CRASH_DUMP | |
| 1304 | + &crash_loaded_attr.attr, | |
| 1305 | + &crash_size_attr.attr, | |
| 1306 | +#ifdef CONFIG_CRASH_RESERVE | |
| 1307 | + &crash_cma_ranges_attr.attr, | |
| 1308 | +#endif | |
| 1309 | +#ifdef CONFIG_CRASH_HOTPLUG | |
| 1310 | + &crash_elfcorehdr_size_attr.attr, | |
| 1311 | +#endif | |
| 1312 | +#endif | |
| 1313 | + NULL | |
| 1314 | +}; | |
| 1315 | + | |
| 1316 | +struct kexec_link_entry { | |
| 1317 | + const char *target; | |
| 1318 | + const char *name; | |
| 1319 | +}; | |
| 1320 | + | |
| 1321 | +static struct kexec_link_entry kexec_links[] = { | |
| 1322 | + { "loaded", "kexec_loaded" }, | |
| 1323 | +#ifdef CONFIG_CRASH_DUMP | |
| 1324 | + { "crash_loaded", "kexec_crash_loaded" }, | |
| 1325 | + { "crash_size", "kexec_crash_size" }, | |
| 1326 | +#ifdef CONFIG_CRASH_RESERVE | |
| 1327 | + {"crash_cma_ranges", "kexec_crash_cma_ranges"}, | |
| 1328 | +#endif | |
| 1329 | +#ifdef CONFIG_CRASH_HOTPLUG | |
| 1330 | + { "crash_elfcorehdr_size", "crash_elfcorehdr_size" }, | |
| 1331 | +#endif | |
| 1332 | +#endif | |
| 1333 | +}; | |
| 1334 | + | |
| 1335 | +static struct kobject *kexec_kobj; | |
| 1336 | +ATTRIBUTE_GROUPS(kexec); | |
| 1337 | + | |
| 1338 | +static int __init init_kexec_sysctl(void) | |
| 1339 | +{ | |
| 1340 | + int error; | |
| 1341 | + int i; | |
| 1342 | + | |
| 1343 | + kexec_kobj = kobject_create_and_add("kexec", kernel_kobj); | |
| 1344 | + if (!kexec_kobj) { | |
| 1345 | + pr_err("failed to create kexec kobject\n"); | |
| 1346 | + return -ENOMEM; | |
| 1347 | + } | |
| 1348 | + | |
| 1349 | + error = sysfs_create_groups(kexec_kobj, kexec_groups); | |
| 1350 | + if (error) | |
| 1351 | + goto kset_exit; | |
| 1352 | + | |
| 1353 | + for (i = 0; i < ARRAY_SIZE(kexec_links); i++) { | |
| 1354 | + error = compat_only_sysfs_link_entry_to_kobj(kernel_kobj, kexec_kobj, | |
| 1355 | + kexec_links[i].target, | |
| 1356 | + kexec_links[i].name); | |
| 1357 | + if (error) | |
| 1358 | + pr_err("Unable to create %s symlink (%d)", kexec_links[i].name, error); | |
| 1359 | + } | |
| 1360 | + | |
| 1361 | + return 0; | |
| 1362 | + | |
| 1363 | +kset_exit: | |
| 1364 | + kobject_put(kexec_kobj); | |
| 1365 | + return error; | |
| 1366 | +} | |
| 1367 | + | |
| 1368 | +subsys_initcall(init_kexec_sysctl); | |
| @@ -1,20 +0,0 @@ | ||
| 1 | −/* SPDX-License-Identifier: GPL-2.0 */ | |
| 2 | −#ifndef LINUX_KEXEC_HANDOVER_INTERNAL_H | |
| 3 | −#define LINUX_KEXEC_HANDOVER_INTERNAL_H | |
| 4 | − | |
| 5 | −#include <linux/kexec_handover.h> | |
| 6 | −#include <linux/types.h> | |
| 7 | − | |
| 8 | −extern struct kho_scratch *kho_scratch; | |
| 9 | −extern unsigned int kho_scratch_cnt; | |
| 10 | − | |
| 11 | −#ifdef CONFIG_KEXEC_HANDOVER_DEBUG | |
| 12 | −bool kho_scratch_overlap(phys_addr_t phys, size_t size); | |
| 13 | −#else | |
| 14 | −static inline bool kho_scratch_overlap(phys_addr_t phys, size_t size) | |
| 15 | −{ | |
| 16 | − return false; | |
| 17 | −} | |
| 18 | −#endif /* CONFIG_KEXEC_HANDOVER_DEBUG */ | |
| 19 | − | |
| 20 | −#endif /* LINUX_KEXEC_HANDOVER_INTERNAL_H */ | |
| @@ -12,7 +12,7 @@ · 11 unmodified lines | ||
| 12 | 12 | #include <linux/sysfs.h> |
| 13 | 13 | #include <linux/export.h> |
| 14 | 14 | #include <linux/init.h> |
| 15 | −#include <linux/kexec.h> | |
| 15 | +#include <linux/vmcore_info.h> | |
| 16 | 16 | #include <linux/profile.h> |
| 17 | 17 | #include <linux/stat.h> |
| 18 | 18 | #include <linux/sched.h> |
| @@ -119,50 +119,6 @@ static ssize_t profiling_store(struct kobject *kobj, · 100 unmodified lines | ||
| 119 | 119 | KERNEL_ATTR_RW(profiling); |
| 120 | 120 | #endif |
| 121 | 121 | |
| 122 | −#ifdef CONFIG_KEXEC_CORE | |
| 123 | −static ssize_t kexec_loaded_show(struct kobject *kobj, | |
| 124 | − struct kobj_attribute *attr, char *buf) | |
| 125 | −{ | |
| 126 | − return sysfs_emit(buf, "%d\n", !!kexec_image); | |
| 127 | −} | |
| 128 | −KERNEL_ATTR_RO(kexec_loaded); | |
| 129 | − | |
| 130 | −#ifdef CONFIG_CRASH_DUMP | |
| 131 | −static ssize_t kexec_crash_loaded_show(struct kobject *kobj, | |
| 132 | − struct kobj_attribute *attr, char *buf) | |
| 133 | −{ | |
| 134 | − return sysfs_emit(buf, "%d\n", kexec_crash_loaded()); | |
| 135 | −} | |
| 136 | −KERNEL_ATTR_RO(kexec_crash_loaded); | |
| 137 | − | |
| 138 | −static ssize_t kexec_crash_size_show(struct kobject *kobj, | |
| 139 | − struct kobj_attribute *attr, char *buf) | |
| 140 | −{ | |
| 141 | − ssize_t size = crash_get_memory_size(); | |
| 142 | − | |
| 143 | − if (size < 0) | |
| 144 | − return size; | |
| 145 | − | |
| 146 | − return sysfs_emit(buf, "%zd\n", size); | |
| 147 | −} | |
| 148 | −static ssize_t kexec_crash_size_store(struct kobject *kobj, | |
| 149 | − struct kobj_attribute *attr, | |
| 150 | − const char *buf, size_t count) | |
| 151 | −{ | |
| 152 | − unsigned long cnt; | |
| 153 | − int ret; | |
| 154 | − | |
| 155 | − if (kstrtoul(buf, 0, &cnt)) | |
| 156 | − return -EINVAL; | |
| 157 | − | |
| 158 | − ret = crash_shrink_memory(cnt); | |
| 159 | − return ret < 0 ? ret : count; | |
| 160 | −} | |
| 161 | −KERNEL_ATTR_RW(kexec_crash_size); | |
| 162 | − | |
| 163 | −#endif /* CONFIG_CRASH_DUMP*/ | |
| 164 | −#endif /* CONFIG_KEXEC_CORE */ | |
| 165 | − | |
| 166 | 122 | #ifdef CONFIG_VMCORE_INFO |
| 167 | 123 | |
| 168 | 124 | static ssize_t vmcoreinfo_show(struct kobject *kobj, |
| @@ -174,18 +130,6 @@ static ssize_t vmcoreinfo_show(struct kobject *kobj, · 5 unmodified lines | ||
| 174 | 130 | } |
| 175 | 131 | KERNEL_ATTR_RO(vmcoreinfo); |
| 176 | 132 | |
| 177 | −#ifdef CONFIG_CRASH_HOTPLUG | |
| 178 | −static ssize_t crash_elfcorehdr_size_show(struct kobject *kobj, | |
| 179 | − struct kobj_attribute *attr, char *buf) | |
| 180 | −{ | |
| 181 | − unsigned int sz = crash_get_elfcorehdr_size(); | |
| 182 | − | |
| 183 | − return sysfs_emit(buf, "%u\n", sz); | |
| 184 | −} | |
| 185 | −KERNEL_ATTR_RO(crash_elfcorehdr_size); | |
| 186 | − | |
| 187 | −#endif | |
| 188 | − | |
| 189 | 133 | #endif /* CONFIG_VMCORE_INFO */ |
| 190 | 134 | |
| 191 | 135 | /* whether file capabilities are enabled */ |
| @@ -255,18 +199,8 @@ static struct attribute * kernel_attrs[] = { · 63 unmodified lines | ||
| 255 | 199 | #ifdef CONFIG_PROFILING |
| 256 | 200 | &profiling_attr.attr, |
| 257 | 201 | #endif |
| 258 | −#ifdef CONFIG_KEXEC_CORE | |
| 259 | − &kexec_loaded_attr.attr, | |
| 260 | −#ifdef CONFIG_CRASH_DUMP | |
| 261 | − &kexec_crash_loaded_attr.attr, | |
| 262 | − &kexec_crash_size_attr.attr, | |
| 263 | −#endif | |
| 264 | −#endif | |
| 265 | 202 | #ifdef CONFIG_VMCORE_INFO |
| 266 | 203 | &vmcoreinfo_attr.attr, |
| 267 | −#ifdef CONFIG_CRASH_HOTPLUG | |
| 268 | − &crash_elfcorehdr_size_attr.attr, | |
| 269 | −#endif | |
| 270 | 204 | #endif |
| 271 | 205 | #ifndef CONFIG_TINY_RCU |
| 272 | 206 | &rcu_expedited_attr.attr, |
| @@ -0,0 +1,75 @@ | ||
| 1 | +# SPDX-License-Identifier: GPL-2.0-only | |
| 2 | + | |
| 3 | +menu "Live Update and Kexec HandOver" | |
| 4 | + depends on !DEFERRED_STRUCT_PAGE_INIT | |
| 5 | + | |
| 6 | +config KEXEC_HANDOVER | |
| 7 | + bool "kexec handover" | |
| 8 | + depends on ARCH_SUPPORTS_KEXEC_HANDOVER && ARCH_SUPPORTS_KEXEC_FILE | |
| 9 | + depends on !DEFERRED_STRUCT_PAGE_INIT | |
| 10 | + select MEMBLOCK_KHO_SCRATCH | |
| 11 | + select KEXEC_FILE | |
| 12 | + select LIBFDT | |
| 13 | + select CMA | |
| 14 | + help | |
| 15 | + Allow kexec to hand over state across kernels by generating and | |
| 16 | + passing additional metadata to the target kernel. This is useful | |
| 17 | + to keep data or state alive across the kexec. For this to work, | |
| 18 | + both source and target kernels need to have this option enabled. | |
| 19 | + | |
| 20 | +config KEXEC_HANDOVER_DEBUG | |
| 21 | + bool "Enable Kexec Handover debug checks" | |
| 22 | + depends on KEXEC_HANDOVER | |
| 23 | + help | |
| 24 | + This option enables extra sanity checks for the Kexec Handover | |
| 25 | + subsystem. Since, KHO performance is crucial in live update | |
| 26 | + scenarios and the extra code might be adding overhead it is | |
| 27 | + only optionally enabled. | |
| 28 | + | |
| 29 | +config KEXEC_HANDOVER_DEBUGFS | |
| 30 | + bool "kexec handover debugfs interface" | |
| 31 | + default KEXEC_HANDOVER | |
| 32 | + depends on KEXEC_HANDOVER | |
| 33 | + select DEBUG_FS | |
| 34 | + help | |
| 35 | + Allow to control kexec handover device tree via debugfs | |
| 36 | + interface, i.e. finalize the state or aborting the finalization. | |
| 37 | + Also, enables inspecting the KHO fdt trees with the debugfs binary | |
| 38 | + blobs. | |
| 39 | + | |
| 40 | +config KEXEC_HANDOVER_ENABLE_DEFAULT | |
| 41 | + bool "Enable kexec handover by default" | |
| 42 | + depends on KEXEC_HANDOVER | |
| 43 | + help | |
| 44 | + Enable Kexec Handover by default. This avoids the need to | |
| 45 | + explicitly pass 'kho=on' on the kernel command line. | |
| 46 | + | |
| 47 | + This is useful for systems where KHO is a prerequisite for other | |
| 48 | + features, such as Live Update, ensuring the mechanism is always | |
| 49 | + active. | |
| 50 | + | |
| 51 | + The default behavior can still be overridden at boot time by | |
| 52 | + passing 'kho=off'. | |
| 53 | + | |
| 54 | +config LIVEUPDATE | |
| 55 | + bool "Live Update Orchestrator" | |
| 56 | + depends on KEXEC_HANDOVER | |
| 57 | + help | |
| 58 | + Enable the Live Update Orchestrator. Live Update is a mechanism, | |
| 59 | + typically based on kexec, that allows the kernel to be updated | |
| 60 | + while keeping selected devices operational across the transition. | |
| 61 | + These devices are intended to be reclaimed by the new kernel and | |
| 62 | + re-attached to their original workload without requiring a device | |
| 63 | + reset. | |
| 64 | + | |
| 65 | + Ability to handover a device from current to the next kernel depends | |
| 66 | + on specific support within device drivers and related kernel | |
| 67 | + subsystems. | |
| 68 | + | |
| 69 | + This feature primarily targets virtual machine hosts to quickly update | |
| 70 | + the kernel hypervisor with minimal disruption to the running virtual | |
| 71 | + machines. | |
| 72 | + | |
| 73 | + If unsure, say N. | |
| 74 | + | |
| 75 | +endmenu | |
| @@ -0,0 +1,12 @@ | ||
| 1 | +# SPDX-License-Identifier: GPL-2.0 | |
| 2 | + | |
| 3 | +luo-y := \ | |
| 4 | + luo_core.o \ | |
| 5 | + luo_file.o \ | |
| 6 | + luo_session.o | |
| 7 | + | |
| 8 | +obj-$(CONFIG_KEXEC_HANDOVER) += kexec_handover.o | |
| 9 | +obj-$(CONFIG_KEXEC_HANDOVER_DEBUG) += kexec_handover_debug.o | |
| 10 | +obj-$(CONFIG_KEXEC_HANDOVER_DEBUGFS) += kexec_handover_debugfs.o | |
| 11 | + | |
| 12 | +obj-$(CONFIG_LIVEUPDATE) += luo.o | |
| @@ -4,21 +4,22 @@ · 3 unmodified lines | ||
| 4 | 4 | * Copyright (C) 2023 Alexander Graf <graf@amazon.com> |
| 5 | 5 | * Copyright (C) 2025 Microsoft Corporation, Mike Rapoport <rppt@kernel.org> |
| 6 | 6 | * Copyright (C) 2025 Google LLC, Changyuan Lyu <changyuanl@google.com> |
| 7 | + * Copyright (C) 2025 Pasha Tatashin <pasha.tatashin@soleen.com> | |
| 7 | 8 | */ |
| 8 | 9 | |
| 9 | 10 | #define pr_fmt(fmt) "KHO: " fmt |
| 10 | 11 | |
| 11 | 12 | #include <linux/cleanup.h> |
| 12 | 13 | #include <linux/cma.h> |
| 14 | +#include <linux/kmemleak.h> | |
| 13 | 15 | #include <linux/count_zeros.h> |
| 14 | −#include <linux/debugfs.h> | |
| 15 | 16 | #include <linux/kexec.h> |
| 16 | 17 | #include <linux/kexec_handover.h> |
| 17 | 18 | #include <linux/libfdt.h> |
| 18 | 19 | #include <linux/list.h> |
| 19 | 20 | #include <linux/memblock.h> |
| 20 | −#include <linux/notifier.h> | |
| 21 | 21 | #include <linux/page-isolation.h> |
| 22 | +#include <linux/unaligned.h> | |
| 22 | 23 | #include <linux/vmalloc.h> |
| 23 | 24 | |
| 24 | 25 | #include <asm/early_ioremap.h> |
| @@ -28,8 +29,9 @@ · 3 unmodified lines | ||
| 28 | 29 | * KHO is tightly coupled with mm init and needs access to some of mm |
| 29 | 30 | * internal APIs. |
| 30 | 31 | */ |
| 31 | −#include "../mm/internal.h" | |
| 32 | −#include "kexec_internal.h" | |
| 32 | +#include "../../mm/internal.h" | |
| 33 | +#include "../kexec_internal.h" | |
| 34 | +#include "kexec_handover_internal.h" | |
| 33 | 35 | |
| 34 | 36 | #define KHO_FDT_COMPATIBLE "kho-v1" |
| 35 | 37 | #define PROP_PRESERVED_MEMORY_MAP "preserved-memory-map" |
| @@ -51,7 +53,7 @@ union kho_page_info { · 15 unmodified lines | ||
| 51 | 53 | |
| 52 | 54 | static_assert(sizeof(union kho_page_info) == sizeof(((struct page *)0)->private)); |
| 53 | 55 | |
| 54 | −static bool kho_enable __ro_after_init; | |
| 56 | +static bool kho_enable __ro_after_init = IS_ENABLED(CONFIG_KEXEC_HANDOVER_ENABLE_DEFAULT); | |
| 55 | 57 | |
| 56 | 58 | bool kho_is_enabled(void) |
| 57 | 59 | { |
| @@ -103,34 +105,19 @@ struct kho_mem_track { · 45 unmodified lines | ||
| 103 | 105 | |
| 104 | 106 | struct khoser_mem_chunk; |
| 105 | 107 | |
| 106 | −struct kho_serialization { | |
| 107 | − struct page *fdt; | |
| 108 | − struct list_head fdt_list; | |
| 109 | − struct dentry *sub_fdt_dir; | |
| 110 | − struct kho_mem_track track; | |
| 111 | − /* First chunk of serialized preserved memory map */ | |
| 112 | − struct khoser_mem_chunk *preserved_mem_map; | |
| 113 | −}; | |
| 114 | − | |
| 115 | 108 | struct kho_out { |
| 116 | − struct blocking_notifier_head chain_head; | |
| 117 | − | |
| 118 | − struct dentry *dir; | |
| 119 | − | |
| 109 | + void *fdt; | |
| 110 | + bool finalized; | |
| 120 | 111 | struct mutex lock; /* protects KHO FDT finalization */ |
| 121 | 112 | |
| 122 | − struct kho_serialization ser; | |
| 123 | − bool finalized; | |
| 113 | + struct kho_mem_track track; | |
| 114 | + struct kho_debugfs dbg; | |
| 124 | 115 | }; |
| 125 | 116 | |
| 126 | 117 | static struct kho_out kho_out = { |
| 127 | − .chain_head = BLOCKING_NOTIFIER_INIT(kho_out.chain_head), | |
| 128 | 118 | .lock = __MUTEX_INITIALIZER(kho_out.lock), |
| 129 | − .ser = { | |
| 130 | − .fdt_list = LIST_HEAD_INIT(kho_out.ser.fdt_list), | |
| 131 | − .track = { | |
| 132 | − .orders = XARRAY_INIT(kho_out.ser.track.orders, 0), | |
| 133 | − }, | |
| 119 | + .track = { | |
| 120 | + .orders = XARRAY_INIT(kho_out.track.orders, 0), | |
| 134 | 121 | }, |
| 135 | 122 | .finalized = false, |
| 136 | 123 | }; |
| @@ -159,26 +146,33 @@ static void *xa_load_or_alloc(struct xarray *xa, unsigned long index) · 22 unmodified lines | ||
| 159 | 146 | return no_free_ptr(elm); |
| 160 | 147 | } |
| 161 | 148 | |
| 162 | −static void __kho_unpreserve(struct kho_mem_track *track, unsigned long pfn, | |
| 163 | − unsigned long end_pfn) | |
| 149 | +static void __kho_unpreserve_order(struct kho_mem_track *track, unsigned long pfn, | |
| 150 | + unsigned int order) | |
| 164 | 151 | { |
| 165 | 152 | struct kho_mem_phys_bits *bits; |
| 166 | 153 | struct kho_mem_phys *physxa; |
| 154 | + const unsigned long pfn_high = pfn >> order; | |
| 167 | 155 | |
| 168 | − while (pfn < end_pfn) { | |
| 169 | − const unsigned int order = | |
| 170 | − min(count_trailing_zeros(pfn), ilog2(end_pfn - pfn)); | |
| 171 | − const unsigned long pfn_high = pfn >> order; | |
| 156 | + physxa = xa_load(&track->orders, order); | |
| 157 | + if (WARN_ON_ONCE(!physxa)) | |
| 158 | + return; | |
| 172 | 159 | |
| 173 | − physxa = xa_load(&track->orders, order); | |
| 174 | − if (WARN_ON_ONCE(!physxa)) | |
| 175 | − return; | |
| 160 | + bits = xa_load(&physxa->phys_bits, pfn_high / PRESERVE_BITS); | |
| 161 | + if (WARN_ON_ONCE(!bits)) | |
| 162 | + return; | |
| 176 | 163 | |
| 177 | − bits = xa_load(&physxa->phys_bits, pfn_high / PRESERVE_BITS); | |
| 178 | − if (WARN_ON_ONCE(!bits)) | |
| 179 | − return; | |
| 164 | + clear_bit(pfn_high % PRESERVE_BITS, bits->preserve); | |
| 165 | +} | |
| 180 | 166 | |
| 181 | − clear_bit(pfn_high % PRESERVE_BITS, bits->preserve); | |
| 167 | +static void __kho_unpreserve(struct kho_mem_track *track, unsigned long pfn, | |
| 168 | + unsigned long end_pfn) | |
| 169 | +{ | |
| 170 | + unsigned int order; | |
| 171 | + | |
| 172 | + while (pfn < end_pfn) { | |
| 173 | + order = min(count_trailing_zeros(pfn), ilog2(end_pfn - pfn)); | |
| 174 | + | |
| 175 | + __kho_unpreserve_order(track, pfn, order); | |
| 182 | 176 | |
| 183 | 177 | pfn += 1 << order; |
| 184 | 178 | } |
| @@ -192,10 +186,6 @@ static int __kho_preserve_order(struct kho_mem_track *track, unsigned long pfn, · 7 unmodified lines | ||
| 192 | 186 | const unsigned long pfn_high = pfn >> order; |
| 193 | 187 | |
| 194 | 188 | might_sleep(); |
| 195 | − | |
| 196 | − if (kho_out.finalized) | |
| 197 | − return -EBUSY; | |
| 198 | − | |
| 199 | 189 | physxa = xa_load(&track->orders, order); |
| 200 | 190 | if (!physxa) { |
| 201 | 191 | int err; |
| @@ -229,11 +219,11 @@ static int __kho_preserve_order(struct kho_mem_track *track, unsigned long pfn, · 27 unmodified lines | ||
| 229 | 219 | return 0; |
| 230 | 220 | } |
| 231 | 221 | |
| 232 | −static struct page *kho_restore_page(phys_addr_t phys) | |
| 222 | +static struct page *kho_restore_page(phys_addr_t phys, bool is_folio) | |
| 233 | 223 | { |
| 234 | 224 | struct page *page = pfn_to_online_page(PHYS_PFN(phys)); |
| 225 | + unsigned int nr_pages, ref_cnt; | |
| 235 | 226 | union kho_page_info info; |
| 236 | − unsigned int nr_pages; | |
| 237 | 227 | |
| 238 | 228 | if (!page) |
| 239 | 229 | return NULL; |
| @@ -253,11 +243,16 @@ static struct page *kho_restore_page(phys_addr_t phys) · 13 unmodified lines | ||
| 253 | 243 | /* Head page gets refcount of 1. */ |
| 254 | 244 | set_page_count(page, 1); |
| 255 | 245 | |
| 256 | − /* For higher order folios, tail pages get a page count of zero. */ | |
| 246 | + /* | |
| 247 | + * For higher order folios, tail pages get a page count of zero. | |
| 248 | + * For physically contiguous order-0 pages every pages gets a page | |
| 249 | + * count of 1 | |
| 250 | + */ | |
| 251 | + ref_cnt = is_folio ? 0 : 1; | |
| 257 | 252 | for (unsigned int i = 1; i < nr_pages; i++) |
| 258 | − set_page_count(page + i, 0); | |
| 253 | + set_page_count(page + i, ref_cnt); | |
| 259 | 254 | |
| 260 | − if (info.order > 0) | |
| 255 | + if (is_folio && info.order) | |
| 261 | 256 | prep_compound_page(page, info.order); |
| 262 | 257 | |
| 263 | 258 | adjust_managed_page_count(page, nr_pages); |
| @@ -272,7 +267,7 @@ static struct page *kho_restore_page(phys_addr_t phys) · 8 unmodified lines | ||
| 272 | 267 | */ |
| 273 | 268 | struct folio *kho_restore_folio(phys_addr_t phys) |
| 274 | 269 | { |
| 275 | − struct page *page = kho_restore_page(phys); | |
| 270 | + struct page *page = kho_restore_page(phys, true); | |
| 276 | 271 | |
| 277 | 272 | return page ? page_folio(page) : NULL; |
| 278 | 273 | } |
| @@ -297,11 +292,10 @@ struct page *kho_restore_pages(phys_addr_t phys, unsigned int nr_pages) · 18 unmodified lines | ||
| 297 | 292 | while (pfn < end_pfn) { |
| 298 | 293 | const unsigned int order = |
| 299 | 294 | min(count_trailing_zeros(pfn), ilog2(end_pfn - pfn)); |
| 300 | − struct page *page = kho_restore_page(PFN_PHYS(pfn)); | |
| 295 | + struct page *page = kho_restore_page(PFN_PHYS(pfn), false); | |
| 301 | 296 | |
| 302 | 297 | if (!page) |
| 303 | 298 | return NULL; |
| 304 | − split_page(page, order); | |
| 305 | 299 | pfn += 1 << order; |
| 306 | 300 | } |
| 307 | 301 | |
| @@ -371,11 +365,32 @@ static void kho_mem_ser_free(struct khoser_mem_chunk *first_chunk) · 63 unmodified lines | ||
| 371 | 365 | struct khoser_mem_chunk *tmp = chunk; |
| 372 | 366 | |
| 373 | 367 | chunk = KHOSER_LOAD_PTR(chunk->hdr.next); |
| 374 | − kfree(tmp); | |
| 368 | + free_page((unsigned long)tmp); | |
| 375 | 369 | } |
| 376 | 370 | } |
| 377 | 371 | |
| 378 | −static int kho_mem_serialize(struct kho_serialization *ser) | |
| 372 | +/* | |
| 373 | + * Update memory map property, if old one is found discard it via | |
| 374 | + * kho_mem_ser_free(). | |
| 375 | + */ | |
| 376 | +static void kho_update_memory_map(struct khoser_mem_chunk *first_chunk) | |
| 377 | +{ | |
| 378 | + void *ptr; | |
| 379 | + u64 phys; | |
| 380 | + | |
| 381 | + ptr = fdt_getprop_w(kho_out.fdt, 0, PROP_PRESERVED_MEMORY_MAP, NULL); | |
| 382 | + | |
| 383 | + /* Check and discard previous memory map */ | |
| 384 | + phys = get_unaligned((u64 *)ptr); | |
| 385 | + if (phys) | |
| 386 | + kho_mem_ser_free((struct khoser_mem_chunk *)phys_to_virt(phys)); | |
| 387 | + | |
| 388 | + /* Update with the new value */ | |
| 389 | + phys = first_chunk ? (u64)virt_to_phys(first_chunk) : 0; | |
| 390 | + put_unaligned(phys, (u64 *)ptr); | |
| 391 | +} | |
| 392 | + | |
| 393 | +static int kho_mem_serialize(struct kho_out *kho_out) | |
| 379 | 394 | { |
| 380 | 395 | struct khoser_mem_chunk *first_chunk = NULL; |
| 381 | 396 | struct khoser_mem_chunk *chunk = NULL; |
| @@ -383,7 +398,7 @@ static int kho_mem_serialize(struct kho_serialization *ser) · 1 unmodified line | ||
| 383 | 398 | unsigned long order; |
| 384 | 399 | int err = -ENOMEM; |
| 385 | 400 | |
| 386 | − xa_for_each(&ser->track.orders, order, physxa) { | |
| 401 | + xa_for_each(&kho_out->track.orders, order, physxa) { | |
| 387 | 402 | struct kho_mem_phys_bits *bits; |
| 388 | 403 | unsigned long phys; |
| 389 | 404 | |
| @@ -415,7 +430,7 @@ static int kho_mem_serialize(struct kho_serialization *ser) · 25 unmodified lines | ||
| 415 | 430 | } |
| 416 | 431 | } |
| 417 | 432 | |
| 418 | − ser->preserved_mem_map = first_chunk; | |
| 433 | + kho_update_memory_map(first_chunk); | |
| 419 | 434 | |
| 420 | 435 | return 0; |
| 421 | 436 | |
| @@ -445,20 +460,27 @@ static void __init deserialize_bitmap(unsigned int order, · 23 unmodified lines | ||
| 445 | 460 | } |
| 446 | 461 | } |
| 447 | 462 | |
| 448 | −static void __init kho_mem_deserialize(const void *fdt) | |
| 463 | +/* Return true if memory was deserizlied */ | |
| 464 | +static bool __init kho_mem_deserialize(const void *fdt) | |
| 449 | 465 | { |
| 450 | 466 | struct khoser_mem_chunk *chunk; |
| 451 | − const phys_addr_t *mem; | |
| 467 | + const void *mem_ptr; | |
| 468 | + u64 mem; | |
| 452 | 469 | int len; |
| 453 | 470 | |
| 454 | − mem = fdt_getprop(fdt, 0, PROP_PRESERVED_MEMORY_MAP, &len); | |
| 455 | − | |
| 456 | − if (!mem || len != sizeof(*mem)) { | |
| 471 | + mem_ptr = fdt_getprop(fdt, 0, PROP_PRESERVED_MEMORY_MAP, &len); | |
| 472 | + if (!mem_ptr || len != sizeof(u64)) { | |
| 457 | 473 | pr_err("failed to get preserved memory bitmaps\n"); |
| 458 | − return; | |
| 474 | + return false; | |
| 459 | 475 | } |
| 460 | 476 | |
| 461 | − chunk = *mem ? phys_to_virt(*mem) : NULL; | |
| 477 | + mem = get_unaligned((const u64 *)mem_ptr); | |
| 478 | + chunk = mem ? phys_to_virt(mem) : NULL; | |
| 479 | + | |
| 480 | + /* No preserved physical pages were passed, no deserialization */ | |
| 481 | + if (!chunk) | |
| 482 | + return false; | |
| 483 | + | |
| 462 | 484 | while (chunk) { |
| 463 | 485 | unsigned int i; |
| 464 | 486 | |
| @@ -467,6 +489,8 @@ static void __init kho_mem_deserialize(const void *fdt) · 2 unmodified lines | ||
| 467 | 489 | &chunk->bitmaps[i]); |
| 468 | 490 | chunk = KHOSER_LOAD_PTR(chunk->hdr.next); |
| 469 | 491 | } |
| 492 | + | |
| 493 | + return true; | |
| 470 | 494 | } |
| 471 | 495 | |
| 472 | 496 | /* |
| @@ -674,40 +698,8 @@ err_disable_kho: · 201 unmodified lines | ||
| 674 | 698 | kho_enable = false; |
| 675 | 699 | } |
| 676 | 700 | |
| 677 | −struct fdt_debugfs { | |
| 678 | − struct list_head list; | |
| 679 | − struct debugfs_blob_wrapper wrapper; | |
| 680 | − struct dentry *file; | |
| 681 | −}; | |
| 682 | − | |
| 683 | −static int kho_debugfs_fdt_add(struct list_head *list, struct dentry *dir, | |
| 684 | − const char *name, const void *fdt) | |
| 685 | −{ | |
| 686 | − struct fdt_debugfs *f; | |
| 687 | − struct dentry *file; | |
| 688 | − | |
| 689 | − f = kmalloc(sizeof(*f), GFP_KERNEL); | |
| 690 | − if (!f) | |
| 691 | − return -ENOMEM; | |
| 692 | − | |
| 693 | − f->wrapper.data = (void *)fdt; | |
| 694 | − f->wrapper.size = fdt_totalsize(fdt); | |
| 695 | − | |
| 696 | − file = debugfs_create_blob(name, 0400, dir, &f->wrapper); | |
| 697 | − if (IS_ERR(file)) { | |
| 698 | − kfree(f); | |
| 699 | − return PTR_ERR(file); | |
| 700 | − } | |
| 701 | − | |
| 702 | − f->file = file; | |
| 703 | − list_add(&f->list, list); | |
| 704 | − | |
| 705 | − return 0; | |
| 706 | −} | |
| 707 | − | |
| 708 | 701 | /** |
| 709 | 702 | * kho_add_subtree - record the physical address of a sub FDT in KHO root tree. |
| 710 | − * @ser: serialization control object passed by KHO notifiers. | |
| 711 | 703 | * @name: name of the sub tree. |
| 712 | 704 | * @fdt: the sub tree blob. |
| 713 | 705 | * |
| @@ -716,38 +708,76 @@ static int kho_debugfs_fdt_add(struct list_head *list, struct dentry *dir, · 2 unmodified lines | ||
| 716 | 708 | * by KHO for the new kernel to retrieve it after kexec. |
| 717 | 709 | * |
| 718 | 710 | * A debugfs blob entry is also created at |
| 719 | − * ``/sys/kernel/debug/kho/out/sub_fdts/@name``. | |
| 711 | + * ``/sys/kernel/debug/kho/out/sub_fdts/@name`` when kernel is configured with | |
| 712 | + * CONFIG_KEXEC_HANDOVER_DEBUGFS | |
| 720 | 713 | * |
| 721 | 714 | * Return: 0 on success, error code on failure |
| 722 | 715 | */ |
| 723 | −int kho_add_subtree(struct kho_serialization *ser, const char *name, void *fdt) | |
| 716 | +int kho_add_subtree(const char *name, void *fdt) | |
| 724 | 717 | { |
| 725 | − int err = 0; | |
| 726 | − u64 phys = (u64)virt_to_phys(fdt); | |
| 727 | − void *root = page_to_virt(ser->fdt); | |
| 718 | + phys_addr_t phys = virt_to_phys(fdt); | |
| 719 | + void *root_fdt = kho_out.fdt; | |
| 720 | + int err = -ENOMEM; | |
| 721 | + int off, fdt_err; | |
| 728 | 722 | |
| 729 | − err |= fdt_begin_node(root, name); | |
| 730 | − err |= fdt_property(root, PROP_SUB_FDT, &phys, sizeof(phys)); | |
| 731 | − err |= fdt_end_node(root); | |
| 723 | + guard(mutex)(&kho_out.lock); | |
| 732 | 724 | |
| 733 | − if (err) | |
| 725 | + fdt_err = fdt_open_into(root_fdt, root_fdt, PAGE_SIZE); | |
| 726 | + if (fdt_err < 0) | |
| 734 | 727 | return err; |
| 735 | 728 | |
| 736 | − return kho_debugfs_fdt_add(&ser->fdt_list, ser->sub_fdt_dir, name, fdt); | |
| 729 | + off = fdt_add_subnode(root_fdt, 0, name); | |
| 730 | + if (off < 0) { | |
| 731 | + if (off == -FDT_ERR_EXISTS) | |
| 732 | + err = -EEXIST; | |
| 733 | + goto out_pack; | |
| 734 | + } | |
| 735 | + | |
| 736 | + err = fdt_setprop(root_fdt, off, PROP_SUB_FDT, &phys, sizeof(phys)); | |
| 737 | + if (err < 0) | |
| 738 | + goto out_pack; | |
| 739 | + | |
| 740 | + WARN_ON_ONCE(kho_debugfs_fdt_add(&kho_out.dbg, name, fdt, false)); | |
| 741 | + | |
| 742 | +out_pack: | |
| 743 | + fdt_pack(root_fdt); | |
| 744 | + | |
| 745 | + return err; | |
| 737 | 746 | } |
| 738 | 747 | EXPORT_SYMBOL_GPL(kho_add_subtree); |
| 739 | 748 | |
| 740 | −int register_kho_notifier(struct notifier_block *nb) | |
| 749 | +void kho_remove_subtree(void *fdt) | |
| 741 | 750 | { |
| 742 | − return blocking_notifier_chain_register(&kho_out.chain_head, nb); | |
| 743 | −} | |
| 744 | −EXPORT_SYMBOL_GPL(register_kho_notifier); | |
| 751 | + phys_addr_t target_phys = virt_to_phys(fdt); | |
| 752 | + void *root_fdt = kho_out.fdt; | |
| 753 | + int off; | |
| 754 | + int err; | |
| 745 | 755 | |
| 746 | −int unregister_kho_notifier(struct notifier_block *nb) | |
| 747 | −{ | |
| 748 | − return blocking_notifier_chain_unregister(&kho_out.chain_head, nb); | |
| 756 | + guard(mutex)(&kho_out.lock); | |
| 757 | + | |
| 758 | + err = fdt_open_into(root_fdt, root_fdt, PAGE_SIZE); | |
| 759 | + if (err < 0) | |
| 760 | + return; | |
| 761 | + | |
| 762 | + for (off = fdt_first_subnode(root_fdt, 0); off >= 0; | |
| 763 | + off = fdt_next_subnode(root_fdt, off)) { | |
| 764 | + const u64 *val; | |
| 765 | + int len; | |
| 766 | + | |
| 767 | + val = fdt_getprop(root_fdt, off, PROP_SUB_FDT, &len); | |
| 768 | + if (!val || len != sizeof(phys_addr_t)) | |
| 769 | + continue; | |
| 770 | + | |
| 771 | + if ((phys_addr_t)*val == target_phys) { | |
| 772 | + fdt_del_node(root_fdt, off); | |
| 773 | + kho_debugfs_fdt_remove(&kho_out.dbg, fdt); | |
| 774 | + break; | |
| 775 | + } | |
| 776 | + } | |
| 777 | + | |
| 778 | + fdt_pack(root_fdt); | |
| 749 | 779 | } |
| 750 | −EXPORT_SYMBOL_GPL(unregister_kho_notifier); | |
| 780 | +EXPORT_SYMBOL_GPL(kho_remove_subtree); | |
| 751 | 781 | |
| 752 | 782 | /** |
| 753 | 783 | * kho_preserve_folio - preserve a folio across kexec. |
| @@ -762,7 +792,7 @@ int kho_preserve_folio(struct folio *folio) · 8 unmodified lines | ||
| 762 | 792 | { |
| 763 | 793 | const unsigned long pfn = folio_pfn(folio); |
| 764 | 794 | const unsigned int order = folio_order(folio); |
| 765 | − struct kho_mem_track *track = &kho_out.ser.track; | |
| 795 | + struct kho_mem_track *track = &kho_out.track; | |
| 766 | 796 | |
| 767 | 797 | if (WARN_ON(kho_scratch_overlap(pfn << PAGE_SHIFT, PAGE_SIZE << order))) |
| 768 | 798 | return -EINVAL; |
| @@ -771,6 +801,24 @@ int kho_preserve_folio(struct folio *folio) · 2 unmodified lines | ||
| 771 | 801 | } |
| 772 | 802 | EXPORT_SYMBOL_GPL(kho_preserve_folio); |
| 773 | 803 | |
| 804 | +/** | |
| 805 | + * kho_unpreserve_folio - unpreserve a folio. | |
| 806 | + * @folio: folio to unpreserve. | |
| 807 | + * | |
| 808 | + * Instructs KHO to unpreserve a folio that was preserved by | |
| 809 | + * kho_preserve_folio() before. The provided @folio (pfn and order) | |
| 810 | + * must exactly match a previously preserved folio. | |
| 811 | + */ | |
| 812 | +void kho_unpreserve_folio(struct folio *folio) | |
| 813 | +{ | |
| 814 | + const unsigned long pfn = folio_pfn(folio); | |
| 815 | + const unsigned int order = folio_order(folio); | |
| 816 | + struct kho_mem_track *track = &kho_out.track; | |
| 817 | + | |
| 818 | + __kho_unpreserve_order(track, pfn, order); | |
| 819 | +} | |
| 820 | +EXPORT_SYMBOL_GPL(kho_unpreserve_folio); | |
| 821 | + | |
| 774 | 822 | /** |
| 775 | 823 | * kho_preserve_pages - preserve contiguous pages across kexec |
| 776 | 824 | * @page: first page in the list. |
| @@ -783,7 +831,7 @@ EXPORT_SYMBOL_GPL(kho_preserve_folio); · 6 unmodified lines | ||
| 783 | 831 | */ |
| 784 | 832 | int kho_preserve_pages(struct page *page, unsigned int nr_pages) |
| 785 | 833 | { |
| 786 | − struct kho_mem_track *track = &kho_out.ser.track; | |
| 834 | + struct kho_mem_track *track = &kho_out.track; | |
| 787 | 835 | const unsigned long start_pfn = page_to_pfn(page); |
| 788 | 836 | const unsigned long end_pfn = start_pfn + nr_pages; |
| 789 | 837 | unsigned long pfn = start_pfn; |
| @@ -815,6 +863,26 @@ int kho_preserve_pages(struct page *page, unsigned int nr_pages) · 25 unmodified lines | ||
| 815 | 863 | } |
| 816 | 864 | EXPORT_SYMBOL_GPL(kho_preserve_pages); |
| 817 | 865 | |
| 866 | +/** | |
| 867 | + * kho_unpreserve_pages - unpreserve contiguous pages. | |
| 868 | + * @page: first page in the list. | |
| 869 | + * @nr_pages: number of pages. | |
| 870 | + * | |
| 871 | + * Instructs KHO to unpreserve @nr_pages contiguous pages starting from @page. | |
| 872 | + * This must be called with the same @page and @nr_pages as the corresponding | |
| 873 | + * kho_preserve_pages() call. Unpreserving arbitrary sub-ranges of larger | |
| 874 | + * preserved blocks is not supported. | |
| 875 | + */ | |
| 876 | +void kho_unpreserve_pages(struct page *page, unsigned int nr_pages) | |
| 877 | +{ | |
| 878 | + struct kho_mem_track *track = &kho_out.track; | |
| 879 | + const unsigned long start_pfn = page_to_pfn(page); | |
| 880 | + const unsigned long end_pfn = start_pfn + nr_pages; | |
| 881 | + | |
| 882 | + __kho_unpreserve(track, start_pfn, end_pfn); | |
| 883 | +} | |
| 884 | +EXPORT_SYMBOL_GPL(kho_unpreserve_pages); | |
| 885 | + | |
| 818 | 886 | struct kho_vmalloc_hdr { |
| 819 | 887 | DECLARE_KHOSER_PTR(next, struct kho_vmalloc_chunk *); |
| 820 | 888 | }; |
| @@ -885,7 +953,7 @@ err_free: · 64 unmodified lines | ||
| 885 | 953 | static void kho_vmalloc_unpreserve_chunk(struct kho_vmalloc_chunk *chunk, |
| 886 | 954 | unsigned short order) |
| 887 | 955 | { |
| 888 | − struct kho_mem_track *track = &kho_out.ser.track; | |
| 956 | + struct kho_mem_track *track = &kho_out.track; | |
| 889 | 957 | unsigned long pfn = PHYS_PFN(virt_to_phys(chunk)); |
| 890 | 958 | |
| 891 | 959 | __kho_unpreserve(track, pfn, pfn + 1); |
| @@ -896,20 +964,6 @@ static void kho_vmalloc_unpreserve_chunk(struct kho_vmalloc_chunk *chunk, · 4 unmodified lines | ||
| 896 | 964 | } |
| 897 | 965 | } |
| 898 | 966 | |
| 899 | −static void kho_vmalloc_free_chunks(struct kho_vmalloc *kho_vmalloc) | |
| 900 | −{ | |
| 901 | − struct kho_vmalloc_chunk *chunk = KHOSER_LOAD_PTR(kho_vmalloc->first); | |
| 902 | − | |
| 903 | − while (chunk) { | |
| 904 | − struct kho_vmalloc_chunk *tmp = chunk; | |
| 905 | − | |
| 906 | − kho_vmalloc_unpreserve_chunk(chunk, kho_vmalloc->order); | |
| 907 | − | |
| 908 | − chunk = KHOSER_LOAD_PTR(chunk->hdr.next); | |
| 909 | − free_page((unsigned long)tmp); | |
| 910 | − } | |
| 911 | −} | |
| 912 | − | |
| 913 | 967 | /** |
| 914 | 968 | * kho_preserve_vmalloc - preserve memory allocated with vmalloc() across kexec |
| 915 | 969 | * @ptr: pointer to the area in vmalloc address space |
| @@ -971,11 +1025,33 @@ int kho_preserve_vmalloc(void *ptr, struct kho_vmalloc *preservation) · 55 unmodified lines | ||
| 971 | 1025 | return 0; |
| 972 | 1026 | |
| 973 | 1027 | err_free: |
| 974 | − kho_vmalloc_free_chunks(preservation); | |
| 1028 | + kho_unpreserve_vmalloc(preservation); | |
| 975 | 1029 | return err; |
| 976 | 1030 | } |
| 977 | 1031 | EXPORT_SYMBOL_GPL(kho_preserve_vmalloc); |
| 978 | 1032 | |
| 1033 | +/** | |
| 1034 | + * kho_unpreserve_vmalloc - unpreserve memory allocated with vmalloc() | |
| 1035 | + * @preservation: preservation metadata returned by kho_preserve_vmalloc() | |
| 1036 | + * | |
| 1037 | + * Instructs KHO to unpreserve the area in vmalloc address space that was | |
| 1038 | + * previously preserved with kho_preserve_vmalloc(). | |
| 1039 | + */ | |
| 1040 | +void kho_unpreserve_vmalloc(struct kho_vmalloc *preservation) | |
| 1041 | +{ | |
| 1042 | + struct kho_vmalloc_chunk *chunk = KHOSER_LOAD_PTR(preservation->first); | |
| 1043 | + | |
| 1044 | + while (chunk) { | |
| 1045 | + struct kho_vmalloc_chunk *tmp = chunk; | |
| 1046 | + | |
| 1047 | + kho_vmalloc_unpreserve_chunk(chunk, preservation->order); | |
| 1048 | + | |
| 1049 | + chunk = KHOSER_LOAD_PTR(chunk->hdr.next); | |
| 1050 | + free_page((unsigned long)tmp); | |
| 1051 | + } | |
| 1052 | +} | |
| 1053 | +EXPORT_SYMBOL_GPL(kho_unpreserve_vmalloc); | |
| 1054 | + | |
| 979 | 1055 | /** |
| 980 | 1056 | * kho_restore_vmalloc - recreates and populates an area in vmalloc address |
| 981 | 1057 | * space from the preserved memory. |
| @@ -1024,7 +1100,7 @@ void *kho_restore_vmalloc(const struct kho_vmalloc *preservation) · 42 unmodified lines | ||
| 1024 | 1100 | goto err_free_pages_array; |
| 1025 | 1101 | |
| 1026 | 1102 | for (int j = 0; j < contig_pages; j++) |
| 1027 | − pages[idx++] = page; | |
| 1103 | + pages[idx++] = page + j; | |
| 1028 | 1104 | |
| 1029 | 1105 | phys += contig_pages * PAGE_SIZE; |
| 1030 | 1106 | } |
| @@ -1065,217 +1141,122 @@ err_free_pages_array: · 34 unmodified lines | ||
| 1065 | 1141 | } |
| 1066 | 1142 | EXPORT_SYMBOL_GPL(kho_restore_vmalloc); |
| 1067 | 1143 | |
| 1068 | −/* Handling for debug/kho/out */ | |
| 1069 | − | |
| 1070 | −static struct dentry *debugfs_root; | |
| 1071 | − | |
| 1072 | −static int kho_out_update_debugfs_fdt(void) | |
| 1073 | −{ | |
| 1074 | − int err = 0; | |
| 1075 | − struct fdt_debugfs *ff, *tmp; | |
| 1076 | − | |
| 1077 | − if (kho_out.finalized) { | |
| 1078 | − err = kho_debugfs_fdt_add(&kho_out.ser.fdt_list, kho_out.dir, | |
| 1079 | − "fdt", page_to_virt(kho_out.ser.fdt)); | |
| 1080 | − } else { | |
| 1081 | − list_for_each_entry_safe(ff, tmp, &kho_out.ser.fdt_list, list) { | |
| 1082 | − debugfs_remove(ff->file); | |
| 1083 | − list_del(&ff->list); | |
| 1084 | − kfree(ff); | |
| 1085 | − } | |
| 1086 | − } | |
| 1087 | − | |
| 1088 | − return err; | |
| 1089 | −} | |
| 1090 | − | |
| 1091 | −static int kho_abort(void) | |
| 1144 | +/** | |
| 1145 | + * kho_alloc_preserve - Allocate, zero, and preserve memory. | |
| 1146 | + * @size: The number of bytes to allocate. | |
| 1147 | + * | |
| 1148 | + * Allocates a physically contiguous block of zeroed pages that is large | |
| 1149 | + * enough to hold @size bytes. The allocated memory is then registered with | |
| 1150 | + * KHO for preservation across a kexec. | |
| 1151 | + * | |
| 1152 | + * Note: The actual allocated size will be rounded up to the nearest | |
| 1153 | + * power-of-two page boundary. | |
| 1154 | + * | |
| 1155 | + * @return A virtual pointer to the allocated and preserved memory on success, | |
| 1156 | + * or an ERR_PTR() encoded error on failure. | |
| 1157 | + */ | |
| 1158 | +void *kho_alloc_preserve(size_t size) | |
| 1092 | 1159 | { |
| 1093 | − int err; | |
| 1094 | − unsigned long order; | |
| 1095 | − struct kho_mem_phys *physxa; | |
| 1160 | + struct folio *folio; | |
| 1161 | + int order, ret; | |
| 1096 | 1162 | |
| 1097 | − xa_for_each(&kho_out.ser.track.orders, order, physxa) { | |
| 1098 | − struct kho_mem_phys_bits *bits; | |
| 1099 | − unsigned long phys; | |
| 1163 | + if (!size) | |
| 1164 | + return ERR_PTR(-EINVAL); | |
| 1100 | 1165 | |
| 1101 | − xa_for_each(&physxa->phys_bits, phys, bits) | |
| 1102 | − kfree(bits); | |
| 1166 | + order = get_order(size); | |
| 1167 | + if (order > MAX_PAGE_ORDER) | |
| 1168 | + return ERR_PTR(-E2BIG); | |
| 1103 | 1169 | |
| 1104 | − xa_destroy(&physxa->phys_bits); | |
| 1105 | − kfree(physxa); | |
| 1106 | − } | |
| 1107 | − xa_destroy(&kho_out.ser.track.orders); | |
| 1170 | + folio = folio_alloc(GFP_KERNEL | __GFP_ZERO, order); | |
| 1171 | + if (!folio) | |
| 1172 | + return ERR_PTR(-ENOMEM); | |
| 1108 | 1173 | |
| 1109 | − if (kho_out.ser.preserved_mem_map) { | |
| 1110 | − kho_mem_ser_free(kho_out.ser.preserved_mem_map); | |
| 1111 | − kho_out.ser.preserved_mem_map = NULL; | |
| 1174 | + ret = kho_preserve_folio(folio); | |
| 1175 | + if (ret) { | |
| 1176 | + folio_put(folio); | |
| 1177 | + return ERR_PTR(ret); | |
| 1112 | 1178 | } |
| 1113 | 1179 | |
| 1114 | − err = blocking_notifier_call_chain(&kho_out.chain_head, KEXEC_KHO_ABORT, | |
| 1115 | − NULL); | |
| 1116 | − err = notifier_to_errno(err); | |
| 1117 | − | |
| 1118 | − if (err) | |
| 1119 | − pr_err("Failed to abort KHO finalization: %d\n", err); | |
| 1120 | − | |
| 1121 | − return err; | |
| 1180 | + return folio_address(folio); | |
| 1122 | 1181 | } |
| 1182 | +EXPORT_SYMBOL_GPL(kho_alloc_preserve); | |
| 1123 | 1183 | |
| 1124 | −static int kho_finalize(void) | |
| 1184 | +/** | |
| 1185 | + * kho_unpreserve_free - Unpreserve and free memory. | |
| 1186 | + * @mem: Pointer to the memory allocated by kho_alloc_preserve(). | |
| 1187 | + * | |
| 1188 | + * Unregisters the memory from KHO preservation and frees the underlying | |
| 1189 | + * pages back to the system. This function should be called to clean up | |
| 1190 | + * memory allocated with kho_alloc_preserve(). | |
| 1191 | + */ | |
| 1192 | +void kho_unpreserve_free(void *mem) | |
| 1125 | 1193 | { |
| 1126 | − int err = 0; | |
| 1127 | − u64 *preserved_mem_map; | |
| 1128 | − void *fdt = page_to_virt(kho_out.ser.fdt); | |
| 1129 | − | |
| 1130 | − err |= fdt_create(fdt, PAGE_SIZE); | |
| 1131 | − err |= fdt_finish_reservemap(fdt); | |
| 1132 | − err |= fdt_begin_node(fdt, ""); | |
| 1133 | − err |= fdt_property_string(fdt, "compatible", KHO_FDT_COMPATIBLE); | |
| 1134 | − /** | |
| 1135 | − * Reserve the preserved-memory-map property in the root FDT, so | |
| 1136 | − * that all property definitions will precede subnodes created by | |
| 1137 | − * KHO callers. | |
| 1138 | − */ | |
| 1139 | − err |= fdt_property_placeholder(fdt, PROP_PRESERVED_MEMORY_MAP, | |
| 1140 | − sizeof(*preserved_mem_map), | |
| 1141 | − (void **)&preserved_mem_map); | |
| 1142 | − if (err) | |
| 1143 | − goto abort; | |
| 1144 | − | |
| 1145 | − err = kho_preserve_folio(page_folio(kho_out.ser.fdt)); | |
| 1146 | − if (err) | |
| 1147 | − goto abort; | |
| 1148 | − | |
| 1149 | − err = blocking_notifier_call_chain(&kho_out.chain_head, | |
| 1150 | − KEXEC_KHO_FINALIZE, &kho_out.ser); | |
| 1151 | − err = notifier_to_errno(err); | |
| 1152 | − if (err) | |
| 1153 | − goto abort; | |
| 1154 | − | |
| 1155 | − err = kho_mem_serialize(&kho_out.ser); | |
| 1156 | − if (err) | |
| 1157 | − goto abort; | |
| 1158 | − | |
| 1159 | − *preserved_mem_map = (u64)virt_to_phys(kho_out.ser.preserved_mem_map); | |
| 1160 | − | |
| 1161 | − err |= fdt_end_node(fdt); | |
| 1162 | − err |= fdt_finish(fdt); | |
| 1194 | + struct folio *folio; | |
| 1163 | 1195 | |
| 1164 | −abort: | |
| 1165 | − if (err) { | |
| 1166 | − pr_err("Failed to convert KHO state tree: %d\n", err); | |
| 1167 | − kho_abort(); | |
| 1168 | − } | |
| 1196 | + if (!mem) | |
| 1197 | + return; | |
| 1169 | 1198 | |
| 1170 | − return err; | |
| 1199 | + folio = virt_to_folio(mem); | |
| 1200 | + kho_unpreserve_folio(folio); | |
| 1201 | + folio_put(folio); | |
| 1171 | 1202 | } |
| 1203 | +EXPORT_SYMBOL_GPL(kho_unpreserve_free); | |
| 1172 | 1204 | |
| 1173 | −static int kho_out_finalize_get(void *data, u64 *val) | |
| 1205 | +/** | |
| 1206 | + * kho_restore_free - Restore and free memory after kexec. | |
| 1207 | + * @mem: Pointer to the memory (in the new kernel's address space) | |
| 1208 | + * that was allocated by the old kernel. | |
| 1209 | + * | |
| 1210 | + * This function is intended to be called in the new kernel (post-kexec) | |
| 1211 | + * to take ownership of and free a memory region that was preserved by the | |
| 1212 | + * old kernel using kho_alloc_preserve(). | |
| 1213 | + * | |
| 1214 | + * It first restores the pages from KHO (using their physical address) | |
| 1215 | + * and then frees the pages back to the new kernel's page allocator. | |
| 1216 | + */ | |
| 1217 | +void kho_restore_free(void *mem) | |
| 1174 | 1218 | { |
| 1175 | − mutex_lock(&kho_out.lock); | |
| 1176 | − *val = kho_out.finalized; | |
| 1177 | − mutex_unlock(&kho_out.lock); | |
| 1219 | + struct folio *folio; | |
| 1178 | 1220 | |
| 1179 | − return 0; | |
| 1221 | + if (!mem) | |
| 1222 | + return; | |
| 1223 | + | |
| 1224 | + folio = kho_restore_folio(__pa(mem)); | |
| 1225 | + if (!WARN_ON(!folio)) | |
| 1226 | + folio_put(folio); | |
| 1180 | 1227 | } |
| 1228 | +EXPORT_SYMBOL_GPL(kho_restore_free); | |
| 1181 | 1229 | |
| 1182 | −static int kho_out_finalize_set(void *data, u64 _val) | |
| 1230 | +int kho_finalize(void) | |
| 1183 | 1231 | { |
| 1184 | − int ret = 0; | |
| 1185 | − bool val = !!_val; | |
| 1186 | − | |
| 1187 | − mutex_lock(&kho_out.lock); | |
| 1188 | − | |
| 1189 | − if (val == kho_out.finalized) { | |
| 1190 | − if (kho_out.finalized) | |
| 1191 | − ret = -EEXIST; | |
| 1192 | − else | |
| 1193 | − ret = -ENOENT; | |
| 1194 | − goto unlock; | |
| 1195 | − } | |
| 1232 | + int ret; | |
| 1196 | 1233 | |
| 1197 | − if (val) | |
| 1198 | − ret = kho_finalize(); | |
| 1199 | − else | |
| 1200 | − ret = kho_abort(); | |
| 1234 | + if (!kho_enable) | |
| 1235 | + return -EOPNOTSUPP; | |
| 1201 | 1236 | |
| 1237 | + guard(mutex)(&kho_out.lock); | |
| 1238 | + ret = kho_mem_serialize(&kho_out); | |
| 1202 | 1239 | if (ret) |
| 1203 | − goto unlock; | |
| 1204 | − | |
| 1205 | − kho_out.finalized = val; | |
| 1206 | − ret = kho_out_update_debugfs_fdt(); | |
| 1207 | − | |
| 1208 | −unlock: | |
| 1209 | − mutex_unlock(&kho_out.lock); | |
| 1210 | − return ret; | |
| 1211 | −} | |
| 1212 | − | |
| 1213 | −DEFINE_DEBUGFS_ATTRIBUTE(fops_kho_out_finalize, kho_out_finalize_get, | |
| 1214 | − kho_out_finalize_set, "%llu\n"); | |
| 1240 | + return ret; | |
| 1215 | 1241 | |
| 1216 | −static int scratch_phys_show(struct seq_file *m, void *v) | |
| 1217 | −{ | |
| 1218 | − for (int i = 0; i < kho_scratch_cnt; i++) | |
| 1219 | − seq_printf(m, "0x%llx\n", kho_scratch[i].addr); | |
| 1242 | + kho_out.finalized = true; | |
| 1220 | 1243 | |
| 1221 | 1244 | return 0; |
| 1222 | 1245 | } |
| 1223 | −DEFINE_SHOW_ATTRIBUTE(scratch_phys); | |
| 1224 | 1246 | |
| 1225 | −static int scratch_len_show(struct seq_file *m, void *v) | |
| 1247 | +bool kho_finalized(void) | |
| 1226 | 1248 | { |
| 1227 | − for (int i = 0; i < kho_scratch_cnt; i++) | |
| 1228 | − seq_printf(m, "0x%llx\n", kho_scratch[i].size); | |
| 1229 | − | |
| 1230 | − return 0; | |
| 1231 | −} | |
| 1232 | −DEFINE_SHOW_ATTRIBUTE(scratch_len); | |
| 1233 | − | |
| 1234 | −static __init int kho_out_debugfs_init(void) | |
| 1235 | −{ | |
| 1236 | − struct dentry *dir, *f, *sub_fdt_dir; | |
| 1237 | − | |
| 1238 | − dir = debugfs_create_dir("out", debugfs_root); | |
| 1239 | − if (IS_ERR(dir)) | |
| 1240 | − return -ENOMEM; | |
| 1241 | − | |
| 1242 | − sub_fdt_dir = debugfs_create_dir("sub_fdts", dir); | |
| 1243 | − if (IS_ERR(sub_fdt_dir)) | |
| 1244 | − goto err_rmdir; | |
| 1245 | − | |
| 1246 | − f = debugfs_create_file("scratch_phys", 0400, dir, NULL, | |
| 1247 | − &scratch_phys_fops); | |
| 1248 | − if (IS_ERR(f)) | |
| 1249 | − goto err_rmdir; | |
| 1250 | − | |
| 1251 | − f = debugfs_create_file("scratch_len", 0400, dir, NULL, | |
| 1252 | − &scratch_len_fops); | |
| 1253 | − if (IS_ERR(f)) | |
| 1254 | − goto err_rmdir; | |
| 1255 | − | |
| 1256 | − f = debugfs_create_file("finalize", 0600, dir, NULL, | |
| 1257 | − &fops_kho_out_finalize); | |
| 1258 | − if (IS_ERR(f)) | |
| 1259 | − goto err_rmdir; | |
| 1260 | − | |
| 1261 | − kho_out.dir = dir; | |
| 1262 | − kho_out.ser.sub_fdt_dir = sub_fdt_dir; | |
| 1263 | − return 0; | |
| 1264 | − | |
| 1265 | −err_rmdir: | |
| 1266 | − debugfs_remove_recursive(dir); | |
| 1267 | − return -ENOENT; | |
| 1249 | + guard(mutex)(&kho_out.lock); | |
| 1250 | + return kho_out.finalized; | |
| 1268 | 1251 | } |
| 1269 | 1252 | |
| 1270 | 1253 | struct kho_in { |
| 1271 | − struct dentry *dir; | |
| 1272 | 1254 | phys_addr_t fdt_phys; |
| 1273 | 1255 | phys_addr_t scratch_phys; |
| 1274 | − struct list_head fdt_list; | |
| 1256 | + struct kho_debugfs dbg; | |
| 1275 | 1257 | }; |
| 1276 | 1258 | |
| 1277 | 1259 | static struct kho_in kho_in = { |
| 1278 | − .fdt_list = LIST_HEAD_INIT(kho_in.fdt_list), | |
| 1279 | 1260 | }; |
| 1280 | 1261 | |
| 1281 | 1262 | static const void *kho_get_fdt(void) |
| @@ -1339,91 +1320,52 @@ int kho_retrieve_subtree(const char *name, phys_addr_t *phys) · 57 unmodified lines | ||
| 1339 | 1320 | } |
| 1340 | 1321 | EXPORT_SYMBOL_GPL(kho_retrieve_subtree); |
| 1341 | 1322 | |
| 1342 | −/* Handling for debugfs/kho/in */ | |
| 1343 | − | |
| 1344 | −static __init int kho_in_debugfs_init(const void *fdt) | |
| 1323 | +static __init int kho_out_fdt_setup(void) | |
| 1345 | 1324 | { |
| 1346 | − struct dentry *sub_fdt_dir; | |
| 1347 | − int err, child; | |
| 1348 | − | |
| 1349 | − kho_in.dir = debugfs_create_dir("in", debugfs_root); | |
| 1350 | − if (IS_ERR(kho_in.dir)) | |
| 1351 | − return PTR_ERR(kho_in.dir); | |
| 1352 | − | |
| 1353 | − sub_fdt_dir = debugfs_create_dir("sub_fdts", kho_in.dir); | |
| 1354 | − if (IS_ERR(sub_fdt_dir)) { | |
| 1355 | − err = PTR_ERR(sub_fdt_dir); | |
| 1356 | − goto err_rmdir; | |
| 1357 | − } | |
| 1358 | − | |
| 1359 | − err = kho_debugfs_fdt_add(&kho_in.fdt_list, kho_in.dir, "fdt", fdt); | |
| 1360 | − if (err) | |
| 1361 | − goto err_rmdir; | |
| 1362 | − | |
| 1363 | − fdt_for_each_subnode(child, fdt, 0) { | |
| 1364 | − int len = 0; | |
| 1365 | − const char *name = fdt_get_name(fdt, child, NULL); | |
| 1366 | − const u64 *fdt_phys; | |
| 1367 | − | |
| 1368 | − fdt_phys = fdt_getprop(fdt, child, "fdt", &len); | |
| 1369 | − if (!fdt_phys) | |
| 1370 | − continue; | |
| 1371 | − if (len != sizeof(*fdt_phys)) { | |
| 1372 | − pr_warn("node `%s`'s prop `fdt` has invalid length: %d\n", | |
| 1373 | − name, len); | |
| 1374 | − continue; | |
| 1375 | − } | |
| 1376 | − err = kho_debugfs_fdt_add(&kho_in.fdt_list, sub_fdt_dir, name, | |
| 1377 | − phys_to_virt(*fdt_phys)); | |
| 1378 | − if (err) { | |
| 1379 | − pr_warn("failed to add fdt `%s` to debugfs: %d\n", name, | |
| 1380 | − err); | |
| 1381 | − continue; | |
| 1382 | − } | |
| 1383 | − } | |
| 1325 | + void *root = kho_out.fdt; | |
| 1326 | + u64 empty_mem_map = 0; | |
| 1327 | + int err; | |
| 1384 | 1328 | |
| 1385 | − return 0; | |
| 1329 | + err = fdt_create(root, PAGE_SIZE); | |
| 1330 | + err |= fdt_finish_reservemap(root); | |
| 1331 | + err |= fdt_begin_node(root, ""); | |
| 1332 | + err |= fdt_property_string(root, "compatible", KHO_FDT_COMPATIBLE); | |
| 1333 | + err |= fdt_property(root, PROP_PRESERVED_MEMORY_MAP, &empty_mem_map, | |
| 1334 | + sizeof(empty_mem_map)); | |
| 1335 | + err |= fdt_end_node(root); | |
| 1336 | + err |= fdt_finish(root); | |
| 1386 | 1337 | |
| 1387 | −err_rmdir: | |
| 1388 | − debugfs_remove_recursive(kho_in.dir); | |
| 1389 | 1338 | return err; |
| 1390 | 1339 | } |
| 1391 | 1340 | |
| 1392 | 1341 | static __init int kho_init(void) |
| 1393 | 1342 | { |
| 1394 | − int err = 0; | |
| 1395 | 1343 | const void *fdt = kho_get_fdt(); |
| 1344 | + int err = 0; | |
| 1396 | 1345 | |
| 1397 | 1346 | if (!kho_enable) |
| 1398 | 1347 | return 0; |
| 1399 | 1348 | |
| 1400 | − kho_out.ser.fdt = alloc_page(GFP_KERNEL); | |
| 1401 | − if (!kho_out.ser.fdt) { | |
| 1402 | − err = -ENOMEM; | |
| 1349 | + kho_out.fdt = kho_alloc_preserve(PAGE_SIZE); | |
| 1350 | + if (IS_ERR(kho_out.fdt)) { | |
| 1351 | + err = PTR_ERR(kho_out.fdt); | |
| 1403 | 1352 | goto err_free_scratch; |
| 1404 | 1353 | } |
| 1405 | 1354 | |
| 1406 | − debugfs_root = debugfs_create_dir("kho", NULL); | |
| 1407 | − if (IS_ERR(debugfs_root)) { | |
| 1408 | − err = -ENOENT; | |
| 1355 | + err = kho_debugfs_init(); | |
| 1356 | + if (err) | |
| 1409 | 1357 | goto err_free_fdt; |
| 1410 | − } | |
| 1411 | 1358 | |
| 1412 | − err = kho_out_debugfs_init(); | |
| 1359 | + err = kho_out_debugfs_init(&kho_out.dbg); | |
| 1413 | 1360 | if (err) |
| 1414 | 1361 | goto err_free_fdt; |
| 1415 | 1362 | |
| 1416 | − if (fdt) { | |
| 1417 | − err = kho_in_debugfs_init(fdt); | |
| 1418 | − /* | |
| 1419 | − * Failure to create /sys/kernel/debug/kho/in does not prevent | |
| 1420 | − * reviving state from KHO and setting up KHO for the next | |
| 1421 | − * kexec. | |
| 1422 | − */ | |
| 1423 | − if (err) | |
| 1424 | − pr_err("failed exposing handover FDT in debugfs: %d\n", | |
| 1425 | − err); | |
| 1363 | + err = kho_out_fdt_setup(); | |
| 1364 | + if (err) | |
| 1365 | + goto err_free_fdt; | |
| 1426 | 1366 | |
| 1367 | + if (fdt) { | |
| 1368 | + kho_in_debugfs_init(&kho_in.dbg, fdt); | |
| 1427 | 1369 | return 0; |
| 1428 | 1370 | } |
| 1429 | 1371 | |
| @@ -1432,17 +1374,29 @@ static __init int kho_init(void) · 2 unmodified lines | ||
| 1432 | 1374 | unsigned long count = kho_scratch[i].size >> PAGE_SHIFT; |
| 1433 | 1375 | unsigned long pfn; |
| 1434 | 1376 | |
| 1377 | + /* | |
| 1378 | + * When debug_pagealloc is enabled, __free_pages() clears the | |
| 1379 | + * corresponding PRESENT bit in the kernel page table. | |
| 1380 | + * Subsequent kmemleak scans of these pages cause the | |
| 1381 | + * non-PRESENT page faults. | |
| 1382 | + * Mark scratch areas with kmemleak_ignore_phys() to exclude | |
| 1383 | + * them from kmemleak scanning. | |
| 1384 | + */ | |
| 1385 | + kmemleak_ignore_phys(kho_scratch[i].addr); | |
| 1435 | 1386 | for (pfn = base_pfn; pfn < base_pfn + count; |
| 1436 | 1387 | pfn += pageblock_nr_pages) |
| 1437 | 1388 | init_cma_reserved_pageblock(pfn_to_page(pfn)); |
| 1438 | 1389 | } |
| 1439 | 1390 | |
| 1391 | + WARN_ON_ONCE(kho_debugfs_fdt_add(&kho_out.dbg, "fdt", | |
| 1392 | + kho_out.fdt, true)); | |
| 1393 | + | |
| 1440 | 1394 | return 0; |
| 1441 | 1395 | |
| 1442 | 1396 | err_free_fdt: |
| 1443 | − put_page(kho_out.ser.fdt); | |
| 1444 | − kho_out.ser.fdt = NULL; | |
| 1397 | + kho_unpreserve_free(kho_out.fdt); | |
| 1445 | 1398 | err_free_scratch: |
| 1399 | + kho_out.fdt = NULL; | |
| 1446 | 1400 | for (int i = 0; i < kho_scratch_cnt; i++) { |
| 1447 | 1401 | void *start = __va(kho_scratch[i].addr); |
| 1448 | 1402 | void *end = start + kho_scratch[i].size; |
| @@ -1452,7 +1406,7 @@ err_free_scratch: · 3 unmodified lines | ||
| 1452 | 1406 | kho_enable = false; |
| 1453 | 1407 | return err; |
| 1454 | 1408 | } |
| 1455 | −late_initcall(kho_init); | |
| 1409 | +fs_initcall(kho_init); | |
| 1456 | 1410 | |
| 1457 | 1411 | static void __init kho_release_scratch(void) |
| 1458 | 1412 | { |
| @@ -1480,16 +1434,12 @@ static void __init kho_release_scratch(void) · 21 unmodified lines | ||
| 1480 | 1434 | |
| 1481 | 1435 | void __init kho_memory_init(void) |
| 1482 | 1436 | { |
| 1483 | − struct folio *folio; | |
| 1484 | − | |
| 1485 | 1437 | if (kho_in.scratch_phys) { |
| 1486 | 1438 | kho_scratch = phys_to_virt(kho_in.scratch_phys); |
| 1487 | 1439 | kho_release_scratch(); |
| 1488 | 1440 | |
| 1489 | − kho_mem_deserialize(kho_get_fdt()); | |
| 1490 | − folio = kho_restore_folio(kho_in.fdt_phys); | |
| 1491 | − if (!folio) | |
| 1492 | − pr_warn("failed to restore folio for KHO fdt\n"); | |
| 1441 | + if (!kho_mem_deserialize(kho_get_fdt())) | |
| 1442 | + kho_in.fdt_phys = 0; | |
| 1493 | 1443 | } else { |
| 1494 | 1444 | kho_reserve_scratch(); |
| 1495 | 1445 | } |
| @@ -1545,8 +1495,8 @@ void __init kho_populate(phys_addr_t fdt_phys, u64 fdt_len, · 49 unmodified lines | ||
| 1545 | 1495 | memblock_add(area->addr, size); |
| 1546 | 1496 | err = memblock_mark_kho_scratch(area->addr, size); |
| 1547 | 1497 | if (WARN_ON(err)) { |
| 1548 | − pr_warn("failed to mark the scratch region 0x%pa+0x%pa: %d", | |
| 1549 | − &area->addr, &size, err); | |
| 1498 | + pr_warn("failed to mark the scratch region 0x%pa+0x%pa: %pe", | |
| 1499 | + &area->addr, &size, ERR_PTR(err)); | |
| 1550 | 1500 | goto out; |
| 1551 | 1501 | } |
| 1552 | 1502 | pr_debug("Marked 0x%pa+0x%pa as scratch", &area->addr, &size); |
| @@ -1566,7 +1516,7 @@ void __init kho_populate(phys_addr_t fdt_phys, u64 fdt_len, · 13 unmodified lines | ||
| 1566 | 1516 | kho_in.fdt_phys = fdt_phys; |
| 1567 | 1517 | kho_in.scratch_phys = scratch_phys; |
| 1568 | 1518 | kho_scratch_cnt = scratch_cnt; |
| 1569 | − pr_info("found kexec handover data. Will skip init for some devices\n"); | |
| 1519 | + pr_info("found kexec handover data.\n"); | |
| 1570 | 1520 | |
| 1571 | 1521 | out: |
| 1572 | 1522 | if (fdt) |
| @@ -1585,10 +1535,10 @@ int kho_fill_kimage(struct kimage *image) · 12 unmodified lines | ||
| 1585 | 1535 | int err = 0; |
| 1586 | 1536 | struct kexec_buf scratch; |
| 1587 | 1537 | |
| 1588 | − if (!kho_out.finalized) | |
| 1538 | + if (!kho_enable) | |
| 1589 | 1539 | return 0; |
| 1590 | 1540 | |
| 1591 | − image->kho.fdt = page_to_phys(kho_out.ser.fdt); | |
| 1541 | + image->kho.fdt = virt_to_phys(kho_out.fdt); | |
| 1592 | 1542 | |
| 1593 | 1543 | scratch_size = sizeof(*kho_scratch) * kho_scratch_cnt; |
| 1594 | 1544 | scratch = (struct kexec_buf){ |
File renamed without changes.
| @@ -0,0 +1,221 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0-only | |
| 2 | +/* | |
| 3 | + * kexec_handover_debugfs.c - kexec handover debugfs interfaces | |
| 4 | + * Copyright (C) 2023 Alexander Graf <graf@amazon.com> | |
| 5 | + * Copyright (C) 2025 Microsoft Corporation, Mike Rapoport <rppt@kernel.org> | |
| 6 | + * Copyright (C) 2025 Google LLC, Changyuan Lyu <changyuanl@google.com> | |
| 7 | + * Copyright (C) 2025 Google LLC, Pasha Tatashin <pasha.tatashin@soleen.com> | |
| 8 | + */ | |
| 9 | + | |
| 10 | +#define pr_fmt(fmt) "KHO: " fmt | |
| 11 | + | |
| 12 | +#include <linux/init.h> | |
| 13 | +#include <linux/io.h> | |
| 14 | +#include <linux/libfdt.h> | |
| 15 | +#include <linux/mm.h> | |
| 16 | +#include "kexec_handover_internal.h" | |
| 17 | + | |
| 18 | +static struct dentry *debugfs_root; | |
| 19 | + | |
| 20 | +struct fdt_debugfs { | |
| 21 | + struct list_head list; | |
| 22 | + struct debugfs_blob_wrapper wrapper; | |
| 23 | + struct dentry *file; | |
| 24 | +}; | |
| 25 | + | |
| 26 | +static int __kho_debugfs_fdt_add(struct list_head *list, struct dentry *dir, | |
| 27 | + const char *name, const void *fdt) | |
| 28 | +{ | |
| 29 | + struct fdt_debugfs *f; | |
| 30 | + struct dentry *file; | |
| 31 | + | |
| 32 | + f = kmalloc(sizeof(*f), GFP_KERNEL); | |
| 33 | + if (!f) | |
| 34 | + return -ENOMEM; | |
| 35 | + | |
| 36 | + f->wrapper.data = (void *)fdt; | |
| 37 | + f->wrapper.size = fdt_totalsize(fdt); | |
| 38 | + | |
| 39 | + file = debugfs_create_blob(name, 0400, dir, &f->wrapper); | |
| 40 | + if (IS_ERR(file)) { | |
| 41 | + kfree(f); | |
| 42 | + return PTR_ERR(file); | |
| 43 | + } | |
| 44 | + | |
| 45 | + f->file = file; | |
| 46 | + list_add(&f->list, list); | |
| 47 | + | |
| 48 | + return 0; | |
| 49 | +} | |
| 50 | + | |
| 51 | +int kho_debugfs_fdt_add(struct kho_debugfs *dbg, const char *name, | |
| 52 | + const void *fdt, bool root) | |
| 53 | +{ | |
| 54 | + struct dentry *dir; | |
| 55 | + | |
| 56 | + if (root) | |
| 57 | + dir = dbg->dir; | |
| 58 | + else | |
| 59 | + dir = dbg->sub_fdt_dir; | |
| 60 | + | |
| 61 | + return __kho_debugfs_fdt_add(&dbg->fdt_list, dir, name, fdt); | |
| 62 | +} | |
| 63 | + | |
| 64 | +void kho_debugfs_fdt_remove(struct kho_debugfs *dbg, void *fdt) | |
| 65 | +{ | |
| 66 | + struct fdt_debugfs *ff; | |
| 67 | + | |
| 68 | + list_for_each_entry(ff, &dbg->fdt_list, list) { | |
| 69 | + if (ff->wrapper.data == fdt) { | |
| 70 | + debugfs_remove(ff->file); | |
| 71 | + list_del(&ff->list); | |
| 72 | + kfree(ff); | |
| 73 | + break; | |
| 74 | + } | |
| 75 | + } | |
| 76 | +} | |
| 77 | + | |
| 78 | +static int kho_out_finalize_get(void *data, u64 *val) | |
| 79 | +{ | |
| 80 | + *val = kho_finalized(); | |
| 81 | + | |
| 82 | + return 0; | |
| 83 | +} | |
| 84 | + | |
| 85 | +static int kho_out_finalize_set(void *data, u64 val) | |
| 86 | +{ | |
| 87 | + if (val) | |
| 88 | + return kho_finalize(); | |
| 89 | + else | |
| 90 | + return -EINVAL; | |
| 91 | +} | |
| 92 | + | |
| 93 | +DEFINE_DEBUGFS_ATTRIBUTE(kho_out_finalize_fops, kho_out_finalize_get, | |
| 94 | + kho_out_finalize_set, "%llu\n"); | |
| 95 | + | |
| 96 | +static int scratch_phys_show(struct seq_file *m, void *v) | |
| 97 | +{ | |
| 98 | + for (int i = 0; i < kho_scratch_cnt; i++) | |
| 99 | + seq_printf(m, "0x%llx\n", kho_scratch[i].addr); | |
| 100 | + | |
| 101 | + return 0; | |
| 102 | +} | |
| 103 | +DEFINE_SHOW_ATTRIBUTE(scratch_phys); | |
| 104 | + | |
| 105 | +static int scratch_len_show(struct seq_file *m, void *v) | |
| 106 | +{ | |
| 107 | + for (int i = 0; i < kho_scratch_cnt; i++) | |
| 108 | + seq_printf(m, "0x%llx\n", kho_scratch[i].size); | |
| 109 | + | |
| 110 | + return 0; | |
| 111 | +} | |
| 112 | +DEFINE_SHOW_ATTRIBUTE(scratch_len); | |
| 113 | + | |
| 114 | +__init void kho_in_debugfs_init(struct kho_debugfs *dbg, const void *fdt) | |
| 115 | +{ | |
| 116 | + struct dentry *dir, *sub_fdt_dir; | |
| 117 | + int err, child; | |
| 118 | + | |
| 119 | + INIT_LIST_HEAD(&dbg->fdt_list); | |
| 120 | + | |
| 121 | + dir = debugfs_create_dir("in", debugfs_root); | |
| 122 | + if (IS_ERR(dir)) { | |
| 123 | + err = PTR_ERR(dir); | |
| 124 | + goto err_out; | |
| 125 | + } | |
| 126 | + | |
| 127 | + sub_fdt_dir = debugfs_create_dir("sub_fdts", dir); | |
| 128 | + if (IS_ERR(sub_fdt_dir)) { | |
| 129 | + err = PTR_ERR(sub_fdt_dir); | |
| 130 | + goto err_rmdir; | |
| 131 | + } | |
| 132 | + | |
| 133 | + err = __kho_debugfs_fdt_add(&dbg->fdt_list, dir, "fdt", fdt); | |
| 134 | + if (err) | |
| 135 | + goto err_rmdir; | |
| 136 | + | |
| 137 | + fdt_for_each_subnode(child, fdt, 0) { | |
| 138 | + int len = 0; | |
| 139 | + const char *name = fdt_get_name(fdt, child, NULL); | |
| 140 | + const u64 *fdt_phys; | |
| 141 | + | |
| 142 | + fdt_phys = fdt_getprop(fdt, child, "fdt", &len); | |
| 143 | + if (!fdt_phys) | |
| 144 | + continue; | |
| 145 | + if (len != sizeof(*fdt_phys)) { | |
| 146 | + pr_warn("node %s prop fdt has invalid length: %d\n", | |
| 147 | + name, len); | |
| 148 | + continue; | |
| 149 | + } | |
| 150 | + err = __kho_debugfs_fdt_add(&dbg->fdt_list, sub_fdt_dir, name, | |
| 151 | + phys_to_virt(*fdt_phys)); | |
| 152 | + if (err) { | |
| 153 | + pr_warn("failed to add fdt %s to debugfs: %pe\n", name, | |
| 154 | + ERR_PTR(err)); | |
| 155 | + continue; | |
| 156 | + } | |
| 157 | + } | |
| 158 | + | |
| 159 | + dbg->dir = dir; | |
| 160 | + dbg->sub_fdt_dir = sub_fdt_dir; | |
| 161 | + | |
| 162 | + return; | |
| 163 | +err_rmdir: | |
| 164 | + debugfs_remove_recursive(dir); | |
| 165 | +err_out: | |
| 166 | + /* | |
| 167 | + * Failure to create /sys/kernel/debug/kho/in does not prevent | |
| 168 | + * reviving state from KHO and setting up KHO for the next | |
| 169 | + * kexec. | |
| 170 | + */ | |
| 171 | + if (err) { | |
| 172 | + pr_err("failed exposing handover FDT in debugfs: %pe\n", | |
| 173 | + ERR_PTR(err)); | |
| 174 | + } | |
| 175 | +} | |
| 176 | + | |
| 177 | +__init int kho_out_debugfs_init(struct kho_debugfs *dbg) | |
| 178 | +{ | |
| 179 | + struct dentry *dir, *f, *sub_fdt_dir; | |
| 180 | + | |
| 181 | + INIT_LIST_HEAD(&dbg->fdt_list); | |
| 182 | + | |
| 183 | + dir = debugfs_create_dir("out", debugfs_root); | |
| 184 | + if (IS_ERR(dir)) | |
| 185 | + return -ENOMEM; | |
| 186 | + | |
| 187 | + sub_fdt_dir = debugfs_create_dir("sub_fdts", dir); | |
| 188 | + if (IS_ERR(sub_fdt_dir)) | |
| 189 | + goto err_rmdir; | |
| 190 | + | |
| 191 | + f = debugfs_create_file("scratch_phys", 0400, dir, NULL, | |
| 192 | + &scratch_phys_fops); | |
| 193 | + if (IS_ERR(f)) | |
| 194 | + goto err_rmdir; | |
| 195 | + | |
| 196 | + f = debugfs_create_file("scratch_len", 0400, dir, NULL, | |
| 197 | + &scratch_len_fops); | |
| 198 | + if (IS_ERR(f)) | |
| 199 | + goto err_rmdir; | |
| 200 | + | |
| 201 | + f = debugfs_create_file("finalize", 0600, dir, NULL, | |
| 202 | + &kho_out_finalize_fops); | |
| 203 | + if (IS_ERR(f)) | |
| 204 | + goto err_rmdir; | |
| 205 | + | |
| 206 | + dbg->dir = dir; | |
| 207 | + dbg->sub_fdt_dir = sub_fdt_dir; | |
| 208 | + return 0; | |
| 209 | + | |
| 210 | +err_rmdir: | |
| 211 | + debugfs_remove_recursive(dir); | |
| 212 | + return -ENOENT; | |
| 213 | +} | |
| 214 | + | |
| 215 | +__init int kho_debugfs_init(void) | |
| 216 | +{ | |
| 217 | + debugfs_root = debugfs_create_dir("kho", NULL); | |
| 218 | + if (IS_ERR(debugfs_root)) | |
| 219 | + return -ENOENT; | |
| 220 | + return 0; | |
| 221 | +} | |
| @@ -0,0 +1,55 @@ | ||
| 1 | +/* SPDX-License-Identifier: GPL-2.0 */ | |
| 2 | +#ifndef LINUX_KEXEC_HANDOVER_INTERNAL_H | |
| 3 | +#define LINUX_KEXEC_HANDOVER_INTERNAL_H | |
| 4 | + | |
| 5 | +#include <linux/kexec_handover.h> | |
| 6 | +#include <linux/list.h> | |
| 7 | +#include <linux/types.h> | |
| 8 | + | |
| 9 | +#ifdef CONFIG_KEXEC_HANDOVER_DEBUGFS | |
| 10 | +#include <linux/debugfs.h> | |
| 11 | + | |
| 12 | +struct kho_debugfs { | |
| 13 | + struct dentry *dir; | |
| 14 | + struct dentry *sub_fdt_dir; | |
| 15 | + struct list_head fdt_list; | |
| 16 | +}; | |
| 17 | + | |
| 18 | +#else | |
| 19 | +struct kho_debugfs {}; | |
| 20 | +#endif | |
| 21 | + | |
| 22 | +extern struct kho_scratch *kho_scratch; | |
| 23 | +extern unsigned int kho_scratch_cnt; | |
| 24 | + | |
| 25 | +bool kho_finalized(void); | |
| 26 | +int kho_finalize(void); | |
| 27 | + | |
| 28 | +#ifdef CONFIG_KEXEC_HANDOVER_DEBUGFS | |
| 29 | +int kho_debugfs_init(void); | |
| 30 | +void kho_in_debugfs_init(struct kho_debugfs *dbg, const void *fdt); | |
| 31 | +int kho_out_debugfs_init(struct kho_debugfs *dbg); | |
| 32 | +int kho_debugfs_fdt_add(struct kho_debugfs *dbg, const char *name, | |
| 33 | + const void *fdt, bool root); | |
| 34 | +void kho_debugfs_fdt_remove(struct kho_debugfs *dbg, void *fdt); | |
| 35 | +#else | |
| 36 | +static inline int kho_debugfs_init(void) { return 0; } | |
| 37 | +static inline void kho_in_debugfs_init(struct kho_debugfs *dbg, | |
| 38 | + const void *fdt) { } | |
| 39 | +static inline int kho_out_debugfs_init(struct kho_debugfs *dbg) { return 0; } | |
| 40 | +static inline int kho_debugfs_fdt_add(struct kho_debugfs *dbg, const char *name, | |
| 41 | + const void *fdt, bool root) { return 0; } | |
| 42 | +static inline void kho_debugfs_fdt_remove(struct kho_debugfs *dbg, | |
| 43 | + void *fdt) { } | |
| 44 | +#endif /* CONFIG_KEXEC_HANDOVER_DEBUGFS */ | |
| 45 | + | |
| 46 | +#ifdef CONFIG_KEXEC_HANDOVER_DEBUG | |
| 47 | +bool kho_scratch_overlap(phys_addr_t phys, size_t size); | |
| 48 | +#else | |
| 49 | +static inline bool kho_scratch_overlap(phys_addr_t phys, size_t size) | |
| 50 | +{ | |
| 51 | + return false; | |
| 52 | +} | |
| 53 | +#endif /* CONFIG_KEXEC_HANDOVER_DEBUG */ | |
| 54 | + | |
| 55 | +#endif /* LINUX_KEXEC_HANDOVER_INTERNAL_H */ | |
| @@ -0,0 +1,450 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0 | |
| 2 | + | |
| 3 | +/* | |
| 4 | + * Copyright (c) 2025, Google LLC. | |
| 5 | + * Pasha Tatashin <pasha.tatashin@soleen.com> | |
| 6 | + */ | |
| 7 | + | |
| 8 | +/** | |
| 9 | + * DOC: Live Update Orchestrator (LUO) | |
| 10 | + * | |
| 11 | + * Live Update is a specialized, kexec-based reboot process that allows a | |
| 12 | + * running kernel to be updated from one version to another while preserving | |
| 13 | + * the state of selected resources and keeping designated hardware devices | |
| 14 | + * operational. For these devices, DMA activity may continue throughout the | |
| 15 | + * kernel transition. | |
| 16 | + * | |
| 17 | + * While the primary use case driving this work is supporting live updates of | |
| 18 | + * the Linux kernel when it is used as a hypervisor in cloud environments, the | |
| 19 | + * LUO framework itself is designed to be workload-agnostic. Live Update | |
| 20 | + * facilitates a full kernel version upgrade for any type of system. | |
| 21 | + * | |
| 22 | + * For example, a non-hypervisor system running an in-memory cache like | |
| 23 | + * memcached with many gigabytes of data can use LUO. The userspace service | |
| 24 | + * can place its cache into a memfd, have its state preserved by LUO, and | |
| 25 | + * restore it immediately after the kernel kexec. | |
| 26 | + * | |
| 27 | + * Whether the system is running virtual machines, containers, a | |
| 28 | + * high-performance database, or networking services, LUO's primary goal is to | |
| 29 | + * enable a full kernel update by preserving critical userspace state and | |
| 30 | + * keeping essential devices operational. | |
| 31 | + * | |
| 32 | + * The core of LUO is a mechanism that tracks the progress of a live update, | |
| 33 | + * along with a callback API that allows other kernel subsystems to participate | |
| 34 | + * in the process. Example subsystems that can hook into LUO include: kvm, | |
| 35 | + * iommu, interrupts, vfio, participating filesystems, and memory management. | |
| 36 | + * | |
| 37 | + * LUO uses Kexec Handover to transfer memory state from the current kernel to | |
| 38 | + * the next kernel. For more details see | |
| 39 | + * Documentation/core-api/kho/concepts.rst. | |
| 40 | + */ | |
| 41 | + | |
| 42 | +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt | |
| 43 | + | |
| 44 | +#include <linux/atomic.h> | |
| 45 | +#include <linux/errno.h> | |
| 46 | +#include <linux/file.h> | |
| 47 | +#include <linux/fs.h> | |
| 48 | +#include <linux/init.h> | |
| 49 | +#include <linux/io.h> | |
| 50 | +#include <linux/kernel.h> | |
| 51 | +#include <linux/kexec_handover.h> | |
| 52 | +#include <linux/kho/abi/luo.h> | |
| 53 | +#include <linux/kobject.h> | |
| 54 | +#include <linux/libfdt.h> | |
| 55 | +#include <linux/liveupdate.h> | |
| 56 | +#include <linux/miscdevice.h> | |
| 57 | +#include <linux/mm.h> | |
| 58 | +#include <linux/sizes.h> | |
| 59 | +#include <linux/string.h> | |
| 60 | +#include <linux/unaligned.h> | |
| 61 | + | |
| 62 | +#include "kexec_handover_internal.h" | |
| 63 | +#include "luo_internal.h" | |
| 64 | + | |
| 65 | +static struct { | |
| 66 | + bool enabled; | |
| 67 | + void *fdt_out; | |
| 68 | + void *fdt_in; | |
| 69 | + u64 liveupdate_num; | |
| 70 | +} luo_global; | |
| 71 | + | |
| 72 | +static int __init early_liveupdate_param(char *buf) | |
| 73 | +{ | |
| 74 | + return kstrtobool(buf, &luo_global.enabled); | |
| 75 | +} | |
| 76 | +early_param("liveupdate", early_liveupdate_param); | |
| 77 | + | |
| 78 | +static int __init luo_early_startup(void) | |
| 79 | +{ | |
| 80 | + phys_addr_t fdt_phys; | |
| 81 | + int err, ln_size; | |
| 82 | + const void *ptr; | |
| 83 | + | |
| 84 | + if (!kho_is_enabled()) { | |
| 85 | + if (liveupdate_enabled()) | |
| 86 | + pr_warn("Disabling liveupdate because KHO is disabled\n"); | |
| 87 | + luo_global.enabled = false; | |
| 88 | + return 0; | |
| 89 | + } | |
| 90 | + | |
| 91 | + /* Retrieve LUO subtree, and verify its format. */ | |
| 92 | + err = kho_retrieve_subtree(LUO_FDT_KHO_ENTRY_NAME, &fdt_phys); | |
| 93 | + if (err) { | |
| 94 | + if (err != -ENOENT) { | |
| 95 | + pr_err("failed to retrieve FDT '%s' from KHO: %pe\n", | |
| 96 | + LUO_FDT_KHO_ENTRY_NAME, ERR_PTR(err)); | |
| 97 | + return err; | |
| 98 | + } | |
| 99 | + | |
| 100 | + return 0; | |
| 101 | + } | |
| 102 | + | |
| 103 | + luo_global.fdt_in = phys_to_virt(fdt_phys); | |
| 104 | + err = fdt_node_check_compatible(luo_global.fdt_in, 0, | |
| 105 | + LUO_FDT_COMPATIBLE); | |
| 106 | + if (err) { | |
| 107 | + pr_err("FDT '%s' is incompatible with '%s' [%d]\n", | |
| 108 | + LUO_FDT_KHO_ENTRY_NAME, LUO_FDT_COMPATIBLE, err); | |
| 109 | + | |
| 110 | + return -EINVAL; | |
| 111 | + } | |
| 112 | + | |
| 113 | + ln_size = 0; | |
| 114 | + ptr = fdt_getprop(luo_global.fdt_in, 0, LUO_FDT_LIVEUPDATE_NUM, | |
| 115 | + &ln_size); | |
| 116 | + if (!ptr || ln_size != sizeof(luo_global.liveupdate_num)) { | |
| 117 | + pr_err("Unable to get live update number '%s' [%d]\n", | |
| 118 | + LUO_FDT_LIVEUPDATE_NUM, ln_size); | |
| 119 | + | |
| 120 | + return -EINVAL; | |
| 121 | + } | |
| 122 | + | |
| 123 | + luo_global.liveupdate_num = get_unaligned((u64 *)ptr); | |
| 124 | + pr_info("Retrieved live update data, liveupdate number: %lld\n", | |
| 125 | + luo_global.liveupdate_num); | |
| 126 | + | |
| 127 | + err = luo_session_setup_incoming(luo_global.fdt_in); | |
| 128 | + if (err) | |
| 129 | + return err; | |
| 130 | + | |
| 131 | + return 0; | |
| 132 | +} | |
| 133 | + | |
| 134 | +static int __init liveupdate_early_init(void) | |
| 135 | +{ | |
| 136 | + int err; | |
| 137 | + | |
| 138 | + err = luo_early_startup(); | |
| 139 | + if (err) { | |
| 140 | + luo_global.enabled = false; | |
| 141 | + luo_restore_fail("The incoming tree failed to initialize properly [%pe], disabling live update\n", | |
| 142 | + ERR_PTR(err)); | |
| 143 | + } | |
| 144 | + | |
| 145 | + return err; | |
| 146 | +} | |
| 147 | +early_initcall(liveupdate_early_init); | |
| 148 | + | |
| 149 | +/* Called during boot to create outgoing LUO fdt tree */ | |
| 150 | +static int __init luo_fdt_setup(void) | |
| 151 | +{ | |
| 152 | + const u64 ln = luo_global.liveupdate_num + 1; | |
| 153 | + void *fdt_out; | |
| 154 | + int err; | |
| 155 | + | |
| 156 | + fdt_out = kho_alloc_preserve(LUO_FDT_SIZE); | |
| 157 | + if (IS_ERR(fdt_out)) { | |
| 158 | + pr_err("failed to allocate/preserve FDT memory\n"); | |
| 159 | + return PTR_ERR(fdt_out); | |
| 160 | + } | |
| 161 | + | |
| 162 | + err = fdt_create(fdt_out, LUO_FDT_SIZE); | |
| 163 | + err |= fdt_finish_reservemap(fdt_out); | |
| 164 | + err |= fdt_begin_node(fdt_out, ""); | |
| 165 | + err |= fdt_property_string(fdt_out, "compatible", LUO_FDT_COMPATIBLE); | |
| 166 | + err |= fdt_property(fdt_out, LUO_FDT_LIVEUPDATE_NUM, &ln, sizeof(ln)); | |
| 167 | + err |= luo_session_setup_outgoing(fdt_out); | |
| 168 | + err |= fdt_end_node(fdt_out); | |
| 169 | + err |= fdt_finish(fdt_out); | |
| 170 | + if (err) | |
| 171 | + goto exit_free; | |
| 172 | + | |
| 173 | + err = kho_add_subtree(LUO_FDT_KHO_ENTRY_NAME, fdt_out); | |
| 174 | + if (err) | |
| 175 | + goto exit_free; | |
| 176 | + luo_global.fdt_out = fdt_out; | |
| 177 | + | |
| 178 | + return 0; | |
| 179 | + | |
| 180 | +exit_free: | |
| 181 | + kho_unpreserve_free(fdt_out); | |
| 182 | + pr_err("failed to prepare LUO FDT: %d\n", err); | |
| 183 | + | |
| 184 | + return err; | |
| 185 | +} | |
| 186 | + | |
| 187 | +/* | |
| 188 | + * late initcall because it initializes the outgoing tree that is needed only | |
| 189 | + * once userspace starts using /dev/liveupdate. | |
| 190 | + */ | |
| 191 | +static int __init luo_late_startup(void) | |
| 192 | +{ | |
| 193 | + int err; | |
| 194 | + | |
| 195 | + if (!liveupdate_enabled()) | |
| 196 | + return 0; | |
| 197 | + | |
| 198 | + err = luo_fdt_setup(); | |
| 199 | + if (err) | |
| 200 | + luo_global.enabled = false; | |
| 201 | + | |
| 202 | + return err; | |
| 203 | +} | |
| 204 | +late_initcall(luo_late_startup); | |
| 205 | + | |
| 206 | +/* Public Functions */ | |
| 207 | + | |
| 208 | +/** | |
| 209 | + * liveupdate_reboot() - Kernel reboot notifier for live update final | |
| 210 | + * serialization. | |
| 211 | + * | |
| 212 | + * This function is invoked directly from the reboot() syscall pathway | |
| 213 | + * if kexec is in progress. | |
| 214 | + * | |
| 215 | + * If any callback fails, this function aborts KHO, undoes the freeze() | |
| 216 | + * callbacks, and returns an error. | |
| 217 | + */ | |
| 218 | +int liveupdate_reboot(void) | |
| 219 | +{ | |
| 220 | + int err; | |
| 221 | + | |
| 222 | + if (!liveupdate_enabled()) | |
| 223 | + return 0; | |
| 224 | + | |
| 225 | + err = luo_session_serialize(); | |
| 226 | + if (err) | |
| 227 | + return err; | |
| 228 | + | |
| 229 | + err = kho_finalize(); | |
| 230 | + if (err) { | |
| 231 | + pr_err("kho_finalize failed %d\n", err); | |
| 232 | + /* | |
| 233 | + * kho_finalize() may return libfdt errors, to aboid passing to | |
| 234 | + * userspace unknown errors, change this to EAGAIN. | |
| 235 | + */ | |
| 236 | + err = -EAGAIN; | |
| 237 | + } | |
| 238 | + | |
| 239 | + return err; | |
| 240 | +} | |
| 241 | + | |
| 242 | +/** | |
| 243 | + * liveupdate_enabled - Check if the live update feature is enabled. | |
| 244 | + * | |
| 245 | + * This function returns the state of the live update feature flag, which | |
| 246 | + * can be controlled via the ``liveupdate`` kernel command-line parameter. | |
| 247 | + * | |
| 248 | + * @return true if live update is enabled, false otherwise. | |
| 249 | + */ | |
| 250 | +bool liveupdate_enabled(void) | |
| 251 | +{ | |
| 252 | + return luo_global.enabled; | |
| 253 | +} | |
| 254 | + | |
| 255 | +/** | |
| 256 | + * DOC: LUO ioctl Interface | |
| 257 | + * | |
| 258 | + * The IOCTL user-space control interface for the LUO subsystem. | |
| 259 | + * It registers a character device, typically found at ``/dev/liveupdate``, | |
| 260 | + * which allows a userspace agent to manage the LUO state machine and its | |
| 261 | + * associated resources, such as preservable file descriptors. | |
| 262 | + * | |
| 263 | + * To ensure that the state machine is controlled by a single entity, access | |
| 264 | + * to this device is exclusive: only one process is permitted to have | |
| 265 | + * ``/dev/liveupdate`` open at any given time. Subsequent open attempts will | |
| 266 | + * fail with -EBUSY until the first process closes its file descriptor. | |
| 267 | + * This singleton model simplifies state management by preventing conflicting | |
| 268 | + * commands from multiple userspace agents. | |
| 269 | + */ | |
| 270 | + | |
| 271 | +struct luo_device_state { | |
| 272 | + struct miscdevice miscdev; | |
| 273 | + atomic_t in_use; | |
| 274 | +}; | |
| 275 | + | |
| 276 | +static int luo_ioctl_create_session(struct luo_ucmd *ucmd) | |
| 277 | +{ | |
| 278 | + struct liveupdate_ioctl_create_session *argp = ucmd->cmd; | |
| 279 | + struct file *file; | |
| 280 | + int err; | |
| 281 | + | |
| 282 | + argp->fd = get_unused_fd_flags(O_CLOEXEC); | |
| 283 | + if (argp->fd < 0) | |
| 284 | + return argp->fd; | |
| 285 | + | |
| 286 | + err = luo_session_create(argp->name, &file); | |
| 287 | + if (err) | |
| 288 | + goto err_put_fd; | |
| 289 | + | |
| 290 | + err = luo_ucmd_respond(ucmd, sizeof(*argp)); | |
| 291 | + if (err) | |
| 292 | + goto err_put_file; | |
| 293 | + | |
| 294 | + fd_install(argp->fd, file); | |
| 295 | + | |
| 296 | + return 0; | |
| 297 | + | |
| 298 | +err_put_file: | |
| 299 | + fput(file); | |
| 300 | +err_put_fd: | |
| 301 | + put_unused_fd(argp->fd); | |
| 302 | + | |
| 303 | + return err; | |
| 304 | +} | |
| 305 | + | |
| 306 | +static int luo_ioctl_retrieve_session(struct luo_ucmd *ucmd) | |
| 307 | +{ | |
| 308 | + struct liveupdate_ioctl_retrieve_session *argp = ucmd->cmd; | |
| 309 | + struct file *file; | |
| 310 | + int err; | |
| 311 | + | |
| 312 | + argp->fd = get_unused_fd_flags(O_CLOEXEC); | |
| 313 | + if (argp->fd < 0) | |
| 314 | + return argp->fd; | |
| 315 | + | |
| 316 | + err = luo_session_retrieve(argp->name, &file); | |
| 317 | + if (err < 0) | |
| 318 | + goto err_put_fd; | |
| 319 | + | |
| 320 | + err = luo_ucmd_respond(ucmd, sizeof(*argp)); | |
| 321 | + if (err) | |
| 322 | + goto err_put_file; | |
| 323 | + | |
| 324 | + fd_install(argp->fd, file); | |
| 325 | + | |
| 326 | + return 0; | |
| 327 | + | |
| 328 | +err_put_file: | |
| 329 | + fput(file); | |
| 330 | +err_put_fd: | |
| 331 | + put_unused_fd(argp->fd); | |
| 332 | + | |
| 333 | + return err; | |
| 334 | +} | |
| 335 | + | |
| 336 | +static int luo_open(struct inode *inodep, struct file *filep) | |
| 337 | +{ | |
| 338 | + struct luo_device_state *ldev = container_of(filep->private_data, | |
| 339 | + struct luo_device_state, | |
| 340 | + miscdev); | |
| 341 | + | |
| 342 | + if (atomic_cmpxchg(&ldev->in_use, 0, 1)) | |
| 343 | + return -EBUSY; | |
| 344 | + | |
| 345 | + /* Always return -EIO to user if deserialization fail */ | |
| 346 | + if (luo_session_deserialize()) { | |
| 347 | + atomic_set(&ldev->in_use, 0); | |
| 348 | + return -EIO; | |
| 349 | + } | |
| 350 | + | |
| 351 | + return 0; | |
| 352 | +} | |
| 353 | + | |
| 354 | +static int luo_release(struct inode *inodep, struct file *filep) | |
| 355 | +{ | |
| 356 | + struct luo_device_state *ldev = container_of(filep->private_data, | |
| 357 | + struct luo_device_state, | |
| 358 | + miscdev); | |
| 359 | + atomic_set(&ldev->in_use, 0); | |
| 360 | + | |
| 361 | + return 0; | |
| 362 | +} | |
| 363 | + | |
| 364 | +union ucmd_buffer { | |
| 365 | + struct liveupdate_ioctl_create_session create; | |
| 366 | + struct liveupdate_ioctl_retrieve_session retrieve; | |
| 367 | +}; | |
| 368 | + | |
| 369 | +struct luo_ioctl_op { | |
| 370 | + unsigned int size; | |
| 371 | + unsigned int min_size; | |
| 372 | + unsigned int ioctl_num; | |
| 373 | + int (*execute)(struct luo_ucmd *ucmd); | |
| 374 | +}; | |
| 375 | + | |
| 376 | +#define IOCTL_OP(_ioctl, _fn, _struct, _last) \ | |
| 377 | + [_IOC_NR(_ioctl) - LIVEUPDATE_CMD_BASE] = { \ | |
| 378 | + .size = sizeof(_struct) + \ | |
| 379 | + BUILD_BUG_ON_ZERO(sizeof(union ucmd_buffer) < \ | |
| 380 | + sizeof(_struct)), \ | |
| 381 | + .min_size = offsetofend(_struct, _last), \ | |
| 382 | + .ioctl_num = _ioctl, \ | |
| 383 | + .execute = _fn, \ | |
| 384 | + } | |
| 385 | + | |
| 386 | +static const struct luo_ioctl_op luo_ioctl_ops[] = { | |
| 387 | + IOCTL_OP(LIVEUPDATE_IOCTL_CREATE_SESSION, luo_ioctl_create_session, | |
| 388 | + struct liveupdate_ioctl_create_session, name), | |
| 389 | + IOCTL_OP(LIVEUPDATE_IOCTL_RETRIEVE_SESSION, luo_ioctl_retrieve_session, | |
| 390 | + struct liveupdate_ioctl_retrieve_session, name), | |
| 391 | +}; | |
| 392 | + | |
| 393 | +static long luo_ioctl(struct file *filep, unsigned int cmd, unsigned long arg) | |
| 394 | +{ | |
| 395 | + const struct luo_ioctl_op *op; | |
| 396 | + struct luo_ucmd ucmd = {}; | |
| 397 | + union ucmd_buffer buf; | |
| 398 | + unsigned int nr; | |
| 399 | + int err; | |
| 400 | + | |
| 401 | + nr = _IOC_NR(cmd); | |
| 402 | + if (nr < LIVEUPDATE_CMD_BASE || | |
| 403 | + (nr - LIVEUPDATE_CMD_BASE) >= ARRAY_SIZE(luo_ioctl_ops)) { | |
| 404 | + return -EINVAL; | |
| 405 | + } | |
| 406 | + | |
| 407 | + ucmd.ubuffer = (void __user *)arg; | |
| 408 | + err = get_user(ucmd.user_size, (u32 __user *)ucmd.ubuffer); | |
| 409 | + if (err) | |
| 410 | + return err; | |
| 411 | + | |
| 412 | + op = &luo_ioctl_ops[nr - LIVEUPDATE_CMD_BASE]; | |
| 413 | + if (op->ioctl_num != cmd) | |
| 414 | + return -ENOIOCTLCMD; | |
| 415 | + if (ucmd.user_size < op->min_size) | |
| 416 | + return -EINVAL; | |
| 417 | + | |
| 418 | + ucmd.cmd = &buf; | |
| 419 | + err = copy_struct_from_user(ucmd.cmd, op->size, ucmd.ubuffer, | |
| 420 | + ucmd.user_size); | |
| 421 | + if (err) | |
| 422 | + return err; | |
| 423 | + | |
| 424 | + return op->execute(&ucmd); | |
| 425 | +} | |
| 426 | + | |
| 427 | +static const struct file_operations luo_fops = { | |
| 428 | + .owner = THIS_MODULE, | |
| 429 | + .open = luo_open, | |
| 430 | + .release = luo_release, | |
| 431 | + .unlocked_ioctl = luo_ioctl, | |
| 432 | +}; | |
| 433 | + | |
| 434 | +static struct luo_device_state luo_dev = { | |
| 435 | + .miscdev = { | |
| 436 | + .minor = MISC_DYNAMIC_MINOR, | |
| 437 | + .name = "liveupdate", | |
| 438 | + .fops = &luo_fops, | |
| 439 | + }, | |
| 440 | + .in_use = ATOMIC_INIT(0), | |
| 441 | +}; | |
| 442 | + | |
| 443 | +static int __init liveupdate_ioctl_init(void) | |
| 444 | +{ | |
| 445 | + if (!liveupdate_enabled()) | |
| 446 | + return 0; | |
| 447 | + | |
| 448 | + return misc_register(&luo_dev.miscdev); | |
| 449 | +} | |
| 450 | +late_initcall(liveupdate_ioctl_init); | |
| @@ -0,0 +1,889 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0 | |
| 2 | + | |
| 3 | +/* | |
| 4 | + * Copyright (c) 2025, Google LLC. | |
| 5 | + * Pasha Tatashin <pasha.tatashin@soleen.com> | |
| 6 | + */ | |
| 7 | + | |
| 8 | +/** | |
| 9 | + * DOC: LUO File Descriptors | |
| 10 | + * | |
| 11 | + * LUO provides the infrastructure to preserve specific, stateful file | |
| 12 | + * descriptors across a kexec-based live update. The primary goal is to allow | |
| 13 | + * workloads, such as virtual machines using vfio, memfd, or iommufd, to | |
| 14 | + * retain access to their essential resources without interruption. | |
| 15 | + * | |
| 16 | + * The framework is built around a callback-based handler model and a well- | |
| 17 | + * defined lifecycle for each preserved file. | |
| 18 | + * | |
| 19 | + * Handler Registration: | |
| 20 | + * Kernel modules responsible for a specific file type (e.g., memfd, vfio) | |
| 21 | + * register a &struct liveupdate_file_handler. This handler provides a set of | |
| 22 | + * callbacks that LUO invokes at different stages of the update process, most | |
| 23 | + * notably: | |
| 24 | + * | |
| 25 | + * - can_preserve(): A lightweight check to determine if the handler is | |
| 26 | + * compatible with a given 'struct file'. | |
| 27 | + * - preserve(): The heavyweight operation that saves the file's state and | |
| 28 | + * returns an opaque u64 handle. This is typically performed while the | |
| 29 | + * workload is still active to minimize the downtime during the | |
| 30 | + * actual reboot transition. | |
| 31 | + * - unpreserve(): Cleans up any resources allocated by .preserve(), called | |
| 32 | + * if the preservation process is aborted before the reboot (i.e. session is | |
| 33 | + * closed). | |
| 34 | + * - freeze(): A final pre-reboot opportunity to prepare the state for kexec. | |
| 35 | + * We are already in reboot syscall, and therefore userspace cannot mutate | |
| 36 | + * the file anymore. | |
| 37 | + * - unfreeze(): Undoes the actions of .freeze(), called if the live update | |
| 38 | + * is aborted after the freeze phase. | |
| 39 | + * - retrieve(): Reconstructs the file in the new kernel from the preserved | |
| 40 | + * handle. | |
| 41 | + * - finish(): Performs final check and cleanup in the new kernel. After | |
| 42 | + * succesul finish call, LUO gives up ownership to this file. | |
| 43 | + * | |
| 44 | + * File Preservation Lifecycle happy path: | |
| 45 | + * | |
| 46 | + * 1. Preserve (Normal Operation): A userspace agent preserves files one by one | |
| 47 | + * via an ioctl. For each file, luo_preserve_file() finds a compatible | |
| 48 | + * handler, calls its .preserve() operation, and creates an internal &struct | |
| 49 | + * luo_file to track the live state. | |
| 50 | + * | |
| 51 | + * 2. Freeze (Pre-Reboot): Just before the kexec, luo_file_freeze() is called. | |
| 52 | + * It iterates through all preserved files, calls their respective .freeze() | |
| 53 | + * operation, and serializes their final metadata (compatible string, token, | |
| 54 | + * and data handle) into a contiguous memory block for KHO. | |
| 55 | + * | |
| 56 | + * 3. Deserialize: After kexec, luo_file_deserialize() runs when session gets | |
| 57 | + * deserialized (which is when /dev/liveupdate is first opened). It reads the | |
| 58 | + * serialized data from the KHO memory region and reconstructs the in-memory | |
| 59 | + * list of &struct luo_file instances for the new kernel, linking them to | |
| 60 | + * their corresponding handlers. | |
| 61 | + * | |
| 62 | + * 4. Retrieve (New Kernel - Userspace Ready): The userspace agent can now | |
| 63 | + * restore file descriptors by providing a token. luo_retrieve_file() | |
| 64 | + * searches for the matching token, calls the handler's .retrieve() op to | |
| 65 | + * re-create the 'struct file', and returns a new FD. Files can be | |
| 66 | + * retrieved in ANY order. | |
| 67 | + * | |
| 68 | + * 5. Finish (New Kernel - Cleanup): Once a session retrival is complete, | |
| 69 | + * luo_file_finish() is called. It iterates through all files, invokes their | |
| 70 | + * .finish() operations for final cleanup, and releases all associated kernel | |
| 71 | + * resources. | |
| 72 | + * | |
| 73 | + * File Preservation Lifecycle unhappy paths: | |
| 74 | + * | |
| 75 | + * 1. Abort Before Reboot: If the userspace agent aborts the live update | |
| 76 | + * process before calling reboot (e.g., by closing the session file | |
| 77 | + * descriptor), the session's release handler calls | |
| 78 | + * luo_file_unpreserve_files(). This invokes the .unpreserve() callback on | |
| 79 | + * all preserved files, ensuring all allocated resources are cleaned up and | |
| 80 | + * returning the system to a clean state. | |
| 81 | + * | |
| 82 | + * 2. Freeze Failure: During the reboot() syscall, if any handler's .freeze() | |
| 83 | + * op fails, the .unfreeze() op is invoked on all previously *successful* | |
| 84 | + * freezes to roll back their state. The reboot() syscall then returns an | |
| 85 | + * error to userspace, canceling the live update. | |
| 86 | + * | |
| 87 | + * 3. Finish Failure: In the new kernel, if a handler's .finish() op fails, | |
| 88 | + * the luo_file_finish() operation is aborted. LUO retains ownership of | |
| 89 | + * all files within that session, including those that were not yet | |
| 90 | + * processed. The userspace agent can attempt to call the finish operation | |
| 91 | + * again later. If the issue cannot be resolved, these resources will be held | |
| 92 | + * by LUO until the next live update cycle, at which point they will be | |
| 93 | + * discarded. | |
| 94 | + */ | |
| 95 | + | |
| 96 | +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt | |
| 97 | + | |
| 98 | +#include <linux/cleanup.h> | |
| 99 | +#include <linux/compiler.h> | |
| 100 | +#include <linux/err.h> | |
| 101 | +#include <linux/errno.h> | |
| 102 | +#include <linux/file.h> | |
| 103 | +#include <linux/fs.h> | |
| 104 | +#include <linux/io.h> | |
| 105 | +#include <linux/kexec_handover.h> | |
| 106 | +#include <linux/kho/abi/luo.h> | |
| 107 | +#include <linux/liveupdate.h> | |
| 108 | +#include <linux/module.h> | |
| 109 | +#include <linux/sizes.h> | |
| 110 | +#include <linux/slab.h> | |
| 111 | +#include <linux/string.h> | |
| 112 | +#include "luo_internal.h" | |
| 113 | + | |
| 114 | +static LIST_HEAD(luo_file_handler_list); | |
| 115 | + | |
| 116 | +/* 2 4K pages, give space for 128 files per file_set */ | |
| 117 | +#define LUO_FILE_PGCNT 2ul | |
| 118 | +#define LUO_FILE_MAX \ | |
| 119 | + ((LUO_FILE_PGCNT << PAGE_SHIFT) / sizeof(struct luo_file_ser)) | |
| 120 | + | |
| 121 | +/** | |
| 122 | + * struct luo_file - Represents a single preserved file instance. | |
| 123 | + * @fh: Pointer to the &struct liveupdate_file_handler that manages | |
| 124 | + * this type of file. | |
| 125 | + * @file: Pointer to the kernel's &struct file that is being preserved. | |
| 126 | + * This is NULL in the new kernel until the file is successfully | |
| 127 | + * retrieved. | |
| 128 | + * @serialized_data: The opaque u64 handle to the serialized state of the file. | |
| 129 | + * This handle is passed back to the handler's .freeze(), | |
| 130 | + * .retrieve(), and .finish() callbacks, allowing it to track | |
| 131 | + * and update its serialized state across phases. | |
| 132 | + * @private_data: Pointer to the private data for the file used to hold runtime | |
| 133 | + * state that is not preserved. Set by the handler's .preserve() | |
| 134 | + * callback, and must be freed in the handler's .unpreserve() | |
| 135 | + * callback. | |
| 136 | + * @retrieved: A flag indicating whether a user/kernel in the new kernel has | |
| 137 | + * successfully called retrieve() on this file. This prevents | |
| 138 | + * multiple retrieval attempts. | |
| 139 | + * @mutex: A mutex that protects the fields of this specific instance | |
| 140 | + * (e.g., @retrieved, @file), ensuring that operations like | |
| 141 | + * retrieving or finishing a file are atomic. | |
| 142 | + * @list: The list_head linking this instance into its parent | |
| 143 | + * file_set's list of preserved files. | |
| 144 | + * @token: The user-provided unique token used to identify this file. | |
| 145 | + * | |
| 146 | + * This structure is the core in-kernel representation of a single file being | |
| 147 | + * managed through a live update. An instance is created by luo_preserve_file() | |
| 148 | + * to link a 'struct file' to its corresponding handler, a user-provided token, | |
| 149 | + * and the serialized state handle returned by the handler's .preserve() | |
| 150 | + * operation. | |
| 151 | + * | |
| 152 | + * These instances are tracked in a per-file_set list. The @serialized_data | |
| 153 | + * field, which holds a handle to the file's serialized state, may be updated | |
| 154 | + * during the .freeze() callback before being serialized for the next kernel. | |
| 155 | + * After reboot, these structures are recreated by luo_file_deserialize() and | |
| 156 | + * are finally cleaned up by luo_file_finish(). | |
| 157 | + */ | |
| 158 | +struct luo_file { | |
| 159 | + struct liveupdate_file_handler *fh; | |
| 160 | + struct file *file; | |
| 161 | + u64 serialized_data; | |
| 162 | + void *private_data; | |
| 163 | + bool retrieved; | |
| 164 | + struct mutex mutex; | |
| 165 | + struct list_head list; | |
| 166 | + u64 token; | |
| 167 | +}; | |
| 168 | + | |
| 169 | +static int luo_alloc_files_mem(struct luo_file_set *file_set) | |
| 170 | +{ | |
| 171 | + size_t size; | |
| 172 | + void *mem; | |
| 173 | + | |
| 174 | + if (file_set->files) | |
| 175 | + return 0; | |
| 176 | + | |
| 177 | + WARN_ON_ONCE(file_set->count); | |
| 178 | + | |
| 179 | + size = LUO_FILE_PGCNT << PAGE_SHIFT; | |
| 180 | + mem = kho_alloc_preserve(size); | |
| 181 | + if (IS_ERR(mem)) | |
| 182 | + return PTR_ERR(mem); | |
| 183 | + | |
| 184 | + file_set->files = mem; | |
| 185 | + | |
| 186 | + return 0; | |
| 187 | +} | |
| 188 | + | |
| 189 | +static void luo_free_files_mem(struct luo_file_set *file_set) | |
| 190 | +{ | |
| 191 | + /* If file_set has files, no need to free preservation memory */ | |
| 192 | + if (file_set->count) | |
| 193 | + return; | |
| 194 | + | |
| 195 | + if (!file_set->files) | |
| 196 | + return; | |
| 197 | + | |
| 198 | + kho_unpreserve_free(file_set->files); | |
| 199 | + file_set->files = NULL; | |
| 200 | +} | |
| 201 | + | |
| 202 | +static bool luo_token_is_used(struct luo_file_set *file_set, u64 token) | |
| 203 | +{ | |
| 204 | + struct luo_file *iter; | |
| 205 | + | |
| 206 | + list_for_each_entry(iter, &file_set->files_list, list) { | |
| 207 | + if (iter->token == token) | |
| 208 | + return true; | |
| 209 | + } | |
| 210 | + | |
| 211 | + return false; | |
| 212 | +} | |
| 213 | + | |
| 214 | +/** | |
| 215 | + * luo_preserve_file - Initiate the preservation of a file descriptor. | |
| 216 | + * @file_set: The file_set to which the preserved file will be added. | |
| 217 | + * @token: A unique, user-provided identifier for the file. | |
| 218 | + * @fd: The file descriptor to be preserved. | |
| 219 | + * | |
| 220 | + * This function orchestrates the first phase of preserving a file. Upon entry, | |
| 221 | + * it takes a reference to the 'struct file' via fget(), effectively making LUO | |
| 222 | + * a co-owner of the file. This reference is held until the file is either | |
| 223 | + * unpreserved or successfully finished in the next kernel, preventing the file | |
| 224 | + * from being prematurely destroyed. | |
| 225 | + * | |
| 226 | + * This function orchestrates the first phase of preserving a file. It performs | |
| 227 | + * the following steps: | |
| 228 | + * | |
| 229 | + * 1. Validates that the @token is not already in use within the file_set. | |
| 230 | + * 2. Ensures the file_set's memory for files serialization is allocated | |
| 231 | + * (allocates if needed). | |
| 232 | + * 3. Iterates through registered handlers, calling can_preserve() to find one | |
| 233 | + * compatible with the given @fd. | |
| 234 | + * 4. Calls the handler's .preserve() operation, which saves the file's state | |
| 235 | + * and returns an opaque private data handle. | |
| 236 | + * 5. Adds the new instance to the file_set's internal list. | |
| 237 | + * | |
| 238 | + * On success, LUO takes a reference to the 'struct file' and considers it | |
| 239 | + * under its management until it is unpreserved or finished. | |
| 240 | + * | |
| 241 | + * In case of any failure, all intermediate allocations (file reference, memory | |
| 242 | + * for the 'luo_file' struct, etc.) are cleaned up before returning an error. | |
| 243 | + * | |
| 244 | + * Context: Can be called from an ioctl handler during normal system operation. | |
| 245 | + * Return: 0 on success. Returns a negative errno on failure: | |
| 246 | + * -EEXIST if the token is already used. | |
| 247 | + * -EBADF if the file descriptor is invalid. | |
| 248 | + * -ENOSPC if the file_set is full. | |
| 249 | + * -ENOENT if no compatible handler is found. | |
| 250 | + * -ENOMEM on memory allocation failure. | |
| 251 | + * Other erros might be returned by .preserve(). | |
| 252 | + */ | |
| 253 | +int luo_preserve_file(struct luo_file_set *file_set, u64 token, int fd) | |
| 254 | +{ | |
| 255 | + struct liveupdate_file_op_args args = {0}; | |
| 256 | + struct liveupdate_file_handler *fh; | |
| 257 | + struct luo_file *luo_file; | |
| 258 | + struct file *file; | |
| 259 | + int err; | |
| 260 | + | |
| 261 | + if (luo_token_is_used(file_set, token)) | |
| 262 | + return -EEXIST; | |
| 263 | + | |
| 264 | + if (file_set->count == LUO_FILE_MAX) | |
| 265 | + return -ENOSPC; | |
| 266 | + | |
| 267 | + file = fget(fd); | |
| 268 | + if (!file) | |
| 269 | + return -EBADF; | |
| 270 | + | |
| 271 | + err = luo_alloc_files_mem(file_set); | |
| 272 | + if (err) | |
| 273 | + goto err_fput; | |
| 274 | + | |
| 275 | + err = -ENOENT; | |
| 276 | + luo_list_for_each_private(fh, &luo_file_handler_list, list) { | |
| 277 | + if (fh->ops->can_preserve(fh, file)) { | |
| 278 | + err = 0; | |
| 279 | + break; | |
| 280 | + } | |
| 281 | + } | |
| 282 | + | |
| 283 | + /* err is still -ENOENT if no handler was found */ | |
| 284 | + if (err) | |
| 285 | + goto err_free_files_mem; | |
| 286 | + | |
| 287 | + luo_file = kzalloc(sizeof(*luo_file), GFP_KERNEL); | |
| 288 | + if (!luo_file) { | |
| 289 | + err = -ENOMEM; | |
| 290 | + goto err_free_files_mem; | |
| 291 | + } | |
| 292 | + | |
| 293 | + luo_file->file = file; | |
| 294 | + luo_file->fh = fh; | |
| 295 | + luo_file->token = token; | |
| 296 | + luo_file->retrieved = false; | |
| 297 | + mutex_init(&luo_file->mutex); | |
| 298 | + | |
| 299 | + args.handler = fh; | |
| 300 | + args.file = file; | |
| 301 | + err = fh->ops->preserve(&args); | |
| 302 | + if (err) | |
| 303 | + goto err_kfree; | |
| 304 | + | |
| 305 | + luo_file->serialized_data = args.serialized_data; | |
| 306 | + luo_file->private_data = args.private_data; | |
| 307 | + list_add_tail(&luo_file->list, &file_set->files_list); | |
| 308 | + file_set->count++; | |
| 309 | + | |
| 310 | + return 0; | |
| 311 | + | |
| 312 | +err_kfree: | |
| 313 | + kfree(luo_file); | |
| 314 | +err_free_files_mem: | |
| 315 | + luo_free_files_mem(file_set); | |
| 316 | +err_fput: | |
| 317 | + fput(file); | |
| 318 | + | |
| 319 | + return err; | |
| 320 | +} | |
| 321 | + | |
| 322 | +/** | |
| 323 | + * luo_file_unpreserve_files - Unpreserves all files from a file_set. | |
| 324 | + * @file_set: The files to be cleaned up. | |
| 325 | + * | |
| 326 | + * This function serves as the primary cleanup path for a file_set. It is | |
| 327 | + * invoked when the userspace agent closes the file_set's file descriptor. | |
| 328 | + * | |
| 329 | + * For each file, it performs the following cleanup actions: | |
| 330 | + * 1. Calls the handler's .unpreserve() callback to allow the handler to | |
| 331 | + * release any resources it allocated. | |
| 332 | + * 2. Removes the file from the file_set's internal tracking list. | |
| 333 | + * 3. Releases the reference to the 'struct file' that was taken by | |
| 334 | + * luo_preserve_file() via fput(), returning ownership. | |
| 335 | + * 4. Frees the memory associated with the internal 'struct luo_file'. | |
| 336 | + * | |
| 337 | + * After all individual files are unpreserved, it frees the contiguous memory | |
| 338 | + * block that was allocated to hold their serialization data. | |
| 339 | + */ | |
| 340 | +void luo_file_unpreserve_files(struct luo_file_set *file_set) | |
| 341 | +{ | |
| 342 | + struct luo_file *luo_file; | |
| 343 | + | |
| 344 | + while (!list_empty(&file_set->files_list)) { | |
| 345 | + struct liveupdate_file_op_args args = {0}; | |
| 346 | + | |
| 347 | + luo_file = list_last_entry(&file_set->files_list, | |
| 348 | + struct luo_file, list); | |
| 349 | + | |
| 350 | + args.handler = luo_file->fh; | |
| 351 | + args.file = luo_file->file; | |
| 352 | + args.serialized_data = luo_file->serialized_data; | |
| 353 | + args.private_data = luo_file->private_data; | |
| 354 | + luo_file->fh->ops->unpreserve(&args); | |
| 355 | + | |
| 356 | + list_del(&luo_file->list); | |
| 357 | + file_set->count--; | |
| 358 | + | |
| 359 | + fput(luo_file->file); | |
| 360 | + mutex_destroy(&luo_file->mutex); | |
| 361 | + kfree(luo_file); | |
| 362 | + } | |
| 363 | + | |
| 364 | + luo_free_files_mem(file_set); | |
| 365 | +} | |
| 366 | + | |
| 367 | +static int luo_file_freeze_one(struct luo_file_set *file_set, | |
| 368 | + struct luo_file *luo_file) | |
| 369 | +{ | |
| 370 | + int err = 0; | |
| 371 | + | |
| 372 | + guard(mutex)(&luo_file->mutex); | |
| 373 | + | |
| 374 | + if (luo_file->fh->ops->freeze) { | |
| 375 | + struct liveupdate_file_op_args args = {0}; | |
| 376 | + | |
| 377 | + args.handler = luo_file->fh; | |
| 378 | + args.file = luo_file->file; | |
| 379 | + args.serialized_data = luo_file->serialized_data; | |
| 380 | + args.private_data = luo_file->private_data; | |
| 381 | + | |
| 382 | + err = luo_file->fh->ops->freeze(&args); | |
| 383 | + if (!err) | |
| 384 | + luo_file->serialized_data = args.serialized_data; | |
| 385 | + } | |
| 386 | + | |
| 387 | + return err; | |
| 388 | +} | |
| 389 | + | |
| 390 | +static void luo_file_unfreeze_one(struct luo_file_set *file_set, | |
| 391 | + struct luo_file *luo_file) | |
| 392 | +{ | |
| 393 | + guard(mutex)(&luo_file->mutex); | |
| 394 | + | |
| 395 | + if (luo_file->fh->ops->unfreeze) { | |
| 396 | + struct liveupdate_file_op_args args = {0}; | |
| 397 | + | |
| 398 | + args.handler = luo_file->fh; | |
| 399 | + args.file = luo_file->file; | |
| 400 | + args.serialized_data = luo_file->serialized_data; | |
| 401 | + args.private_data = luo_file->private_data; | |
| 402 | + | |
| 403 | + luo_file->fh->ops->unfreeze(&args); | |
| 404 | + } | |
| 405 | + | |
| 406 | + luo_file->serialized_data = 0; | |
| 407 | +} | |
| 408 | + | |
| 409 | +static void __luo_file_unfreeze(struct luo_file_set *file_set, | |
| 410 | + struct luo_file *failed_entry) | |
| 411 | +{ | |
| 412 | + struct list_head *files_list = &file_set->files_list; | |
| 413 | + struct luo_file *luo_file; | |
| 414 | + | |
| 415 | + list_for_each_entry(luo_file, files_list, list) { | |
| 416 | + if (luo_file == failed_entry) | |
| 417 | + break; | |
| 418 | + | |
| 419 | + luo_file_unfreeze_one(file_set, luo_file); | |
| 420 | + } | |
| 421 | + | |
| 422 | + memset(file_set->files, 0, LUO_FILE_PGCNT << PAGE_SHIFT); | |
| 423 | +} | |
| 424 | + | |
| 425 | +/** | |
| 426 | + * luo_file_freeze - Freezes all preserved files and serializes their metadata. | |
| 427 | + * @file_set: The file_set whose files are to be frozen. | |
| 428 | + * @file_set_ser: Where to put the serialized file_set. | |
| 429 | + * | |
| 430 | + * This function is called from the reboot() syscall path, just before the | |
| 431 | + * kernel transitions to the new image via kexec. Its purpose is to perform the | |
| 432 | + * final preparation and serialization of all preserved files in the file_set. | |
| 433 | + * | |
| 434 | + * It iterates through each preserved file in FIFO order (the order of | |
| 435 | + * preservation) and performs two main actions: | |
| 436 | + * | |
| 437 | + * 1. Freezes the File: It calls the handler's .freeze() callback for each | |
| 438 | + * file. This gives the handler a final opportunity to quiesce the device or | |
| 439 | + * prepare its state for the upcoming reboot. The handler may update its | |
| 440 | + * private data handle during this step. | |
| 441 | + * | |
| 442 | + * 2. Serializes Metadata: After a successful freeze, it copies the final file | |
| 443 | + * metadata—the handler's compatible string, the user token, and the final | |
| 444 | + * private data handle—into the pre-allocated contiguous memory buffer | |
| 445 | + * (file_set->files) that will be handed over to the next kernel via KHO. | |
| 446 | + * | |
| 447 | + * Error Handling (Rollback): | |
| 448 | + * This function is atomic. If any handler's .freeze() operation fails, the | |
| 449 | + * entire live update is aborted. The __luo_file_unfreeze() helper is | |
| 450 | + * immediately called to invoke the .unfreeze() op on all files that were | |
| 451 | + * successfully frozen before the point of failure, rolling them back to a | |
| 452 | + * running state. The function then returns an error, causing the reboot() | |
| 453 | + * syscall to fail. | |
| 454 | + * | |
| 455 | + * Context: Called only from the liveupdate_reboot() path. | |
| 456 | + * Return: 0 on success, or a negative errno on failure. | |
| 457 | + */ | |
| 458 | +int luo_file_freeze(struct luo_file_set *file_set, | |
| 459 | + struct luo_file_set_ser *file_set_ser) | |
| 460 | +{ | |
| 461 | + struct luo_file_ser *file_ser = file_set->files; | |
| 462 | + struct luo_file *luo_file; | |
| 463 | + int err; | |
| 464 | + int i; | |
| 465 | + | |
| 466 | + if (!file_set->count) | |
| 467 | + return 0; | |
| 468 | + | |
| 469 | + if (WARN_ON(!file_ser)) | |
| 470 | + return -EINVAL; | |
| 471 | + | |
| 472 | + i = 0; | |
| 473 | + list_for_each_entry(luo_file, &file_set->files_list, list) { | |
| 474 | + err = luo_file_freeze_one(file_set, luo_file); | |
| 475 | + if (err < 0) { | |
| 476 | + pr_warn("Freeze failed for token[%#0llx] handler[%s] err[%pe]\n", | |
| 477 | + luo_file->token, luo_file->fh->compatible, | |
| 478 | + ERR_PTR(err)); | |
| 479 | + goto err_unfreeze; | |
| 480 | + } | |
| 481 | + | |
| 482 | + strscpy(file_ser[i].compatible, luo_file->fh->compatible, | |
| 483 | + sizeof(file_ser[i].compatible)); | |
| 484 | + file_ser[i].data = luo_file->serialized_data; | |
| 485 | + file_ser[i].token = luo_file->token; | |
| 486 | + i++; | |
| 487 | + } | |
| 488 | + | |
| 489 | + file_set_ser->count = file_set->count; | |
| 490 | + if (file_set->files) | |
| 491 | + file_set_ser->files = virt_to_phys(file_set->files); | |
| 492 | + | |
| 493 | + return 0; | |
| 494 | + | |
| 495 | +err_unfreeze: | |
| 496 | + __luo_file_unfreeze(file_set, luo_file); | |
| 497 | + | |
| 498 | + return err; | |
| 499 | +} | |
| 500 | + | |
| 501 | +/** | |
| 502 | + * luo_file_unfreeze - Unfreezes all files in a file_set and clear serialization | |
| 503 | + * @file_set: The file_set whose files are to be unfrozen. | |
| 504 | + * @file_set_ser: Serialized file_set. | |
| 505 | + * | |
| 506 | + * This function rolls back the state of all files in a file_set after the | |
| 507 | + * freeze phase has begun but must be aborted. It is the counterpart to | |
| 508 | + * luo_file_freeze(). | |
| 509 | + * | |
| 510 | + * It invokes the __luo_file_unfreeze() helper with a NULL argument, which | |
| 511 | + * signals the helper to iterate through all files in the file_set and call | |
| 512 | + * their respective .unfreeze() handler callbacks. | |
| 513 | + * | |
| 514 | + * Context: This is called when the live update is aborted during | |
| 515 | + * the reboot() syscall, after luo_file_freeze() has been called. | |
| 516 | + */ | |
| 517 | +void luo_file_unfreeze(struct luo_file_set *file_set, | |
| 518 | + struct luo_file_set_ser *file_set_ser) | |
| 519 | +{ | |
| 520 | + if (!file_set->count) | |
| 521 | + return; | |
| 522 | + | |
| 523 | + __luo_file_unfreeze(file_set, NULL); | |
| 524 | + memset(file_set_ser, 0, sizeof(*file_set_ser)); | |
| 525 | +} | |
| 526 | + | |
| 527 | +/** | |
| 528 | + * luo_retrieve_file - Restores a preserved file from a file_set by its token. | |
| 529 | + * @file_set: The file_set from which to retrieve the file. | |
| 530 | + * @token: The unique token identifying the file to be restored. | |
| 531 | + * @filep: Output parameter; on success, this is populated with a pointer | |
| 532 | + * to the newly retrieved 'struct file'. | |
| 533 | + * | |
| 534 | + * This function is the primary mechanism for recreating a file in the new | |
| 535 | + * kernel after a live update. It searches the file_set's list of deserialized | |
| 536 | + * files for an entry matching the provided @token. | |
| 537 | + * | |
| 538 | + * The operation is idempotent: if a file has already been successfully | |
| 539 | + * retrieved, this function will simply return a pointer to the existing | |
| 540 | + * 'struct file' and report success without re-executing the retrieve | |
| 541 | + * operation. This is handled by checking the 'retrieved' flag under a lock. | |
| 542 | + * | |
| 543 | + * File retrieval can happen in any order; it is not bound by the order of | |
| 544 | + * preservation. | |
| 545 | + * | |
| 546 | + * Context: Can be called from an ioctl or other in-kernel code in the new | |
| 547 | + * kernel. | |
| 548 | + * Return: 0 on success. Returns a negative errno on failure: | |
| 549 | + * -ENOENT if no file with the matching token is found. | |
| 550 | + * Any error code returned by the handler's .retrieve() op. | |
| 551 | + */ | |
| 552 | +int luo_retrieve_file(struct luo_file_set *file_set, u64 token, | |
| 553 | + struct file **filep) | |
| 554 | +{ | |
| 555 | + struct liveupdate_file_op_args args = {0}; | |
| 556 | + struct luo_file *luo_file; | |
| 557 | + int err; | |
| 558 | + | |
| 559 | + if (list_empty(&file_set->files_list)) | |
| 560 | + return -ENOENT; | |
| 561 | + | |
| 562 | + list_for_each_entry(luo_file, &file_set->files_list, list) { | |
| 563 | + if (luo_file->token == token) | |
| 564 | + break; | |
| 565 | + } | |
| 566 | + | |
| 567 | + if (luo_file->token != token) | |
| 568 | + return -ENOENT; | |
| 569 | + | |
| 570 | + guard(mutex)(&luo_file->mutex); | |
| 571 | + if (luo_file->retrieved) { | |
| 572 | + /* | |
| 573 | + * Someone is asking for this file again, so get a reference | |
| 574 | + * for them. | |
| 575 | + */ | |
| 576 | + get_file(luo_file->file); | |
| 577 | + *filep = luo_file->file; | |
| 578 | + return 0; | |
| 579 | + } | |
| 580 | + | |
| 581 | + args.handler = luo_file->fh; | |
| 582 | + args.serialized_data = luo_file->serialized_data; | |
| 583 | + err = luo_file->fh->ops->retrieve(&args); | |
| 584 | + if (!err) { | |
| 585 | + luo_file->file = args.file; | |
| 586 | + | |
| 587 | + /* Get reference so we can keep this file in LUO until finish */ | |
| 588 | + get_file(luo_file->file); | |
| 589 | + *filep = luo_file->file; | |
| 590 | + luo_file->retrieved = true; | |
| 591 | + } | |
| 592 | + | |
| 593 | + return err; | |
| 594 | +} | |
| 595 | + | |
| 596 | +static int luo_file_can_finish_one(struct luo_file_set *file_set, | |
| 597 | + struct luo_file *luo_file) | |
| 598 | +{ | |
| 599 | + bool can_finish = true; | |
| 600 | + | |
| 601 | + guard(mutex)(&luo_file->mutex); | |
| 602 | + | |
| 603 | + if (luo_file->fh->ops->can_finish) { | |
| 604 | + struct liveupdate_file_op_args args = {0}; | |
| 605 | + | |
| 606 | + args.handler = luo_file->fh; | |
| 607 | + args.file = luo_file->file; | |
| 608 | + args.serialized_data = luo_file->serialized_data; | |
| 609 | + args.retrieved = luo_file->retrieved; | |
| 610 | + can_finish = luo_file->fh->ops->can_finish(&args); | |
| 611 | + } | |
| 612 | + | |
| 613 | + return can_finish ? 0 : -EBUSY; | |
| 614 | +} | |
| 615 | + | |
| 616 | +static void luo_file_finish_one(struct luo_file_set *file_set, | |
| 617 | + struct luo_file *luo_file) | |
| 618 | +{ | |
| 619 | + struct liveupdate_file_op_args args = {0}; | |
| 620 | + | |
| 621 | + guard(mutex)(&luo_file->mutex); | |
| 622 | + | |
| 623 | + args.handler = luo_file->fh; | |
| 624 | + args.file = luo_file->file; | |
| 625 | + args.serialized_data = luo_file->serialized_data; | |
| 626 | + args.retrieved = luo_file->retrieved; | |
| 627 | + | |
| 628 | + luo_file->fh->ops->finish(&args); | |
| 629 | +} | |
| 630 | + | |
| 631 | +/** | |
| 632 | + * luo_file_finish - Completes the lifecycle for all files in a file_set. | |
| 633 | + * @file_set: The file_set to be finalized. | |
| 634 | + * | |
| 635 | + * This function orchestrates the final teardown of a live update file_set in | |
| 636 | + * the new kernel. It should be called after all necessary files have been | |
| 637 | + * retrieved and the userspace agent is ready to release the preserved state. | |
| 638 | + * | |
| 639 | + * The function iterates through all tracked files. For each file, it performs | |
| 640 | + * the following sequence of cleanup actions: | |
| 641 | + * | |
| 642 | + * 1. If file is not yet retrieved, retrieves it, and calls can_finish() on | |
| 643 | + * every file in the file_set. If all can_finish return true, continue to | |
| 644 | + * finish. | |
| 645 | + * 2. Calls the handler's .finish() callback (via luo_file_finish_one) to | |
| 646 | + * allow for final resource cleanup within the handler. | |
| 647 | + * 3. Releases LUO's ownership reference on the 'struct file' via fput(). This | |
| 648 | + * is the counterpart to the get_file() call in luo_retrieve_file(). | |
| 649 | + * 4. Removes the 'struct luo_file' from the file_set's internal list. | |
| 650 | + * 5. Frees the memory for the 'struct luo_file' instance itself. | |
| 651 | + * | |
| 652 | + * After successfully finishing all individual files, it frees the | |
| 653 | + * contiguous memory block that was used to transfer the serialized metadata | |
| 654 | + * from the previous kernel. | |
| 655 | + * | |
| 656 | + * Error Handling (Atomic Failure): | |
| 657 | + * This operation is atomic. If any handler's .can_finish() op fails, the entire | |
| 658 | + * function aborts immediately and returns an error. | |
| 659 | + * | |
| 660 | + * Context: Can be called from an ioctl handler in the new kernel. | |
| 661 | + * Return: 0 on success, or a negative errno on failure. | |
| 662 | + */ | |
| 663 | +int luo_file_finish(struct luo_file_set *file_set) | |
| 664 | +{ | |
| 665 | + struct list_head *files_list = &file_set->files_list; | |
| 666 | + struct luo_file *luo_file; | |
| 667 | + int err; | |
| 668 | + | |
| 669 | + if (!file_set->count) | |
| 670 | + return 0; | |
| 671 | + | |
| 672 | + list_for_each_entry(luo_file, files_list, list) { | |
| 673 | + err = luo_file_can_finish_one(file_set, luo_file); | |
| 674 | + if (err) | |
| 675 | + return err; | |
| 676 | + } | |
| 677 | + | |
| 678 | + while (!list_empty(&file_set->files_list)) { | |
| 679 | + luo_file = list_last_entry(&file_set->files_list, | |
| 680 | + struct luo_file, list); | |
| 681 | + | |
| 682 | + luo_file_finish_one(file_set, luo_file); | |
| 683 | + | |
| 684 | + if (luo_file->file) | |
| 685 | + fput(luo_file->file); | |
| 686 | + list_del(&luo_file->list); | |
| 687 | + file_set->count--; | |
| 688 | + mutex_destroy(&luo_file->mutex); | |
| 689 | + kfree(luo_file); | |
| 690 | + } | |
| 691 | + | |
| 692 | + if (file_set->files) { | |
| 693 | + kho_restore_free(file_set->files); | |
| 694 | + file_set->files = NULL; | |
| 695 | + } | |
| 696 | + | |
| 697 | + return 0; | |
| 698 | +} | |
| 699 | + | |
| 700 | +/** | |
| 701 | + * luo_file_deserialize - Reconstructs the list of preserved files in the new kernel. | |
| 702 | + * @file_set: The incoming file_set to fill with deserialized data. | |
| 703 | + * @file_set_ser: Serialized KHO file_set data from the previous kernel. | |
| 704 | + * | |
| 705 | + * This function is called during the early boot process of the new kernel. It | |
| 706 | + * takes the raw, contiguous memory block of 'struct luo_file_ser' entries, | |
| 707 | + * provided by the previous kernel, and transforms it back into a live, | |
| 708 | + * in-memory linked list of 'struct luo_file' instances. | |
| 709 | + * | |
| 710 | + * For each serialized entry, it performs the following steps: | |
| 711 | + * 1. Reads the 'compatible' string. | |
| 712 | + * 2. Searches the global list of registered file handlers for one that | |
| 713 | + * matches the compatible string. | |
| 714 | + * 3. Allocates a new 'struct luo_file'. | |
| 715 | + * 4. Populates the new structure with the deserialized data (token, private | |
| 716 | + * data handle) and links it to the found handler. The 'file' pointer is | |
| 717 | + * initialized to NULL, as the file has not been retrieved yet. | |
| 718 | + * 5. Adds the new 'struct luo_file' to the file_set's files_list. | |
| 719 | + * | |
| 720 | + * This prepares the file_set for userspace, which can later call | |
| 721 | + * luo_retrieve_file() to restore the actual file descriptors. | |
| 722 | + * | |
| 723 | + * Context: Called from session deserialization. | |
| 724 | + */ | |
| 725 | +int luo_file_deserialize(struct luo_file_set *file_set, | |
| 726 | + struct luo_file_set_ser *file_set_ser) | |
| 727 | +{ | |
| 728 | + struct luo_file_ser *file_ser; | |
| 729 | + u64 i; | |
| 730 | + | |
| 731 | + if (!file_set_ser->files) { | |
| 732 | + WARN_ON(file_set_ser->count); | |
| 733 | + return 0; | |
| 734 | + } | |
| 735 | + | |
| 736 | + file_set->count = file_set_ser->count; | |
| 737 | + file_set->files = phys_to_virt(file_set_ser->files); | |
| 738 | + | |
| 739 | + /* | |
| 740 | + * Note on error handling: | |
| 741 | + * | |
| 742 | + * If deserialization fails (e.g., allocation failure or corrupt data), | |
| 743 | + * we intentionally skip cleanup of files that were already restored. | |
| 744 | + * | |
| 745 | + * A partial failure leaves the preserved state inconsistent. | |
| 746 | + * Implementing a safe "undo" to unwind complex dependencies (sessions, | |
| 747 | + * files, hardware state) is error-prone and provides little value, as | |
| 748 | + * the system is effectively in a broken state. | |
| 749 | + * | |
| 750 | + * We treat these resources as leaked. The expected recovery path is for | |
| 751 | + * userspace to detect the failure and trigger a reboot, which will | |
| 752 | + * reliably reset devices and reclaim memory. | |
| 753 | + */ | |
| 754 | + file_ser = file_set->files; | |
| 755 | + for (i = 0; i < file_set->count; i++) { | |
| 756 | + struct liveupdate_file_handler *fh; | |
| 757 | + bool handler_found = false; | |
| 758 | + struct luo_file *luo_file; | |
| 759 | + | |
| 760 | + luo_list_for_each_private(fh, &luo_file_handler_list, list) { | |
| 761 | + if (!strcmp(fh->compatible, file_ser[i].compatible)) { | |
| 762 | + handler_found = true; | |
| 763 | + break; | |
| 764 | + } | |
| 765 | + } | |
| 766 | + | |
| 767 | + if (!handler_found) { | |
| 768 | + pr_warn("No registered handler for compatible '%s'\n", | |
| 769 | + file_ser[i].compatible); | |
| 770 | + return -ENOENT; | |
| 771 | + } | |
| 772 | + | |
| 773 | + luo_file = kzalloc(sizeof(*luo_file), GFP_KERNEL); | |
| 774 | + if (!luo_file) | |
| 775 | + return -ENOMEM; | |
| 776 | + | |
| 777 | + luo_file->fh = fh; | |
| 778 | + luo_file->file = NULL; | |
| 779 | + luo_file->serialized_data = file_ser[i].data; | |
| 780 | + luo_file->token = file_ser[i].token; | |
| 781 | + luo_file->retrieved = false; | |
| 782 | + mutex_init(&luo_file->mutex); | |
| 783 | + list_add_tail(&luo_file->list, &file_set->files_list); | |
| 784 | + } | |
| 785 | + | |
| 786 | + return 0; | |
| 787 | +} | |
| 788 | + | |
| 789 | +void luo_file_set_init(struct luo_file_set *file_set) | |
| 790 | +{ | |
| 791 | + INIT_LIST_HEAD(&file_set->files_list); | |
| 792 | +} | |
| 793 | + | |
| 794 | +void luo_file_set_destroy(struct luo_file_set *file_set) | |
| 795 | +{ | |
| 796 | + WARN_ON(file_set->count); | |
| 797 | + WARN_ON(!list_empty(&file_set->files_list)); | |
| 798 | +} | |
| 799 | + | |
| 800 | +/** | |
| 801 | + * liveupdate_register_file_handler - Register a file handler with LUO. | |
| 802 | + * @fh: Pointer to a caller-allocated &struct liveupdate_file_handler. | |
| 803 | + * The caller must initialize this structure, including a unique | |
| 804 | + * 'compatible' string and a valid 'fh' callbacks. This function adds the | |
| 805 | + * handler to the global list of supported file handlers. | |
| 806 | + * | |
| 807 | + * Context: Typically called during module initialization for file types that | |
| 808 | + * support live update preservation. | |
| 809 | + * | |
| 810 | + * Return: 0 on success. Negative errno on failure. | |
| 811 | + */ | |
| 812 | +int liveupdate_register_file_handler(struct liveupdate_file_handler *fh) | |
| 813 | +{ | |
| 814 | + struct liveupdate_file_handler *fh_iter; | |
| 815 | + int err; | |
| 816 | + | |
| 817 | + if (!liveupdate_enabled()) | |
| 818 | + return -EOPNOTSUPP; | |
| 819 | + | |
| 820 | + /* Sanity check that all required callbacks are set */ | |
| 821 | + if (!fh->ops->preserve || !fh->ops->unpreserve || !fh->ops->retrieve || | |
| 822 | + !fh->ops->finish || !fh->ops->can_preserve) { | |
| 823 | + return -EINVAL; | |
| 824 | + } | |
| 825 | + | |
| 826 | + /* | |
| 827 | + * Ensure the system is quiescent (no active sessions). | |
| 828 | + * This prevents registering new handlers while sessions are active or | |
| 829 | + * while deserialization is in progress. | |
| 830 | + */ | |
| 831 | + if (!luo_session_quiesce()) | |
| 832 | + return -EBUSY; | |
| 833 | + | |
| 834 | + /* Check for duplicate compatible strings */ | |
| 835 | + luo_list_for_each_private(fh_iter, &luo_file_handler_list, list) { | |
| 836 | + if (!strcmp(fh_iter->compatible, fh->compatible)) { | |
| 837 | + pr_err("File handler registration failed: Compatible string '%s' already registered.\n", | |
| 838 | + fh->compatible); | |
| 839 | + err = -EEXIST; | |
| 840 | + goto err_resume; | |
| 841 | + } | |
| 842 | + } | |
| 843 | + | |
| 844 | + /* Pin the module implementing the handler */ | |
| 845 | + if (!try_module_get(fh->ops->owner)) { | |
| 846 | + err = -EAGAIN; | |
| 847 | + goto err_resume; | |
| 848 | + } | |
| 849 | + | |
| 850 | + INIT_LIST_HEAD(&ACCESS_PRIVATE(fh, list)); | |
| 851 | + list_add_tail(&ACCESS_PRIVATE(fh, list), &luo_file_handler_list); | |
| 852 | + luo_session_resume(); | |
| 853 | + | |
| 854 | + return 0; | |
| 855 | + | |
| 856 | +err_resume: | |
| 857 | + luo_session_resume(); | |
| 858 | + return err; | |
| 859 | +} | |
| 860 | + | |
| 861 | +/** | |
| 862 | + * liveupdate_unregister_file_handler - Unregister a liveupdate file handler | |
| 863 | + * @fh: The file handler to unregister | |
| 864 | + * | |
| 865 | + * Unregisters the file handler from the liveupdate core. This function | |
| 866 | + * reverses the operations of liveupdate_register_file_handler(). | |
| 867 | + * | |
| 868 | + * It ensures safe removal by checking that: | |
| 869 | + * No live update session is currently in progress. | |
| 870 | + * | |
| 871 | + * If the unregistration fails, the internal test state is reverted. | |
| 872 | + * | |
| 873 | + * Return: 0 Success. -EOPNOTSUPP when live update is not enabled. -EBUSY A live | |
| 874 | + * update is in progress, can't quiesce live update. | |
| 875 | + */ | |
| 876 | +int liveupdate_unregister_file_handler(struct liveupdate_file_handler *fh) | |
| 877 | +{ | |
| 878 | + if (!liveupdate_enabled()) | |
| 879 | + return -EOPNOTSUPP; | |
| 880 | + | |
| 881 | + if (!luo_session_quiesce()) | |
| 882 | + return -EBUSY; | |
| 883 | + | |
| 884 | + list_del(&ACCESS_PRIVATE(fh, list)); | |
| 885 | + module_put(fh->ops->owner); | |
| 886 | + luo_session_resume(); | |
| 887 | + | |
| 888 | + return 0; | |
| 889 | +} | |
| @@ -0,0 +1,110 @@ | ||
| 1 | +/* SPDX-License-Identifier: GPL-2.0 */ | |
| 2 | + | |
| 3 | +/* | |
| 4 | + * Copyright (c) 2025, Google LLC. | |
| 5 | + * Pasha Tatashin <pasha.tatashin@soleen.com> | |
| 6 | + */ | |
| 7 | + | |
| 8 | +#ifndef _LINUX_LUO_INTERNAL_H | |
| 9 | +#define _LINUX_LUO_INTERNAL_H | |
| 10 | + | |
| 11 | +#include <linux/liveupdate.h> | |
| 12 | +#include <linux/uaccess.h> | |
| 13 | + | |
| 14 | +struct luo_ucmd { | |
| 15 | + void __user *ubuffer; | |
| 16 | + u32 user_size; | |
| 17 | + void *cmd; | |
| 18 | +}; | |
| 19 | + | |
| 20 | +static inline int luo_ucmd_respond(struct luo_ucmd *ucmd, | |
| 21 | + size_t kernel_cmd_size) | |
| 22 | +{ | |
| 23 | + /* | |
| 24 | + * Copy the minimum of what the user provided and what we actually | |
| 25 | + * have. | |
| 26 | + */ | |
| 27 | + if (copy_to_user(ucmd->ubuffer, ucmd->cmd, | |
| 28 | + min_t(size_t, ucmd->user_size, kernel_cmd_size))) { | |
| 29 | + return -EFAULT; | |
| 30 | + } | |
| 31 | + return 0; | |
| 32 | +} | |
| 33 | + | |
| 34 | +/* | |
| 35 | + * Handles a deserialization failure: devices and memory is in unpredictable | |
| 36 | + * state. | |
| 37 | + * | |
| 38 | + * Continuing the boot process after a failure is dangerous because it could | |
| 39 | + * lead to leaks of private data. | |
| 40 | + */ | |
| 41 | +#define luo_restore_fail(__fmt, ...) panic(__fmt, ##__VA_ARGS__) | |
| 42 | + | |
| 43 | +/* Mimics list_for_each_entry() but for private list head entries */ | |
| 44 | +#define luo_list_for_each_private(pos, head, member) \ | |
| 45 | + for (struct list_head *__iter = (head)->next; \ | |
| 46 | + __iter != (head) && \ | |
| 47 | + ({ pos = container_of(__iter, typeof(*(pos)), member); 1; }); \ | |
| 48 | + __iter = __iter->next) | |
| 49 | + | |
| 50 | +/** | |
| 51 | + * struct luo_file_set - A set of files that belong to the same sessions. | |
| 52 | + * @files_list: An ordered list of files associated with this session, it is | |
| 53 | + * ordered by preservation time. | |
| 54 | + * @files: The physically contiguous memory block that holds the serialized | |
| 55 | + * state of files. | |
| 56 | + * @count: A counter tracking the number of files currently stored in the | |
| 57 | + * @files_list for this session. | |
| 58 | + */ | |
| 59 | +struct luo_file_set { | |
| 60 | + struct list_head files_list; | |
| 61 | + struct luo_file_ser *files; | |
| 62 | + long count; | |
| 63 | +}; | |
| 64 | + | |
| 65 | +/** | |
| 66 | + * struct luo_session - Represents an active or incoming Live Update session. | |
| 67 | + * @name: A unique name for this session, used for identification and | |
| 68 | + * retrieval. | |
| 69 | + * @ser: Pointer to the serialized data for this session. | |
| 70 | + * @list: A list_head member used to link this session into a global list | |
| 71 | + * of either outgoing (to be preserved) or incoming (restored from | |
| 72 | + * previous kernel) sessions. | |
| 73 | + * @retrieved: A boolean flag indicating whether this session has been | |
| 74 | + * retrieved by a consumer in the new kernel. | |
| 75 | + * @file_set: A set of files that belong to this session. | |
| 76 | + * @mutex: protects fields in the luo_session. | |
| 77 | + */ | |
| 78 | +struct luo_session { | |
| 79 | + char name[LIVEUPDATE_SESSION_NAME_LENGTH]; | |
| 80 | + struct luo_session_ser *ser; | |
| 81 | + struct list_head list; | |
| 82 | + bool retrieved; | |
| 83 | + struct luo_file_set file_set; | |
| 84 | + struct mutex mutex; | |
| 85 | +}; | |
| 86 | + | |
| 87 | +int luo_session_create(const char *name, struct file **filep); | |
| 88 | +int luo_session_retrieve(const char *name, struct file **filep); | |
| 89 | +int __init luo_session_setup_outgoing(void *fdt); | |
| 90 | +int __init luo_session_setup_incoming(void *fdt); | |
| 91 | +int luo_session_serialize(void); | |
| 92 | +int luo_session_deserialize(void); | |
| 93 | +bool luo_session_quiesce(void); | |
| 94 | +void luo_session_resume(void); | |
| 95 | + | |
| 96 | +int luo_preserve_file(struct luo_file_set *file_set, u64 token, int fd); | |
| 97 | +void luo_file_unpreserve_files(struct luo_file_set *file_set); | |
| 98 | +int luo_file_freeze(struct luo_file_set *file_set, | |
| 99 | + struct luo_file_set_ser *file_set_ser); | |
| 100 | +void luo_file_unfreeze(struct luo_file_set *file_set, | |
| 101 | + struct luo_file_set_ser *file_set_ser); | |
| 102 | +int luo_retrieve_file(struct luo_file_set *file_set, u64 token, | |
| 103 | + struct file **filep); | |
| 104 | +int luo_file_finish(struct luo_file_set *file_set); | |
| 105 | +int luo_file_deserialize(struct luo_file_set *file_set, | |
| 106 | + struct luo_file_set_ser *file_set_ser); | |
| 107 | +void luo_file_set_init(struct luo_file_set *file_set); | |
| 108 | +void luo_file_set_destroy(struct luo_file_set *file_set); | |
| 109 | + | |
| 110 | +#endif /* _LINUX_LUO_INTERNAL_H */ | |
| @@ -0,0 +1,646 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0 | |
| 2 | + | |
| 3 | +/* | |
| 4 | + * Copyright (c) 2025, Google LLC. | |
| 5 | + * Pasha Tatashin <pasha.tatashin@soleen.com> | |
| 6 | + */ | |
| 7 | + | |
| 8 | +/** | |
| 9 | + * DOC: LUO Sessions | |
| 10 | + * | |
| 11 | + * LUO Sessions provide the core mechanism for grouping and managing `struct | |
| 12 | + * file *` instances that need to be preserved across a kexec-based live | |
| 13 | + * update. Each session acts as a named container for a set of file objects, | |
| 14 | + * allowing a userspace agent to manage the lifecycle of resources critical to a | |
| 15 | + * workload. | |
| 16 | + * | |
| 17 | + * Core Concepts: | |
| 18 | + * | |
| 19 | + * - Named Containers: Sessions are identified by a unique, user-provided name, | |
| 20 | + * which is used for both creation in the current kernel and retrieval in the | |
| 21 | + * next kernel. | |
| 22 | + * | |
| 23 | + * - Userspace Interface: Session management is driven from userspace via | |
| 24 | + * ioctls on /dev/liveupdate. | |
| 25 | + * | |
| 26 | + * - Serialization: Session metadata is preserved using the KHO framework. When | |
| 27 | + * a live update is triggered via kexec, an array of `struct luo_session_ser` | |
| 28 | + * is populated and placed in a preserved memory region. An FDT node is also | |
| 29 | + * created, containing the count of sessions and the physical address of this | |
| 30 | + * array. | |
| 31 | + * | |
| 32 | + * Session Lifecycle: | |
| 33 | + * | |
| 34 | + * 1. Creation: A userspace agent calls `luo_session_create()` to create a | |
| 35 | + * new, empty session and receives a file descriptor for it. | |
| 36 | + * | |
| 37 | + * 2. Serialization: When the `reboot(LINUX_REBOOT_CMD_KEXEC)` syscall is | |
| 38 | + * made, `luo_session_serialize()` is called. It iterates through all | |
| 39 | + * active sessions and writes their metadata into a memory area preserved | |
| 40 | + * by KHO. | |
| 41 | + * | |
| 42 | + * 3. Deserialization (in new kernel): After kexec, `luo_session_deserialize()` | |
| 43 | + * runs, reading the serialized data and creating a list of `struct | |
| 44 | + * luo_session` objects representing the preserved sessions. | |
| 45 | + * | |
| 46 | + * 4. Retrieval: A userspace agent in the new kernel can then call | |
| 47 | + * `luo_session_retrieve()` with a session name to get a new file | |
| 48 | + * descriptor and access the preserved state. | |
| 49 | + */ | |
| 50 | + | |
| 51 | +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt | |
| 52 | + | |
| 53 | +#include <linux/anon_inodes.h> | |
| 54 | +#include <linux/cleanup.h> | |
| 55 | +#include <linux/err.h> | |
| 56 | +#include <linux/errno.h> | |
| 57 | +#include <linux/file.h> | |
| 58 | +#include <linux/fs.h> | |
| 59 | +#include <linux/io.h> | |
| 60 | +#include <linux/kexec_handover.h> | |
| 61 | +#include <linux/kho/abi/luo.h> | |
| 62 | +#include <linux/libfdt.h> | |
| 63 | +#include <linux/list.h> | |
| 64 | +#include <linux/liveupdate.h> | |
| 65 | +#include <linux/mutex.h> | |
| 66 | +#include <linux/rwsem.h> | |
| 67 | +#include <linux/slab.h> | |
| 68 | +#include <linux/unaligned.h> | |
| 69 | +#include <uapi/linux/liveupdate.h> | |
| 70 | +#include "luo_internal.h" | |
| 71 | + | |
| 72 | +/* 16 4K pages, give space for 744 sessions */ | |
| 73 | +#define LUO_SESSION_PGCNT 16ul | |
| 74 | +#define LUO_SESSION_MAX (((LUO_SESSION_PGCNT << PAGE_SHIFT) - \ | |
| 75 | + sizeof(struct luo_session_header_ser)) / \ | |
| 76 | + sizeof(struct luo_session_ser)) | |
| 77 | + | |
| 78 | +/** | |
| 79 | + * struct luo_session_header - Header struct for managing LUO sessions. | |
| 80 | + * @count: The number of sessions currently tracked in the @list. | |
| 81 | + * @list: The head of the linked list of `struct luo_session` instances. | |
| 82 | + * @rwsem: A read-write semaphore providing synchronized access to the | |
| 83 | + * session list and other fields in this structure. | |
| 84 | + * @header_ser: The header data of serialization array. | |
| 85 | + * @ser: The serialized session data (an array of | |
| 86 | + * `struct luo_session_ser`). | |
| 87 | + * @active: Set to true when first initialized. If previous kernel did not | |
| 88 | + * send session data, active stays false for incoming. | |
| 89 | + */ | |
| 90 | +struct luo_session_header { | |
| 91 | + long count; | |
| 92 | + struct list_head list; | |
| 93 | + struct rw_semaphore rwsem; | |
| 94 | + struct luo_session_header_ser *header_ser; | |
| 95 | + struct luo_session_ser *ser; | |
| 96 | + bool active; | |
| 97 | +}; | |
| 98 | + | |
| 99 | +/** | |
| 100 | + * struct luo_session_global - Global container for managing LUO sessions. | |
| 101 | + * @incoming: The sessions passed from the previous kernel. | |
| 102 | + * @outgoing: The sessions that are going to be passed to the next kernel. | |
| 103 | + */ | |
| 104 | +struct luo_session_global { | |
| 105 | + struct luo_session_header incoming; | |
| 106 | + struct luo_session_header outgoing; | |
| 107 | +}; | |
| 108 | + | |
| 109 | +static struct luo_session_global luo_session_global = { | |
| 110 | + .incoming = { | |
| 111 | + .list = LIST_HEAD_INIT(luo_session_global.incoming.list), | |
| 112 | + .rwsem = __RWSEM_INITIALIZER(luo_session_global.incoming.rwsem), | |
| 113 | + }, | |
| 114 | + .outgoing = { | |
| 115 | + .list = LIST_HEAD_INIT(luo_session_global.outgoing.list), | |
| 116 | + .rwsem = __RWSEM_INITIALIZER(luo_session_global.outgoing.rwsem), | |
| 117 | + }, | |
| 118 | +}; | |
| 119 | + | |
| 120 | +static struct luo_session *luo_session_alloc(const char *name) | |
| 121 | +{ | |
| 122 | + struct luo_session *session = kzalloc(sizeof(*session), GFP_KERNEL); | |
| 123 | + | |
| 124 | + if (!session) | |
| 125 | + return ERR_PTR(-ENOMEM); | |
| 126 | + | |
| 127 | + strscpy(session->name, name, sizeof(session->name)); | |
| 128 | + INIT_LIST_HEAD(&session->file_set.files_list); | |
| 129 | + luo_file_set_init(&session->file_set); | |
| 130 | + INIT_LIST_HEAD(&session->list); | |
| 131 | + mutex_init(&session->mutex); | |
| 132 | + | |
| 133 | + return session; | |
| 134 | +} | |
| 135 | + | |
| 136 | +static void luo_session_free(struct luo_session *session) | |
| 137 | +{ | |
| 138 | + luo_file_set_destroy(&session->file_set); | |
| 139 | + mutex_destroy(&session->mutex); | |
| 140 | + kfree(session); | |
| 141 | +} | |
| 142 | + | |
| 143 | +static int luo_session_insert(struct luo_session_header *sh, | |
| 144 | + struct luo_session *session) | |
| 145 | +{ | |
| 146 | + struct luo_session *it; | |
| 147 | + | |
| 148 | + guard(rwsem_write)(&sh->rwsem); | |
| 149 | + | |
| 150 | + /* | |
| 151 | + * For outgoing we should make sure there is room in serialization array | |
| 152 | + * for new session. | |
| 153 | + */ | |
| 154 | + if (sh == &luo_session_global.outgoing) { | |
| 155 | + if (sh->count == LUO_SESSION_MAX) | |
| 156 | + return -ENOMEM; | |
| 157 | + } | |
| 158 | + | |
| 159 | + /* | |
| 160 | + * For small number of sessions this loop won't hurt performance | |
| 161 | + * but if we ever start using a lot of sessions, this might | |
| 162 | + * become a bottle neck during deserialization time, as it would | |
| 163 | + * cause O(n*n) complexity. | |
| 164 | + */ | |
| 165 | + list_for_each_entry(it, &sh->list, list) { | |
| 166 | + if (!strncmp(it->name, session->name, sizeof(it->name))) | |
| 167 | + return -EEXIST; | |
| 168 | + } | |
| 169 | + list_add_tail(&session->list, &sh->list); | |
| 170 | + sh->count++; | |
| 171 | + | |
| 172 | + return 0; | |
| 173 | +} | |
| 174 | + | |
| 175 | +static void luo_session_remove(struct luo_session_header *sh, | |
| 176 | + struct luo_session *session) | |
| 177 | +{ | |
| 178 | + guard(rwsem_write)(&sh->rwsem); | |
| 179 | + list_del(&session->list); | |
| 180 | + sh->count--; | |
| 181 | +} | |
| 182 | + | |
| 183 | +static int luo_session_finish_one(struct luo_session *session) | |
| 184 | +{ | |
| 185 | + guard(mutex)(&session->mutex); | |
| 186 | + return luo_file_finish(&session->file_set); | |
| 187 | +} | |
| 188 | + | |
| 189 | +static void luo_session_unfreeze_one(struct luo_session *session, | |
| 190 | + struct luo_session_ser *ser) | |
| 191 | +{ | |
| 192 | + guard(mutex)(&session->mutex); | |
| 193 | + luo_file_unfreeze(&session->file_set, &ser->file_set_ser); | |
| 194 | +} | |
| 195 | + | |
| 196 | +static int luo_session_freeze_one(struct luo_session *session, | |
| 197 | + struct luo_session_ser *ser) | |
| 198 | +{ | |
| 199 | + guard(mutex)(&session->mutex); | |
| 200 | + return luo_file_freeze(&session->file_set, &ser->file_set_ser); | |
| 201 | +} | |
| 202 | + | |
| 203 | +static int luo_session_release(struct inode *inodep, struct file *filep) | |
| 204 | +{ | |
| 205 | + struct luo_session *session = filep->private_data; | |
| 206 | + struct luo_session_header *sh; | |
| 207 | + | |
| 208 | + /* If retrieved is set, it means this session is from incoming list */ | |
| 209 | + if (session->retrieved) { | |
| 210 | + int err = luo_session_finish_one(session); | |
| 211 | + | |
| 212 | + if (err) { | |
| 213 | + pr_warn("Unable to finish session [%s] on release\n", | |
| 214 | + session->name); | |
| 215 | + return err; | |
| 216 | + } | |
| 217 | + sh = &luo_session_global.incoming; | |
| 218 | + } else { | |
| 219 | + scoped_guard(mutex, &session->mutex) | |
| 220 | + luo_file_unpreserve_files(&session->file_set); | |
| 221 | + sh = &luo_session_global.outgoing; | |
| 222 | + } | |
| 223 | + | |
| 224 | + luo_session_remove(sh, session); | |
| 225 | + luo_session_free(session); | |
| 226 | + | |
| 227 | + return 0; | |
| 228 | +} | |
| 229 | + | |
| 230 | +static int luo_session_preserve_fd(struct luo_session *session, | |
| 231 | + struct luo_ucmd *ucmd) | |
| 232 | +{ | |
| 233 | + struct liveupdate_session_preserve_fd *argp = ucmd->cmd; | |
| 234 | + int err; | |
| 235 | + | |
| 236 | + guard(mutex)(&session->mutex); | |
| 237 | + err = luo_preserve_file(&session->file_set, argp->token, argp->fd); | |
| 238 | + if (err) | |
| 239 | + return err; | |
| 240 | + | |
| 241 | + err = luo_ucmd_respond(ucmd, sizeof(*argp)); | |
| 242 | + if (err) | |
| 243 | + pr_warn("The file was successfully preserved, but response to user failed\n"); | |
| 244 | + | |
| 245 | + return err; | |
| 246 | +} | |
| 247 | + | |
| 248 | +static int luo_session_retrieve_fd(struct luo_session *session, | |
| 249 | + struct luo_ucmd *ucmd) | |
| 250 | +{ | |
| 251 | + struct liveupdate_session_retrieve_fd *argp = ucmd->cmd; | |
| 252 | + struct file *file; | |
| 253 | + int err; | |
| 254 | + | |
| 255 | + argp->fd = get_unused_fd_flags(O_CLOEXEC); | |
| 256 | + if (argp->fd < 0) | |
| 257 | + return argp->fd; | |
| 258 | + | |
| 259 | + guard(mutex)(&session->mutex); | |
| 260 | + err = luo_retrieve_file(&session->file_set, argp->token, &file); | |
| 261 | + if (err < 0) | |
| 262 | + goto err_put_fd; | |
| 263 | + | |
| 264 | + err = luo_ucmd_respond(ucmd, sizeof(*argp)); | |
| 265 | + if (err) | |
| 266 | + goto err_put_file; | |
| 267 | + | |
| 268 | + fd_install(argp->fd, file); | |
| 269 | + | |
| 270 | + return 0; | |
| 271 | + | |
| 272 | +err_put_file: | |
| 273 | + fput(file); | |
| 274 | +err_put_fd: | |
| 275 | + put_unused_fd(argp->fd); | |
| 276 | + | |
| 277 | + return err; | |
| 278 | +} | |
| 279 | + | |
| 280 | +static int luo_session_finish(struct luo_session *session, | |
| 281 | + struct luo_ucmd *ucmd) | |
| 282 | +{ | |
| 283 | + struct liveupdate_session_finish *argp = ucmd->cmd; | |
| 284 | + int err = luo_session_finish_one(session); | |
| 285 | + | |
| 286 | + if (err) | |
| 287 | + return err; | |
| 288 | + | |
| 289 | + return luo_ucmd_respond(ucmd, sizeof(*argp)); | |
| 290 | +} | |
| 291 | + | |
| 292 | +union ucmd_buffer { | |
| 293 | + struct liveupdate_session_finish finish; | |
| 294 | + struct liveupdate_session_preserve_fd preserve; | |
| 295 | + struct liveupdate_session_retrieve_fd retrieve; | |
| 296 | +}; | |
| 297 | + | |
| 298 | +struct luo_ioctl_op { | |
| 299 | + unsigned int size; | |
| 300 | + unsigned int min_size; | |
| 301 | + unsigned int ioctl_num; | |
| 302 | + int (*execute)(struct luo_session *session, struct luo_ucmd *ucmd); | |
| 303 | +}; | |
| 304 | + | |
| 305 | +#define IOCTL_OP(_ioctl, _fn, _struct, _last) \ | |
| 306 | + [_IOC_NR(_ioctl) - LIVEUPDATE_CMD_SESSION_BASE] = { \ | |
| 307 | + .size = sizeof(_struct) + \ | |
| 308 | + BUILD_BUG_ON_ZERO(sizeof(union ucmd_buffer) < \ | |
| 309 | + sizeof(_struct)), \ | |
| 310 | + .min_size = offsetofend(_struct, _last), \ | |
| 311 | + .ioctl_num = _ioctl, \ | |
| 312 | + .execute = _fn, \ | |
| 313 | + } | |
| 314 | + | |
| 315 | +static const struct luo_ioctl_op luo_session_ioctl_ops[] = { | |
| 316 | + IOCTL_OP(LIVEUPDATE_SESSION_FINISH, luo_session_finish, | |
| 317 | + struct liveupdate_session_finish, reserved), | |
| 318 | + IOCTL_OP(LIVEUPDATE_SESSION_PRESERVE_FD, luo_session_preserve_fd, | |
| 319 | + struct liveupdate_session_preserve_fd, token), | |
| 320 | + IOCTL_OP(LIVEUPDATE_SESSION_RETRIEVE_FD, luo_session_retrieve_fd, | |
| 321 | + struct liveupdate_session_retrieve_fd, token), | |
| 322 | +}; | |
| 323 | + | |
| 324 | +static long luo_session_ioctl(struct file *filep, unsigned int cmd, | |
| 325 | + unsigned long arg) | |
| 326 | +{ | |
| 327 | + struct luo_session *session = filep->private_data; | |
| 328 | + const struct luo_ioctl_op *op; | |
| 329 | + struct luo_ucmd ucmd = {}; | |
| 330 | + union ucmd_buffer buf; | |
| 331 | + unsigned int nr; | |
| 332 | + int ret; | |
| 333 | + | |
| 334 | + nr = _IOC_NR(cmd); | |
| 335 | + if (nr < LIVEUPDATE_CMD_SESSION_BASE || (nr - LIVEUPDATE_CMD_SESSION_BASE) >= | |
| 336 | + ARRAY_SIZE(luo_session_ioctl_ops)) { | |
| 337 | + return -EINVAL; | |
| 338 | + } | |
| 339 | + | |
| 340 | + ucmd.ubuffer = (void __user *)arg; | |
| 341 | + ret = get_user(ucmd.user_size, (u32 __user *)ucmd.ubuffer); | |
| 342 | + if (ret) | |
| 343 | + return ret; | |
| 344 | + | |
| 345 | + op = &luo_session_ioctl_ops[nr - LIVEUPDATE_CMD_SESSION_BASE]; | |
| 346 | + if (op->ioctl_num != cmd) | |
| 347 | + return -ENOIOCTLCMD; | |
| 348 | + if (ucmd.user_size < op->min_size) | |
| 349 | + return -EINVAL; | |
| 350 | + | |
| 351 | + ucmd.cmd = &buf; | |
| 352 | + ret = copy_struct_from_user(ucmd.cmd, op->size, ucmd.ubuffer, | |
| 353 | + ucmd.user_size); | |
| 354 | + if (ret) | |
| 355 | + return ret; | |
| 356 | + | |
| 357 | + return op->execute(session, &ucmd); | |
| 358 | +} | |
| 359 | + | |
| 360 | +static const struct file_operations luo_session_fops = { | |
| 361 | + .owner = THIS_MODULE, | |
| 362 | + .release = luo_session_release, | |
| 363 | + .unlocked_ioctl = luo_session_ioctl, | |
| 364 | +}; | |
| 365 | + | |
| 366 | +/* Create a "struct file" for session */ | |
| 367 | +static int luo_session_getfile(struct luo_session *session, struct file **filep) | |
| 368 | +{ | |
| 369 | + char name_buf[128]; | |
| 370 | + struct file *file; | |
| 371 | + | |
| 372 | + lockdep_assert_held(&session->mutex); | |
| 373 | + snprintf(name_buf, sizeof(name_buf), "[luo_session] %s", session->name); | |
| 374 | + file = anon_inode_getfile(name_buf, &luo_session_fops, session, O_RDWR); | |
| 375 | + if (IS_ERR(file)) | |
| 376 | + return PTR_ERR(file); | |
| 377 | + | |
| 378 | + *filep = file; | |
| 379 | + | |
| 380 | + return 0; | |
| 381 | +} | |
| 382 | + | |
| 383 | +int luo_session_create(const char *name, struct file **filep) | |
| 384 | +{ | |
| 385 | + struct luo_session *session; | |
| 386 | + int err; | |
| 387 | + | |
| 388 | + session = luo_session_alloc(name); | |
| 389 | + if (IS_ERR(session)) | |
| 390 | + return PTR_ERR(session); | |
| 391 | + | |
| 392 | + err = luo_session_insert(&luo_session_global.outgoing, session); | |
| 393 | + if (err) | |
| 394 | + goto err_free; | |
| 395 | + | |
| 396 | + scoped_guard(mutex, &session->mutex) | |
| 397 | + err = luo_session_getfile(session, filep); | |
| 398 | + if (err) | |
| 399 | + goto err_remove; | |
| 400 | + | |
| 401 | + return 0; | |
| 402 | + | |
| 403 | +err_remove: | |
| 404 | + luo_session_remove(&luo_session_global.outgoing, session); | |
| 405 | +err_free: | |
| 406 | + luo_session_free(session); | |
| 407 | + | |
| 408 | + return err; | |
| 409 | +} | |
| 410 | + | |
| 411 | +int luo_session_retrieve(const char *name, struct file **filep) | |
| 412 | +{ | |
| 413 | + struct luo_session_header *sh = &luo_session_global.incoming; | |
| 414 | + struct luo_session *session = NULL; | |
| 415 | + struct luo_session *it; | |
| 416 | + int err; | |
| 417 | + | |
| 418 | + scoped_guard(rwsem_read, &sh->rwsem) { | |
| 419 | + list_for_each_entry(it, &sh->list, list) { | |
| 420 | + if (!strncmp(it->name, name, sizeof(it->name))) { | |
| 421 | + session = it; | |
| 422 | + break; | |
| 423 | + } | |
| 424 | + } | |
| 425 | + } | |
| 426 | + | |
| 427 | + if (!session) | |
| 428 | + return -ENOENT; | |
| 429 | + | |
| 430 | + guard(mutex)(&session->mutex); | |
| 431 | + if (session->retrieved) | |
| 432 | + return -EINVAL; | |
| 433 | + | |
| 434 | + err = luo_session_getfile(session, filep); | |
| 435 | + if (!err) | |
| 436 | + session->retrieved = true; | |
| 437 | + | |
| 438 | + return err; | |
| 439 | +} | |
| 440 | + | |
| 441 | +int __init luo_session_setup_outgoing(void *fdt_out) | |
| 442 | +{ | |
| 443 | + struct luo_session_header_ser *header_ser; | |
| 444 | + u64 header_ser_pa; | |
| 445 | + int err; | |
| 446 | + | |
| 447 | + header_ser = kho_alloc_preserve(LUO_SESSION_PGCNT << PAGE_SHIFT); | |
| 448 | + if (IS_ERR(header_ser)) | |
| 449 | + return PTR_ERR(header_ser); | |
| 450 | + header_ser_pa = virt_to_phys(header_ser); | |
| 451 | + | |
| 452 | + err = fdt_begin_node(fdt_out, LUO_FDT_SESSION_NODE_NAME); | |
| 453 | + err |= fdt_property_string(fdt_out, "compatible", | |
| 454 | + LUO_FDT_SESSION_COMPATIBLE); | |
| 455 | + err |= fdt_property(fdt_out, LUO_FDT_SESSION_HEADER, &header_ser_pa, | |
| 456 | + sizeof(header_ser_pa)); | |
| 457 | + err |= fdt_end_node(fdt_out); | |
| 458 | + | |
| 459 | + if (err) | |
| 460 | + goto err_unpreserve; | |
| 461 | + | |
| 462 | + luo_session_global.outgoing.header_ser = header_ser; | |
| 463 | + luo_session_global.outgoing.ser = (void *)(header_ser + 1); | |
| 464 | + luo_session_global.outgoing.active = true; | |
| 465 | + | |
| 466 | + return 0; | |
| 467 | + | |
| 468 | +err_unpreserve: | |
| 469 | + kho_unpreserve_free(header_ser); | |
| 470 | + return err; | |
| 471 | +} | |
| 472 | + | |
| 473 | +int __init luo_session_setup_incoming(void *fdt_in) | |
| 474 | +{ | |
| 475 | + struct luo_session_header_ser *header_ser; | |
| 476 | + int err, header_size, offset; | |
| 477 | + u64 header_ser_pa; | |
| 478 | + const void *ptr; | |
| 479 | + | |
| 480 | + offset = fdt_subnode_offset(fdt_in, 0, LUO_FDT_SESSION_NODE_NAME); | |
| 481 | + if (offset < 0) { | |
| 482 | + pr_err("Unable to get session node: [%s]\n", | |
| 483 | + LUO_FDT_SESSION_NODE_NAME); | |
| 484 | + return -EINVAL; | |
| 485 | + } | |
| 486 | + | |
| 487 | + err = fdt_node_check_compatible(fdt_in, offset, | |
| 488 | + LUO_FDT_SESSION_COMPATIBLE); | |
| 489 | + if (err) { | |
| 490 | + pr_err("Session node incompatible [%s]\n", | |
| 491 | + LUO_FDT_SESSION_COMPATIBLE); | |
| 492 | + return -EINVAL; | |
| 493 | + } | |
| 494 | + | |
| 495 | + header_size = 0; | |
| 496 | + ptr = fdt_getprop(fdt_in, offset, LUO_FDT_SESSION_HEADER, &header_size); | |
| 497 | + if (!ptr || header_size != sizeof(u64)) { | |
| 498 | + pr_err("Unable to get session header '%s' [%d]\n", | |
| 499 | + LUO_FDT_SESSION_HEADER, header_size); | |
| 500 | + return -EINVAL; | |
| 501 | + } | |
| 502 | + | |
| 503 | + header_ser_pa = get_unaligned((u64 *)ptr); | |
| 504 | + header_ser = phys_to_virt(header_ser_pa); | |
| 505 | + | |
| 506 | + luo_session_global.incoming.header_ser = header_ser; | |
| 507 | + luo_session_global.incoming.ser = (void *)(header_ser + 1); | |
| 508 | + luo_session_global.incoming.active = true; | |
| 509 | + | |
| 510 | + return 0; | |
| 511 | +} | |
| 512 | + | |
| 513 | +int luo_session_deserialize(void) | |
| 514 | +{ | |
| 515 | + struct luo_session_header *sh = &luo_session_global.incoming; | |
| 516 | + static bool is_deserialized; | |
| 517 | + static int err; | |
| 518 | + | |
| 519 | + /* If has been deserialized, always return the same error code */ | |
| 520 | + if (is_deserialized) | |
| 521 | + return err; | |
| 522 | + | |
| 523 | + is_deserialized = true; | |
| 524 | + if (!sh->active) | |
| 525 | + return 0; | |
| 526 | + | |
| 527 | + /* | |
| 528 | + * Note on error handling: | |
| 529 | + * | |
| 530 | + * If deserialization fails (e.g., allocation failure or corrupt data), | |
| 531 | + * we intentionally skip cleanup of sessions that were already restored. | |
| 532 | + * | |
| 533 | + * A partial failure leaves the preserved state inconsistent. | |
| 534 | + * Implementing a safe "undo" to unwind complex dependencies (sessions, | |
| 535 | + * files, hardware state) is error-prone and provides little value, as | |
| 536 | + * the system is effectively in a broken state. | |
| 537 | + * | |
| 538 | + * We treat these resources as leaked. The expected recovery path is for | |
| 539 | + * userspace to detect the failure and trigger a reboot, which will | |
| 540 | + * reliably reset devices and reclaim memory. | |
| 541 | + */ | |
| 542 | + for (int i = 0; i < sh->header_ser->count; i++) { | |
| 543 | + struct luo_session *session; | |
| 544 | + | |
| 545 | + session = luo_session_alloc(sh->ser[i].name); | |
| 546 | + if (IS_ERR(session)) { | |
| 547 | + pr_warn("Failed to allocate session [%s] during deserialization %pe\n", | |
| 548 | + sh->ser[i].name, session); | |
| 549 | + return PTR_ERR(session); | |
| 550 | + } | |
| 551 | + | |
| 552 | + err = luo_session_insert(sh, session); | |
| 553 | + if (err) { | |
| 554 | + pr_warn("Failed to insert session [%s] %pe\n", | |
| 555 | + session->name, ERR_PTR(err)); | |
| 556 | + luo_session_free(session); | |
| 557 | + return err; | |
| 558 | + } | |
| 559 | + | |
| 560 | + scoped_guard(mutex, &session->mutex) { | |
| 561 | + luo_file_deserialize(&session->file_set, | |
| 562 | + &sh->ser[i].file_set_ser); | |
| 563 | + } | |
| 564 | + } | |
| 565 | + | |
| 566 | + kho_restore_free(sh->header_ser); | |
| 567 | + sh->header_ser = NULL; | |
| 568 | + sh->ser = NULL; | |
| 569 | + | |
| 570 | + return 0; | |
| 571 | +} | |
| 572 | + | |
| 573 | +int luo_session_serialize(void) | |
| 574 | +{ | |
| 575 | + struct luo_session_header *sh = &luo_session_global.outgoing; | |
| 576 | + struct luo_session *session; | |
| 577 | + int i = 0; | |
| 578 | + int err; | |
| 579 | + | |
| 580 | + guard(rwsem_write)(&sh->rwsem); | |
| 581 | + list_for_each_entry(session, &sh->list, list) { | |
| 582 | + err = luo_session_freeze_one(session, &sh->ser[i]); | |
| 583 | + if (err) | |
| 584 | + goto err_undo; | |
| 585 | + | |
| 586 | + strscpy(sh->ser[i].name, session->name, | |
| 587 | + sizeof(sh->ser[i].name)); | |
| 588 | + i++; | |
| 589 | + } | |
| 590 | + sh->header_ser->count = sh->count; | |
| 591 | + | |
| 592 | + return 0; | |
| 593 | + | |
| 594 | +err_undo: | |
| 595 | + list_for_each_entry_continue_reverse(session, &sh->list, list) { | |
| 596 | + i--; | |
| 597 | + luo_session_unfreeze_one(session, &sh->ser[i]); | |
| 598 | + memset(sh->ser[i].name, 0, sizeof(sh->ser[i].name)); | |
| 599 | + } | |
| 600 | + | |
| 601 | + return err; | |
| 602 | +} | |
| 603 | + | |
| 604 | +/** | |
| 605 | + * luo_session_quiesce - Ensure no active sessions exist and lock session lists. | |
| 606 | + * | |
| 607 | + * Acquires exclusive write locks on both incoming and outgoing session lists. | |
| 608 | + * It then validates no sessions exist in either list. | |
| 609 | + * | |
| 610 | + * This mechanism is used during file handler un/registration to ensure that no | |
| 611 | + * sessions are currently using the handler, and no new sessions can be created | |
| 612 | + * while un/registration is in progress. | |
| 613 | + * | |
| 614 | + * This prevents registering new handlers while sessions are active or | |
| 615 | + * while deserialization is in progress. | |
| 616 | + * | |
| 617 | + * Return: | |
| 618 | + * true - System is quiescent (0 sessions) and locked. | |
| 619 | + * false - Active sessions exist. The locks are released internally. | |
| 620 | + */ | |
| 621 | +bool luo_session_quiesce(void) | |
| 622 | +{ | |
| 623 | + down_write(&luo_session_global.incoming.rwsem); | |
| 624 | + down_write(&luo_session_global.outgoing.rwsem); | |
| 625 | + | |
| 626 | + if (luo_session_global.incoming.count || | |
| 627 | + luo_session_global.outgoing.count) { | |
| 628 | + up_write(&luo_session_global.outgoing.rwsem); | |
| 629 | + up_write(&luo_session_global.incoming.rwsem); | |
| 630 | + return false; | |
| 631 | + } | |
| 632 | + | |
| 633 | + return true; | |
| 634 | +} | |
| 635 | + | |
| 636 | +/** | |
| 637 | + * luo_session_resume - Unlock session lists and resume normal activity. | |
| 638 | + * | |
| 639 | + * Releases the exclusive locks acquired by a successful call to | |
| 640 | + * luo_session_quiesce(). | |
| 641 | + */ | |
| 642 | +void luo_session_resume(void) | |
| 643 | +{ | |
| 644 | + up_write(&luo_session_global.outgoing.rwsem); | |
| 645 | + up_write(&luo_session_global.incoming.rwsem); | |
| 646 | +} | |
| @@ -954,7 +954,7 @@ size_t module_flags_taint(unsigned long taints, char *buf) · 953 unmodified lines | ||
| 954 | 954 | int i; |
| 955 | 955 | |
| 956 | 956 | for (i = 0; i < TAINT_FLAGS_COUNT; i++) { |
| 957 | − if (taint_flags[i].module && test_bit(i, &taints)) | |
| 957 | + if (test_bit(i, &taints)) | |
| 958 | 958 | buf[l++] = taint_flags[i].c_true; |
| 959 | 959 | } |
| 960 | 960 | |
| @@ -401,7 +401,7 @@ static void panic_trigger_all_cpu_backtrace(void) · 400 unmodified lines | ||
| 401 | 401 | */ |
| 402 | 402 | static void panic_other_cpus_shutdown(bool crash_kexec) |
| 403 | 403 | { |
| 404 | − if (panic_print & SYS_INFO_ALL_CPU_BT) | |
| 404 | + if (panic_print & SYS_INFO_ALL_BT) | |
| 405 | 405 | panic_trigger_all_cpu_backtrace(); |
| 406 | 406 | |
| 407 | 407 | /* |
| @@ -628,38 +628,40 @@ void panic(const char *fmt, ...) · 220 unmodified lines | ||
| 628 | 628 | } |
| 629 | 629 | EXPORT_SYMBOL(panic); |
| 630 | 630 | |
| 631 | −#define TAINT_FLAG(taint, _c_true, _c_false, _module) \ | |
| 631 | +#define TAINT_FLAG(taint, _c_true, _c_false) \ | |
| 632 | 632 | [ TAINT_##taint ] = { \ |
| 633 | 633 | .c_true = _c_true, .c_false = _c_false, \ |
| 634 | − .module = _module, \ | |
| 635 | 634 | .desc = #taint, \ |
| 636 | 635 | } |
| 637 | 636 | |
| 638 | 637 | /* |
| 639 | − * TAINT_FORCED_RMMOD could be a per-module flag but the module | |
| 640 | − * is being removed anyway. | |
| 638 | + * NOTE: if you modify the taint_flags or TAINT_FLAGS_COUNT, | |
| 639 | + * please also modify tools/debugging/kernel-chktaint and | |
| 640 | + * Documentation/admin-guide/tainted-kernels.rst, including its | |
| 641 | + * small shell script that prints the TAINT_FLAGS_COUNT bits of | |
| 642 | + * /proc/sys/kernel/tainted. | |
| 641 | 643 | */ |
| 642 | 644 | const struct taint_flag taint_flags[TAINT_FLAGS_COUNT] = { |
| 643 | − TAINT_FLAG(PROPRIETARY_MODULE, 'P', 'G', true), | |
| 644 | − TAINT_FLAG(FORCED_MODULE, 'F', ' ', true), | |
| 645 | − TAINT_FLAG(CPU_OUT_OF_SPEC, 'S', ' ', false), | |
| 646 | − TAINT_FLAG(FORCED_RMMOD, 'R', ' ', false), | |
| 647 | − TAINT_FLAG(MACHINE_CHECK, 'M', ' ', false), | |
| 648 | − TAINT_FLAG(BAD_PAGE, 'B', ' ', false), | |
| 649 | − TAINT_FLAG(USER, 'U', ' ', false), | |
| 650 | − TAINT_FLAG(DIE, 'D', ' ', false), | |
| 651 | − TAINT_FLAG(OVERRIDDEN_ACPI_TABLE, 'A', ' ', false), | |
| 652 | − TAINT_FLAG(WARN, 'W', ' ', false), | |
| 653 | − TAINT_FLAG(CRAP, 'C', ' ', true), | |
| 654 | − TAINT_FLAG(FIRMWARE_WORKAROUND, 'I', ' ', false), | |
| 655 | − TAINT_FLAG(OOT_MODULE, 'O', ' ', true), | |
| 656 | − TAINT_FLAG(UNSIGNED_MODULE, 'E', ' ', true), | |
| 657 | − TAINT_FLAG(SOFTLOCKUP, 'L', ' ', false), | |
| 658 | − TAINT_FLAG(LIVEPATCH, 'K', ' ', true), | |
| 659 | − TAINT_FLAG(AUX, 'X', ' ', true), | |
| 660 | − TAINT_FLAG(RANDSTRUCT, 'T', ' ', true), | |
| 661 | − TAINT_FLAG(TEST, 'N', ' ', true), | |
| 662 | − TAINT_FLAG(FWCTL, 'J', ' ', true), | |
| 645 | + TAINT_FLAG(PROPRIETARY_MODULE, 'P', 'G'), | |
| 646 | + TAINT_FLAG(FORCED_MODULE, 'F', ' '), | |
| 647 | + TAINT_FLAG(CPU_OUT_OF_SPEC, 'S', ' '), | |
| 648 | + TAINT_FLAG(FORCED_RMMOD, 'R', ' '), | |
| 649 | + TAINT_FLAG(MACHINE_CHECK, 'M', ' '), | |
| 650 | + TAINT_FLAG(BAD_PAGE, 'B', ' '), | |
| 651 | + TAINT_FLAG(USER, 'U', ' '), | |
| 652 | + TAINT_FLAG(DIE, 'D', ' '), | |
| 653 | + TAINT_FLAG(OVERRIDDEN_ACPI_TABLE, 'A', ' '), | |
| 654 | + TAINT_FLAG(WARN, 'W', ' '), | |
| 655 | + TAINT_FLAG(CRAP, 'C', ' '), | |
| 656 | + TAINT_FLAG(FIRMWARE_WORKAROUND, 'I', ' '), | |
| 657 | + TAINT_FLAG(OOT_MODULE, 'O', ' '), | |
| 658 | + TAINT_FLAG(UNSIGNED_MODULE, 'E', ' '), | |
| 659 | + TAINT_FLAG(SOFTLOCKUP, 'L', ' '), | |
| 660 | + TAINT_FLAG(LIVEPATCH, 'K', ' '), | |
| 661 | + TAINT_FLAG(AUX, 'X', ' '), | |
| 662 | + TAINT_FLAG(RANDSTRUCT, 'T', ' '), | |
| 663 | + TAINT_FLAG(TEST, 'N', ' '), | |
| 664 | + TAINT_FLAG(FWCTL, 'J', ' '), | |
| 663 | 665 | }; |
| 664 | 666 | |
| 665 | 667 | #undef TAINT_FLAG |
| @@ -341,6 +341,8 @@ static int find_next_iomem_res(resource_size_t start, resource_size_t end, · 340 unmodified lines | ||
| 341 | 341 | unsigned long flags, unsigned long desc, |
| 342 | 342 | struct resource *res) |
| 343 | 343 | { |
| 344 | + /* Skip children until we find a top level range that matches */ | |
| 345 | + bool skip_children = true; | |
| 344 | 346 | struct resource *p; |
| 345 | 347 | |
| 346 | 348 | if (!res) |
| @@ -351,7 +353,7 @@ static int find_next_iomem_res(resource_size_t start, resource_size_t end, · 4 unmodified lines | ||
| 351 | 353 | |
| 352 | 354 | read_lock(&resource_lock); |
| 353 | 355 | |
| 354 | − for_each_resource(&iomem_resource, p, false) { | |
| 356 | + for_each_resource(&iomem_resource, p, skip_children) { | |
| 355 | 357 | /* If we passed the resource we are looking for, stop */ |
| 356 | 358 | if (p->start > end) { |
| 357 | 359 | p = NULL; |
| @@ -362,6 +364,12 @@ static int find_next_iomem_res(resource_size_t start, resource_size_t end, · 4 unmodified lines | ||
| 362 | 364 | if (p->end < start) |
| 363 | 365 | continue; |
| 364 | 366 | |
| 367 | + /* | |
| 368 | + * We found a top level range that matches what we are looking | |
| 369 | + * for. Time to start checking children too. | |
| 370 | + */ | |
| 371 | + skip_children = false; | |
| 372 | + | |
| 365 | 373 | /* Found a match, break */ |
| 366 | 374 | if (is_type_match(p, flags, desc)) |
| 367 | 375 | break; |
| @@ -135,7 +135,7 @@ static void scs_check_usage(struct task_struct *tsk) · 134 unmodified lines | ||
| 135 | 135 | if (!IS_ENABLED(CONFIG_DEBUG_STACK_USAGE)) |
| 136 | 136 | return; |
| 137 | 137 | |
| 138 | − for (p = task_scs(tsk); p < __scs_magic(tsk); ++p) { | |
| 138 | + for (p = task_scs(tsk); p < __scs_magic(task_scs(tsk)); ++p) { | |
| 139 | 139 | if (!READ_ONCE_NOCHECK(*p)) |
| 140 | 140 | break; |
| 141 | 141 | used += sizeof(*p); |
| @@ -31,6 +31,13 @@ u32 *vmcoreinfo_note; · 30 unmodified lines | ||
| 31 | 31 | /* trusted vmcoreinfo, e.g. we can make a copy in the crash memory */ |
| 32 | 32 | static unsigned char *vmcoreinfo_data_safecopy; |
| 33 | 33 | |
| 34 | +struct hwerr_info { | |
| 35 | + atomic_t count; | |
| 36 | + time64_t timestamp; | |
| 37 | +}; | |
| 38 | + | |
| 39 | +static struct hwerr_info hwerr_data[HWERR_RECOV_MAX]; | |
| 40 | + | |
| 34 | 41 | Elf_Word *append_elf_note(Elf_Word *buf, char *name, unsigned int type, |
| 35 | 42 | void *data, size_t data_len) |
| 36 | 43 | { |
| @@ -118,6 +125,16 @@ phys_addr_t __weak paddr_vmcoreinfo_note(void) · 81 unmodified lines | ||
| 118 | 125 | } |
| 119 | 126 | EXPORT_SYMBOL(paddr_vmcoreinfo_note); |
| 120 | 127 | |
| 128 | +void hwerr_log_error_type(enum hwerr_error_type src) | |
| 129 | +{ | |
| 130 | + if (src < 0 || src >= HWERR_RECOV_MAX) | |
| 131 | + return; | |
| 132 | + | |
| 133 | + atomic_inc(&hwerr_data[src].count); | |
| 134 | + WRITE_ONCE(hwerr_data[src].timestamp, ktime_get_real_seconds()); | |
| 135 | +} | |
| 136 | +EXPORT_SYMBOL_GPL(hwerr_log_error_type); | |
| 137 | + | |
| 121 | 138 | static int __init crash_save_vmcoreinfo_init(void) |
| 122 | 139 | { |
| 123 | 140 | vmcoreinfo_data = (unsigned char *)get_zeroed_page(GFP_KERNEL); |
| @@ -25,6 +25,7 @@ · 24 unmodified lines | ||
| 25 | 25 | #include <linux/stop_machine.h> |
| 26 | 26 | #include <linux/sysctl.h> |
| 27 | 27 | #include <linux/tick.h> |
| 28 | +#include <linux/sys_info.h> | |
| 28 | 29 | |
| 29 | 30 | #include <linux/sched/clock.h> |
| 30 | 31 | #include <linux/sched/debug.h> |
| @@ -65,6 +66,13 @@ int __read_mostly sysctl_hardlockup_all_cpu_backtrace; · 34 unmodified lines | ||
| 65 | 66 | unsigned int __read_mostly hardlockup_panic = |
| 66 | 67 | IS_ENABLED(CONFIG_BOOTPARAM_HARDLOCKUP_PANIC); |
| 67 | 68 | |
| 69 | +/* | |
| 70 | + * bitmasks to control what kinds of system info to be printed when | |
| 71 | + * hard lockup is detected, it could be task, memory, lock etc. | |
| 72 | + * Refer include/linux/sys_info.h for detailed bit definition. | |
| 73 | + */ | |
| 74 | +static unsigned long hardlockup_si_mask; | |
| 75 | + | |
| 68 | 76 | #ifdef CONFIG_SYSFS |
| 69 | 77 | |
| 70 | 78 | static unsigned int hardlockup_count; |
| @@ -178,11 +186,15 @@ static void watchdog_hardlockup_kick(void) · 107 unmodified lines | ||
| 178 | 186 | |
| 179 | 187 | void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs) |
| 180 | 188 | { |
| 189 | + int hardlockup_all_cpu_backtrace; | |
| 190 | + | |
| 181 | 191 | if (per_cpu(watchdog_hardlockup_touched, cpu)) { |
| 182 | 192 | per_cpu(watchdog_hardlockup_touched, cpu) = false; |
| 183 | 193 | return; |
| 184 | 194 | } |
| 185 | 195 | |
| 196 | + hardlockup_all_cpu_backtrace = (hardlockup_si_mask & SYS_INFO_ALL_BT) ? | |
| 197 | + 1 : sysctl_hardlockup_all_cpu_backtrace; | |
| 186 | 198 | /* |
| 187 | 199 | * Check for a hardlockup by making sure the CPU's timer |
| 188 | 200 | * interrupt is incrementing. The timer interrupt should have |
| @@ -214,7 +226,7 @@ void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs) · 25 unmodified lines | ||
| 214 | 226 | * Prevent multiple hard-lockup reports if one cpu is already |
| 215 | 227 | * engaged in dumping all cpu back traces. |
| 216 | 228 | */ |
| 217 | − if (sysctl_hardlockup_all_cpu_backtrace) { | |
| 229 | + if (hardlockup_all_cpu_backtrace) { | |
| 218 | 230 | if (test_and_set_bit_lock(0, &hard_lockup_nmi_warn)) |
| 219 | 231 | return; |
| 220 | 232 | } |
| @@ -243,12 +255,13 @@ void watchdog_hardlockup_check(unsigned int cpu, struct pt_regs *regs) · 22 unmodified lines | ||
| 243 | 255 | trigger_single_cpu_backtrace(cpu); |
| 244 | 256 | } |
| 245 | 257 | |
| 246 | − if (sysctl_hardlockup_all_cpu_backtrace) { | |
| 258 | + if (hardlockup_all_cpu_backtrace) { | |
| 247 | 259 | trigger_allbutcpu_cpu_backtrace(cpu); |
| 248 | 260 | if (!hardlockup_panic) |
| 249 | 261 | clear_bit_unlock(0, &hard_lockup_nmi_warn); |
| 250 | 262 | } |
| 251 | 263 | |
| 264 | + sys_info(hardlockup_si_mask & ~SYS_INFO_ALL_BT); | |
| 252 | 265 | if (hardlockup_panic) |
| 253 | 266 | nmi_panic(regs, "Hard LOCKUP"); |
| 254 | 267 | |
| @@ -339,6 +352,13 @@ static void lockup_detector_update_enable(void) · 84 unmodified lines | ||
| 339 | 352 | int __read_mostly sysctl_softlockup_all_cpu_backtrace; |
| 340 | 353 | #endif |
| 341 | 354 | |
| 355 | +/* | |
| 356 | + * bitmasks to control what kinds of system info to be printed when | |
| 357 | + * soft lockup is detected, it could be task, memory, lock etc. | |
| 358 | + * Refer include/linux/sys_info.h for detailed bit definition. | |
| 359 | + */ | |
| 360 | +static unsigned long softlockup_si_mask; | |
| 361 | + | |
| 342 | 362 | static struct cpumask watchdog_allowed_mask __read_mostly; |
| 343 | 363 | |
| 344 | 364 | /* Global variables, exported for sysctl */ |
| @@ -755,7 +775,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) · 410 unmodified lines | ||
| 755 | 775 | unsigned long touch_ts, period_ts, now; |
| 756 | 776 | struct pt_regs *regs = get_irq_regs(); |
| 757 | 777 | int duration; |
| 758 | − int softlockup_all_cpu_backtrace = sysctl_softlockup_all_cpu_backtrace; | |
| 778 | + int softlockup_all_cpu_backtrace; | |
| 759 | 779 | unsigned long flags; |
| 760 | 780 | |
| 761 | 781 | if (!watchdog_enabled) |
| @@ -767,6 +787,9 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) · 5 unmodified lines | ||
| 767 | 787 | if (panic_in_progress()) |
| 768 | 788 | return HRTIMER_NORESTART; |
| 769 | 789 | |
| 790 | + softlockup_all_cpu_backtrace = (softlockup_si_mask & SYS_INFO_ALL_BT) ? | |
| 791 | + 1 : sysctl_softlockup_all_cpu_backtrace; | |
| 792 | + | |
| 770 | 793 | watchdog_hardlockup_kick(); |
| 771 | 794 | |
| 772 | 795 | /* kick the softlockup detector */ |
| @@ -855,6 +878,7 @@ static enum hrtimer_restart watchdog_timer_fn(struct hrtimer *hrtimer) · 82 unmodified lines | ||
| 855 | 878 | } |
| 856 | 879 | |
| 857 | 880 | add_taint(TAINT_SOFTLOCKUP, LOCKDEP_STILL_OK); |
| 881 | + sys_info(softlockup_si_mask & ~SYS_INFO_ALL_BT); | |
| 858 | 882 | if (softlockup_panic) |
| 859 | 883 | panic("softlockup: hung tasks"); |
| 860 | 884 | } |
| @@ -1206,6 +1230,13 @@ static const struct ctl_table watchdog_sysctls[] = { · 345 unmodified lines | ||
| 1206 | 1230 | .extra1 = SYSCTL_ZERO, |
| 1207 | 1231 | .extra2 = SYSCTL_ONE, |
| 1208 | 1232 | }, |
| 1233 | + { | |
| 1234 | + .procname = "softlockup_sys_info", | |
| 1235 | + .data = &softlockup_si_mask, | |
| 1236 | + .maxlen = sizeof(softlockup_si_mask), | |
| 1237 | + .mode = 0644, | |
| 1238 | + .proc_handler = sysctl_sys_info_handler, | |
| 1239 | + }, | |
| 1209 | 1240 | #ifdef CONFIG_SMP |
| 1210 | 1241 | { |
| 1211 | 1242 | .procname = "softlockup_all_cpu_backtrace", |
| @@ -1228,6 +1259,13 @@ static const struct ctl_table watchdog_sysctls[] = { · 16 unmodified lines | ||
| 1228 | 1259 | .extra1 = SYSCTL_ZERO, |
| 1229 | 1260 | .extra2 = SYSCTL_ONE, |
| 1230 | 1261 | }, |
| 1262 | + { | |
| 1263 | + .procname = "hardlockup_sys_info", | |
| 1264 | + .data = &hardlockup_si_mask, | |
| 1265 | + .maxlen = sizeof(hardlockup_si_mask), | |
| 1266 | + .mode = 0644, | |
| 1267 | + .proc_handler = sysctl_sys_info_handler, | |
| 1268 | + }, | |
| 1231 | 1269 | #ifdef CONFIG_SMP |
| 1232 | 1270 | { |
| 1233 | 1271 | .procname = "hardlockup_all_cpu_backtrace", |
| @@ -342,8 +342,7 @@ config DEBUG_INFO_COMPRESSED_ZLIB · 341 unmodified lines | ||
| 342 | 342 | depends on $(cc-option,-gz=zlib) |
| 343 | 343 | depends on $(ld-option,--compress-debug-sections=zlib) |
| 344 | 344 | help |
| 345 | − Compress the debug information using zlib. Requires GCC 5.0+ or Clang | |
| 346 | − 5.0+, binutils 2.26+, and zlib. | |
| 345 | + Compress the debug information using zlib. | |
| 347 | 346 | |
| 348 | 347 | Users of dpkg-deb via debian/rules may find an increase in |
| 349 | 348 | size of their debug .deb packages with this config set, due to the |
| @@ -493,23 +492,23 @@ config DEBUG_SECTION_MISMATCH · 143 unmodified lines | ||
| 493 | 492 | bool "Enable full Section mismatch analysis" |
| 494 | 493 | depends on CC_IS_GCC |
| 495 | 494 | help |
| 496 | − The section mismatch analysis checks if there are illegal | |
| 497 | − references from one section to another section. | |
| 498 | − During linktime or runtime, some sections are dropped; | |
| 499 | − any use of code/data previously in these sections would | |
| 500 | − most likely result in an oops. | |
| 501 | − In the code, functions and variables are annotated with | |
| 502 | − __init,, etc. (see the full list in include/linux/init.h), | |
| 503 | − which results in the code/data being placed in specific sections. | |
| 495 | + The section mismatch analysis checks if there are illegal references | |
| 496 | + from one section to another. During linktime or runtime, some | |
| 497 | + sections are dropped; any use of code/data previously in these | |
| 498 | + sections would most likely result in an oops. | |
| 499 | + | |
| 500 | + In the code, functions and variables are annotated with __init, | |
| 501 | + __initdata, and so on (see the full list in include/linux/init.h). | |
| 502 | + This directs the toolchain to place code/data in specific sections. | |
| 503 | + | |
| 504 | 504 | The section mismatch analysis is always performed after a full |
| 505 | − kernel build, and enabling this option causes the following | |
| 506 | − additional step to occur: | |
| 507 | − - Add the option -fno-inline-functions-called-once to gcc commands. | |
| 508 | − When inlining a function annotated with __init in a non-init | |
| 509 | − function, we would lose the section information and thus | |
| 510 | − the analysis would not catch the illegal reference. | |
| 511 | − This option tells gcc to inline less (but it does result in | |
| 512 | − a larger kernel). | |
| 505 | + kernel build, and enabling this option causes the option | |
| 506 | + -fno-inline-functions-called-once to be added to gcc commands. | |
| 507 | + | |
| 508 | + However, when inlining a function annotated with __init in | |
| 509 | + a non-init function, we would lose the section information and thus | |
| 510 | + the analysis would not catch the illegal reference. This option | |
| 511 | + tells gcc to inline less (but it does result in a larger kernel). | |
| 513 | 512 | |
| 514 | 513 | config SECTION_MISMATCH_WARN_ONLY |
| 515 | 514 | bool "Make section mismatch errors non-fatal" |
| @@ -1260,12 +1259,13 @@ config DEFAULT_HUNG_TASK_TIMEOUT · 744 unmodified lines | ||
| 1260 | 1259 | Keeping the default should be fine in most cases. |
| 1261 | 1260 | |
| 1262 | 1261 | config BOOTPARAM_HUNG_TASK_PANIC |
| 1263 | − bool "Panic (Reboot) On Hung Tasks" | |
| 1262 | + int "Number of hung tasks to trigger kernel panic" | |
| 1264 | 1263 | depends on DETECT_HUNG_TASK |
| 1264 | + default 0 | |
| 1265 | 1265 | help |
| 1266 | − Say Y here to enable the kernel to panic on "hung tasks", | |
| 1267 | − which are bugs that cause the kernel to leave a task stuck | |
| 1268 | − in uninterruptible "D" state. | |
| 1266 | + When set to a non-zero value, a kernel panic will be triggered | |
| 1267 | + if the number of hung tasks found during a single scan reaches | |
| 1268 | + this value. | |
| 1269 | 1269 | |
| 1270 | 1270 | The panic can be used in combination with panic_timeout, |
| 1271 | 1271 | to cause the system to reboot automatically after a |
| @@ -2817,8 +2817,25 @@ config CMDLINE_KUNIT_TEST · 1545 unmodified lines | ||
| 2817 | 2817 | |
| 2818 | 2818 | If unsure, say N. |
| 2819 | 2819 | |
| 2820 | +config BASE64_KUNIT | |
| 2821 | + tristate "KUnit test for base64 decoding and encoding" if !KUNIT_ALL_TESTS | |
| 2822 | + depends on KUNIT | |
| 2823 | + default KUNIT_ALL_TESTS | |
| 2824 | + help | |
| 2825 | + This builds the base64 unit tests. | |
| 2826 | + | |
| 2827 | + The tests cover the encoding and decoding logic of Base64 functions | |
| 2828 | + in the kernel. | |
| 2829 | + In addition to correctness checks, simple performance benchmarks | |
| 2830 | + for both encoding and decoding are also included. | |
| 2831 | + | |
| 2832 | + For more information on KUnit and unit tests in general please refer | |
| 2833 | + to the KUnit documentation in Documentation/dev-tools/kunit/. | |
| 2834 | + | |
| 2835 | + If unsure, say N. | |
| 2836 | + | |
| 2820 | 2837 | config BITS_TEST |
| 2821 | − tristate "KUnit test for bits.h" if !KUNIT_ALL_TESTS | |
| 2838 | + tristate "KUnit test for bit functions and macros" if !KUNIT_ALL_TESTS | |
| 2822 | 2839 | depends on KUNIT |
| 2823 | 2840 | default KUNIT_ALL_TESTS |
| 2824 | 2841 | help |
| @@ -1,12 +1,12 @@ | ||
| 1 | 1 | // SPDX-License-Identifier: GPL-2.0 |
| 2 | 2 | /* |
| 3 | − * base64.c - RFC4648-compliant base64 encoding | |
| 3 | + * base64.c - Base64 with support for multiple variants | |
| 4 | 4 | * |
| 5 | 5 | * Copyright (c) 2020 Hannes Reinecke, SUSE |
| 6 | 6 | * |
| 7 | 7 | * Based on the base64url routines from fs/crypto/fname.c |
| 8 | − * (which are using the URL-safe base64 encoding), | |
| 9 | − * modified to use the standard coding table from RFC4648 section 4. | |
| 8 | + * (which are using the URL-safe Base64 encoding), | |
| 9 | + * modified to support multiple Base64 variants. | |
| 10 | 10 | */ |
| 11 | 11 | |
| 12 | 12 | #include <linux/kernel.h> |
| @@ -15,89 +15,170 @@ · 2 unmodified lines | ||
| 15 | 15 | #include <linux/string.h> |
| 16 | 16 | #include <linux/base64.h> |
| 17 | 17 | |
| 18 | −static const char base64_table[65] = | |
| 19 | − "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; | |
| 18 | +static const char base64_tables[][65] = { | |
| 19 | + [BASE64_STD] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/", | |
| 20 | + [BASE64_URLSAFE] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_", | |
| 21 | + [BASE64_IMAP] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+,", | |
| 22 | +}; | |
| 20 | 23 | |
| 24 | +/* | |
| 25 | + * Initialize the base64 reverse mapping for a single character | |
| 26 | + * This macro maps a character to its corresponding base64 value, | |
| 27 | + * returning -1 if the character is invalid. | |
| 28 | + * char 'A'-'Z' maps to 0-25, 'a'-'z' maps to 26-51, '0'-'9' maps to 52-61, | |
| 29 | + * ch_62 maps to 62, ch_63 maps to 63, and other characters return -1 | |
| 30 | + */ | |
| 31 | +#define INIT_1(v, ch_62, ch_63) \ | |
| 32 | + [v] = (v) >= 'A' && (v) <= 'Z' ? (v) - 'A' \ | |
| 33 | + : (v) >= 'a' && (v) <= 'z' ? (v) - 'a' + 26 \ | |
| 34 | + : (v) >= '0' && (v) <= '9' ? (v) - '0' + 52 \ | |
| 35 | + : (v) == (ch_62) ? 62 : (v) == (ch_63) ? 63 : -1 | |
| 36 | + | |
| 37 | +/* | |
| 38 | + * Recursive macros to generate multiple Base64 reverse mapping table entries. | |
| 39 | + * Each macro generates a sequence of entries in the lookup table: | |
| 40 | + * INIT_2 generates 2 entries, INIT_4 generates 4, INIT_8 generates 8, and so on up to INIT_32. | |
| 41 | + */ | |
| 42 | +#define INIT_2(v, ...) INIT_1(v, __VA_ARGS__), INIT_1((v) + 1, __VA_ARGS__) | |
| 43 | +#define INIT_4(v, ...) INIT_2(v, __VA_ARGS__), INIT_2((v) + 2, __VA_ARGS__) | |
| 44 | +#define INIT_8(v, ...) INIT_4(v, __VA_ARGS__), INIT_4((v) + 4, __VA_ARGS__) | |
| 45 | +#define INIT_16(v, ...) INIT_8(v, __VA_ARGS__), INIT_8((v) + 8, __VA_ARGS__) | |
| 46 | +#define INIT_32(v, ...) INIT_16(v, __VA_ARGS__), INIT_16((v) + 16, __VA_ARGS__) | |
| 47 | + | |
| 48 | +#define BASE64_REV_INIT(ch_62, ch_63) { \ | |
| 49 | + [0 ... 0x1f] = -1, \ | |
| 50 | + INIT_32(0x20, ch_62, ch_63), \ | |
| 51 | + INIT_32(0x40, ch_62, ch_63), \ | |
| 52 | + INIT_32(0x60, ch_62, ch_63), \ | |
| 53 | + [0x80 ... 0xff] = -1 } | |
| 54 | + | |
| 55 | +static const s8 base64_rev_maps[][256] = { | |
| 56 | + [BASE64_STD] = BASE64_REV_INIT('+', '/'), | |
| 57 | + [BASE64_URLSAFE] = BASE64_REV_INIT('-', '_'), | |
| 58 | + [BASE64_IMAP] = BASE64_REV_INIT('+', ',') | |
| 59 | +}; | |
| 60 | + | |
| 61 | +#undef BASE64_REV_INIT | |
| 62 | +#undef INIT_32 | |
| 63 | +#undef INIT_16 | |
| 64 | +#undef INIT_8 | |
| 65 | +#undef INIT_4 | |
| 66 | +#undef INIT_2 | |
| 67 | +#undef INIT_1 | |
| 21 | 68 | /** |
| 22 | − * base64_encode() - base64-encode some binary data | |
| 69 | + * base64_encode() - Base64-encode some binary data | |
| 23 | 70 | * @src: the binary data to encode |
| 24 | 71 | * @srclen: the length of @src in bytes |
| 25 | − * @dst: (output) the base64-encoded string. Not NUL-terminated. | |
| 72 | + * @dst: (output) the Base64-encoded string. Not NUL-terminated. | |
| 73 | + * @padding: whether to append '=' padding characters | |
| 74 | + * @variant: which base64 variant to use | |
| 26 | 75 | * |
| 27 | − * Encodes data using base64 encoding, i.e. the "Base 64 Encoding" specified | |
| 28 | − * by RFC 4648, including the '='-padding. | |
| 76 | + * Encodes data using the selected Base64 variant. | |
| 29 | 77 | * |
| 30 | − * Return: the length of the resulting base64-encoded string in bytes. | |
| 78 | + * Return: the length of the resulting Base64-encoded string in bytes. | |
| 31 | 79 | */ |
| 32 | −int base64_encode(const u8 *src, int srclen, char *dst) | |
| 80 | +int base64_encode(const u8 *src, int srclen, char *dst, bool padding, enum base64_variant variant) | |
| 33 | 81 | { |
| 34 | 82 | u32 ac = 0; |
| 35 | − int bits = 0; | |
| 36 | − int i; | |
| 37 | 83 | char *cp = dst; |
| 84 | + const char *base64_table = base64_tables[variant]; | |
| 38 | 85 | |
| 39 | − for (i = 0; i < srclen; i++) { | |
| 40 | − ac = (ac << 8) | src[i]; | |
| 41 | − bits += 8; | |
| 42 | − do { | |
| 43 | − bits -= 6; | |
| 44 | − *cp++ = base64_table[(ac >> bits) & 0x3f]; | |
| 45 | − } while (bits >= 6); | |
| 46 | − } | |
| 47 | − if (bits) { | |
| 48 | − *cp++ = base64_table[(ac << (6 - bits)) & 0x3f]; | |
| 49 | − bits -= 6; | |
| 86 | + while (srclen >= 3) { | |
| 87 | + ac = src[0] << 16 | src[1] << 8 | src[2]; | |
| 88 | + *cp++ = base64_table[ac >> 18]; | |
| 89 | + *cp++ = base64_table[(ac >> 12) & 0x3f]; | |
| 90 | + *cp++ = base64_table[(ac >> 6) & 0x3f]; | |
| 91 | + *cp++ = base64_table[ac & 0x3f]; | |
| 92 | + | |
| 93 | + src += 3; | |
| 94 | + srclen -= 3; | |
| 50 | 95 | } |
| 51 | − while (bits < 0) { | |
| 52 | − *cp++ = '='; | |
| 53 | − bits += 2; | |
| 96 | + | |
| 97 | + switch (srclen) { | |
| 98 | + case 2: | |
| 99 | + ac = src[0] << 16 | src[1] << 8; | |
| 100 | + *cp++ = base64_table[ac >> 18]; | |
| 101 | + *cp++ = base64_table[(ac >> 12) & 0x3f]; | |
| 102 | + *cp++ = base64_table[(ac >> 6) & 0x3f]; | |
| 103 | + if (padding) | |
| 104 | + *cp++ = '='; | |
| 105 | + break; | |
| 106 | + case 1: | |
| 107 | + ac = src[0] << 16; | |
| 108 | + *cp++ = base64_table[ac >> 18]; | |
| 109 | + *cp++ = base64_table[(ac >> 12) & 0x3f]; | |
| 110 | + if (padding) { | |
| 111 | + *cp++ = '='; | |
| 112 | + *cp++ = '='; | |
| 113 | + } | |
| 114 | + break; | |
| 54 | 115 | } |
| 55 | 116 | return cp - dst; |
| 56 | 117 | } |
| 57 | 118 | EXPORT_SYMBOL_GPL(base64_encode); |
| 58 | 119 | |
| 59 | 120 | /** |
| 60 | − * base64_decode() - base64-decode a string | |
| 121 | + * base64_decode() - Base64-decode a string | |
| 61 | 122 | * @src: the string to decode. Doesn't need to be NUL-terminated. |
| 62 | 123 | * @srclen: the length of @src in bytes |
| 63 | 124 | * @dst: (output) the decoded binary data |
| 125 | + * @padding: whether to append '=' padding characters | |
| 126 | + * @variant: which base64 variant to use | |
| 64 | 127 | * |
| 65 | − * Decodes a string using base64 encoding, i.e. the "Base 64 Encoding" | |
| 66 | − * specified by RFC 4648, including the '='-padding. | |
| 67 | − * | |
| 68 | − * This implementation hasn't been optimized for performance. | |
| 128 | + * Decodes a string using the selected Base64 variant. | |
| 69 | 129 | * |
| 70 | 130 | * Return: the length of the resulting decoded binary data in bytes, |
| 71 | − * or -1 if the string isn't a valid base64 string. | |
| 131 | + * or -1 if the string isn't a valid Base64 string. | |
| 72 | 132 | */ |
| 73 | −int base64_decode(const char *src, int srclen, u8 *dst) | |
| 133 | +int base64_decode(const char *src, int srclen, u8 *dst, bool padding, enum base64_variant variant) | |
| 74 | 134 | { |
| 75 | − u32 ac = 0; | |
| 76 | − int bits = 0; | |
| 77 | − int i; | |
| 78 | 135 | u8 *bp = dst; |
| 136 | + s8 input[4]; | |
| 137 | + s32 val; | |
| 138 | + const u8 *s = (const u8 *)src; | |
| 139 | + const s8 *base64_rev_tables = base64_rev_maps[variant]; | |
| 79 | 140 | |
| 80 | − for (i = 0; i < srclen; i++) { | |
| 81 | − const char *p = strchr(base64_table, src[i]); | |
| 141 | + while (srclen >= 4) { | |
| 142 | + input[0] = base64_rev_tables[s[0]]; | |
| 143 | + input[1] = base64_rev_tables[s[1]]; | |
| 144 | + input[2] = base64_rev_tables[s[2]]; | |
| 145 | + input[3] = base64_rev_tables[s[3]]; | |
| 82 | 146 | |
| 83 | − if (src[i] == '=') { | |
| 84 | − ac = (ac << 6); | |
| 85 | − bits += 6; | |
| 86 | − if (bits >= 8) | |
| 87 | − bits -= 8; | |
| 88 | − continue; | |
| 89 | − } | |
| 90 | − if (p == NULL || src[i] == 0) | |
| 91 | − return -1; | |
| 92 | − ac = (ac << 6) | (p - base64_table); | |
| 93 | − bits += 6; | |
| 94 | − if (bits >= 8) { | |
| 95 | − bits -= 8; | |
| 96 | − *bp++ = (u8)(ac >> bits); | |
| 147 | + val = input[0] << 18 | input[1] << 12 | input[2] << 6 | input[3]; | |
| 148 | + | |
| 149 | + if (unlikely(val < 0)) { | |
| 150 | + if (!padding || srclen != 4 || s[3] != '=') | |
| 151 | + return -1; | |
| 152 | + padding = 0; | |
| 153 | + srclen = s[2] == '=' ? 2 : 3; | |
| 154 | + break; | |
| 97 | 155 | } |
| 156 | + | |
| 157 | + *bp++ = val >> 16; | |
| 158 | + *bp++ = val >> 8; | |
| 159 | + *bp++ = val; | |
| 160 | + | |
| 161 | + s += 4; | |
| 162 | + srclen -= 4; | |
| 98 | 163 | } |
| 99 | − if (ac & ((1 << bits) - 1)) | |
| 164 | + | |
| 165 | + if (likely(!srclen)) | |
| 166 | + return bp - dst; | |
| 167 | + if (padding || srclen == 1) | |
| 100 | 168 | return -1; |
| 169 | + | |
| 170 | + val = (base64_rev_tables[s[0]] << 12) | (base64_rev_tables[s[1]] << 6); | |
| 171 | + *bp++ = val >> 10; | |
| 172 | + | |
| 173 | + if (srclen == 2) { | |
| 174 | + if (val & 0x800003ff) | |
| 175 | + return -1; | |
| 176 | + } else { | |
| 177 | + val |= base64_rev_tables[s[2]]; | |
| 178 | + if (val & 0x80000003) | |
| 179 | + return -1; | |
| 180 | + *bp++ = val >> 2; | |
| 181 | + } | |
| 101 | 182 | return bp - dst; |
| 102 | 183 | } |
| 103 | 184 | EXPORT_SYMBOL_GPL(base64_decode); |
| @@ -95,6 +95,7 @@ static const struct { unsigned flag:8; char opt_char; } opt_array[] = { · 94 unmodified lines | ||
| 95 | 95 | { _DPRINTK_FLAGS_INCL_SOURCENAME, 's' }, |
| 96 | 96 | { _DPRINTK_FLAGS_INCL_LINENO, 'l' }, |
| 97 | 97 | { _DPRINTK_FLAGS_INCL_TID, 't' }, |
| 98 | + { _DPRINTK_FLAGS_INCL_STACK, 'd' }, | |
| 98 | 99 | { _DPRINTK_FLAGS_NONE, '_' }, |
| 99 | 100 | }; |
| 100 | 101 | |
| @@ -177,94 +177,157 @@ EXPORT_SYMBOL(div64_s64); · 176 unmodified lines | ||
| 177 | 177 | * Iterative div/mod for use when dividend is not expected to be much |
| 178 | 178 | * bigger than divisor. |
| 179 | 179 | */ |
| 180 | +#ifndef iter_div_u64_rem | |
| 180 | 181 | u32 iter_div_u64_rem(u64 dividend, u32 divisor, u64 *remainder) |
| 181 | 182 | { |
| 182 | 183 | return __iter_div_u64_rem(dividend, divisor, remainder); |
| 183 | 184 | } |
| 184 | 185 | EXPORT_SYMBOL(iter_div_u64_rem); |
| 186 | +#endif | |
| 185 | 187 | |
| 186 | −#ifndef mul_u64_u64_div_u64 | |
| 187 | −u64 mul_u64_u64_div_u64(u64 a, u64 b, u64 c) | |
| 188 | −{ | |
| 189 | − if (ilog2(a) + ilog2(b) <= 62) | |
| 190 | − return div64_u64(a * b, c); | |
| 188 | +#if !defined(mul_u64_add_u64_div_u64) || defined(test_mul_u64_add_u64_div_u64) | |
| 191 | 189 | |
| 192 | −#if defined(__SIZEOF_INT128__) | |
| 190 | +#define mul_add(a, b, c) add_u64_u32(mul_u32_u32(a, b), c) | |
| 193 | 191 | |
| 192 | +#if defined(__SIZEOF_INT128__) && !defined(test_mul_u64_add_u64_div_u64) | |
| 193 | +static inline u64 mul_u64_u64_add_u64(u64 *p_lo, u64 a, u64 b, u64 c) | |
| 194 | +{ | |
| 194 | 195 | /* native 64x64=128 bits multiplication */ |
| 195 | − u128 prod = (u128)a * b; | |
| 196 | − u64 n_lo = prod, n_hi = prod >> 64; | |
| 196 | + u128 prod = (u128)a * b + c; | |
| 197 | 197 | |
| 198 | + *p_lo = prod; | |
| 199 | + return prod >> 64; | |
| 200 | +} | |
| 198 | 201 | #else |
| 199 | − | |
| 200 | − /* perform a 64x64=128 bits multiplication manually */ | |
| 201 | − u32 a_lo = a, a_hi = a >> 32, b_lo = b, b_hi = b >> 32; | |
| 202 | +static inline u64 mul_u64_u64_add_u64(u64 *p_lo, u64 a, u64 b, u64 c) | |
| 203 | +{ | |
| 204 | + /* perform a 64x64=128 bits multiplication in 32bit chunks */ | |
| 202 | 205 | u64 x, y, z; |
| 203 | 206 | |
| 204 | − x = (u64)a_lo * b_lo; | |
| 205 | − y = (u64)a_lo * b_hi + (u32)(x >> 32); | |
| 206 | − z = (u64)a_hi * b_hi + (u32)(y >> 32); | |
| 207 | − y = (u64)a_hi * b_lo + (u32)y; | |
| 208 | − z += (u32)(y >> 32); | |
| 209 | − x = (y << 32) + (u32)x; | |
| 210 | − | |
| 211 | − u64 n_lo = x, n_hi = z; | |
| 207 | + /* Since (x-1)(x-1) + 2(x-1) == x.x - 1 two u32 can be added to a u64 */ | |
| 208 | + x = mul_add(a, b, c); | |
| 209 | + y = mul_add(a, b >> 32, c >> 32); | |
| 210 | + y = add_u64_u32(y, x >> 32); | |
| 211 | + z = mul_add(a >> 32, b >> 32, y >> 32); | |
| 212 | + y = mul_add(a >> 32, b, y); | |
| 213 | + *p_lo = (y << 32) + (u32)x; | |
| 214 | + return add_u64_u32(z, y >> 32); | |
| 215 | +} | |
| 216 | +#endif | |
| 212 | 217 | |
| 218 | +#ifndef BITS_PER_ITER | |
| 219 | +#define BITS_PER_ITER (__LONG_WIDTH__ >= 64 ? 32 : 16) | |
| 213 | 220 | #endif |
| 214 | 221 | |
| 215 | − /* make sure c is not zero, trigger runtime exception otherwise */ | |
| 216 | − if (unlikely(c == 0)) { | |
| 217 | − unsigned long zero = 0; | |
| 222 | +#if BITS_PER_ITER == 32 | |
| 223 | +#define mul_u64_long_add_u64(p_lo, a, b, c) mul_u64_u64_add_u64(p_lo, a, b, c) | |
| 224 | +#define add_u64_long(a, b) ((a) + (b)) | |
| 225 | +#else | |
| 226 | +#undef BITS_PER_ITER | |
| 227 | +#define BITS_PER_ITER 16 | |
| 228 | +static inline u32 mul_u64_long_add_u64(u64 *p_lo, u64 a, u32 b, u64 c) | |
| 229 | +{ | |
| 230 | + u64 n_lo = mul_add(a, b, c); | |
| 231 | + u64 n_med = mul_add(a >> 32, b, c >> 32); | |
| 218 | 232 | |
| 219 | − OPTIMIZER_HIDE_VAR(zero); | |
| 220 | − return ~0UL/zero; | |
| 221 | − } | |
| 233 | + n_med = add_u64_u32(n_med, n_lo >> 32); | |
| 234 | + *p_lo = n_med << 32 | (u32)n_lo; | |
| 235 | + return n_med >> 32; | |
| 236 | +} | |
| 237 | + | |
| 238 | +#define add_u64_long(a, b) add_u64_u32(a, b) | |
| 239 | +#endif | |
| 240 | + | |
| 241 | +u64 mul_u64_add_u64_div_u64(u64 a, u64 b, u64 c, u64 d) | |
| 242 | +{ | |
| 243 | + unsigned long d_msig, q_digit; | |
| 244 | + unsigned int reps, d_z_hi; | |
| 245 | + u64 quotient, n_lo, n_hi; | |
| 246 | + u32 overflow; | |
| 222 | 247 | |
| 223 | − int shift = __builtin_ctzll(c); | |
| 248 | + n_hi = mul_u64_u64_add_u64(&n_lo, a, b, c); | |
| 224 | 249 | |
| 225 | − /* try reducing the fraction in case the dividend becomes <= 64 bits */ | |
| 226 | − if ((n_hi >> shift) == 0) { | |
| 227 | − u64 n = shift ? (n_lo >> shift) | (n_hi << (64 - shift)) : n_lo; | |
| 250 | + if (!n_hi) | |
| 251 | + return div64_u64(n_lo, d); | |
| 228 | 252 | |
| 229 | − return div64_u64(n, c >> shift); | |
| 230 | − /* | |
| 231 | − * The remainder value if needed would be: | |
| 232 | − * res = div64_u64_rem(n, c >> shift, &rem); | |
| 233 | − * rem = (rem << shift) + (n_lo - (n << shift)); | |
| 234 | − */ | |
| 235 | − } | |
| 253 | + if (unlikely(n_hi >= d)) { | |
| 254 | + /* trigger runtime exception if divisor is zero */ | |
| 255 | + if (d == 0) { | |
| 256 | + unsigned long zero = 0; | |
| 236 | 257 | |
| 237 | − if (n_hi >= c) { | |
| 258 | + OPTIMIZER_HIDE_VAR(zero); | |
| 259 | + return ~0UL/zero; | |
| 260 | + } | |
| 238 | 261 | /* overflow: result is unrepresentable in a u64 */ |
| 239 | − return -1; | |
| 262 | + return ~0ULL; | |
| 240 | 263 | } |
| 241 | 264 | |
| 242 | − /* Do the full 128 by 64 bits division */ | |
| 243 | − | |
| 244 | − shift = __builtin_clzll(c); | |
| 245 | − c <<= shift; | |
| 265 | + /* Left align the divisor, shifting the dividend to match */ | |
| 266 | + d_z_hi = __builtin_clzll(d); | |
| 267 | + if (d_z_hi) { | |
| 268 | + d <<= d_z_hi; | |
| 269 | + n_hi = n_hi << d_z_hi | n_lo >> (64 - d_z_hi); | |
| 270 | + n_lo <<= d_z_hi; | |
| 271 | + } | |
| 246 | 272 | |
| 247 | − int p = 64 + shift; | |
| 248 | − u64 res = 0; | |
| 249 | − bool carry; | |
| 273 | + reps = 64 / BITS_PER_ITER; | |
| 274 | + /* Optimise loop count for small dividends */ | |
| 275 | + if (!(u32)(n_hi >> 32)) { | |
| 276 | + reps -= 32 / BITS_PER_ITER; | |
| 277 | + n_hi = n_hi << 32 | n_lo >> 32; | |
| 278 | + n_lo <<= 32; | |
| 279 | + } | |
| 280 | +#if BITS_PER_ITER == 16 | |
| 281 | + if (!(u32)(n_hi >> 48)) { | |
| 282 | + reps--; | |
| 283 | + n_hi = add_u64_u32(n_hi << 16, n_lo >> 48); | |
| 284 | + n_lo <<= 16; | |
| 285 | + } | |
| 286 | +#endif | |
| 250 | 287 | |
| 251 | − do { | |
| 252 | − carry = n_hi >> 63; | |
| 253 | − shift = carry ? 1 : __builtin_clzll(n_hi); | |
| 254 | − if (p < shift) | |
| 255 | − break; | |
| 256 | − p -= shift; | |
| 257 | − n_hi <<= shift; | |
| 258 | − n_hi |= n_lo >> (64 - shift); | |
| 259 | − n_lo <<= shift; | |
| 260 | − if (carry || (n_hi >= c)) { | |
| 261 | − n_hi -= c; | |
| 262 | − res |= 1ULL << p; | |
| 288 | + /* Invert the dividend so we can use add instead of subtract. */ | |
| 289 | + n_lo = ~n_lo; | |
| 290 | + n_hi = ~n_hi; | |
| 291 | + | |
| 292 | + /* | |
| 293 | + * Get the most significant BITS_PER_ITER bits of the divisor. | |
| 294 | + * This is used to get a low 'guestimate' of the quotient digit. | |
| 295 | + */ | |
| 296 | + d_msig = (d >> (64 - BITS_PER_ITER)) + 1; | |
| 297 | + | |
| 298 | + /* | |
| 299 | + * Now do a 'long division' with BITS_PER_ITER bit 'digits'. | |
| 300 | + * The 'guess' quotient digit can be low and BITS_PER_ITER+1 bits. | |
| 301 | + * The worst case is dividing ~0 by 0x8000 which requires two subtracts. | |
| 302 | + */ | |
| 303 | + quotient = 0; | |
| 304 | + while (reps--) { | |
| 305 | + q_digit = (unsigned long)(~n_hi >> (64 - 2 * BITS_PER_ITER)) / d_msig; | |
| 306 | + /* Shift 'n' left to align with the product q_digit * d */ | |
| 307 | + overflow = n_hi >> (64 - BITS_PER_ITER); | |
| 308 | + n_hi = add_u64_u32(n_hi << BITS_PER_ITER, n_lo >> (64 - BITS_PER_ITER)); | |
| 309 | + n_lo <<= BITS_PER_ITER; | |
| 310 | + /* Add product to negated divisor */ | |
| 311 | + overflow += mul_u64_long_add_u64(&n_hi, d, q_digit, n_hi); | |
| 312 | + /* Adjust for the q_digit 'guestimate' being low */ | |
| 313 | + while (overflow < 0xffffffff >> (32 - BITS_PER_ITER)) { | |
| 314 | + q_digit++; | |
| 315 | + n_hi += d; | |
| 316 | + overflow += n_hi < d; | |
| 263 | 317 | } |
| 264 | − } while (n_hi); | |
| 265 | − /* The remainder value if needed would be n_hi << p */ | |
| 318 | + quotient = add_u64_long(quotient << BITS_PER_ITER, q_digit); | |
| 319 | + } | |
| 266 | 320 | |
| 267 | − return res; | |
| 321 | + /* | |
| 322 | + * The above only ensures the remainder doesn't overflow, | |
| 323 | + * it can still be possible to add (aka subtract) another copy | |
| 324 | + * of the divisor. | |
| 325 | + */ | |
| 326 | + if ((n_hi + d) > n_hi) | |
| 327 | + quotient++; | |
| 328 | + return quotient; | |
| 268 | 329 | } |
| 269 | −EXPORT_SYMBOL(mul_u64_u64_div_u64); | |
| 330 | +#if !defined(test_mul_u64_add_u64_div_u64) | |
| 331 | +EXPORT_SYMBOL(mul_u64_add_u64_div_u64); | |
| 332 | +#endif | |
| 270 | 333 | #endif |
| @@ -10,80 +10,141 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <linux/printk.h> |
| 11 | 11 | #include <linux/math64.h> |
| 12 | 12 | |
| 13 | −typedef struct { u64 a; u64 b; u64 c; u64 result; } test_params; | |
| 13 | +typedef struct { u64 a; u64 b; u64 d; u64 result; uint round_up;} test_params; | |
| 14 | 14 | |
| 15 | 15 | static test_params test_values[] = { |
| 16 | 16 | /* this contains many edge values followed by a couple random values */ |
| 17 | −{ 0xb, 0x7, 0x3, 0x19 }, | |
| 18 | −{ 0xffff0000, 0xffff0000, 0xf, 0x1110eeef00000000 }, | |
| 19 | −{ 0xffffffff, 0xffffffff, 0x1, 0xfffffffe00000001 }, | |
| 20 | −{ 0xffffffff, 0xffffffff, 0x2, 0x7fffffff00000000 }, | |
| 21 | −{ 0x1ffffffff, 0xffffffff, 0x2, 0xfffffffe80000000 }, | |
| 22 | −{ 0x1ffffffff, 0xffffffff, 0x3, 0xaaaaaaa9aaaaaaab }, | |
| 23 | −{ 0x1ffffffff, 0x1ffffffff, 0x4, 0xffffffff00000000 }, | |
| 24 | −{ 0xffff000000000000, 0xffff000000000000, 0xffff000000000001, 0xfffeffffffffffff }, | |
| 25 | −{ 0x3333333333333333, 0x3333333333333333, 0x5555555555555555, 0x1eb851eb851eb851 }, | |
| 26 | −{ 0x7fffffffffffffff, 0x2, 0x3, 0x5555555555555554 }, | |
| 27 | −{ 0xffffffffffffffff, 0x2, 0x8000000000000000, 0x3 }, | |
| 28 | −{ 0xffffffffffffffff, 0x2, 0xc000000000000000, 0x2 }, | |
| 29 | −{ 0xffffffffffffffff, 0x4000000000000004, 0x8000000000000000, 0x8000000000000007 }, | |
| 30 | −{ 0xffffffffffffffff, 0x4000000000000001, 0x8000000000000000, 0x8000000000000001 }, | |
| 31 | −{ 0xffffffffffffffff, 0x8000000000000001, 0xffffffffffffffff, 0x8000000000000001 }, | |
| 32 | −{ 0xfffffffffffffffe, 0x8000000000000001, 0xffffffffffffffff, 0x8000000000000000 }, | |
| 33 | −{ 0xffffffffffffffff, 0x8000000000000001, 0xfffffffffffffffe, 0x8000000000000001 }, | |
| 34 | −{ 0xffffffffffffffff, 0x8000000000000001, 0xfffffffffffffffd, 0x8000000000000002 }, | |
| 35 | −{ 0x7fffffffffffffff, 0xffffffffffffffff, 0xc000000000000000, 0xaaaaaaaaaaaaaaa8 }, | |
| 36 | −{ 0xffffffffffffffff, 0x7fffffffffffffff, 0xa000000000000000, 0xccccccccccccccca }, | |
| 37 | −{ 0xffffffffffffffff, 0x7fffffffffffffff, 0x9000000000000000, 0xe38e38e38e38e38b }, | |
| 38 | −{ 0x7fffffffffffffff, 0x7fffffffffffffff, 0x5000000000000000, 0xccccccccccccccc9 }, | |
| 39 | −{ 0xffffffffffffffff, 0xfffffffffffffffe, 0xffffffffffffffff, 0xfffffffffffffffe }, | |
| 40 | −{ 0xe6102d256d7ea3ae, 0x70a77d0be4c31201, 0xd63ec35ab3220357, 0x78f8bf8cc86c6e18 }, | |
| 41 | −{ 0xf53bae05cb86c6e1, 0x3847b32d2f8d32e0, 0xcfd4f55a647f403c, 0x42687f79d8998d35 }, | |
| 42 | −{ 0x9951c5498f941092, 0x1f8c8bfdf287a251, 0xa3c8dc5f81ea3fe2, 0x1d887cb25900091f }, | |
| 43 | −{ 0x374fee9daa1bb2bb, 0x0d0bfbff7b8ae3ef, 0xc169337bd42d5179, 0x03bb2dbaffcbb961 }, | |
| 44 | −{ 0xeac0d03ac10eeaf0, 0x89be05dfa162ed9b, 0x92bb1679a41f0e4b, 0xdc5f5cc9e270d216 }, | |
| 17 | +{ 0xb, 0x7, 0x3, 0x19, 1 }, | |
| 18 | +{ 0xffff0000, 0xffff0000, 0xf, 0x1110eeef00000000, 0 }, | |
| 19 | +{ 0xffffffff, 0xffffffff, 0x1, 0xfffffffe00000001, 0 }, | |
| 20 | +{ 0xffffffff, 0xffffffff, 0x2, 0x7fffffff00000000, 1 }, | |
| 21 | +{ 0x1ffffffff, 0xffffffff, 0x2, 0xfffffffe80000000, 1 }, | |
| 22 | +{ 0x1ffffffff, 0xffffffff, 0x3, 0xaaaaaaa9aaaaaaab, 0 }, | |
| 23 | +{ 0x1ffffffff, 0x1ffffffff, 0x4, 0xffffffff00000000, 1 }, | |
| 24 | +{ 0xffff000000000000, 0xffff000000000000, 0xffff000000000001, 0xfffeffffffffffff, 1 }, | |
| 25 | +{ 0x3333333333333333, 0x3333333333333333, 0x5555555555555555, 0x1eb851eb851eb851, 1 }, | |
| 26 | +{ 0x7fffffffffffffff, 0x2, 0x3, 0x5555555555555554, 1 }, | |
| 27 | +{ 0xffffffffffffffff, 0x2, 0x8000000000000000, 0x3, 1 }, | |
| 28 | +{ 0xffffffffffffffff, 0x2, 0xc000000000000000, 0x2, 1 }, | |
| 29 | +{ 0xffffffffffffffff, 0x4000000000000004, 0x8000000000000000, 0x8000000000000007, 1 }, | |
| 30 | +{ 0xffffffffffffffff, 0x4000000000000001, 0x8000000000000000, 0x8000000000000001, 1 }, | |
| 31 | +{ 0xffffffffffffffff, 0x8000000000000001, 0xffffffffffffffff, 0x8000000000000001, 0 }, | |
| 32 | +{ 0xfffffffffffffffe, 0x8000000000000001, 0xffffffffffffffff, 0x8000000000000000, 1 }, | |
| 33 | +{ 0xffffffffffffffff, 0x8000000000000001, 0xfffffffffffffffe, 0x8000000000000001, 1 }, | |
| 34 | +{ 0xffffffffffffffff, 0x8000000000000001, 0xfffffffffffffffd, 0x8000000000000002, 1 }, | |
| 35 | +{ 0x7fffffffffffffff, 0xffffffffffffffff, 0xc000000000000000, 0xaaaaaaaaaaaaaaa8, 1 }, | |
| 36 | +{ 0xffffffffffffffff, 0x7fffffffffffffff, 0xa000000000000000, 0xccccccccccccccca, 1 }, | |
| 37 | +{ 0xffffffffffffffff, 0x7fffffffffffffff, 0x9000000000000000, 0xe38e38e38e38e38b, 1 }, | |
| 38 | +{ 0x7fffffffffffffff, 0x7fffffffffffffff, 0x5000000000000000, 0xccccccccccccccc9, 1 }, | |
| 39 | +{ 0xffffffffffffffff, 0xfffffffffffffffe, 0xffffffffffffffff, 0xfffffffffffffffe, 0 }, | |
| 40 | +{ 0xe6102d256d7ea3ae, 0x70a77d0be4c31201, 0xd63ec35ab3220357, 0x78f8bf8cc86c6e18, 1 }, | |
| 41 | +{ 0xf53bae05cb86c6e1, 0x3847b32d2f8d32e0, 0xcfd4f55a647f403c, 0x42687f79d8998d35, 1 }, | |
| 42 | +{ 0x9951c5498f941092, 0x1f8c8bfdf287a251, 0xa3c8dc5f81ea3fe2, 0x1d887cb25900091f, 1 }, | |
| 43 | +{ 0x374fee9daa1bb2bb, 0x0d0bfbff7b8ae3ef, 0xc169337bd42d5179, 0x03bb2dbaffcbb961, 1 }, | |
| 44 | +{ 0xeac0d03ac10eeaf0, 0x89be05dfa162ed9b, 0x92bb1679a41f0e4b, 0xdc5f5cc9e270d216, 1 }, | |
| 45 | 45 | }; |
| 46 | 46 | |
| 47 | 47 | /* |
| 48 | 48 | * The above table can be verified with the following shell script: |
| 49 | − * | |
| 50 | − * #!/bin/sh | |
| 51 | − * sed -ne 's/^{ \+\(.*\), \+\(.*\), \+\(.*\), \+\(.*\) },$/\1 \2 \3 \4/p' \ | |
| 52 | − * lib/math/test_mul_u64_u64_div_u64.c | | |
| 53 | − * while read a b c r; do | |
| 54 | − * expected=$( printf "obase=16; ibase=16; %X * %X / %X\n" $a $b $c | bc ) | |
| 55 | − * given=$( printf "%X\n" $r ) | |
| 56 | − * if [ "$expected" = "$given" ]; then | |
| 57 | − * echo "$a * $b / $c = $r OK" | |
| 58 | − * else | |
| 59 | − * echo "$a * $b / $c = $r is wrong" >&2 | |
| 60 | − * echo "should be equivalent to 0x$expected" >&2 | |
| 61 | − * exit 1 | |
| 62 | − * fi | |
| 63 | − * done | |
| 49 | + | |
| 50 | +#!/bin/sh | |
| 51 | +sed -ne 's/^{ \+\(.*\), \+\(.*\), \+\(.*\), \+\(.*\), \+\(.*\) },$/\1 \2 \3 \4 \5/p' \ | |
| 52 | + lib/math/test_mul_u64_u64_div_u64.c | | |
| 53 | +while read a b d r e; do | |
| 54 | + expected=$( printf "obase=16; ibase=16; %X * %X / %X\n" $a $b $d | bc ) | |
| 55 | + given=$( printf "%X\n" $r ) | |
| 56 | + if [ "$expected" = "$given" ]; then | |
| 57 | + echo "$a * $b / $d = $r OK" | |
| 58 | + else | |
| 59 | + echo "$a * $b / $d = $r is wrong" >&2 | |
| 60 | + echo "should be equivalent to 0x$expected" >&2 | |
| 61 | + exit 1 | |
| 62 | + fi | |
| 63 | + expected=$( printf "obase=16; ibase=16; (%X * %X + %X) / %X\n" $a $b $((d-1)) $d | bc ) | |
| 64 | + given=$( printf "%X\n" $((r + e)) ) | |
| 65 | + if [ "$expected" = "$given" ]; then | |
| 66 | + echo "$a * $b +/ $d = $(printf '%#x' $((r + e))) OK" | |
| 67 | + else | |
| 68 | + echo "$a * $b +/ $d = $(printf '%#x' $((r + e))) is wrong" >&2 | |
| 69 | + echo "should be equivalent to 0x$expected" >&2 | |
| 70 | + exit 1 | |
| 71 | + fi | |
| 72 | +done | |
| 73 | + | |
| 64 | 74 | */ |
| 65 | 75 | |
| 66 | −static int __init test_init(void) | |
| 76 | +static u64 test_mul_u64_add_u64_div_u64(u64 a, u64 b, u64 c, u64 d); | |
| 77 | +#if __LONG_WIDTH__ >= 64 | |
| 78 | +#define TEST_32BIT_DIV | |
| 79 | +static u64 test_mul_u64_add_u64_div_u64_32bit(u64 a, u64 b, u64 c, u64 d); | |
| 80 | +#endif | |
| 81 | + | |
| 82 | +static int __init test_run(unsigned int fn_no, const char *fn_name) | |
| 67 | 83 | { |
| 84 | + u64 start_time; | |
| 85 | + int errors = 0; | |
| 86 | + int tests = 0; | |
| 68 | 87 | int i; |
| 69 | 88 | |
| 70 | − pr_info("Starting mul_u64_u64_div_u64() test\n"); | |
| 89 | + start_time = ktime_get_ns(); | |
| 71 | 90 | |
| 72 | 91 | for (i = 0; i < ARRAY_SIZE(test_values); i++) { |
| 73 | 92 | u64 a = test_values[i].a; |
| 74 | 93 | u64 b = test_values[i].b; |
| 75 | − u64 c = test_values[i].c; | |
| 94 | + u64 d = test_values[i].d; | |
| 76 | 95 | u64 expected_result = test_values[i].result; |
| 77 | − u64 result = mul_u64_u64_div_u64(a, b, c); | |
| 96 | + u64 result, result_up; | |
| 97 | + | |
| 98 | + switch (fn_no) { | |
| 99 | + default: | |
| 100 | + result = mul_u64_u64_div_u64(a, b, d); | |
| 101 | + result_up = mul_u64_u64_div_u64_roundup(a, b, d); | |
| 102 | + break; | |
| 103 | + case 1: | |
| 104 | + result = test_mul_u64_add_u64_div_u64(a, b, 0, d); | |
| 105 | + result_up = test_mul_u64_add_u64_div_u64(a, b, d - 1, d); | |
| 106 | + break; | |
| 107 | +#ifdef TEST_32BIT_DIV | |
| 108 | + case 2: | |
| 109 | + result = test_mul_u64_add_u64_div_u64_32bit(a, b, 0, d); | |
| 110 | + result_up = test_mul_u64_add_u64_div_u64_32bit(a, b, d - 1, d); | |
| 111 | + break; | |
| 112 | +#endif | |
| 113 | + } | |
| 114 | + | |
| 115 | + tests += 2; | |
| 78 | 116 | |
| 79 | 117 | if (result != expected_result) { |
| 80 | − pr_err("ERROR: 0x%016llx * 0x%016llx / 0x%016llx\n", a, b, c); | |
| 118 | + pr_err("ERROR: 0x%016llx * 0x%016llx / 0x%016llx\n", a, b, d); | |
| 81 | 119 | pr_err("ERROR: expected result: %016llx\n", expected_result); |
| 82 | 120 | pr_err("ERROR: obtained result: %016llx\n", result); |
| 121 | + errors++; | |
| 122 | + } | |
| 123 | + expected_result += test_values[i].round_up; | |
| 124 | + if (result_up != expected_result) { | |
| 125 | + pr_err("ERROR: 0x%016llx * 0x%016llx +/ 0x%016llx\n", a, b, d); | |
| 126 | + pr_err("ERROR: expected result: %016llx\n", expected_result); | |
| 127 | + pr_err("ERROR: obtained result: %016llx\n", result_up); | |
| 128 | + errors++; | |
| 83 | 129 | } |
| 84 | 130 | } |
| 85 | 131 | |
| 86 | − pr_info("Completed mul_u64_u64_div_u64() test\n"); | |
| 132 | + pr_info("Completed %s() test, %d tests, %d errors, %llu ns\n", | |
| 133 | + fn_name, tests, errors, ktime_get_ns() - start_time); | |
| 134 | + return errors; | |
| 135 | +} | |
| 136 | + | |
| 137 | +static int __init test_init(void) | |
| 138 | +{ | |
| 139 | + pr_info("Starting mul_u64_u64_div_u64() test\n"); | |
| 140 | + if (test_run(0, "mul_u64_u64_div_u64")) | |
| 141 | + return -EINVAL; | |
| 142 | + if (test_run(1, "test_mul_u64_u64_div_u64")) | |
| 143 | + return -EINVAL; | |
| 144 | +#ifdef TEST_32BIT_DIV | |
| 145 | + if (test_run(2, "test_mul_u64_u64_div_u64_32bit")) | |
| 146 | + return -EINVAL; | |
| 147 | +#endif | |
| 87 | 148 | return 0; |
| 88 | 149 | } |
| 89 | 150 | |
| @@ -91,6 +152,36 @@ static void __exit test_exit(void) · 1 unmodified line | ||
| 91 | 152 | { |
| 92 | 153 | } |
| 93 | 154 | |
| 155 | +/* Compile the generic mul_u64_add_u64_div_u64() code */ | |
| 156 | +#undef __div64_32 | |
| 157 | +#define __div64_32 __div64_32 | |
| 158 | +#define div_s64_rem div_s64_rem | |
| 159 | +#define div64_u64_rem div64_u64_rem | |
| 160 | +#define div64_u64 div64_u64 | |
| 161 | +#define div64_s64 div64_s64 | |
| 162 | +#define iter_div_u64_rem iter_div_u64_rem | |
| 163 | + | |
| 164 | +#undef mul_u64_add_u64_div_u64 | |
| 165 | +#define mul_u64_add_u64_div_u64 test_mul_u64_add_u64_div_u64 | |
| 166 | +#define test_mul_u64_add_u64_div_u64 test_mul_u64_add_u64_div_u64 | |
| 167 | + | |
| 168 | +#include "div64.c" | |
| 169 | + | |
| 170 | +#ifdef TEST_32BIT_DIV | |
| 171 | +/* Recompile the generic code for 32bit long */ | |
| 172 | +#undef test_mul_u64_add_u64_div_u64 | |
| 173 | +#define test_mul_u64_add_u64_div_u64 test_mul_u64_add_u64_div_u64_32bit | |
| 174 | +#undef BITS_PER_ITER | |
| 175 | +#define BITS_PER_ITER 16 | |
| 176 | + | |
| 177 | +#define mul_u64_u64_add_u64 mul_u64_u64_add_u64_32bit | |
| 178 | +#undef mul_u64_long_add_u64 | |
| 179 | +#undef add_u64_long | |
| 180 | +#undef mul_add | |
| 181 | + | |
| 182 | +#include "div64.c" | |
| 183 | +#endif | |
| 184 | + | |
| 94 | 185 | module_init(test_init); |
| 95 | 186 | module_exit(test_exit); |
| 96 | 187 | |
| @@ -47,8 +47,8 @@ static void plist_check_list(struct list_head *top) · 46 unmodified lines | ||
| 47 | 47 | |
| 48 | 48 | plist_check_prev_next(top, prev, next); |
| 49 | 49 | while (next != top) { |
| 50 | − WRITE_ONCE(prev, next); | |
| 51 | − WRITE_ONCE(next, prev->next); | |
| 50 | + prev = next; | |
| 51 | + next = prev->next; | |
| 52 | 52 | plist_check_prev_next(top, prev, next); |
| 53 | 53 | } |
| 54 | 54 | } |
| @@ -27,7 +27,7 @@ · 26 unmodified lines | ||
| 27 | 27 | int ___ratelimit(struct ratelimit_state *rs, const char *func) |
| 28 | 28 | { |
| 29 | 29 | /* Paired with WRITE_ONCE() in .proc_handler(). |
| 30 | − * Changing two values seperately could be inconsistent | |
| 30 | + * Changing two values separately could be inconsistent | |
| 31 | 31 | * and some message could be lost. (See: net_ratelimit_state). |
| 32 | 32 | */ |
| 33 | 33 | int interval = READ_ONCE(rs->interval); |
| @@ -460,35 +460,6 @@ void __rb_insert_augmented(struct rb_node *node, struct rb_root *root, · 459 unmodified lines | ||
| 460 | 460 | } |
| 461 | 461 | EXPORT_SYMBOL(__rb_insert_augmented); |
| 462 | 462 | |
| 463 | −/* | |
| 464 | − * This function returns the first node (in sort order) of the tree. | |
| 465 | − */ | |
| 466 | −struct rb_node *rb_first(const struct rb_root *root) | |
| 467 | −{ | |
| 468 | − struct rb_node *n; | |
| 469 | − | |
| 470 | − n = root->rb_node; | |
| 471 | − if (!n) | |
| 472 | − return NULL; | |
| 473 | − while (n->rb_left) | |
| 474 | − n = n->rb_left; | |
| 475 | − return n; | |
| 476 | −} | |
| 477 | −EXPORT_SYMBOL(rb_first); | |
| 478 | − | |
| 479 | −struct rb_node *rb_last(const struct rb_root *root) | |
| 480 | −{ | |
| 481 | − struct rb_node *n; | |
| 482 | − | |
| 483 | − n = root->rb_node; | |
| 484 | − if (!n) | |
| 485 | − return NULL; | |
| 486 | − while (n->rb_right) | |
| 487 | − n = n->rb_right; | |
| 488 | − return n; | |
| 489 | −} | |
| 490 | −EXPORT_SYMBOL(rb_last); | |
| 491 | − | |
| 492 | 463 | struct rb_node *rb_next(const struct rb_node *node) |
| 493 | 464 | { |
| 494 | 465 | struct rb_node *parent; |
| @@ -1,31 +1,35 @@ | ||
| 1 | 1 | // SPDX-License-Identifier: GPL-2.0-only |
| 2 | −#include <linux/sched/debug.h> | |
| 2 | +#include <linux/array_size.h> | |
| 3 | +#include <linux/bitops.h> | |
| 4 | +#include <linux/cleanup.h> | |
| 3 | 5 | #include <linux/console.h> |
| 6 | +#include <linux/log2.h> | |
| 4 | 7 | #include <linux/kernel.h> |
| 5 | 8 | #include <linux/ftrace.h> |
| 6 | −#include <linux/sysctl.h> | |
| 7 | 9 | #include <linux/nmi.h> |
| 10 | +#include <linux/sched/debug.h> | |
| 11 | +#include <linux/string.h> | |
| 12 | +#include <linux/sysctl.h> | |
| 8 | 13 | |
| 9 | 14 | #include <linux/sys_info.h> |
| 10 | 15 | |
| 11 | −struct sys_info_name { | |
| 12 | − unsigned long bit; | |
| 13 | − const char *name; | |
| 16 | +static const char * const si_names[] = { | |
| 17 | + [ilog2(SYS_INFO_TASKS)] = "tasks", | |
| 18 | + [ilog2(SYS_INFO_MEM)] = "mem", | |
| 19 | + [ilog2(SYS_INFO_TIMERS)] = "timers", | |
| 20 | + [ilog2(SYS_INFO_LOCKS)] = "locks", | |
| 21 | + [ilog2(SYS_INFO_FTRACE)] = "ftrace", | |
| 22 | + [ilog2(SYS_INFO_PANIC_CONSOLE_REPLAY)] = "", | |
| 23 | + [ilog2(SYS_INFO_ALL_BT)] = "all_bt", | |
| 24 | + [ilog2(SYS_INFO_BLOCKED_TASKS)] = "blocked_tasks", | |
| 14 | 25 | }; |
| 15 | 26 | |
| 16 | 27 | /* |
| 17 | − * When 'si_names' gets updated, please make sure the 'sys_info_avail' | |
| 18 | − * below is updated accordingly. | |
| 28 | + * Default kernel sys_info mask. | |
| 29 | + * If a kernel module calls sys_info() with "parameter == 0", then | |
| 30 | + * this mask will be used. | |
| 19 | 31 | */ |
| 20 | −static const struct sys_info_name si_names[] = { | |
| 21 | − { SYS_INFO_TASKS, "tasks" }, | |
| 22 | − { SYS_INFO_MEM, "mem" }, | |
| 23 | − { SYS_INFO_TIMERS, "timers" }, | |
| 24 | − { SYS_INFO_LOCKS, "locks" }, | |
| 25 | − { SYS_INFO_FTRACE, "ftrace" }, | |
| 26 | − { SYS_INFO_ALL_CPU_BT, "all_bt" }, | |
| 27 | − { SYS_INFO_BLOCKED_TASKS, "blocked_tasks" }, | |
| 28 | −}; | |
| 32 | +static unsigned long kernel_si_mask; | |
| 29 | 33 | |
| 30 | 34 | /* Expecting string like "xxx_sys_info=tasks,mem,timers,locks,ftrace,..." */ |
| 31 | 35 | unsigned long sys_info_parse_param(char *str) |
| @@ -36,12 +40,9 @@ unsigned long sys_info_parse_param(char *str) · 4 unmodified lines | ||
| 36 | 40 | |
| 37 | 41 | s = str; |
| 38 | 42 | while ((name = strsep(&s, ",")) && *name) { |
| 39 | − for (i = 0; i < ARRAY_SIZE(si_names); i++) { | |
| 40 | − if (!strcmp(name, si_names[i].name)) { | |
| 41 | − si_bits |= si_names[i].bit; | |
| 42 | − break; | |
| 43 | − } | |
| 44 | − } | |
| 43 | + i = match_string(si_names, ARRAY_SIZE(si_names), name); | |
| 44 | + if (i >= 0) | |
| 45 | + __set_bit(i, &si_bits); | |
| 45 | 46 | } |
| 46 | 47 | |
| 47 | 48 | return si_bits; |
| @@ -49,56 +50,93 @@ unsigned long sys_info_parse_param(char *str) · 1 unmodified line | ||
| 49 | 50 | |
| 50 | 51 | #ifdef CONFIG_SYSCTL |
| 51 | 52 | |
| 52 | −static const char sys_info_avail[] __maybe_unused = "tasks,mem,timers,locks,ftrace,all_bt,blocked_tasks"; | |
| 53 | +static int sys_info_write_handler(const struct ctl_table *table, | |
| 54 | + void *buffer, size_t *lenp, loff_t *ppos, | |
| 55 | + unsigned long *si_bits_global) | |
| 56 | +{ | |
| 57 | + unsigned long si_bits; | |
| 58 | + int ret; | |
| 59 | + | |
| 60 | + ret = proc_dostring(table, 1, buffer, lenp, ppos); | |
| 61 | + if (ret) | |
| 62 | + return ret; | |
| 63 | + | |
| 64 | + si_bits = sys_info_parse_param(table->data); | |
| 65 | + | |
| 66 | + /* The access to the global value is not synchronized. */ | |
| 67 | + WRITE_ONCE(*si_bits_global, si_bits); | |
| 68 | + | |
| 69 | + return 0; | |
| 70 | +} | |
| 71 | + | |
| 72 | +static int sys_info_read_handler(const struct ctl_table *table, | |
| 73 | + void *buffer, size_t *lenp, loff_t *ppos, | |
| 74 | + unsigned long *si_bits_global) | |
| 75 | +{ | |
| 76 | + unsigned long si_bits; | |
| 77 | + unsigned int len = 0; | |
| 78 | + char *delim = ""; | |
| 79 | + unsigned int i; | |
| 80 | + | |
| 81 | + /* The access to the global value is not synchronized. */ | |
| 82 | + si_bits = READ_ONCE(*si_bits_global); | |
| 83 | + | |
| 84 | + for_each_set_bit(i, &si_bits, ARRAY_SIZE(si_names)) { | |
| 85 | + if (*si_names[i]) { | |
| 86 | + len += scnprintf(table->data + len, table->maxlen - len, | |
| 87 | + "%s%s", delim, si_names[i]); | |
| 88 | + delim = ","; | |
| 89 | + } | |
| 90 | + } | |
| 91 | + | |
| 92 | + return proc_dostring(table, 0, buffer, lenp, ppos); | |
| 93 | +} | |
| 53 | 94 | |
| 54 | 95 | int sysctl_sys_info_handler(const struct ctl_table *ro_table, int write, |
| 55 | 96 | void *buffer, size_t *lenp, |
| 56 | 97 | loff_t *ppos) |
| 57 | 98 | { |
| 58 | − char names[sizeof(sys_info_avail)]; | |
| 59 | 99 | struct ctl_table table; |
| 60 | − unsigned long *si_bits_global; | |
| 61 | − | |
| 62 | − si_bits_global = ro_table->data; | |
| 63 | − | |
| 64 | − if (write) { | |
| 65 | − unsigned long si_bits; | |
| 66 | − int ret; | |
| 67 | − | |
| 68 | − table = *ro_table; | |
| 69 | − table.data = names; | |
| 70 | − table.maxlen = sizeof(names); | |
| 71 | − ret = proc_dostring(&table, write, buffer, lenp, ppos); | |
| 72 | − if (ret) | |
| 73 | − return ret; | |
| 74 | − | |
| 75 | − si_bits = sys_info_parse_param(names); | |
| 76 | − /* The access to the global value is not synchronized. */ | |
| 77 | − WRITE_ONCE(*si_bits_global, si_bits); | |
| 78 | − return 0; | |
| 79 | − } else { | |
| 80 | − /* for 'read' operation */ | |
| 81 | − char *delim = ""; | |
| 82 | − int i, len = 0; | |
| 83 | − | |
| 84 | − names[0] = '\0'; | |
| 85 | − for (i = 0; i < ARRAY_SIZE(si_names); i++) { | |
| 86 | − if (*si_bits_global & si_names[i].bit) { | |
| 87 | − len += scnprintf(names + len, sizeof(names) - len, | |
| 88 | − "%s%s", delim, si_names[i].name); | |
| 89 | − delim = ","; | |
| 90 | − } | |
| 91 | − } | |
| 100 | + unsigned int i; | |
| 101 | + size_t maxlen; | |
| 92 | 102 | |
| 93 | − table = *ro_table; | |
| 94 | − table.data = names; | |
| 95 | − table.maxlen = sizeof(names); | |
| 96 | − return proc_dostring(&table, write, buffer, lenp, ppos); | |
| 97 | − } | |
| 103 | + maxlen = 0; | |
| 104 | + for (i = 0; i < ARRAY_SIZE(si_names); i++) | |
| 105 | + maxlen += strlen(si_names[i]) + 1; | |
| 106 | + | |
| 107 | + char *names __free(kfree) = kzalloc(maxlen, GFP_KERNEL); | |
| 108 | + if (!names) | |
| 109 | + return -ENOMEM; | |
| 110 | + | |
| 111 | + table = *ro_table; | |
| 112 | + table.data = names; | |
| 113 | + table.maxlen = maxlen; | |
| 114 | + | |
| 115 | + if (write) | |
| 116 | + return sys_info_write_handler(&table, buffer, lenp, ppos, ro_table->data); | |
| 117 | + else | |
| 118 | + return sys_info_read_handler(&table, buffer, lenp, ppos, ro_table->data); | |
| 119 | +} | |
| 120 | + | |
| 121 | +static const struct ctl_table sys_info_sysctls[] = { | |
| 122 | + { | |
| 123 | + .procname = "kernel_sys_info", | |
| 124 | + .data = &kernel_si_mask, | |
| 125 | + .maxlen = sizeof(kernel_si_mask), | |
| 126 | + .mode = 0644, | |
| 127 | + .proc_handler = sysctl_sys_info_handler, | |
| 128 | + }, | |
| 129 | +}; | |
| 130 | + | |
| 131 | +static int __init sys_info_sysctl_init(void) | |
| 132 | +{ | |
| 133 | + register_sysctl_init("kernel", sys_info_sysctls); | |
| 134 | + return 0; | |
| 98 | 135 | } |
| 136 | +subsys_initcall(sys_info_sysctl_init); | |
| 99 | 137 | #endif |
| 100 | 138 | |
| 101 | −void sys_info(unsigned long si_mask) | |
| 139 | +static void __sys_info(unsigned long si_mask) | |
| 102 | 140 | { |
| 103 | 141 | if (si_mask & SYS_INFO_TASKS) |
| 104 | 142 | show_state(); |
| @@ -115,9 +153,14 @@ void sys_info(unsigned long si_mask) · 10 unmodified lines | ||
| 115 | 153 | if (si_mask & SYS_INFO_FTRACE) |
| 116 | 154 | ftrace_dump(DUMP_ALL); |
| 117 | 155 | |
| 118 | − if (si_mask & SYS_INFO_ALL_CPU_BT) | |
| 156 | + if (si_mask & SYS_INFO_ALL_BT) | |
| 119 | 157 | trigger_all_cpu_backtrace(); |
| 120 | 158 | |
| 121 | 159 | if (si_mask & SYS_INFO_BLOCKED_TASKS) |
| 122 | 160 | show_state_filter(TASK_UNINTERRUPTIBLE); |
| 123 | 161 | } |
| 162 | + | |
| 163 | +void sys_info(unsigned long si_mask) | |
| 164 | +{ | |
| 165 | + __sys_info(si_mask ? : kernel_si_mask); | |
| 166 | +} | |
| @@ -33,44 +33,28 @@ struct kho_test_state { · 32 unmodified lines | ||
| 33 | 33 | unsigned int nr_folios; |
| 34 | 34 | struct folio **folios; |
| 35 | 35 | phys_addr_t *folios_info; |
| 36 | + struct kho_vmalloc folios_info_phys; | |
| 37 | + int nr_folios_preserved; | |
| 36 | 38 | struct folio *fdt; |
| 37 | 39 | __wsum csum; |
| 38 | 40 | }; |
| 39 | 41 | |
| 40 | 42 | static struct kho_test_state kho_test_state; |
| 41 | 43 | |
| 42 | −static int kho_test_notifier(struct notifier_block *self, unsigned long cmd, | |
| 43 | − void *v) | |
| 44 | +static void kho_test_unpreserve_data(struct kho_test_state *state) | |
| 44 | 45 | { |
| 45 | − struct kho_test_state *state = &kho_test_state; | |
| 46 | − struct kho_serialization *ser = v; | |
| 47 | − int err = 0; | |
| 48 | − | |
| 49 | − switch (cmd) { | |
| 50 | − case KEXEC_KHO_ABORT: | |
| 51 | − return NOTIFY_DONE; | |
| 52 | − case KEXEC_KHO_FINALIZE: | |
| 53 | − /* Handled below */ | |
| 54 | − break; | |
| 55 | − default: | |
| 56 | − return NOTIFY_BAD; | |
| 57 | − } | |
| 58 | − | |
| 59 | − err |= kho_preserve_folio(state->fdt); | |
| 60 | − err |= kho_add_subtree(ser, KHO_TEST_FDT, folio_address(state->fdt)); | |
| 46 | + for (int i = 0; i < state->nr_folios_preserved; i++) | |
| 47 | + kho_unpreserve_folio(state->folios[i]); | |
| 61 | 48 | |
| 62 | − return err ? NOTIFY_BAD : NOTIFY_DONE; | |
| 49 | + kho_unpreserve_vmalloc(&state->folios_info_phys); | |
| 50 | + vfree(state->folios_info); | |
| 63 | 51 | } |
| 64 | 52 | |
| 65 | −static struct notifier_block kho_test_nb = { | |
| 66 | − .notifier_call = kho_test_notifier, | |
| 67 | −}; | |
| 68 | − | |
| 69 | −static int kho_test_save_data(struct kho_test_state *state, void *fdt) | |
| 53 | +static int kho_test_preserve_data(struct kho_test_state *state) | |
| 70 | 54 | { |
| 71 | − phys_addr_t *folios_info __free(kvfree) = NULL; | |
| 72 | 55 | struct kho_vmalloc folios_info_phys; |
| 73 | − int err = 0; | |
| 56 | + phys_addr_t *folios_info; | |
| 57 | + int err; | |
| 74 | 58 | |
| 75 | 59 | folios_info = vmalloc_array(state->nr_folios, sizeof(*folios_info)); |
| 76 | 60 | if (!folios_info) |
| @@ -78,62 +62,98 @@ static int kho_test_save_data(struct kho_test_state *state, void *fdt) · 1 unmodified line | ||
| 78 | 62 | |
| 79 | 63 | err = kho_preserve_vmalloc(folios_info, &folios_info_phys); |
| 80 | 64 | if (err) |
| 81 | − return err; | |
| 65 | + goto err_free_info; | |
| 66 | + | |
| 67 | + state->folios_info_phys = folios_info_phys; | |
| 68 | + state->folios_info = folios_info; | |
| 82 | 69 | |
| 83 | 70 | for (int i = 0; i < state->nr_folios; i++) { |
| 84 | 71 | struct folio *folio = state->folios[i]; |
| 85 | 72 | unsigned int order = folio_order(folio); |
| 86 | 73 | |
| 87 | 74 | folios_info[i] = virt_to_phys(folio_address(folio)) | order; |
| 88 | − | |
| 89 | 75 | err = kho_preserve_folio(folio); |
| 90 | 76 | if (err) |
| 91 | − break; | |
| 77 | + goto err_unpreserve; | |
| 78 | + state->nr_folios_preserved++; | |
| 92 | 79 | } |
| 93 | 80 | |
| 81 | + return 0; | |
| 82 | + | |
| 83 | +err_unpreserve: | |
| 84 | + /* | |
| 85 | + * kho_test_unpreserve_data frees folio_info, bail out immediately to | |
| 86 | + * avoid double free | |
| 87 | + */ | |
| 88 | + kho_test_unpreserve_data(state); | |
| 89 | + return err; | |
| 90 | + | |
| 91 | +err_free_info: | |
| 92 | + vfree(folios_info); | |
| 93 | + return err; | |
| 94 | +} | |
| 95 | + | |
| 96 | +static int kho_test_prepare_fdt(struct kho_test_state *state, ssize_t fdt_size) | |
| 97 | +{ | |
| 98 | + const char compatible[] = KHO_TEST_COMPAT; | |
| 99 | + unsigned int magic = KHO_TEST_MAGIC; | |
| 100 | + void *fdt = folio_address(state->fdt); | |
| 101 | + int err; | |
| 102 | + | |
| 103 | + err = fdt_create(fdt, fdt_size); | |
| 104 | + err |= fdt_finish_reservemap(fdt); | |
| 105 | + err |= fdt_begin_node(fdt, ""); | |
| 106 | + err |= fdt_property(fdt, "compatible", compatible, sizeof(compatible)); | |
| 107 | + err |= fdt_property(fdt, "magic", &magic, sizeof(magic)); | |
| 108 | + | |
| 94 | 109 | err |= fdt_begin_node(fdt, "data"); |
| 95 | 110 | err |= fdt_property(fdt, "nr_folios", &state->nr_folios, |
| 96 | 111 | sizeof(state->nr_folios)); |
| 97 | − err |= fdt_property(fdt, "folios_info", &folios_info_phys, | |
| 98 | − sizeof(folios_info_phys)); | |
| 112 | + err |= fdt_property(fdt, "folios_info", &state->folios_info_phys, | |
| 113 | + sizeof(state->folios_info_phys)); | |
| 99 | 114 | err |= fdt_property(fdt, "csum", &state->csum, sizeof(state->csum)); |
| 100 | 115 | err |= fdt_end_node(fdt); |
| 101 | 116 | |
| 102 | − if (!err) | |
| 103 | − state->folios_info = no_free_ptr(folios_info); | |
| 117 | + err |= fdt_end_node(fdt); | |
| 118 | + err |= fdt_finish(fdt); | |
| 104 | 119 | |
| 105 | 120 | return err; |
| 106 | 121 | } |
| 107 | 122 | |
| 108 | −static int kho_test_prepare_fdt(struct kho_test_state *state) | |
| 123 | +static int kho_test_preserve(struct kho_test_state *state) | |
| 109 | 124 | { |
| 110 | − const char compatible[] = KHO_TEST_COMPAT; | |
| 111 | − unsigned int magic = KHO_TEST_MAGIC; | |
| 112 | 125 | ssize_t fdt_size; |
| 113 | − int err = 0; | |
| 114 | − void *fdt; | |
| 126 | + int err; | |
| 115 | 127 | |
| 116 | 128 | fdt_size = state->nr_folios * sizeof(phys_addr_t) + PAGE_SIZE; |
| 117 | 129 | state->fdt = folio_alloc(GFP_KERNEL, get_order(fdt_size)); |
| 118 | 130 | if (!state->fdt) |
| 119 | 131 | return -ENOMEM; |
| 120 | 132 | |
| 121 | − fdt = folio_address(state->fdt); | |
| 122 | − | |
| 123 | − err |= fdt_create(fdt, fdt_size); | |
| 124 | − err |= fdt_finish_reservemap(fdt); | |
| 133 | + err = kho_preserve_folio(state->fdt); | |
| 134 | + if (err) | |
| 135 | + goto err_free_fdt; | |
| 125 | 136 | |
| 126 | − err |= fdt_begin_node(fdt, ""); | |
| 127 | − err |= fdt_property(fdt, "compatible", compatible, sizeof(compatible)); | |
| 128 | − err |= fdt_property(fdt, "magic", &magic, sizeof(magic)); | |
| 129 | − err |= kho_test_save_data(state, fdt); | |
| 130 | − err |= fdt_end_node(fdt); | |
| 137 | + err = kho_test_preserve_data(state); | |
| 138 | + if (err) | |
| 139 | + goto err_unpreserve_fdt; | |
| 131 | 140 | |
| 132 | − err |= fdt_finish(fdt); | |
| 141 | + err = kho_test_prepare_fdt(state, fdt_size); | |
| 142 | + if (err) | |
| 143 | + goto err_unpreserve_data; | |
| 133 | 144 | |
| 145 | + err = kho_add_subtree(KHO_TEST_FDT, folio_address(state->fdt)); | |
| 134 | 146 | if (err) |
| 135 | − folio_put(state->fdt); | |
| 147 | + goto err_unpreserve_data; | |
| 148 | + | |
| 149 | + return 0; | |
| 136 | 150 | |
| 151 | +err_unpreserve_data: | |
| 152 | + kho_test_unpreserve_data(state); | |
| 153 | +err_unpreserve_fdt: | |
| 154 | + kho_unpreserve_folio(state->fdt); | |
| 155 | +err_free_fdt: | |
| 156 | + folio_put(state->fdt); | |
| 137 | 157 | return err; |
| 138 | 158 | } |
| 139 | 159 | |
| @@ -199,18 +219,12 @@ static int kho_test_save(void) · 59 unmodified lines | ||
| 199 | 219 | if (err) |
| 200 | 220 | goto err_free_folios; |
| 201 | 221 | |
| 202 | − err = kho_test_prepare_fdt(state); | |
| 222 | + err = kho_test_preserve(state); | |
| 203 | 223 | if (err) |
| 204 | 224 | goto err_free_folios; |
| 205 | 225 | |
| 206 | − err = register_kho_notifier(&kho_test_nb); | |
| 207 | − if (err) | |
| 208 | − goto err_free_fdt; | |
| 209 | − | |
| 210 | 226 | return 0; |
| 211 | 227 | |
| 212 | −err_free_fdt: | |
| 213 | − folio_put(state->fdt); | |
| 214 | 228 | err_free_folios: |
| 215 | 229 | kvfree(folios); |
| 216 | 230 | return err; |
| @@ -292,7 +306,6 @@ static int kho_test_restore(phys_addr_t fdt_phys) · 75 unmodified lines | ||
| 292 | 306 | if (err) |
| 293 | 307 | return err; |
| 294 | 308 | |
| 295 | − pr_info("KHO restore succeeded\n"); | |
| 296 | 309 | return 0; |
| 297 | 310 | } |
| 298 | 311 | |
| @@ -305,8 +318,15 @@ static int __init kho_test_init(void) · 6 unmodified lines | ||
| 305 | 318 | return 0; |
| 306 | 319 | |
| 307 | 320 | err = kho_retrieve_subtree(KHO_TEST_FDT, &fdt_phys); |
| 308 | − if (!err) | |
| 309 | − return kho_test_restore(fdt_phys); | |
| 321 | + if (!err) { | |
| 322 | + err = kho_test_restore(fdt_phys); | |
| 323 | + if (err) | |
| 324 | + pr_err("KHO restore failed\n"); | |
| 325 | + else | |
| 326 | + pr_info("KHO restore succeeded\n"); | |
| 327 | + | |
| 328 | + return err; | |
| 329 | + } | |
| 310 | 330 | |
| 311 | 331 | if (err != -ENOENT) { |
| 312 | 332 | pr_warn("failed to retrieve %s FDT: %d\n", KHO_TEST_FDT, err); |
| @@ -329,7 +349,7 @@ static void kho_test_cleanup(void) · 16 unmodified lines | ||
| 329 | 349 | |
| 330 | 350 | static void __exit kho_test_exit(void) |
| 331 | 351 | { |
| 332 | − unregister_kho_notifier(&kho_test_nb); | |
| 352 | + kho_remove_subtree(folio_address(kho_test_state.fdt)); | |
| 333 | 353 | kho_test_cleanup(); |
| 334 | 354 | } |
| 335 | 355 | module_exit(kho_test_exit); |
| @@ -4,6 +4,7 @@ · 3 unmodified lines | ||
| 4 | 4 | |
| 5 | 5 | # KUnit tests |
| 6 | 6 | CFLAGS_bitfield_kunit.o := $(DISABLE_STRUCTLEAK_PLUGIN) |
| 7 | +obj-$(CONFIG_BASE64_KUNIT) += base64_kunit.o | |
| 7 | 8 | obj-$(CONFIG_BITFIELD_KUNIT) += bitfield_kunit.o |
| 8 | 9 | obj-$(CONFIG_BITS_TEST) += test_bits.o |
| 9 | 10 | obj-$(CONFIG_BLACKHOLE_DEV_KUNIT_TEST) += blackhole_dev_kunit.o |
| @@ -0,0 +1,294 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0 | |
| 2 | +/* | |
| 3 | + * base64_kunit_test.c - KUnit tests for base64 encoding and decoding functions | |
| 4 | + * | |
| 5 | + * Copyright (c) 2025, Guan-Chun Wu <409411716@gms.tku.edu.tw> | |
| 6 | + */ | |
| 7 | + | |
| 8 | +#include <kunit/test.h> | |
| 9 | +#include <linux/base64.h> | |
| 10 | + | |
| 11 | +/* ---------- Benchmark helpers ---------- */ | |
| 12 | +static u64 bench_encode_ns(const u8 *data, int len, char *dst, int reps, | |
| 13 | + enum base64_variant variant) | |
| 14 | +{ | |
| 15 | + u64 t0, t1; | |
| 16 | + | |
| 17 | + t0 = ktime_get_ns(); | |
| 18 | + for (int i = 0; i < reps; i++) | |
| 19 | + base64_encode(data, len, dst, true, variant); | |
| 20 | + t1 = ktime_get_ns(); | |
| 21 | + | |
| 22 | + return div64_u64(t1 - t0, (u64)reps); | |
| 23 | +} | |
| 24 | + | |
| 25 | +static u64 bench_decode_ns(const char *data, int len, u8 *dst, int reps, | |
| 26 | + enum base64_variant variant) | |
| 27 | +{ | |
| 28 | + u64 t0, t1; | |
| 29 | + | |
| 30 | + t0 = ktime_get_ns(); | |
| 31 | + for (int i = 0; i < reps; i++) | |
| 32 | + base64_decode(data, len, dst, true, variant); | |
| 33 | + t1 = ktime_get_ns(); | |
| 34 | + | |
| 35 | + return div64_u64(t1 - t0, (u64)reps); | |
| 36 | +} | |
| 37 | + | |
| 38 | +static void run_perf_and_check(struct kunit *test, const char *label, int size, | |
| 39 | + enum base64_variant variant) | |
| 40 | +{ | |
| 41 | + const int reps = 1000; | |
| 42 | + size_t outlen = DIV_ROUND_UP(size, 3) * 4; | |
| 43 | + u8 *in = kmalloc(size, GFP_KERNEL); | |
| 44 | + char *enc = kmalloc(outlen, GFP_KERNEL); | |
| 45 | + u8 *decoded = kmalloc(size, GFP_KERNEL); | |
| 46 | + | |
| 47 | + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, in); | |
| 48 | + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, enc); | |
| 49 | + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, decoded); | |
| 50 | + | |
| 51 | + get_random_bytes(in, size); | |
| 52 | + int enc_len = base64_encode(in, size, enc, true, variant); | |
| 53 | + int dec_len = base64_decode(enc, enc_len, decoded, true, variant); | |
| 54 | + | |
| 55 | + /* correctness sanity check */ | |
| 56 | + KUNIT_EXPECT_EQ(test, dec_len, size); | |
| 57 | + KUNIT_EXPECT_MEMEQ(test, decoded, in, size); | |
| 58 | + | |
| 59 | + /* benchmark encode */ | |
| 60 | + | |
| 61 | + u64 t1 = bench_encode_ns(in, size, enc, reps, variant); | |
| 62 | + | |
| 63 | + kunit_info(test, "[%s] encode run : %lluns", label, t1); | |
| 64 | + | |
| 65 | + u64 t2 = bench_decode_ns(enc, enc_len, decoded, reps, variant); | |
| 66 | + | |
| 67 | + kunit_info(test, "[%s] decode run : %lluns", label, t2); | |
| 68 | + | |
| 69 | + kfree(in); | |
| 70 | + kfree(enc); | |
| 71 | + kfree(decoded); | |
| 72 | +} | |
| 73 | + | |
| 74 | +static void base64_performance_tests(struct kunit *test) | |
| 75 | +{ | |
| 76 | + /* run on STD variant only */ | |
| 77 | + run_perf_and_check(test, "64B", 64, BASE64_STD); | |
| 78 | + run_perf_and_check(test, "1KB", 1024, BASE64_STD); | |
| 79 | +} | |
| 80 | + | |
| 81 | +/* ---------- Helpers for encode ---------- */ | |
| 82 | +static void expect_encode_ok(struct kunit *test, const u8 *src, int srclen, | |
| 83 | + const char *expected, bool padding, | |
| 84 | + enum base64_variant variant) | |
| 85 | +{ | |
| 86 | + char buf[128]; | |
| 87 | + int encoded_len = base64_encode(src, srclen, buf, padding, variant); | |
| 88 | + | |
| 89 | + buf[encoded_len] = '\0'; | |
| 90 | + | |
| 91 | + KUNIT_EXPECT_EQ(test, encoded_len, strlen(expected)); | |
| 92 | + KUNIT_EXPECT_STREQ(test, buf, expected); | |
| 93 | +} | |
| 94 | + | |
| 95 | +/* ---------- Helpers for decode ---------- */ | |
| 96 | +static void expect_decode_ok(struct kunit *test, const char *src, | |
| 97 | + const u8 *expected, int expected_len, bool padding, | |
| 98 | + enum base64_variant variant) | |
| 99 | +{ | |
| 100 | + u8 buf[128]; | |
| 101 | + int decoded_len = base64_decode(src, strlen(src), buf, padding, variant); | |
| 102 | + | |
| 103 | + KUNIT_EXPECT_EQ(test, decoded_len, expected_len); | |
| 104 | + KUNIT_EXPECT_MEMEQ(test, buf, expected, expected_len); | |
| 105 | +} | |
| 106 | + | |
| 107 | +static void expect_decode_err(struct kunit *test, const char *src, | |
| 108 | + int srclen, bool padding, | |
| 109 | + enum base64_variant variant) | |
| 110 | +{ | |
| 111 | + u8 buf[64]; | |
| 112 | + int decoded_len = base64_decode(src, srclen, buf, padding, variant); | |
| 113 | + | |
| 114 | + KUNIT_EXPECT_EQ(test, decoded_len, -1); | |
| 115 | +} | |
| 116 | + | |
| 117 | +/* ---------- Encode Tests ---------- */ | |
| 118 | +static void base64_std_encode_tests(struct kunit *test) | |
| 119 | +{ | |
| 120 | + /* With padding */ | |
| 121 | + expect_encode_ok(test, (const u8 *)"", 0, "", true, BASE64_STD); | |
| 122 | + expect_encode_ok(test, (const u8 *)"f", 1, "Zg==", true, BASE64_STD); | |
| 123 | + expect_encode_ok(test, (const u8 *)"fo", 2, "Zm8=", true, BASE64_STD); | |
| 124 | + expect_encode_ok(test, (const u8 *)"foo", 3, "Zm9v", true, BASE64_STD); | |
| 125 | + expect_encode_ok(test, (const u8 *)"foob", 4, "Zm9vYg==", true, BASE64_STD); | |
| 126 | + expect_encode_ok(test, (const u8 *)"fooba", 5, "Zm9vYmE=", true, BASE64_STD); | |
| 127 | + expect_encode_ok(test, (const u8 *)"foobar", 6, "Zm9vYmFy", true, BASE64_STD); | |
| 128 | + | |
| 129 | + /* Extra cases with padding */ | |
| 130 | + expect_encode_ok(test, (const u8 *)"Hello, world!", 13, "SGVsbG8sIHdvcmxkIQ==", | |
| 131 | + true, BASE64_STD); | |
| 132 | + expect_encode_ok(test, (const u8 *)"ABCDEFGHIJKLMNOPQRSTUVWXYZ", 26, | |
| 133 | + "QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVo=", true, BASE64_STD); | |
| 134 | + expect_encode_ok(test, (const u8 *)"abcdefghijklmnopqrstuvwxyz", 26, | |
| 135 | + "YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXo=", true, BASE64_STD); | |
| 136 | + expect_encode_ok(test, (const u8 *)"0123456789+/", 12, "MDEyMzQ1Njc4OSsv", | |
| 137 | + true, BASE64_STD); | |
| 138 | + | |
| 139 | + /* Without padding */ | |
| 140 | + expect_encode_ok(test, (const u8 *)"", 0, "", false, BASE64_STD); | |
| 141 | + expect_encode_ok(test, (const u8 *)"f", 1, "Zg", false, BASE64_STD); | |
| 142 | + expect_encode_ok(test, (const u8 *)"fo", 2, "Zm8", false, BASE64_STD); | |
| 143 | + expect_encode_ok(test, (const u8 *)"foo", 3, "Zm9v", false, BASE64_STD); | |
| 144 | + expect_encode_ok(test, (const u8 *)"foob", 4, "Zm9vYg", false, BASE64_STD); | |
| 145 | + expect_encode_ok(test, (const u8 *)"fooba", 5, "Zm9vYmE", false, BASE64_STD); | |
| 146 | + expect_encode_ok(test, (const u8 *)"foobar", 6, "Zm9vYmFy", false, BASE64_STD); | |
| 147 | + | |
| 148 | + /* Extra cases without padding */ | |
| 149 | + expect_encode_ok(test, (const u8 *)"Hello, world!", 13, "SGVsbG8sIHdvcmxkIQ", | |
| 150 | + false, BASE64_STD); | |
| 151 | + expect_encode_ok(test, (const u8 *)"ABCDEFGHIJKLMNOPQRSTUVWXYZ", 26, | |
| 152 | + "QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVo", false, BASE64_STD); | |
| 153 | + expect_encode_ok(test, (const u8 *)"abcdefghijklmnopqrstuvwxyz", 26, | |
| 154 | + "YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXo", false, BASE64_STD); | |
| 155 | + expect_encode_ok(test, (const u8 *)"0123456789+/", 12, "MDEyMzQ1Njc4OSsv", | |
| 156 | + false, BASE64_STD); | |
| 157 | +} | |
| 158 | + | |
| 159 | +/* ---------- Decode Tests ---------- */ | |
| 160 | +static void base64_std_decode_tests(struct kunit *test) | |
| 161 | +{ | |
| 162 | + /* -------- With padding --------*/ | |
| 163 | + expect_decode_ok(test, "", (const u8 *)"", 0, true, BASE64_STD); | |
| 164 | + expect_decode_ok(test, "Zg==", (const u8 *)"f", 1, true, BASE64_STD); | |
| 165 | + expect_decode_ok(test, "Zm8=", (const u8 *)"fo", 2, true, BASE64_STD); | |
| 166 | + expect_decode_ok(test, "Zm9v", (const u8 *)"foo", 3, true, BASE64_STD); | |
| 167 | + expect_decode_ok(test, "Zm9vYg==", (const u8 *)"foob", 4, true, BASE64_STD); | |
| 168 | + expect_decode_ok(test, "Zm9vYmE=", (const u8 *)"fooba", 5, true, BASE64_STD); | |
| 169 | + expect_decode_ok(test, "Zm9vYmFy", (const u8 *)"foobar", 6, true, BASE64_STD); | |
| 170 | + expect_decode_ok(test, "SGVsbG8sIHdvcmxkIQ==", (const u8 *)"Hello, world!", 13, | |
| 171 | + true, BASE64_STD); | |
| 172 | + expect_decode_ok(test, "QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVo=", | |
| 173 | + (const u8 *)"ABCDEFGHIJKLMNOPQRSTUVWXYZ", 26, true, BASE64_STD); | |
| 174 | + expect_decode_ok(test, "YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXo=", | |
| 175 | + (const u8 *)"abcdefghijklmnopqrstuvwxyz", 26, true, BASE64_STD); | |
| 176 | + | |
| 177 | + /* Error cases */ | |
| 178 | + expect_decode_err(test, "Zg=!", 4, true, BASE64_STD); | |
| 179 | + expect_decode_err(test, "Zm$=", 4, true, BASE64_STD); | |
| 180 | + expect_decode_err(test, "Z===", 4, true, BASE64_STD); | |
| 181 | + expect_decode_err(test, "Zg", 2, true, BASE64_STD); | |
| 182 | + expect_decode_err(test, "Zm9v====", 8, true, BASE64_STD); | |
| 183 | + expect_decode_err(test, "Zm==A", 5, true, BASE64_STD); | |
| 184 | + | |
| 185 | + { | |
| 186 | + char with_nul[4] = { 'Z', 'g', '\0', '=' }; | |
| 187 | + | |
| 188 | + expect_decode_err(test, with_nul, 4, true, BASE64_STD); | |
| 189 | + } | |
| 190 | + | |
| 191 | + /* -------- Without padding --------*/ | |
| 192 | + expect_decode_ok(test, "", (const u8 *)"", 0, false, BASE64_STD); | |
| 193 | + expect_decode_ok(test, "Zg", (const u8 *)"f", 1, false, BASE64_STD); | |
| 194 | + expect_decode_ok(test, "Zm8", (const u8 *)"fo", 2, false, BASE64_STD); | |
| 195 | + expect_decode_ok(test, "Zm9v", (const u8 *)"foo", 3, false, BASE64_STD); | |
| 196 | + expect_decode_ok(test, "Zm9vYg", (const u8 *)"foob", 4, false, BASE64_STD); | |
| 197 | + expect_decode_ok(test, "Zm9vYmE", (const u8 *)"fooba", 5, false, BASE64_STD); | |
| 198 | + expect_decode_ok(test, "Zm9vYmFy", (const u8 *)"foobar", 6, false, BASE64_STD); | |
| 199 | + expect_decode_ok(test, "TWFu", (const u8 *)"Man", 3, false, BASE64_STD); | |
| 200 | + expect_decode_ok(test, "SGVsbG8sIHdvcmxkIQ", (const u8 *)"Hello, world!", 13, | |
| 201 | + false, BASE64_STD); | |
| 202 | + expect_decode_ok(test, "QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVo", | |
| 203 | + (const u8 *)"ABCDEFGHIJKLMNOPQRSTUVWXYZ", 26, false, BASE64_STD); | |
| 204 | + expect_decode_ok(test, "YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXo", | |
| 205 | + (const u8 *)"abcdefghijklmnopqrstuvwxyz", 26, false, BASE64_STD); | |
| 206 | + expect_decode_ok(test, "MDEyMzQ1Njc4OSsv", (const u8 *)"0123456789+/", 12, | |
| 207 | + false, BASE64_STD); | |
| 208 | + | |
| 209 | + /* Error cases */ | |
| 210 | + expect_decode_err(test, "Zg=!", 4, false, BASE64_STD); | |
| 211 | + expect_decode_err(test, "Zm$=", 4, false, BASE64_STD); | |
| 212 | + expect_decode_err(test, "Z===", 4, false, BASE64_STD); | |
| 213 | + expect_decode_err(test, "Zg=", 3, false, BASE64_STD); | |
| 214 | + expect_decode_err(test, "Zm9v====", 8, false, BASE64_STD); | |
| 215 | + expect_decode_err(test, "Zm==v", 4, false, BASE64_STD); | |
| 216 | + | |
| 217 | + { | |
| 218 | + char with_nul[4] = { 'Z', 'g', '\0', '=' }; | |
| 219 | + | |
| 220 | + expect_decode_err(test, with_nul, 4, false, BASE64_STD); | |
| 221 | + } | |
| 222 | +} | |
| 223 | + | |
| 224 | +/* ---------- Variant tests (URLSAFE / IMAP) ---------- */ | |
| 225 | +static void base64_variant_tests(struct kunit *test) | |
| 226 | +{ | |
| 227 | + const u8 sample1[] = { 0x00, 0xfb, 0xff, 0x7f, 0x80 }; | |
| 228 | + char std_buf[128], url_buf[128], imap_buf[128]; | |
| 229 | + u8 back[128]; | |
| 230 | + int n_std, n_url, n_imap, m; | |
| 231 | + int i; | |
| 232 | + | |
| 233 | + n_std = base64_encode(sample1, sizeof(sample1), std_buf, false, BASE64_STD); | |
| 234 | + n_url = base64_encode(sample1, sizeof(sample1), url_buf, false, BASE64_URLSAFE); | |
| 235 | + std_buf[n_std] = '\0'; | |
| 236 | + url_buf[n_url] = '\0'; | |
| 237 | + | |
| 238 | + for (i = 0; i < n_std; i++) { | |
| 239 | + if (std_buf[i] == '+') | |
| 240 | + std_buf[i] = '-'; | |
| 241 | + else if (std_buf[i] == '/') | |
| 242 | + std_buf[i] = '_'; | |
| 243 | + } | |
| 244 | + KUNIT_EXPECT_STREQ(test, std_buf, url_buf); | |
| 245 | + | |
| 246 | + m = base64_decode(url_buf, n_url, back, false, BASE64_URLSAFE); | |
| 247 | + KUNIT_EXPECT_EQ(test, m, (int)sizeof(sample1)); | |
| 248 | + KUNIT_EXPECT_MEMEQ(test, back, sample1, sizeof(sample1)); | |
| 249 | + | |
| 250 | + n_std = base64_encode(sample1, sizeof(sample1), std_buf, false, BASE64_STD); | |
| 251 | + n_imap = base64_encode(sample1, sizeof(sample1), imap_buf, false, BASE64_IMAP); | |
| 252 | + std_buf[n_std] = '\0'; | |
| 253 | + imap_buf[n_imap] = '\0'; | |
| 254 | + | |
| 255 | + for (i = 0; i < n_std; i++) | |
| 256 | + if (std_buf[i] == '/') | |
| 257 | + std_buf[i] = ','; | |
| 258 | + KUNIT_EXPECT_STREQ(test, std_buf, imap_buf); | |
| 259 | + | |
| 260 | + m = base64_decode(imap_buf, n_imap, back, false, BASE64_IMAP); | |
| 261 | + KUNIT_EXPECT_EQ(test, m, (int)sizeof(sample1)); | |
| 262 | + KUNIT_EXPECT_MEMEQ(test, back, sample1, sizeof(sample1)); | |
| 263 | + | |
| 264 | + { | |
| 265 | + const char *bad = "Zg=="; | |
| 266 | + u8 tmp[8]; | |
| 267 | + | |
| 268 | + m = base64_decode(bad, strlen(bad), tmp, false, BASE64_URLSAFE); | |
| 269 | + KUNIT_EXPECT_EQ(test, m, -1); | |
| 270 | + | |
| 271 | + m = base64_decode(bad, strlen(bad), tmp, false, BASE64_IMAP); | |
| 272 | + KUNIT_EXPECT_EQ(test, m, -1); | |
| 273 | + } | |
| 274 | +} | |
| 275 | + | |
| 276 | +/* ---------- Test registration ---------- */ | |
| 277 | +static struct kunit_case base64_test_cases[] = { | |
| 278 | + KUNIT_CASE(base64_performance_tests), | |
| 279 | + KUNIT_CASE(base64_std_encode_tests), | |
| 280 | + KUNIT_CASE(base64_std_decode_tests), | |
| 281 | + KUNIT_CASE(base64_variant_tests), | |
| 282 | + {} | |
| 283 | +}; | |
| 284 | + | |
| 285 | +static struct kunit_suite base64_test_suite = { | |
| 286 | + .name = "base64", | |
| 287 | + .test_cases = base64_test_cases, | |
| 288 | +}; | |
| 289 | + | |
| 290 | +kunit_test_suite(base64_test_suite); | |
| 291 | + | |
| 292 | +MODULE_AUTHOR("Guan-Chun Wu <409411716@gms.tku.edu.tw>"); | |
| 293 | +MODULE_DESCRIPTION("KUnit tests for Base64 encoding/decoding, including performance checks"); | |
| 294 | +MODULE_LICENSE("GPL"); | |
| @@ -12,7 +12,7 @@ · 11 unmodified lines | ||
| 12 | 12 | |
| 13 | 13 | /* out-of-line parts */ |
| 14 | 14 | |
| 15 | −#if !defined(INLINE_COPY_FROM_USER) || defined(CONFIG_RUST) | |
| 15 | +#if !defined(INLINE_COPY_FROM_USER) | |
| 16 | 16 | unsigned long _copy_from_user(void *to, const void __user *from, unsigned long n) |
| 17 | 17 | { |
| 18 | 18 | return _inline_copy_from_user(to, from, n); |
| @@ -20,7 +20,7 @@ unsigned long _copy_from_user(void *to, const void __user *from, unsigned long n · 1 unmodified line | ||
| 20 | 20 | EXPORT_SYMBOL(_copy_from_user); |
| 21 | 21 | #endif |
| 22 | 22 | |
| 23 | −#if !defined(INLINE_COPY_TO_USER) || defined(CONFIG_RUST) | |
| 23 | +#if !defined(INLINE_COPY_TO_USER) | |
| 24 | 24 | unsigned long _copy_to_user(void __user *to, const void *from, unsigned long n) |
| 25 | 25 | { |
| 26 | 26 | return _inline_copy_to_user(to, from, n); |
| @@ -73,21 +73,6 @@ static const uint64_t PRIME64_3 = 1609587929392839161ULL; · 72 unmodified lines | ||
| 73 | 73 | static const uint64_t PRIME64_4 = 9650029242287828579ULL; |
| 74 | 74 | static const uint64_t PRIME64_5 = 2870177450012600261ULL; |
| 75 | 75 | |
| 76 | −/*-************************** | |
| 77 | − * Utils | |
| 78 | − ***************************/ | |
| 79 | −void xxh32_copy_state(struct xxh32_state *dst, const struct xxh32_state *src) | |
| 80 | −{ | |
| 81 | − memcpy(dst, src, sizeof(*dst)); | |
| 82 | −} | |
| 83 | −EXPORT_SYMBOL(xxh32_copy_state); | |
| 84 | − | |
| 85 | −void xxh64_copy_state(struct xxh64_state *dst, const struct xxh64_state *src) | |
| 86 | −{ | |
| 87 | − memcpy(dst, src, sizeof(*dst)); | |
| 88 | −} | |
| 89 | −EXPORT_SYMBOL(xxh64_copy_state); | |
| 90 | − | |
| 91 | 76 | /*-*************************** |
| 92 | 77 | * Simple Hash Functions |
| 93 | 78 | ****************************/ |
| @@ -239,20 +224,6 @@ EXPORT_SYMBOL(xxh64); · 145 unmodified lines | ||
| 239 | 224 | /*-************************************************** |
| 240 | 225 | * Advanced Hash Functions |
| 241 | 226 | ***************************************************/ |
| 242 | −void xxh32_reset(struct xxh32_state *statePtr, const uint32_t seed) | |
| 243 | −{ | |
| 244 | − /* use a local state for memcpy() to avoid strict-aliasing warnings */ | |
| 245 | − struct xxh32_state state; | |
| 246 | − | |
| 247 | − memset(&state, 0, sizeof(state)); | |
| 248 | − state.v1 = seed + PRIME32_1 + PRIME32_2; | |
| 249 | − state.v2 = seed + PRIME32_2; | |
| 250 | − state.v3 = seed + 0; | |
| 251 | − state.v4 = seed - PRIME32_1; | |
| 252 | − memcpy(statePtr, &state, sizeof(state)); | |
| 253 | −} | |
| 254 | −EXPORT_SYMBOL(xxh32_reset); | |
| 255 | − | |
| 256 | 227 | void xxh64_reset(struct xxh64_state *statePtr, const uint64_t seed) |
| 257 | 228 | { |
| 258 | 229 | /* use a local state for memcpy() to avoid strict-aliasing warnings */ |
| @@ -20,7 +20,6 @@ struct xz_dec_bcj { · 19 unmodified lines | ||
| 20 | 20 | enum { |
| 21 | 21 | BCJ_X86 = 4, /* x86 or x86-64 */ |
| 22 | 22 | BCJ_POWERPC = 5, /* Big endian only */ |
| 23 | − BCJ_IA64 = 6, /* Big or little endian */ | |
| 24 | 23 | BCJ_ARM = 7, /* Little endian only */ |
| 25 | 24 | BCJ_ARMTHUMB = 8, /* Little endian only */ |
| 26 | 25 | BCJ_SPARC = 9, /* Big or little endian */ |
| @@ -180,92 +179,6 @@ static size_t bcj_powerpc(struct xz_dec_bcj *s, uint8_t *buf, size_t size) · 153 unmodified lines | ||
| 180 | 179 | } |
| 181 | 180 | #endif |
| 182 | 181 | |
| 183 | −#ifdef XZ_DEC_IA64 | |
| 184 | −static size_t bcj_ia64(struct xz_dec_bcj *s, uint8_t *buf, size_t size) | |
| 185 | −{ | |
| 186 | − static const uint8_t branch_table[32] = { | |
| 187 | − 0, 0, 0, 0, 0, 0, 0, 0, | |
| 188 | − 0, 0, 0, 0, 0, 0, 0, 0, | |
| 189 | − 4, 4, 6, 6, 0, 0, 7, 7, | |
| 190 | − 4, 4, 0, 0, 4, 4, 0, 0 | |
| 191 | − }; | |
| 192 | − | |
| 193 | − /* | |
| 194 | − * The local variables take a little bit stack space, but it's less | |
| 195 | − * than what LZMA2 decoder takes, so it doesn't make sense to reduce | |
| 196 | − * stack usage here without doing that for the LZMA2 decoder too. | |
| 197 | − */ | |
| 198 | − | |
| 199 | − /* Loop counters */ | |
| 200 | − size_t i; | |
| 201 | − size_t j; | |
| 202 | − | |
| 203 | − /* Instruction slot (0, 1, or 2) in the 128-bit instruction word */ | |
| 204 | − uint32_t slot; | |
| 205 | − | |
| 206 | − /* Bitwise offset of the instruction indicated by slot */ | |
| 207 | − uint32_t bit_pos; | |
| 208 | − | |
| 209 | − /* bit_pos split into byte and bit parts */ | |
| 210 | − uint32_t byte_pos; | |
| 211 | − uint32_t bit_res; | |
| 212 | − | |
| 213 | − /* Address part of an instruction */ | |
| 214 | − uint32_t addr; | |
| 215 | − | |
| 216 | − /* Mask used to detect which instructions to convert */ | |
| 217 | − uint32_t mask; | |
| 218 | − | |
| 219 | − /* 41-bit instruction stored somewhere in the lowest 48 bits */ | |
| 220 | − uint64_t instr; | |
| 221 | − | |
| 222 | − /* Instruction normalized with bit_res for easier manipulation */ | |
| 223 | − uint64_t norm; | |
| 224 | − | |
| 225 | − size &= ~(size_t)15; | |
| 226 | − | |
| 227 | − for (i = 0; i < size; i += 16) { | |
| 228 | − mask = branch_table[buf[i] & 0x1F]; | |
| 229 | − for (slot = 0, bit_pos = 5; slot < 3; ++slot, bit_pos += 41) { | |
| 230 | − if (((mask >> slot) & 1) == 0) | |
| 231 | − continue; | |
| 232 | − | |
| 233 | − byte_pos = bit_pos >> 3; | |
| 234 | − bit_res = bit_pos & 7; | |
| 235 | − instr = 0; | |
| 236 | − for (j = 0; j < 6; ++j) | |
| 237 | − instr |= (uint64_t)(buf[i + j + byte_pos]) | |
| 238 | − << (8 * j); | |
| 239 | − | |
| 240 | − norm = instr >> bit_res; | |
| 241 | − | |
| 242 | − if (((norm >> 37) & 0x0F) == 0x05 | |
| 243 | − && ((norm >> 9) & 0x07) == 0) { | |
| 244 | − addr = (norm >> 13) & 0x0FFFFF; | |
| 245 | − addr |= ((uint32_t)(norm >> 36) & 1) << 20; | |
| 246 | − addr <<= 4; | |
| 247 | − addr -= s->pos + (uint32_t)i; | |
| 248 | − addr >>= 4; | |
| 249 | − | |
| 250 | − norm &= ~((uint64_t)0x8FFFFF << 13); | |
| 251 | − norm |= (uint64_t)(addr & 0x0FFFFF) << 13; | |
| 252 | − norm |= (uint64_t)(addr & 0x100000) | |
| 253 | − << (36 - 20); | |
| 254 | − | |
| 255 | − instr &= (1 << bit_res) - 1; | |
| 256 | − instr |= norm << bit_res; | |
| 257 | − | |
| 258 | − for (j = 0; j < 6; j++) | |
| 259 | − buf[i + j + byte_pos] | |
| 260 | − = (uint8_t)(instr >> (8 * j)); | |
| 261 | − } | |
| 262 | − } | |
| 263 | − } | |
| 264 | − | |
| 265 | − return i; | |
| 266 | −} | |
| 267 | −#endif | |
| 268 | − | |
| 269 | 182 | #ifdef XZ_DEC_ARM |
| 270 | 183 | static size_t bcj_arm(struct xz_dec_bcj *s, uint8_t *buf, size_t size) |
| 271 | 184 | { |
| @@ -509,11 +422,6 @@ static void bcj_apply(struct xz_dec_bcj *s, · 237 unmodified lines | ||
| 509 | 422 | filtered = bcj_powerpc(s, buf, size); |
| 510 | 423 | break; |
| 511 | 424 | #endif |
| 512 | −#ifdef XZ_DEC_IA64 | |
| 513 | − case BCJ_IA64: | |
| 514 | − filtered = bcj_ia64(s, buf, size); | |
| 515 | − break; | |
| 516 | −#endif | |
| 517 | 425 | #ifdef XZ_DEC_ARM |
| 518 | 426 | case BCJ_ARM: |
| 519 | 427 | filtered = bcj_arm(s, buf, size); |
| @@ -699,9 +607,6 @@ enum xz_ret xz_dec_bcj_reset(struct xz_dec_bcj *s, uint8_t id) · 179 unmodified lines | ||
| 699 | 607 | #ifdef XZ_DEC_POWERPC |
| 700 | 608 | case BCJ_POWERPC: |
| 701 | 609 | #endif |
| 702 | −#ifdef XZ_DEC_IA64 | |
| 703 | − case BCJ_IA64: | |
| 704 | −#endif | |
| 705 | 610 | #ifdef XZ_DEC_ARM |
| 706 | 611 | case BCJ_ARM: |
| 707 | 612 | #endif |
| @@ -24,9 +24,6 @@ · 23 unmodified lines | ||
| 24 | 24 | # ifdef CONFIG_XZ_DEC_POWERPC |
| 25 | 25 | # define XZ_DEC_POWERPC |
| 26 | 26 | # endif |
| 27 | −# ifdef CONFIG_XZ_DEC_IA64 | |
| 28 | −# define XZ_DEC_IA64 | |
| 29 | −# endif | |
| 30 | 27 | # ifdef CONFIG_XZ_DEC_ARM |
| 31 | 28 | # define XZ_DEC_ARM |
| 32 | 29 | # endif |
| @@ -103,7 +100,6 @@ · 70 unmodified lines | ||
| 103 | 100 | */ |
| 104 | 101 | #ifndef XZ_DEC_BCJ |
| 105 | 102 | # if defined(XZ_DEC_X86) || defined(XZ_DEC_POWERPC) \ |
| 106 | − || defined(XZ_DEC_IA64) \ | |
| 107 | 103 | || defined(XZ_DEC_ARM) || defined(XZ_DEC_ARMTHUMB) \ |
| 108 | 104 | || defined(XZ_DEC_SPARC) || defined(XZ_DEC_ARM64) \ |
| 109 | 105 | || defined(XZ_DEC_RISCV) |
| @@ -100,6 +100,7 @@ obj-$(CONFIG_NUMA) += memory-tiers.o · 99 unmodified lines | ||
| 100 | 100 | obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o |
| 101 | 101 | obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o |
| 102 | 102 | obj-$(CONFIG_PAGE_COUNTER) += page_counter.o |
| 103 | +obj-$(CONFIG_LIVEUPDATE) += memfd_luo.o | |
| 103 | 104 | obj-$(CONFIG_MEMCG_V1) += memcontrol-v1.o |
| 104 | 105 | obj-$(CONFIG_MEMCG) += memcontrol.o vmpressure.o |
| 105 | 106 | ifdef CONFIG_SWAP |
| @@ -1582,6 +1582,12 @@ void __meminit __init_page_from_nid(unsigned long pfn, int nid); · 1581 unmodified lines | ||
| 1582 | 1582 | unsigned long shrink_slab(gfp_t gfp_mask, int nid, struct mem_cgroup *memcg, |
| 1583 | 1583 | int priority); |
| 1584 | 1584 | |
| 1585 | +int shmem_add_to_page_cache(struct folio *folio, | |
| 1586 | + struct address_space *mapping, | |
| 1587 | + pgoff_t index, void *expected, gfp_t gfp); | |
| 1588 | +int shmem_inode_acct_blocks(struct inode *inode, long pages); | |
| 1589 | +bool shmem_recalc_inode(struct inode *inode, long alloced, long swapped); | |
| 1590 | + | |
| 1585 | 1591 | #ifdef CONFIG_SHRINKER_DEBUG |
| 1586 | 1592 | static inline __printf(2, 0) int shrinker_debugfs_name_alloc( |
| 1587 | 1593 | struct shrinker *shrinker, const char *fmt, va_list ap) |
| @@ -2445,60 +2445,59 @@ int reserve_mem_release_by_name(const char *name) · 2444 unmodified lines | ||
| 2445 | 2445 | #define MEMBLOCK_KHO_FDT "memblock" |
| 2446 | 2446 | #define MEMBLOCK_KHO_NODE_COMPATIBLE "memblock-v1" |
| 2447 | 2447 | #define RESERVE_MEM_KHO_NODE_COMPATIBLE "reserve-mem-v1" |
| 2448 | −static struct page *kho_fdt; | |
| 2449 | 2448 | |
| 2450 | −static int reserve_mem_kho_finalize(struct kho_serialization *ser) | |
| 2449 | +static int __init reserved_mem_preserve(void) | |
| 2451 | 2450 | { |
| 2452 | − int err = 0, i; | |
| 2451 | + unsigned int nr_preserved = 0; | |
| 2452 | + int err; | |
| 2453 | 2453 | |
| 2454 | − for (i = 0; i < reserved_mem_count; i++) { | |
| 2454 | + for (unsigned int i = 0; i < reserved_mem_count; i++, nr_preserved++) { | |
| 2455 | 2455 | struct reserve_mem_table *map = &reserved_mem_table[i]; |
| 2456 | 2456 | struct page *page = phys_to_page(map->start); |
| 2457 | 2457 | unsigned int nr_pages = map->size >> PAGE_SHIFT; |
| 2458 | 2458 | |
| 2459 | − err |= kho_preserve_pages(page, nr_pages); | |
| 2459 | + err = kho_preserve_pages(page, nr_pages); | |
| 2460 | + if (err) | |
| 2461 | + goto err_unpreserve; | |
| 2460 | 2462 | } |
| 2461 | 2463 | |
| 2462 | − err |= kho_preserve_folio(page_folio(kho_fdt)); | |
| 2463 | − err |= kho_add_subtree(ser, MEMBLOCK_KHO_FDT, page_to_virt(kho_fdt)); | |
| 2464 | + return 0; | |
| 2464 | 2465 | |
| 2465 | − return notifier_from_errno(err); | |
| 2466 | −} | |
| 2466 | +err_unpreserve: | |
| 2467 | + for (unsigned int i = 0; i < nr_preserved; i++) { | |
| 2468 | + struct reserve_mem_table *map = &reserved_mem_table[i]; | |
| 2469 | + struct page *page = phys_to_page(map->start); | |
| 2470 | + unsigned int nr_pages = map->size >> PAGE_SHIFT; | |
| 2467 | 2471 | |
| 2468 | −static int reserve_mem_kho_notifier(struct notifier_block *self, | |
| 2469 | − unsigned long cmd, void *v) | |
| 2470 | −{ | |
| 2471 | − switch (cmd) { | |
| 2472 | − case KEXEC_KHO_FINALIZE: | |
| 2473 | − return reserve_mem_kho_finalize((struct kho_serialization *)v); | |
| 2474 | − case KEXEC_KHO_ABORT: | |
| 2475 | − return NOTIFY_DONE; | |
| 2476 | − default: | |
| 2477 | − return NOTIFY_BAD; | |
| 2472 | + kho_unpreserve_pages(page, nr_pages); | |
| 2478 | 2473 | } |
| 2479 | −} | |
| 2480 | 2474 | |
| 2481 | −static struct notifier_block reserve_mem_kho_nb = { | |
| 2482 | − .notifier_call = reserve_mem_kho_notifier, | |
| 2483 | −}; | |
| 2475 | + return err; | |
| 2476 | +} | |
| 2484 | 2477 | |
| 2485 | 2478 | static int __init prepare_kho_fdt(void) |
| 2486 | 2479 | { |
| 2487 | − int err = 0, i; | |
| 2480 | + struct page *fdt_page; | |
| 2488 | 2481 | void *fdt; |
| 2482 | + int err; | |
| 2489 | 2483 | |
| 2490 | − kho_fdt = alloc_page(GFP_KERNEL); | |
| 2491 | − if (!kho_fdt) | |
| 2492 | − return -ENOMEM; | |
| 2484 | + fdt_page = alloc_page(GFP_KERNEL); | |
| 2485 | + if (!fdt_page) { | |
| 2486 | + err = -ENOMEM; | |
| 2487 | + goto err_report; | |
| 2488 | + } | |
| 2493 | 2489 | |
| 2494 | − fdt = page_to_virt(kho_fdt); | |
| 2490 | + fdt = page_to_virt(fdt_page); | |
| 2491 | + err = kho_preserve_pages(fdt_page, 1); | |
| 2492 | + if (err) | |
| 2493 | + goto err_free_fdt; | |
| 2495 | 2494 | |
| 2496 | 2495 | err |= fdt_create(fdt, PAGE_SIZE); |
| 2497 | 2496 | err |= fdt_finish_reservemap(fdt); |
| 2498 | − | |
| 2499 | 2497 | err |= fdt_begin_node(fdt, ""); |
| 2500 | 2498 | err |= fdt_property_string(fdt, "compatible", MEMBLOCK_KHO_NODE_COMPATIBLE); |
| 2501 | − for (i = 0; i < reserved_mem_count; i++) { | |
| 2499 | + | |
| 2500 | + for (unsigned int i = 0; !err && i < reserved_mem_count; i++) { | |
| 2502 | 2501 | struct reserve_mem_table *map = &reserved_mem_table[i]; |
| 2503 | 2502 | |
| 2504 | 2503 | err |= fdt_begin_node(fdt, map->name); |
| @@ -2508,14 +2507,29 @@ static int __init prepare_kho_fdt(void) · 3 unmodified lines | ||
| 2508 | 2507 | err |= fdt_end_node(fdt); |
| 2509 | 2508 | } |
| 2510 | 2509 | err |= fdt_end_node(fdt); |
| 2511 | − | |
| 2512 | 2510 | err |= fdt_finish(fdt); |
| 2513 | 2511 | |
| 2514 | − if (err) { | |
| 2515 | − pr_err("failed to prepare memblock FDT for KHO: %d\n", err); | |
| 2516 | − put_page(kho_fdt); | |
| 2517 | − kho_fdt = NULL; | |
| 2518 | − } | |
| 2512 | + if (err) | |
| 2513 | + goto err_unpreserve_fdt; | |
| 2514 | + | |
| 2515 | + err = kho_add_subtree(MEMBLOCK_KHO_FDT, fdt); | |
| 2516 | + if (err) | |
| 2517 | + goto err_unpreserve_fdt; | |
| 2518 | + | |
| 2519 | + err = reserved_mem_preserve(); | |
| 2520 | + if (err) | |
| 2521 | + goto err_remove_subtree; | |
| 2522 | + | |
| 2523 | + return 0; | |
| 2524 | + | |
| 2525 | +err_remove_subtree: | |
| 2526 | + kho_remove_subtree(fdt); | |
| 2527 | +err_unpreserve_fdt: | |
| 2528 | + kho_unpreserve_pages(fdt_page, 1); | |
| 2529 | +err_free_fdt: | |
| 2530 | + put_page(fdt_page); | |
| 2531 | +err_report: | |
| 2532 | + pr_err("failed to prepare memblock FDT for KHO: %d\n", err); | |
| 2519 | 2533 | |
| 2520 | 2534 | return err; |
| 2521 | 2535 | } |
| @@ -2530,13 +2544,6 @@ static int __init reserve_mem_init(void) · 8 unmodified lines | ||
| 2530 | 2544 | err = prepare_kho_fdt(); |
| 2531 | 2545 | if (err) |
| 2532 | 2546 | return err; |
| 2533 | − | |
| 2534 | − err = register_kho_notifier(&reserve_mem_kho_nb); | |
| 2535 | − if (err) { | |
| 2536 | − put_page(kho_fdt); | |
| 2537 | − kho_fdt = NULL; | |
| 2538 | − } | |
| 2539 | − | |
| 2540 | 2547 | return err; |
| 2541 | 2548 | } |
| 2542 | 2549 | late_initcall(reserve_mem_init); |
| @@ -0,0 +1,516 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0 | |
| 2 | + | |
| 3 | +/* | |
| 4 | + * Copyright (c) 2025, Google LLC. | |
| 5 | + * Pasha Tatashin <pasha.tatashin@soleen.com> | |
| 6 | + * | |
| 7 | + * Copyright (C) 2025 Amazon.com Inc. or its affiliates. | |
| 8 | + * Pratyush Yadav <ptyadav@amazon.de> | |
| 9 | + */ | |
| 10 | + | |
| 11 | +/** | |
| 12 | + * DOC: Memfd Preservation via LUO | |
| 13 | + * | |
| 14 | + * Overview | |
| 15 | + * ======== | |
| 16 | + * | |
| 17 | + * Memory file descriptors (memfd) can be preserved over a kexec using the Live | |
| 18 | + * Update Orchestrator (LUO) file preservation. This allows userspace to | |
| 19 | + * transfer its memory contents to the next kernel after a kexec. | |
| 20 | + * | |
| 21 | + * The preservation is not intended to be transparent. Only select properties of | |
| 22 | + * the file are preserved. All others are reset to default. The preserved | |
| 23 | + * properties are described below. | |
| 24 | + * | |
| 25 | + * .. note:: | |
| 26 | + * The LUO API is not stabilized yet, so the preserved properties of a memfd | |
| 27 | + * are also not stable and are subject to backwards incompatible changes. | |
| 28 | + * | |
| 29 | + * .. note:: | |
| 30 | + * Currently a memfd backed by Hugetlb is not supported. Memfds created | |
| 31 | + * with ``MFD_HUGETLB`` will be rejected. | |
| 32 | + * | |
| 33 | + * Preserved Properties | |
| 34 | + * ==================== | |
| 35 | + * | |
| 36 | + * The following properties of the memfd are preserved across kexec: | |
| 37 | + * | |
| 38 | + * File Contents | |
| 39 | + * All data stored in the file is preserved. | |
| 40 | + * | |
| 41 | + * File Size | |
| 42 | + * The size of the file is preserved. Holes in the file are filled by | |
| 43 | + * allocating pages for them during preservation. | |
| 44 | + * | |
| 45 | + * File Position | |
| 46 | + * The current file position is preserved, allowing applications to continue | |
| 47 | + * reading/writing from their last position. | |
| 48 | + * | |
| 49 | + * File Status Flags | |
| 50 | + * memfds are always opened with ``O_RDWR`` and ``O_LARGEFILE``. This property | |
| 51 | + * is maintained. | |
| 52 | + * | |
| 53 | + * Non-Preserved Properties | |
| 54 | + * ======================== | |
| 55 | + * | |
| 56 | + * All properties which are not preserved must be assumed to be reset to | |
| 57 | + * default. This section describes some of those properties which may be more of | |
| 58 | + * note. | |
| 59 | + * | |
| 60 | + * ``FD_CLOEXEC`` flag | |
| 61 | + * A memfd can be created with the ``MFD_CLOEXEC`` flag that sets the | |
| 62 | + * ``FD_CLOEXEC`` on the file. This flag is not preserved and must be set | |
| 63 | + * again after restore via ``fcntl()``. | |
| 64 | + * | |
| 65 | + * Seals | |
| 66 | + * File seals are not preserved. The file is unsealed on restore and if | |
| 67 | + * needed, must be sealed again via ``fcntl()``. | |
| 68 | + */ | |
| 69 | + | |
| 70 | +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt | |
| 71 | + | |
| 72 | +#include <linux/bits.h> | |
| 73 | +#include <linux/err.h> | |
| 74 | +#include <linux/file.h> | |
| 75 | +#include <linux/io.h> | |
| 76 | +#include <linux/kexec_handover.h> | |
| 77 | +#include <linux/kho/abi/memfd.h> | |
| 78 | +#include <linux/liveupdate.h> | |
| 79 | +#include <linux/shmem_fs.h> | |
| 80 | +#include <linux/vmalloc.h> | |
| 81 | +#include "internal.h" | |
| 82 | + | |
| 83 | +static int memfd_luo_preserve_folios(struct file *file, | |
| 84 | + struct kho_vmalloc *kho_vmalloc, | |
| 85 | + struct memfd_luo_folio_ser **out_folios_ser, | |
| 86 | + u64 *nr_foliosp) | |
| 87 | +{ | |
| 88 | + struct inode *inode = file_inode(file); | |
| 89 | + struct memfd_luo_folio_ser *folios_ser; | |
| 90 | + unsigned int max_folios; | |
| 91 | + long i, size, nr_pinned; | |
| 92 | + struct folio **folios; | |
| 93 | + int err = -EINVAL; | |
| 94 | + pgoff_t offset; | |
| 95 | + u64 nr_folios; | |
| 96 | + | |
| 97 | + size = i_size_read(inode); | |
| 98 | + /* | |
| 99 | + * If the file has zero size, then the folios and nr_folios properties | |
| 100 | + * are not set. | |
| 101 | + */ | |
| 102 | + if (!size) { | |
| 103 | + *nr_foliosp = 0; | |
| 104 | + *out_folios_ser = NULL; | |
| 105 | + memset(kho_vmalloc, 0, sizeof(*kho_vmalloc)); | |
| 106 | + return 0; | |
| 107 | + } | |
| 108 | + | |
| 109 | + /* | |
| 110 | + * Guess the number of folios based on inode size. Real number might end | |
| 111 | + * up being smaller if there are higher order folios. | |
| 112 | + */ | |
| 113 | + max_folios = PAGE_ALIGN(size) / PAGE_SIZE; | |
| 114 | + folios = kvmalloc_array(max_folios, sizeof(*folios), GFP_KERNEL); | |
| 115 | + if (!folios) | |
| 116 | + return -ENOMEM; | |
| 117 | + | |
| 118 | + /* | |
| 119 | + * Pin the folios so they don't move around behind our back. This also | |
| 120 | + * ensures none of the folios are in CMA -- which ensures they don't | |
| 121 | + * fall in KHO scratch memory. It also moves swapped out folios back to | |
| 122 | + * memory. | |
| 123 | + * | |
| 124 | + * A side effect of doing this is that it allocates a folio for all | |
| 125 | + * indices in the file. This might waste memory on sparse memfds. If | |
| 126 | + * that is really a problem in the future, we can have a | |
| 127 | + * memfd_pin_folios() variant that does not allocate a page on empty | |
| 128 | + * slots. | |
| 129 | + */ | |
| 130 | + nr_pinned = memfd_pin_folios(file, 0, size - 1, folios, max_folios, | |
| 131 | + &offset); | |
| 132 | + if (nr_pinned < 0) { | |
| 133 | + err = nr_pinned; | |
| 134 | + pr_err("failed to pin folios: %d\n", err); | |
| 135 | + goto err_free_folios; | |
| 136 | + } | |
| 137 | + nr_folios = nr_pinned; | |
| 138 | + | |
| 139 | + folios_ser = vcalloc(nr_folios, sizeof(*folios_ser)); | |
| 140 | + if (!folios_ser) { | |
| 141 | + err = -ENOMEM; | |
| 142 | + goto err_unpin; | |
| 143 | + } | |
| 144 | + | |
| 145 | + for (i = 0; i < nr_folios; i++) { | |
| 146 | + struct memfd_luo_folio_ser *pfolio = &folios_ser[i]; | |
| 147 | + struct folio *folio = folios[i]; | |
| 148 | + unsigned int flags = 0; | |
| 149 | + | |
| 150 | + err = kho_preserve_folio(folio); | |
| 151 | + if (err) | |
| 152 | + goto err_unpreserve; | |
| 153 | + | |
| 154 | + if (folio_test_dirty(folio)) | |
| 155 | + flags |= MEMFD_LUO_FOLIO_DIRTY; | |
| 156 | + if (folio_test_uptodate(folio)) | |
| 157 | + flags |= MEMFD_LUO_FOLIO_UPTODATE; | |
| 158 | + | |
| 159 | + pfolio->pfn = folio_pfn(folio); | |
| 160 | + pfolio->flags = flags; | |
| 161 | + pfolio->index = folio->index; | |
| 162 | + } | |
| 163 | + | |
| 164 | + err = kho_preserve_vmalloc(folios_ser, kho_vmalloc); | |
| 165 | + if (err) | |
| 166 | + goto err_unpreserve; | |
| 167 | + | |
| 168 | + kvfree(folios); | |
| 169 | + *nr_foliosp = nr_folios; | |
| 170 | + *out_folios_ser = folios_ser; | |
| 171 | + | |
| 172 | + /* | |
| 173 | + * Note: folios_ser is purposely not freed here. It is preserved | |
| 174 | + * memory (via KHO). In the 'unpreserve' path, we use the vmap pointer | |
| 175 | + * that is passed via private_data. | |
| 176 | + */ | |
| 177 | + return 0; | |
| 178 | + | |
| 179 | +err_unpreserve: | |
| 180 | + for (i = i - 1; i >= 0; i--) | |
| 181 | + kho_unpreserve_folio(folios[i]); | |
| 182 | + vfree(folios_ser); | |
| 183 | +err_unpin: | |
| 184 | + unpin_folios(folios, nr_folios); | |
| 185 | +err_free_folios: | |
| 186 | + kvfree(folios); | |
| 187 | + | |
| 188 | + return err; | |
| 189 | +} | |
| 190 | + | |
| 191 | +static void memfd_luo_unpreserve_folios(struct kho_vmalloc *kho_vmalloc, | |
| 192 | + struct memfd_luo_folio_ser *folios_ser, | |
| 193 | + u64 nr_folios) | |
| 194 | +{ | |
| 195 | + long i; | |
| 196 | + | |
| 197 | + if (!nr_folios) | |
| 198 | + return; | |
| 199 | + | |
| 200 | + kho_unpreserve_vmalloc(kho_vmalloc); | |
| 201 | + | |
| 202 | + for (i = 0; i < nr_folios; i++) { | |
| 203 | + const struct memfd_luo_folio_ser *pfolio = &folios_ser[i]; | |
| 204 | + struct folio *folio; | |
| 205 | + | |
| 206 | + if (!pfolio->pfn) | |
| 207 | + continue; | |
| 208 | + | |
| 209 | + folio = pfn_folio(pfolio->pfn); | |
| 210 | + | |
| 211 | + kho_unpreserve_folio(folio); | |
| 212 | + unpin_folio(folio); | |
| 213 | + } | |
| 214 | + | |
| 215 | + vfree(folios_ser); | |
| 216 | +} | |
| 217 | + | |
| 218 | +static int memfd_luo_preserve(struct liveupdate_file_op_args *args) | |
| 219 | +{ | |
| 220 | + struct inode *inode = file_inode(args->file); | |
| 221 | + struct memfd_luo_folio_ser *folios_ser; | |
| 222 | + struct memfd_luo_ser *ser; | |
| 223 | + u64 nr_folios; | |
| 224 | + int err = 0; | |
| 225 | + | |
| 226 | + inode_lock(inode); | |
| 227 | + shmem_freeze(inode, true); | |
| 228 | + | |
| 229 | + /* Allocate the main serialization structure in preserved memory */ | |
| 230 | + ser = kho_alloc_preserve(sizeof(*ser)); | |
| 231 | + if (IS_ERR(ser)) { | |
| 232 | + err = PTR_ERR(ser); | |
| 233 | + goto err_unlock; | |
| 234 | + } | |
| 235 | + | |
| 236 | + ser->pos = args->file->f_pos; | |
| 237 | + ser->size = i_size_read(inode); | |
| 238 | + | |
| 239 | + err = memfd_luo_preserve_folios(args->file, &ser->folios, | |
| 240 | + &folios_ser, &nr_folios); | |
| 241 | + if (err) | |
| 242 | + goto err_free_ser; | |
| 243 | + | |
| 244 | + ser->nr_folios = nr_folios; | |
| 245 | + inode_unlock(inode); | |
| 246 | + | |
| 247 | + args->private_data = folios_ser; | |
| 248 | + args->serialized_data = virt_to_phys(ser); | |
| 249 | + | |
| 250 | + return 0; | |
| 251 | + | |
| 252 | +err_free_ser: | |
| 253 | + kho_unpreserve_free(ser); | |
| 254 | +err_unlock: | |
| 255 | + shmem_freeze(inode, false); | |
| 256 | + inode_unlock(inode); | |
| 257 | + return err; | |
| 258 | +} | |
| 259 | + | |
| 260 | +static int memfd_luo_freeze(struct liveupdate_file_op_args *args) | |
| 261 | +{ | |
| 262 | + struct memfd_luo_ser *ser; | |
| 263 | + | |
| 264 | + if (WARN_ON_ONCE(!args->serialized_data)) | |
| 265 | + return -EINVAL; | |
| 266 | + | |
| 267 | + ser = phys_to_virt(args->serialized_data); | |
| 268 | + | |
| 269 | + /* | |
| 270 | + * The pos might have changed since prepare. Everything else stays the | |
| 271 | + * same. | |
| 272 | + */ | |
| 273 | + ser->pos = args->file->f_pos; | |
| 274 | + | |
| 275 | + return 0; | |
| 276 | +} | |
| 277 | + | |
| 278 | +static void memfd_luo_unpreserve(struct liveupdate_file_op_args *args) | |
| 279 | +{ | |
| 280 | + struct inode *inode = file_inode(args->file); | |
| 281 | + struct memfd_luo_ser *ser; | |
| 282 | + | |
| 283 | + if (WARN_ON_ONCE(!args->serialized_data)) | |
| 284 | + return; | |
| 285 | + | |
| 286 | + inode_lock(inode); | |
| 287 | + shmem_freeze(inode, false); | |
| 288 | + | |
| 289 | + ser = phys_to_virt(args->serialized_data); | |
| 290 | + | |
| 291 | + memfd_luo_unpreserve_folios(&ser->folios, args->private_data, | |
| 292 | + ser->nr_folios); | |
| 293 | + | |
| 294 | + kho_unpreserve_free(ser); | |
| 295 | + inode_unlock(inode); | |
| 296 | +} | |
| 297 | + | |
| 298 | +static void memfd_luo_discard_folios(const struct memfd_luo_folio_ser *folios_ser, | |
| 299 | + u64 nr_folios) | |
| 300 | +{ | |
| 301 | + u64 i; | |
| 302 | + | |
| 303 | + for (i = 0; i < nr_folios; i++) { | |
| 304 | + const struct memfd_luo_folio_ser *pfolio = &folios_ser[i]; | |
| 305 | + struct folio *folio; | |
| 306 | + phys_addr_t phys; | |
| 307 | + | |
| 308 | + if (!pfolio->pfn) | |
| 309 | + continue; | |
| 310 | + | |
| 311 | + phys = PFN_PHYS(pfolio->pfn); | |
| 312 | + folio = kho_restore_folio(phys); | |
| 313 | + if (!folio) { | |
| 314 | + pr_warn_ratelimited("Unable to restore folio at physical address: %llx\n", | |
| 315 | + phys); | |
| 316 | + continue; | |
| 317 | + } | |
| 318 | + | |
| 319 | + folio_put(folio); | |
| 320 | + } | |
| 321 | +} | |
| 322 | + | |
| 323 | +static void memfd_luo_finish(struct liveupdate_file_op_args *args) | |
| 324 | +{ | |
| 325 | + struct memfd_luo_folio_ser *folios_ser; | |
| 326 | + struct memfd_luo_ser *ser; | |
| 327 | + | |
| 328 | + if (args->retrieved) | |
| 329 | + return; | |
| 330 | + | |
| 331 | + ser = phys_to_virt(args->serialized_data); | |
| 332 | + if (!ser) | |
| 333 | + return; | |
| 334 | + | |
| 335 | + if (ser->nr_folios) { | |
| 336 | + folios_ser = kho_restore_vmalloc(&ser->folios); | |
| 337 | + if (!folios_ser) | |
| 338 | + goto out; | |
| 339 | + | |
| 340 | + memfd_luo_discard_folios(folios_ser, ser->nr_folios); | |
| 341 | + vfree(folios_ser); | |
| 342 | + } | |
| 343 | + | |
| 344 | +out: | |
| 345 | + kho_restore_free(ser); | |
| 346 | +} | |
| 347 | + | |
| 348 | +static int memfd_luo_retrieve_folios(struct file *file, | |
| 349 | + struct memfd_luo_folio_ser *folios_ser, | |
| 350 | + u64 nr_folios) | |
| 351 | +{ | |
| 352 | + struct inode *inode = file_inode(file); | |
| 353 | + struct address_space *mapping = inode->i_mapping; | |
| 354 | + struct folio *folio; | |
| 355 | + int err = -EIO; | |
| 356 | + long i; | |
| 357 | + | |
| 358 | + for (i = 0; i < nr_folios; i++) { | |
| 359 | + const struct memfd_luo_folio_ser *pfolio = &folios_ser[i]; | |
| 360 | + phys_addr_t phys; | |
| 361 | + u64 index; | |
| 362 | + int flags; | |
| 363 | + | |
| 364 | + if (!pfolio->pfn) | |
| 365 | + continue; | |
| 366 | + | |
| 367 | + phys = PFN_PHYS(pfolio->pfn); | |
| 368 | + folio = kho_restore_folio(phys); | |
| 369 | + if (!folio) { | |
| 370 | + pr_err("Unable to restore folio at physical address: %llx\n", | |
| 371 | + phys); | |
| 372 | + goto put_folios; | |
| 373 | + } | |
| 374 | + index = pfolio->index; | |
| 375 | + flags = pfolio->flags; | |
| 376 | + | |
| 377 | + /* Set up the folio for insertion. */ | |
| 378 | + __folio_set_locked(folio); | |
| 379 | + __folio_set_swapbacked(folio); | |
| 380 | + | |
| 381 | + err = mem_cgroup_charge(folio, NULL, mapping_gfp_mask(mapping)); | |
| 382 | + if (err) { | |
| 383 | + pr_err("shmem: failed to charge folio index %ld: %d\n", | |
| 384 | + i, err); | |
| 385 | + goto unlock_folio; | |
| 386 | + } | |
| 387 | + | |
| 388 | + err = shmem_add_to_page_cache(folio, mapping, index, NULL, | |
| 389 | + mapping_gfp_mask(mapping)); | |
| 390 | + if (err) { | |
| 391 | + pr_err("shmem: failed to add to page cache folio index %ld: %d\n", | |
| 392 | + i, err); | |
| 393 | + goto unlock_folio; | |
| 394 | + } | |
| 395 | + | |
| 396 | + if (flags & MEMFD_LUO_FOLIO_UPTODATE) | |
| 397 | + folio_mark_uptodate(folio); | |
| 398 | + if (flags & MEMFD_LUO_FOLIO_DIRTY) | |
| 399 | + folio_mark_dirty(folio); | |
| 400 | + | |
| 401 | + err = shmem_inode_acct_blocks(inode, 1); | |
| 402 | + if (err) { | |
| 403 | + pr_err("shmem: failed to account folio index %ld: %d\n", | |
| 404 | + i, err); | |
| 405 | + goto unlock_folio; | |
| 406 | + } | |
| 407 | + | |
| 408 | + shmem_recalc_inode(inode, 1, 0); | |
| 409 | + folio_add_lru(folio); | |
| 410 | + folio_unlock(folio); | |
| 411 | + folio_put(folio); | |
| 412 | + } | |
| 413 | + | |
| 414 | + return 0; | |
| 415 | + | |
| 416 | +unlock_folio: | |
| 417 | + folio_unlock(folio); | |
| 418 | + folio_put(folio); | |
| 419 | +put_folios: | |
| 420 | + /* | |
| 421 | + * Note: don't free the folios already added to the file. They will be | |
| 422 | + * freed when the file is freed. Free the ones not added yet here. | |
| 423 | + */ | |
| 424 | + for (long j = i + 1; j < nr_folios; j++) { | |
| 425 | + const struct memfd_luo_folio_ser *pfolio = &folios_ser[j]; | |
| 426 | + | |
| 427 | + folio = kho_restore_folio(pfolio->pfn); | |
| 428 | + if (folio) | |
| 429 | + folio_put(folio); | |
| 430 | + } | |
| 431 | + | |
| 432 | + return err; | |
| 433 | +} | |
| 434 | + | |
| 435 | +static int memfd_luo_retrieve(struct liveupdate_file_op_args *args) | |
| 436 | +{ | |
| 437 | + struct memfd_luo_folio_ser *folios_ser; | |
| 438 | + struct memfd_luo_ser *ser; | |
| 439 | + struct file *file; | |
| 440 | + int err; | |
| 441 | + | |
| 442 | + ser = phys_to_virt(args->serialized_data); | |
| 443 | + if (!ser) | |
| 444 | + return -EINVAL; | |
| 445 | + | |
| 446 | + file = shmem_file_setup("", 0, VM_NORESERVE); | |
| 447 | + | |
| 448 | + if (IS_ERR(file)) { | |
| 449 | + pr_err("failed to setup file: %pe\n", file); | |
| 450 | + return PTR_ERR(file); | |
| 451 | + } | |
| 452 | + | |
| 453 | + vfs_setpos(file, ser->pos, MAX_LFS_FILESIZE); | |
| 454 | + file->f_inode->i_size = ser->size; | |
| 455 | + | |
| 456 | + if (ser->nr_folios) { | |
| 457 | + folios_ser = kho_restore_vmalloc(&ser->folios); | |
| 458 | + if (!folios_ser) { | |
| 459 | + err = -EINVAL; | |
| 460 | + goto put_file; | |
| 461 | + } | |
| 462 | + | |
| 463 | + err = memfd_luo_retrieve_folios(file, folios_ser, ser->nr_folios); | |
| 464 | + vfree(folios_ser); | |
| 465 | + if (err) | |
| 466 | + goto put_file; | |
| 467 | + } | |
| 468 | + | |
| 469 | + args->file = file; | |
| 470 | + kho_restore_free(ser); | |
| 471 | + | |
| 472 | + return 0; | |
| 473 | + | |
| 474 | +put_file: | |
| 475 | + fput(file); | |
| 476 | + | |
| 477 | + return err; | |
| 478 | +} | |
| 479 | + | |
| 480 | +static bool memfd_luo_can_preserve(struct liveupdate_file_handler *handler, | |
| 481 | + struct file *file) | |
| 482 | +{ | |
| 483 | + struct inode *inode = file_inode(file); | |
| 484 | + | |
| 485 | + return shmem_file(file) && !inode->i_nlink; | |
| 486 | +} | |
| 487 | + | |
| 488 | +static const struct liveupdate_file_ops memfd_luo_file_ops = { | |
| 489 | + .freeze = memfd_luo_freeze, | |
| 490 | + .finish = memfd_luo_finish, | |
| 491 | + .retrieve = memfd_luo_retrieve, | |
| 492 | + .preserve = memfd_luo_preserve, | |
| 493 | + .unpreserve = memfd_luo_unpreserve, | |
| 494 | + .can_preserve = memfd_luo_can_preserve, | |
| 495 | + .owner = THIS_MODULE, | |
| 496 | +}; | |
| 497 | + | |
| 498 | +static struct liveupdate_file_handler memfd_luo_handler = { | |
| 499 | + .ops = &memfd_luo_file_ops, | |
| 500 | + .compatible = MEMFD_LUO_FH_COMPATIBLE, | |
| 501 | +}; | |
| 502 | + | |
| 503 | +static int __init memfd_luo_init(void) | |
| 504 | +{ | |
| 505 | + int err = liveupdate_register_file_handler(&memfd_luo_handler); | |
| 506 | + | |
| 507 | + if (err && err != -EOPNOTSUPP) { | |
| 508 | + pr_err("Could not register luo filesystem handler: %pe\n", | |
| 509 | + ERR_PTR(err)); | |
| 510 | + | |
| 511 | + return err; | |
| 512 | + } | |
| 513 | + | |
| 514 | + return 0; | |
| 515 | +} | |
| 516 | +late_initcall(memfd_luo_init); | |
| @@ -174,20 +174,20 @@ static inline struct shmem_sb_info *SHMEM_SB(struct super_block *sb) · 173 unmodified lines | ||
| 174 | 174 | */ |
| 175 | 175 | static inline int shmem_acct_size(unsigned long flags, loff_t size) |
| 176 | 176 | { |
| 177 | − return (flags & VM_NORESERVE) ? | |
| 177 | + return (flags & SHMEM_F_NORESERVE) ? | |
| 178 | 178 | 0 : security_vm_enough_memory_mm(current->mm, VM_ACCT(size)); |
| 179 | 179 | } |
| 180 | 180 | |
| 181 | 181 | static inline void shmem_unacct_size(unsigned long flags, loff_t size) |
| 182 | 182 | { |
| 183 | − if (!(flags & VM_NORESERVE)) | |
| 183 | + if (!(flags & SHMEM_F_NORESERVE)) | |
| 184 | 184 | vm_unacct_memory(VM_ACCT(size)); |
| 185 | 185 | } |
| 186 | 186 | |
| 187 | 187 | static inline int shmem_reacct_size(unsigned long flags, |
| 188 | 188 | loff_t oldsize, loff_t newsize) |
| 189 | 189 | { |
| 190 | − if (!(flags & VM_NORESERVE)) { | |
| 190 | + if (!(flags & SHMEM_F_NORESERVE)) { | |
| 191 | 191 | if (VM_ACCT(newsize) > VM_ACCT(oldsize)) |
| 192 | 192 | return security_vm_enough_memory_mm(current->mm, |
| 193 | 193 | VM_ACCT(newsize) - VM_ACCT(oldsize)); |
| @@ -205,7 +205,7 @@ static inline int shmem_reacct_size(unsigned long flags, · 11 unmodified lines | ||
| 205 | 205 | */ |
| 206 | 206 | static inline int shmem_acct_blocks(unsigned long flags, long pages) |
| 207 | 207 | { |
| 208 | − if (!(flags & VM_NORESERVE)) | |
| 208 | + if (!(flags & SHMEM_F_NORESERVE)) | |
| 209 | 209 | return 0; |
| 210 | 210 | |
| 211 | 211 | return security_vm_enough_memory_mm(current->mm, |
| @@ -214,11 +214,11 @@ static inline int shmem_acct_blocks(unsigned long flags, long pages) · 2 unmodified lines | ||
| 214 | 214 | |
| 215 | 215 | static inline void shmem_unacct_blocks(unsigned long flags, long pages) |
| 216 | 216 | { |
| 217 | − if (flags & VM_NORESERVE) | |
| 217 | + if (flags & SHMEM_F_NORESERVE) | |
| 218 | 218 | vm_unacct_memory(pages * VM_ACCT(PAGE_SIZE)); |
| 219 | 219 | } |
| 220 | 220 | |
| 221 | −static int shmem_inode_acct_blocks(struct inode *inode, long pages) | |
| 221 | +int shmem_inode_acct_blocks(struct inode *inode, long pages) | |
| 222 | 222 | { |
| 223 | 223 | struct shmem_inode_info *info = SHMEM_I(inode); |
| 224 | 224 | struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb); |
| @@ -434,7 +434,7 @@ static void shmem_free_inode(struct super_block *sb, size_t freed_ispace) · 209 unmodified lines | ||
| 434 | 434 | * |
| 435 | 435 | * Return: true if swapped was incremented from 0, for shmem_writeout(). |
| 436 | 436 | */ |
| 437 | −static bool shmem_recalc_inode(struct inode *inode, long alloced, long swapped) | |
| 437 | +bool shmem_recalc_inode(struct inode *inode, long alloced, long swapped) | |
| 438 | 438 | { |
| 439 | 439 | struct shmem_inode_info *info = SHMEM_I(inode); |
| 440 | 440 | bool first_swapped = false; |
| @@ -878,9 +878,9 @@ static void shmem_update_stats(struct folio *folio, int nr_pages) · 437 unmodified lines | ||
| 878 | 878 | /* |
| 879 | 879 | * Somewhat like filemap_add_folio, but error if expected item has gone. |
| 880 | 880 | */ |
| 881 | −static int shmem_add_to_page_cache(struct folio *folio, | |
| 882 | − struct address_space *mapping, | |
| 883 | − pgoff_t index, void *expected, gfp_t gfp) | |
| 881 | +int shmem_add_to_page_cache(struct folio *folio, | |
| 882 | + struct address_space *mapping, | |
| 883 | + pgoff_t index, void *expected, gfp_t gfp) | |
| 884 | 884 | { |
| 885 | 885 | XA_STATE_ORDER(xas, &mapping->i_pages, index, folio_order(folio)); |
| 886 | 886 | unsigned long nr = folio_nr_pages(folio); |
| @@ -1314,6 +1314,8 @@ static int shmem_setattr(struct mnt_idmap *idmap, · 427 unmodified lines | ||
| 1314 | 1314 | return -EPERM; |
| 1315 | 1315 | |
| 1316 | 1316 | if (newsize != oldsize) { |
| 1317 | + if (info->flags & SHMEM_F_MAPPING_FROZEN) | |
| 1318 | + return -EPERM; | |
| 1317 | 1319 | error = shmem_reacct_size(SHMEM_I(inode)->flags, |
| 1318 | 1320 | oldsize, newsize); |
| 1319 | 1321 | if (error) |
| @@ -1568,7 +1570,7 @@ int shmem_writeout(struct folio *folio, struct swap_iocb **plug, · 248 unmodified lines | ||
| 1568 | 1570 | int nr_pages; |
| 1569 | 1571 | bool split = false; |
| 1570 | 1572 | |
| 1571 | − if ((info->flags & VM_LOCKED) || sbinfo->noswap) | |
| 1573 | + if ((info->flags & SHMEM_F_LOCKED) || sbinfo->noswap) | |
| 1572 | 1574 | goto redirty; |
| 1573 | 1575 | |
| 1574 | 1576 | if (!total_swap_pages) |
| @@ -2926,15 +2928,15 @@ int shmem_lock(struct file *file, int lock, struct ucounts *ucounts) · 1351 unmodified lines | ||
| 2926 | 2928 | * ipc_lock_object() when called from shmctl_do_lock(), |
| 2927 | 2929 | * no serialization needed when called from shm_destroy(). |
| 2928 | 2930 | */ |
| 2929 | − if (lock && !(info->flags & VM_LOCKED)) { | |
| 2931 | + if (lock && !(info->flags & SHMEM_F_LOCKED)) { | |
| 2930 | 2932 | if (!user_shm_lock(inode->i_size, ucounts)) |
| 2931 | 2933 | goto out_nomem; |
| 2932 | − info->flags |= VM_LOCKED; | |
| 2934 | + info->flags |= SHMEM_F_LOCKED; | |
| 2933 | 2935 | mapping_set_unevictable(file->f_mapping); |
| 2934 | 2936 | } |
| 2935 | − if (!lock && (info->flags & VM_LOCKED) && ucounts) { | |
| 2937 | + if (!lock && (info->flags & SHMEM_F_LOCKED) && ucounts) { | |
| 2936 | 2938 | user_shm_unlock(inode->i_size, ucounts); |
| 2937 | − info->flags &= ~VM_LOCKED; | |
| 2939 | + info->flags &= ~SHMEM_F_LOCKED; | |
| 2938 | 2940 | mapping_clear_unevictable(file->f_mapping); |
| 2939 | 2941 | } |
| 2940 | 2942 | retval = 0; |
| @@ -3079,7 +3081,7 @@ static struct inode *__shmem_get_inode(struct mnt_idmap *idmap, · 138 unmodified lines | ||
| 3079 | 3081 | spin_lock_init(&info->lock); |
| 3080 | 3082 | atomic_set(&info->stop_eviction, 0); |
| 3081 | 3083 | info->seals = F_SEAL_SEAL; |
| 3082 | − info->flags = flags & VM_NORESERVE; | |
| 3084 | + info->flags = (flags & VM_NORESERVE) ? SHMEM_F_NORESERVE : 0; | |
| 3083 | 3085 | info->i_crtime = inode_get_mtime(inode); |
| 3084 | 3086 | info->fsflags = (dir == NULL) ? 0 : |
| 3085 | 3087 | SHMEM_I(dir)->fsflags & SHMEM_FL_INHERITED; |
| @@ -3306,6 +3308,10 @@ shmem_write_begin(const struct kiocb *iocb, struct address_space *mapping, · 220 unmodified lines | ||
| 3306 | 3308 | return -EPERM; |
| 3307 | 3309 | } |
| 3308 | 3310 | |
| 3311 | + if (unlikely((info->flags & SHMEM_F_MAPPING_FROZEN) && | |
| 3312 | + pos + len > inode->i_size)) | |
| 3313 | + return -EPERM; | |
| 3314 | + | |
| 3309 | 3315 | ret = shmem_get_folio(inode, index, pos + len, &folio, SGP_WRITE); |
| 3310 | 3316 | if (ret) |
| 3311 | 3317 | return ret; |
| @@ -3679,6 +3685,11 @@ static long shmem_fallocate(struct file *file, int mode, loff_t offset, · 367 unmodified lines | ||
| 3679 | 3685 | |
| 3680 | 3686 | inode_lock(inode); |
| 3681 | 3687 | |
| 3688 | + if (info->flags & SHMEM_F_MAPPING_FROZEN) { | |
| 3689 | + error = -EPERM; | |
| 3690 | + goto out; | |
| 3691 | + } | |
| 3692 | + | |
| 3682 | 3693 | if (mode & FALLOC_FL_PUNCH_HOLE) { |
| 3683 | 3694 | struct address_space *mapping = file->f_mapping; |
| 3684 | 3695 | loff_t unmap_start = round_up(offset, PAGE_SIZE); |
| @@ -5799,8 +5810,10 @@ static inline struct inode *shmem_get_inode(struct mnt_idmap *idmap, · 2114 unmodified lines | ||
| 5799 | 5810 | /* common code */ |
| 5800 | 5811 | |
| 5801 | 5812 | static struct file *__shmem_file_setup(struct vfsmount *mnt, const char *name, |
| 5802 | − loff_t size, unsigned long flags, unsigned int i_flags) | |
| 5813 | + loff_t size, unsigned long vm_flags, | |
| 5814 | + unsigned int i_flags) | |
| 5803 | 5815 | { |
| 5816 | + unsigned long flags = (vm_flags & VM_NORESERVE) ? SHMEM_F_NORESERVE : 0; | |
| 5804 | 5817 | struct inode *inode; |
| 5805 | 5818 | struct file *res; |
| 5806 | 5819 | |
| @@ -5817,7 +5830,7 @@ static struct file *__shmem_file_setup(struct vfsmount *mnt, const char *name, · 10 unmodified lines | ||
| 5817 | 5830 | return ERR_PTR(-ENOMEM); |
| 5818 | 5831 | |
| 5819 | 5832 | inode = shmem_get_inode(&nop_mnt_idmap, mnt->mnt_sb, NULL, |
| 5820 | − S_IFREG | S_IRWXUGO, 0, flags); | |
| 5833 | + S_IFREG | S_IRWXUGO, 0, vm_flags); | |
| 5821 | 5834 | if (IS_ERR(inode)) { |
| 5822 | 5835 | shmem_unacct_size(flags, size); |
| 5823 | 5836 | return ERR_CAST(inode); |
| @@ -7,3 +7,13 @@ void rust_helper_rb_link_node(struct rb_node *node, struct rb_node *parent, · 6 unmodified lines | ||
| 7 | 7 | { |
| 8 | 8 | rb_link_node(node, parent, rb_link); |
| 9 | 9 | } |
| 10 | + | |
| 11 | +struct rb_node *rust_helper_rb_first(const struct rb_root *root) | |
| 12 | +{ | |
| 13 | + return rb_first(root); | |
| 14 | +} | |
| 15 | + | |
| 16 | +struct rb_node *rust_helper_rb_last(const struct rb_root *root) | |
| 17 | +{ | |
| 18 | + return rb_last(root); | |
| 19 | +} | |
| @@ -13,3 +13,15 @@ unsigned long rust_helper_copy_to_user(void __user *to, const void *from, · 12 unmodified lines | ||
| 13 | 13 | { |
| 14 | 14 | return copy_to_user(to, from, n); |
| 15 | 15 | } |
| 16 | + | |
| 17 | +#ifdef INLINE_COPY_FROM_USER | |
| 18 | +unsigned long rust_helper__copy_from_user(void *to, const void __user *from, unsigned long n) | |
| 19 | +{ | |
| 20 | + return _inline_copy_from_user(to, from, n); | |
| 21 | +} | |
| 22 | + | |
| 23 | +unsigned long rust_helper__copy_to_user(void __user *to, const void *from, unsigned long n) | |
| 24 | +{ | |
| 25 | + return _inline_copy_to_user(to, from, n); | |
| 26 | +} | |
| 27 | +#endif | |
| @@ -23,11 +23,11 @@ config SAMPLE_TRACE_CUSTOM_EVENTS · 22 unmodified lines | ||
| 23 | 23 | This builds the custom trace event example module. |
| 24 | 24 | |
| 25 | 25 | config SAMPLE_TRACE_PRINTK |
| 26 | − tristate "Build trace_printk module - tests various trace_printk formats" | |
| 26 | + tristate "Build trace_printk module - tests various trace_printk formats" | |
| 27 | 27 | depends on EVENT_TRACING && m |
| 28 | 28 | help |
| 29 | − This builds a module that calls trace_printk() and can be used to | |
| 30 | − test various trace_printk() calls from a module. | |
| 29 | + This builds a module that calls trace_printk() and can be used to | |
| 30 | + test various trace_printk() calls from a module. | |
| 31 | 31 | |
| 32 | 32 | config SAMPLE_FTRACE_DIRECT |
| 33 | 33 | tristate "Build register_ftrace_direct() example" |
| @@ -54,11 +54,11 @@ config SAMPLE_FTRACE_OPS · 20 unmodified lines | ||
| 54 | 54 | measures the time taken to invoke one function a number of times. |
| 55 | 55 | |
| 56 | 56 | config SAMPLE_TRACE_ARRAY |
| 57 | − tristate "Build sample module for kernel access to Ftrace instances" | |
| 57 | + tristate "Build sample module for kernel access to Ftrace instances" | |
| 58 | 58 | depends on EVENT_TRACING && m |
| 59 | 59 | help |
| 60 | − This builds a module that demonstrates the use of various APIs to | |
| 61 | − access Ftrace instances from within the kernel. | |
| 60 | + This builds a module that demonstrates the use of various APIs to | |
| 61 | + access Ftrace instances from within the kernel. | |
| 62 | 62 | |
| 63 | 63 | config SAMPLE_KOBJECT |
| 64 | 64 | tristate "Build kobject examples" |
| @@ -290,11 +290,11 @@ config SAMPLE_CORESIGHT_SYSCFG · 225 unmodified lines | ||
| 290 | 290 | configurations and easily load them into the system at runtime. |
| 291 | 291 | |
| 292 | 292 | config SAMPLE_KMEMLEAK |
| 293 | − tristate "Simple test for the kernel memory leak detector" | |
| 294 | − depends on DEBUG_KMEMLEAK && m | |
| 295 | − help | |
| 296 | − Build a sample program which have explicitly leaks memory to test | |
| 297 | − kmemleak | |
| 293 | + tristate "Simple test for the kernel memory leak detector" | |
| 294 | + depends on DEBUG_KMEMLEAK && m | |
| 295 | + help | |
| 296 | + Build a sample program which have explicitly leaks memory to test | |
| 297 | + kmemleak. | |
| 298 | 298 | |
| 299 | 299 | config SAMPLE_CGROUP |
| 300 | 300 | bool "Build cgroup sample code" |
| @@ -1,4 +1,5 @@ | ||
| 1 | 1 | # SPDX-License-Identifier: GPL-2.0-only |
| 2 | 2 | userprogs-always-y += test-fsmount test-statx mountinfo test-list-all-mounts |
| 3 | 3 | |
| 4 | +userccflags += -I $(srctree)/tools/testing/selftests/ | |
| 4 | 5 | userccflags += -I usr/include |
| @@ -860,6 +860,10 @@ our %deprecated_apis = ( · 859 unmodified lines | ||
| 860 | 860 | "kunmap" => "kunmap_local", |
| 861 | 861 | "kmap_atomic" => "kmap_local_page", |
| 862 | 862 | "kunmap_atomic" => "kunmap_local", |
| 863 | + #These should be enough to drive away new IDR users | |
| 864 | + "DEFINE_IDR" => "DEFINE_XARRAY", | |
| 865 | + "idr_init" => "xa_init", | |
| 866 | + "idr_init_base" => "xa_init_flags" | |
| 863 | 867 | ); |
| 864 | 868 | |
| 865 | 869 | #Create a search pattern for all these strings to speed up a loop below |
| @@ -3345,6 +3349,13 @@ sub process { · 2479 unmodified lines | ||
| 3345 | 3349 | } |
| 3346 | 3350 | } |
| 3347 | 3351 | |
| 3352 | +# Check for auto-generated unhandled placeholder text (mostly for cover letters) | |
| 3353 | + if (($in_commit_log || $in_header_lines) && | |
| 3354 | + $rawline =~ /(?:SUBJECT|BLURB) HERE/) { | |
| 3355 | + ERROR("PLACEHOLDER_USE", | |
| 3356 | + "Placeholder text detected\n" . $herecurr); | |
| 3357 | + } | |
| 3358 | + | |
| 3348 | 3359 | # Check for git id commit length and improperly formed commit descriptions |
| 3349 | 3360 | # A correctly formed commit description is: |
| 3350 | 3361 | # commit <SHA-1 hash length 12+ chars> ("Complete commit subject") |
| @@ -0,0 +1,253 @@ | ||
| 1 | +# SPDX-License-Identifier: GPL-2.0 | |
| 2 | + | |
| 3 | +import json | |
| 4 | +import subprocess | |
| 5 | +import tempfile | |
| 6 | + | |
| 7 | +import gdb | |
| 8 | + | |
| 9 | +from linux import constants, lists, radixtree, utils | |
| 10 | + | |
| 11 | + | |
| 12 | +if constants.LX_CONFIG_BPF and constants.LX_CONFIG_BPF_JIT: | |
| 13 | + bpf_ksym_type = utils.CachedType("struct bpf_ksym") | |
| 14 | +if constants.LX_CONFIG_BPF_SYSCALL: | |
| 15 | + bpf_prog_type = utils.CachedType("struct bpf_prog") | |
| 16 | + | |
| 17 | + | |
| 18 | +def get_ksym_name(ksym): | |
| 19 | + name = ksym["name"].bytes | |
| 20 | + end = name.find(b"\x00") | |
| 21 | + if end != -1: | |
| 22 | + name = name[:end] | |
| 23 | + return name.decode() | |
| 24 | + | |
| 25 | + | |
| 26 | +def list_ksyms(): | |
| 27 | + if not (constants.LX_CONFIG_BPF and constants.LX_CONFIG_BPF_JIT): | |
| 28 | + return [] | |
| 29 | + bpf_kallsyms = gdb.parse_and_eval("&bpf_kallsyms") | |
| 30 | + bpf_ksym_ptr_type = bpf_ksym_type.get_type().pointer() | |
| 31 | + return list(lists.list_for_each_entry(bpf_kallsyms, | |
| 32 | + bpf_ksym_ptr_type, | |
| 33 | + "lnode")) | |
| 34 | + | |
| 35 | + | |
| 36 | +class KsymAddBreakpoint(gdb.Breakpoint): | |
| 37 | + def __init__(self, monitor): | |
| 38 | + super(KsymAddBreakpoint, self).__init__("bpf_ksym_add", internal=True) | |
| 39 | + self.silent = True | |
| 40 | + self.monitor = monitor | |
| 41 | + | |
| 42 | + def stop(self): | |
| 43 | + self.monitor.add(gdb.parse_and_eval("ksym")) | |
| 44 | + return False | |
| 45 | + | |
| 46 | + | |
| 47 | +class KsymRemoveBreakpoint(gdb.Breakpoint): | |
| 48 | + def __init__(self, monitor): | |
| 49 | + super(KsymRemoveBreakpoint, self).__init__("bpf_ksym_del", | |
| 50 | + internal=True) | |
| 51 | + self.silent = True | |
| 52 | + self.monitor = monitor | |
| 53 | + | |
| 54 | + def stop(self): | |
| 55 | + self.monitor.remove(gdb.parse_and_eval("ksym")) | |
| 56 | + return False | |
| 57 | + | |
| 58 | + | |
| 59 | +class KsymMonitor: | |
| 60 | + def __init__(self, add, remove): | |
| 61 | + self.add = add | |
| 62 | + self.remove = remove | |
| 63 | + | |
| 64 | + self.add_bp = KsymAddBreakpoint(self) | |
| 65 | + self.remove_bp = KsymRemoveBreakpoint(self) | |
| 66 | + | |
| 67 | + self.notify_initial() | |
| 68 | + | |
| 69 | + def notify_initial(self): | |
| 70 | + for ksym in list_ksyms(): | |
| 71 | + self.add(ksym) | |
| 72 | + | |
| 73 | + def delete(self): | |
| 74 | + self.add_bp.delete() | |
| 75 | + self.remove_bp.delete() | |
| 76 | + | |
| 77 | + | |
| 78 | +def list_progs(): | |
| 79 | + if not constants.LX_CONFIG_BPF_SYSCALL: | |
| 80 | + return [] | |
| 81 | + idr_rt = gdb.parse_and_eval("&prog_idr.idr_rt") | |
| 82 | + bpf_prog_ptr_type = bpf_prog_type.get_type().pointer() | |
| 83 | + progs = [] | |
| 84 | + for _, slot in radixtree.for_each_slot(idr_rt): | |
| 85 | + prog = slot.dereference().cast(bpf_prog_ptr_type) | |
| 86 | + progs.append(prog) | |
| 87 | + # Subprogs are not registered in prog_idr, fetch them manually. | |
| 88 | + # func[0] is the current prog. | |
| 89 | + aux = prog["aux"] | |
| 90 | + func = aux["func"] | |
| 91 | + real_func_cnt = int(aux["real_func_cnt"]) | |
| 92 | + for i in range(1, real_func_cnt): | |
| 93 | + progs.append(func[i]) | |
| 94 | + return progs | |
| 95 | + | |
| 96 | + | |
| 97 | +class ProgAddBreakpoint(gdb.Breakpoint): | |
| 98 | + def __init__(self, monitor): | |
| 99 | + super(ProgAddBreakpoint, self).__init__("bpf_prog_kallsyms_add", | |
| 100 | + internal=True) | |
| 101 | + self.silent = True | |
| 102 | + self.monitor = monitor | |
| 103 | + | |
| 104 | + def stop(self): | |
| 105 | + self.monitor.add(gdb.parse_and_eval("fp")) | |
| 106 | + return False | |
| 107 | + | |
| 108 | + | |
| 109 | +class ProgRemoveBreakpoint(gdb.Breakpoint): | |
| 110 | + def __init__(self, monitor): | |
| 111 | + super(ProgRemoveBreakpoint, self).__init__("bpf_prog_free_id", | |
| 112 | + internal=True) | |
| 113 | + self.silent = True | |
| 114 | + self.monitor = monitor | |
| 115 | + | |
| 116 | + def stop(self): | |
| 117 | + self.monitor.remove(gdb.parse_and_eval("prog")) | |
| 118 | + return False | |
| 119 | + | |
| 120 | + | |
| 121 | +class ProgMonitor: | |
| 122 | + def __init__(self, add, remove): | |
| 123 | + self.add = add | |
| 124 | + self.remove = remove | |
| 125 | + | |
| 126 | + self.add_bp = ProgAddBreakpoint(self) | |
| 127 | + self.remove_bp = ProgRemoveBreakpoint(self) | |
| 128 | + | |
| 129 | + self.notify_initial() | |
| 130 | + | |
| 131 | + def notify_initial(self): | |
| 132 | + for prog in list_progs(): | |
| 133 | + self.add(prog) | |
| 134 | + | |
| 135 | + def delete(self): | |
| 136 | + self.add_bp.delete() | |
| 137 | + self.remove_bp.delete() | |
| 138 | + | |
| 139 | + | |
| 140 | +def btf_str_by_offset(btf, offset): | |
| 141 | + while offset < btf["start_str_off"]: | |
| 142 | + btf = btf["base_btf"] | |
| 143 | + | |
| 144 | + offset -= btf["start_str_off"] | |
| 145 | + if offset < btf["hdr"]["str_len"]: | |
| 146 | + return (btf["strings"] + offset).string() | |
| 147 | + | |
| 148 | + return None | |
| 149 | + | |
| 150 | + | |
| 151 | +def bpf_line_info_line_num(line_col): | |
| 152 | + return line_col >> 10 | |
| 153 | + | |
| 154 | + | |
| 155 | +def bpf_line_info_line_col(line_col): | |
| 156 | + return line_col & 0x3ff | |
| 157 | + | |
| 158 | + | |
| 159 | +class LInfoIter: | |
| 160 | + def __init__(self, prog): | |
| 161 | + # See bpf_prog_get_file_line() for details. | |
| 162 | + self.pos = 0 | |
| 163 | + self.nr_linfo = 0 | |
| 164 | + | |
| 165 | + if prog is None: | |
| 166 | + return | |
| 167 | + | |
| 168 | + self.bpf_func = int(prog["bpf_func"]) | |
| 169 | + aux = prog["aux"] | |
| 170 | + self.btf = aux["btf"] | |
| 171 | + linfo_idx = aux["linfo_idx"] | |
| 172 | + self.nr_linfo = int(aux["nr_linfo"]) - linfo_idx | |
| 173 | + if self.nr_linfo == 0: | |
| 174 | + return | |
| 175 | + | |
| 176 | + linfo_ptr = aux["linfo"] | |
| 177 | + tpe = linfo_ptr.type.target().array(self.nr_linfo).pointer() | |
| 178 | + self.linfo = (linfo_ptr + linfo_idx).cast(tpe).dereference() | |
| 179 | + jited_linfo_ptr = aux["jited_linfo"] | |
| 180 | + tpe = jited_linfo_ptr.type.target().array(self.nr_linfo).pointer() | |
| 181 | + self.jited_linfo = (jited_linfo_ptr + linfo_idx).cast(tpe).dereference() | |
| 182 | + | |
| 183 | + self.filenos = {} | |
| 184 | + | |
| 185 | + def get_code_off(self): | |
| 186 | + if self.pos >= self.nr_linfo: | |
| 187 | + return -1 | |
| 188 | + return self.jited_linfo[self.pos] - self.bpf_func | |
| 189 | + | |
| 190 | + def advance(self): | |
| 191 | + self.pos += 1 | |
| 192 | + | |
| 193 | + def get_fileno(self): | |
| 194 | + file_name_off = int(self.linfo[self.pos]["file_name_off"]) | |
| 195 | + fileno = self.filenos.get(file_name_off) | |
| 196 | + if fileno is not None: | |
| 197 | + return fileno, None | |
| 198 | + file_name = btf_str_by_offset(self.btf, file_name_off) | |
| 199 | + fileno = len(self.filenos) + 1 | |
| 200 | + self.filenos[file_name_off] = fileno | |
| 201 | + return fileno, file_name | |
| 202 | + | |
| 203 | + def get_line_col(self): | |
| 204 | + line_col = int(self.linfo[self.pos]["line_col"]) | |
| 205 | + return bpf_line_info_line_num(line_col), \ | |
| 206 | + bpf_line_info_line_col(line_col) | |
| 207 | + | |
| 208 | + | |
| 209 | +def generate_debug_obj(ksym, prog): | |
| 210 | + name = get_ksym_name(ksym) | |
| 211 | + # Avoid read_memory(); it throws bogus gdb.MemoryError in some contexts. | |
| 212 | + start = ksym["start"] | |
| 213 | + code = start.cast(gdb.lookup_type("unsigned char") | |
| 214 | + .array(int(ksym["end"]) - int(start)) | |
| 215 | + .pointer()).dereference().bytes | |
| 216 | + linfo_iter = LInfoIter(prog) | |
| 217 | + | |
| 218 | + result = tempfile.NamedTemporaryFile(suffix=".o", mode="wb") | |
| 219 | + try: | |
| 220 | + with tempfile.NamedTemporaryFile(suffix=".s", mode="w") as src: | |
| 221 | + # ".loc" does not apply to ".byte"s, only to ".insn"s, but since | |
| 222 | + # this needs to work for all architectures, the latter are not an | |
| 223 | + # option. Ask the assembler to apply ".loc"s to labels as well, | |
| 224 | + # and generate dummy labels after each ".loc". | |
| 225 | + src.write(".loc_mark_labels 1\n") | |
| 226 | + | |
| 227 | + src.write(".globl {}\n".format(name)) | |
| 228 | + src.write(".type {},@function\n".format(name)) | |
| 229 | + src.write("{}:\n".format(name)) | |
| 230 | + for code_off, code_byte in enumerate(code): | |
| 231 | + if linfo_iter.get_code_off() == code_off: | |
| 232 | + fileno, file_name = linfo_iter.get_fileno() | |
| 233 | + if file_name is not None: | |
| 234 | + src.write(".file {} {}\n".format( | |
| 235 | + fileno, json.dumps(file_name))) | |
| 236 | + line, col = linfo_iter.get_line_col() | |
| 237 | + src.write(".loc {} {} {}\n".format(fileno, line, col)) | |
| 238 | + src.write("0:\n") | |
| 239 | + linfo_iter.advance() | |
| 240 | + src.write(".byte {}\n".format(code_byte)) | |
| 241 | + src.write(".size {},{}\n".format(name, len(code))) | |
| 242 | + src.flush() | |
| 243 | + | |
| 244 | + try: | |
| 245 | + subprocess.check_call(["as", "-c", src.name, "-o", result.name]) | |
| 246 | + except FileNotFoundError: | |
| 247 | + # "as" is not installed. | |
| 248 | + result.close() | |
| 249 | + return None | |
| 250 | + return result | |
| 251 | + except: | |
| 252 | + result.close() | |
| 253 | + raise | |
| @@ -170,3 +170,6 @@ LX_CONFIG(CONFIG_PAGE_OWNER) · 169 unmodified lines | ||
| 170 | 170 | LX_CONFIG(CONFIG_SLUB_DEBUG) |
| 171 | 171 | LX_CONFIG(CONFIG_SLAB_FREELIST_HARDENED) |
| 172 | 172 | LX_CONFIG(CONFIG_MMU) |
| 173 | +LX_CONFIG(CONFIG_BPF) | |
| 174 | +LX_CONFIG(CONFIG_BPF_JIT) | |
| 175 | +LX_CONFIG(CONFIG_BPF_SYSCALL) | |
| @@ -30,13 +30,16 @@ def entry_to_node(node): · 29 unmodified lines | ||
| 30 | 30 | def node_maxindex(node): |
| 31 | 31 | return (constants.LX_RADIX_TREE_MAP_SIZE << node['shift']) - 1 |
| 32 | 32 | |
| 33 | −def lookup(root, index): | |
| 33 | +def resolve_root(root): | |
| 34 | + if root.type == radix_tree_root_type.get_type(): | |
| 35 | + return root | |
| 34 | 36 | if root.type == radix_tree_root_type.get_type().pointer(): |
| 35 | − node = root.dereference() | |
| 36 | − elif root.type != radix_tree_root_type.get_type(): | |
| 37 | − raise gdb.GdbError("must be {} not {}" | |
| 38 | − .format(radix_tree_root_type.get_type(), root.type)) | |
| 37 | + return root.dereference() | |
| 38 | + raise gdb.GdbError("must be {} not {}" | |
| 39 | + .format(radix_tree_root_type.get_type(), root.type)) | |
| 39 | 40 | |
| 41 | +def lookup(root, index): | |
| 42 | + root = resolve_root(root) | |
| 40 | 43 | node = root['xa_head'] |
| 41 | 44 | if node == 0: |
| 42 | 45 | return None |
| @@ -71,14 +74,120 @@ def lookup(root, index): · 28 unmodified lines | ||
| 71 | 74 | |
| 72 | 75 | return node |
| 73 | 76 | |
| 74 | −class LxRadixTree(gdb.Function): | |
| 77 | +def descend(parent, index): | |
| 78 | + offset = (index >> int(parent["shift"])) & constants.LX_RADIX_TREE_MAP_MASK | |
| 79 | + return offset, parent["slots"][offset] | |
| 80 | + | |
| 81 | +def load_root(root): | |
| 82 | + node = root["xa_head"] | |
| 83 | + nodep = node | |
| 84 | + | |
| 85 | + if is_internal_node(node): | |
| 86 | + node = entry_to_node(node) | |
| 87 | + maxindex = node_maxindex(node) | |
| 88 | + return int(node["shift"]) + constants.LX_RADIX_TREE_MAP_SHIFT, \ | |
| 89 | + nodep, maxindex | |
| 90 | + | |
| 91 | + return 0, nodep, 0 | |
| 92 | + | |
| 93 | +class RadixTreeIter: | |
| 94 | + def __init__(self, start): | |
| 95 | + self.index = 0 | |
| 96 | + self.next_index = start | |
| 97 | + self.node = None | |
| 98 | + | |
| 99 | +def xa_mk_internal(v): | |
| 100 | + return (v << 2) | 2 | |
| 101 | + | |
| 102 | +LX_XA_RETRY_ENTRY = xa_mk_internal(256) | |
| 103 | +LX_RADIX_TREE_RETRY = LX_XA_RETRY_ENTRY | |
| 104 | + | |
| 105 | +def next_chunk(root, iter): | |
| 106 | + mask = (1 << (utils.get_ulong_type().sizeof * 8)) - 1 | |
| 107 | + | |
| 108 | + index = iter.next_index | |
| 109 | + if index == 0 and iter.index != 0: | |
| 110 | + return None | |
| 111 | + | |
| 112 | + restart = True | |
| 113 | + while restart: | |
| 114 | + restart = False | |
| 115 | + | |
| 116 | + _, child, maxindex = load_root(root) | |
| 117 | + if index > maxindex: | |
| 118 | + return None | |
| 119 | + if not child: | |
| 120 | + return None | |
| 121 | + | |
| 122 | + if not is_internal_node(child): | |
| 123 | + iter.index = index | |
| 124 | + iter.next_index = (maxindex + 1) & mask | |
| 125 | + iter.node = None | |
| 126 | + return root["xa_head"].address | |
| 127 | + | |
| 128 | + while True: | |
| 129 | + node = entry_to_node(child) | |
| 130 | + offset, child = descend(node, index) | |
| 131 | + | |
| 132 | + if not child: | |
| 133 | + while True: | |
| 134 | + offset += 1 | |
| 135 | + if offset >= constants.LX_RADIX_TREE_MAP_SIZE: | |
| 136 | + break | |
| 137 | + slot = node["slots"][offset] | |
| 138 | + if slot: | |
| 139 | + break | |
| 140 | + index &= ~node_maxindex(node) | |
| 141 | + index = (index + (offset << int(node["shift"]))) & mask | |
| 142 | + if index == 0: | |
| 143 | + return None | |
| 144 | + if offset == constants.LX_RADIX_TREE_MAP_SIZE: | |
| 145 | + restart = True | |
| 146 | + break | |
| 147 | + child = node["slots"][offset] | |
| 148 | + | |
| 149 | + if not child: | |
| 150 | + restart = True | |
| 151 | + break | |
| 152 | + if child == LX_XA_RETRY_ENTRY: | |
| 153 | + break | |
| 154 | + if not node["shift"] or not is_internal_node(child): | |
| 155 | + break | |
| 156 | + | |
| 157 | + iter.index = (index & ~node_maxindex(node)) | offset | |
| 158 | + iter.next_index = ((index | node_maxindex(node)) + 1) & mask | |
| 159 | + iter.node = node | |
| 160 | + | |
| 161 | + return node["slots"][offset].address | |
| 162 | + | |
| 163 | +def next_slot(slot, iter): | |
| 164 | + mask = (1 << (utils.get_ulong_type().sizeof * 8)) - 1 | |
| 165 | + for _ in range(iter.next_index - iter.index - 1): | |
| 166 | + slot += 1 | |
| 167 | + iter.index = (iter.index + 1) & mask | |
| 168 | + if slot.dereference(): | |
| 169 | + return slot | |
| 170 | + return None | |
| 171 | + | |
| 172 | +def for_each_slot(root, start=0): | |
| 173 | + iter = RadixTreeIter(start) | |
| 174 | + slot = None | |
| 175 | + while True: | |
| 176 | + if not slot: | |
| 177 | + slot = next_chunk(root, iter) | |
| 178 | + if not slot: | |
| 179 | + break | |
| 180 | + yield iter.index, slot | |
| 181 | + slot = next_slot(slot, iter) | |
| 182 | + | |
| 183 | +class LxRadixTreeLookup(gdb.Function): | |
| 75 | 184 | """ Lookup and return a node from a RadixTree. |
| 76 | 185 | |
| 77 | 186 | $lx_radix_tree_lookup(root_node [, index]): Return the node at the given index. |
| 78 | 187 | If index is omitted, the root node is dereference and returned.""" |
| 79 | 188 | |
| 80 | 189 | def __init__(self): |
| 81 | − super(LxRadixTree, self).__init__("lx_radix_tree_lookup") | |
| 190 | + super(LxRadixTreeLookup, self).__init__("lx_radix_tree_lookup") | |
| 82 | 191 | |
| 83 | 192 | def invoke(self, root, index=0): |
| 84 | 193 | result = lookup(root, index) |
| @@ -87,4 +196,20 @@ If index is omitted, the root node is dereference and returned.""" · 2 unmodified lines | ||
| 87 | 196 | |
| 88 | 197 | return result |
| 89 | 198 | |
| 199 | +class LxRadixTree(gdb.Command): | |
| 200 | + """Show all values stored in a RadixTree.""" | |
| 201 | + | |
| 202 | + def __init__(self): | |
| 203 | + super(LxRadixTree, self).__init__("lx-radix-tree", gdb.COMMAND_DATA, | |
| 204 | + gdb.COMPLETE_NONE) | |
| 205 | + | |
| 206 | + def invoke(self, argument, from_tty): | |
| 207 | + args = gdb.string_to_argv(argument) | |
| 208 | + if len(args) != 1: | |
| 209 | + raise gdb.GdbError("Usage: lx-radix-tree ROOT") | |
| 210 | + root = gdb.parse_and_eval(args[0]) | |
| 211 | + for index, slot in for_each_slot(root): | |
| 212 | + gdb.write("[{}] = {}\n".format(index, slot.dereference())) | |
| 213 | + | |
| 90 | 214 | LxRadixTree() |
| 215 | +LxRadixTreeLookup() | |
| @@ -11,13 +11,14 @@ · 10 unmodified lines | ||
| 11 | 11 | # This work is licensed under the terms of the GNU GPL version 2. |
| 12 | 12 | # |
| 13 | 13 | |
| 14 | +import atexit | |
| 14 | 15 | import gdb |
| 15 | 16 | import os |
| 16 | 17 | import re |
| 17 | 18 | import struct |
| 18 | 19 | |
| 19 | 20 | from itertools import count |
| 20 | −from linux import modules, utils, constants | |
| 21 | +from linux import bpf, constants, modules, utils | |
| 21 | 22 | |
| 22 | 23 | |
| 23 | 24 | if hasattr(gdb, 'Breakpoint'): |
| @@ -114,17 +115,27 @@ class LxSymbols(gdb.Command): · 90 unmodified lines | ||
| 114 | 115 | The kernel (vmlinux) is taken from the current working directly. Modules (.ko) |
| 115 | 116 | are scanned recursively, starting in the same directory. Optionally, the module |
| 116 | 117 | search path can be extended by a space separated list of paths passed to the |
| 117 | −lx-symbols command.""" | |
| 118 | +lx-symbols command. | |
| 119 | + | |
| 120 | +When the -bpf flag is specified, symbols from the currently loaded BPF programs | |
| 121 | +are loaded as well.""" | |
| 118 | 122 | |
| 119 | 123 | module_paths = [] |
| 120 | 124 | module_files = [] |
| 121 | 125 | module_files_updated = False |
| 122 | 126 | loaded_modules = [] |
| 123 | 127 | breakpoint = None |
| 128 | + bpf_prog_monitor = None | |
| 129 | + bpf_ksym_monitor = None | |
| 130 | + bpf_progs = {} | |
| 131 | + # The remove-symbol-file command, even when invoked with -a, requires the | |
| 132 | + # respective object file to exist, so keep them around. | |
| 133 | + bpf_debug_objs = {} | |
| 124 | 134 | |
| 125 | 135 | def __init__(self): |
| 126 | 136 | super(LxSymbols, self).__init__("lx-symbols", gdb.COMMAND_FILES, |
| 127 | 137 | gdb.COMPLETE_FILENAME) |
| 138 | + atexit.register(self.cleanup_bpf) | |
| 128 | 139 | |
| 129 | 140 | def _update_module_files(self): |
| 130 | 141 | self.module_files = [] |
| @@ -197,6 +208,51 @@ lx-symbols command.""" · 66 unmodified lines | ||
| 197 | 208 | else: |
| 198 | 209 | gdb.write("no module object found for '{0}'\n".format(module_name)) |
| 199 | 210 | |
| 211 | + def add_bpf_prog(self, prog): | |
| 212 | + if prog["jited"]: | |
| 213 | + self.bpf_progs[int(prog["bpf_func"])] = prog | |
| 214 | + | |
| 215 | + def remove_bpf_prog(self, prog): | |
| 216 | + self.bpf_progs.pop(int(prog["bpf_func"]), None) | |
| 217 | + | |
| 218 | + def add_bpf_ksym(self, ksym): | |
| 219 | + addr = int(ksym["start"]) | |
| 220 | + name = bpf.get_ksym_name(ksym) | |
| 221 | + with utils.pagination_off(): | |
| 222 | + gdb.write("loading @{addr}: {name}\n".format( | |
| 223 | + addr=hex(addr), name=name)) | |
| 224 | + debug_obj = bpf.generate_debug_obj(ksym, self.bpf_progs.get(addr)) | |
| 225 | + if debug_obj is None: | |
| 226 | + return | |
| 227 | + try: | |
| 228 | + cmdline = "add-symbol-file {obj} {addr}".format( | |
| 229 | + obj=debug_obj.name, addr=hex(addr)) | |
| 230 | + gdb.execute(cmdline, to_string=True) | |
| 231 | + except: | |
| 232 | + debug_obj.close() | |
| 233 | + raise | |
| 234 | + self.bpf_debug_objs[addr] = debug_obj | |
| 235 | + | |
| 236 | + def remove_bpf_ksym(self, ksym): | |
| 237 | + addr = int(ksym["start"]) | |
| 238 | + debug_obj = self.bpf_debug_objs.pop(addr, None) | |
| 239 | + if debug_obj is None: | |
| 240 | + return | |
| 241 | + try: | |
| 242 | + name = bpf.get_ksym_name(ksym) | |
| 243 | + gdb.write("unloading @{addr}: {name}\n".format( | |
| 244 | + addr=hex(addr), name=name)) | |
| 245 | + cmdline = "remove-symbol-file {path}".format(path=debug_obj.name) | |
| 246 | + gdb.execute(cmdline, to_string=True) | |
| 247 | + finally: | |
| 248 | + debug_obj.close() | |
| 249 | + | |
| 250 | + def cleanup_bpf(self): | |
| 251 | + self.bpf_progs = {} | |
| 252 | + while len(self.bpf_debug_objs) > 0: | |
| 253 | + self.bpf_debug_objs.popitem()[1].close() | |
| 254 | + | |
| 255 | + | |
| 200 | 256 | def load_all_symbols(self): |
| 201 | 257 | gdb.write("loading vmlinux\n") |
| 202 | 258 | |
| @@ -224,34 +280,59 @@ lx-symbols command.""" · 21 unmodified lines | ||
| 224 | 280 | else: |
| 225 | 281 | [self.load_module_symbols(module) for module in module_list] |
| 226 | 282 | |
| 283 | + self.cleanup_bpf() | |
| 284 | + if self.bpf_prog_monitor is not None: | |
| 285 | + self.bpf_prog_monitor.notify_initial() | |
| 286 | + if self.bpf_ksym_monitor is not None: | |
| 287 | + self.bpf_ksym_monitor.notify_initial() | |
| 288 | + | |
| 227 | 289 | for saved_state in saved_states: |
| 228 | 290 | saved_state['breakpoint'].enabled = saved_state['enabled'] |
| 229 | 291 | |
| 230 | 292 | def invoke(self, arg, from_tty): |
| 231 | 293 | skip_decompressor() |
| 232 | 294 | |
| 233 | − self.module_paths = [os.path.abspath(os.path.expanduser(p)) | |
| 234 | − for p in arg.split()] | |
| 295 | + monitor_bpf = False | |
| 296 | + self.module_paths = [] | |
| 297 | + for p in arg.split(): | |
| 298 | + if p == "-bpf": | |
| 299 | + monitor_bpf = True | |
| 300 | + else: | |
| 301 | + p.append(os.path.abspath(os.path.expanduser(p))) | |
| 235 | 302 | self.module_paths.append(os.getcwd()) |
| 236 | 303 | |
| 304 | + if self.breakpoint is not None: | |
| 305 | + self.breakpoint.delete() | |
| 306 | + self.breakpoint = None | |
| 307 | + if self.bpf_prog_monitor is not None: | |
| 308 | + self.bpf_prog_monitor.delete() | |
| 309 | + self.bpf_prog_monitor = None | |
| 310 | + if self.bpf_ksym_monitor is not None: | |
| 311 | + self.bpf_ksym_monitor.delete() | |
| 312 | + self.bpf_ksym_monitor = None | |
| 313 | + | |
| 237 | 314 | # enforce update |
| 238 | 315 | self.module_files = [] |
| 239 | 316 | self.module_files_updated = False |
| 240 | 317 | |
| 241 | 318 | self.load_all_symbols() |
| 242 | 319 | |
| 243 | − if not modules.has_modules(): | |
| 320 | + if not hasattr(gdb, 'Breakpoint'): | |
| 321 | + gdb.write("Note: symbol update on module and BPF loading not " | |
| 322 | + "supported with this gdb version\n") | |
| 244 | 323 | return |
| 245 | 324 | |
| 246 | − if hasattr(gdb, 'Breakpoint'): | |
| 247 | − if self.breakpoint is not None: | |
| 248 | − self.breakpoint.delete() | |
| 249 | − self.breakpoint = None | |
| 325 | + if modules.has_modules(): | |
| 250 | 326 | self.breakpoint = LoadModuleBreakpoint( |
| 251 | 327 | "kernel/module/main.c:do_init_module", self) |
| 252 | − else: | |
| 253 | − gdb.write("Note: symbol update on module loading not supported " | |
| 254 | − "with this gdb version\n") | |
| 328 | + | |
| 329 | + if monitor_bpf: | |
| 330 | + if constants.LX_CONFIG_BPF_SYSCALL: | |
| 331 | + self.bpf_prog_monitor = bpf.ProgMonitor(self.add_bpf_prog, | |
| 332 | + self.remove_bpf_prog) | |
| 333 | + if constants.LX_CONFIG_BPF and constants.LX_CONFIG_BPF_JIT: | |
| 334 | + self.bpf_ksym_monitor = bpf.KsymMonitor(self.add_bpf_ksym, | |
| 335 | + self.remove_bpf_ksym) | |
| 255 | 336 | |
| 256 | 337 | |
| 257 | 338 | LxSymbols() |
| @@ -54,6 +54,7 @@ TARGETS += kvm · 53 unmodified lines | ||
| 54 | 54 | TARGETS += landlock |
| 55 | 55 | TARGETS += lib |
| 56 | 56 | TARGETS += livepatch |
| 57 | +TARGETS += liveupdate | |
| 57 | 58 | TARGETS += lkdtm |
| 58 | 59 | TARGETS += lsm |
| 59 | 60 | TARGETS += membarrier |
| @@ -9,7 +9,7 @@ · 8 unmodified lines | ||
| 9 | 9 | #include <string.h> |
| 10 | 10 | #include <sys/wait.h> |
| 11 | 11 | |
| 12 | −#include "../kselftest.h" | |
| 12 | +#include "kselftest.h" | |
| 13 | 13 | |
| 14 | 14 | int main(void) |
| 15 | 15 | { |
| @@ -14,7 +14,7 @@ · 13 unmodified lines | ||
| 14 | 14 | #include <regex.h> |
| 15 | 15 | #include <sys/stat.h> |
| 16 | 16 | |
| 17 | −#include "../kselftest.h" | |
| 17 | +#include "kselftest.h" | |
| 18 | 18 | #include "alsa-local.h" |
| 19 | 19 | |
| 20 | 20 | #define SYSFS_ROOT "/sys" |
| @@ -25,7 +25,7 @@ · 24 unmodified lines | ||
| 25 | 25 | #include <poll.h> |
| 26 | 26 | #include <stdint.h> |
| 27 | 27 | |
| 28 | −#include "../kselftest.h" | |
| 28 | +#include "kselftest.h" | |
| 29 | 29 | #include "alsa-local.h" |
| 30 | 30 | |
| 31 | 31 | #define TESTS_PER_CONTROL 7 |
| @@ -17,7 +17,7 @@ · 16 unmodified lines | ||
| 17 | 17 | #include <assert.h> |
| 18 | 18 | #include <pthread.h> |
| 19 | 19 | |
| 20 | −#include "../kselftest.h" | |
| 20 | +#include "kselftest.h" | |
| 21 | 21 | #include "alsa-local.h" |
| 22 | 22 | |
| 23 | 23 | typedef struct timespec timestamp_t; |
| @@ -7,7 +7,7 @@ · 6 unmodified lines | ||
| 7 | 7 | */ |
| 8 | 8 | #include <string.h> |
| 9 | 9 | #include <alsa/asoundlib.h> |
| 10 | −#include "../kselftest_harness.h" | |
| 10 | +#include "kselftest_harness.h" | |
| 11 | 11 | |
| 12 | 12 | #define CH_NUM 4 |
| 13 | 13 | |
| @@ -6,7 +6,7 @@ · 5 unmodified lines | ||
| 6 | 6 | * |
| 7 | 7 | * Author: Ivan Orlov <ivan.orlov0322@gmail.com> |
| 8 | 8 | */ |
| 9 | −#include "../kselftest_harness.h" | |
| 9 | +#include "kselftest_harness.h" | |
| 10 | 10 | #include <sound/asound.h> |
| 11 | 11 | #include <unistd.h> |
| 12 | 12 | #include <fcntl.h> |
| @@ -19,7 +19,7 @@ · 18 unmodified lines | ||
| 19 | 19 | |
| 20 | 20 | #include <linux/auxvec.h> |
| 21 | 21 | |
| 22 | −#include "../../kselftest.h" | |
| 22 | +#include "kselftest.h" | |
| 23 | 23 | |
| 24 | 24 | #define TESTS_PER_HWCAP 3 |
| 25 | 25 | |
| @@ -18,7 +18,7 @@ · 17 unmodified lines | ||
| 18 | 18 | #include <asm/sigcontext.h> |
| 19 | 19 | #include <asm/ptrace.h> |
| 20 | 20 | |
| 21 | −#include "../../kselftest.h" | |
| 21 | +#include "kselftest.h" | |
| 22 | 22 | |
| 23 | 23 | #define EXPECTED_TESTS 11 |
| 24 | 24 | |
| @@ -16,7 +16,7 @@ · 15 unmodified lines | ||
| 16 | 16 | #include <asm/sigcontext.h> |
| 17 | 17 | #include <asm/unistd.h> |
| 18 | 18 | |
| 19 | −#include "../../kselftest.h" | |
| 19 | +#include "kselftest.h" | |
| 20 | 20 | |
| 21 | 21 | #include "syscall-abi.h" |
| 22 | 22 | |
| @@ -27,7 +27,7 @@ · 26 unmodified lines | ||
| 27 | 27 | #include <asm/sve_context.h> |
| 28 | 28 | #include <asm/ptrace.h> |
| 29 | 29 | |
| 30 | −#include "../../kselftest.h" | |
| 30 | +#include "kselftest.h" | |
| 31 | 31 | |
| 32 | 32 | #include "fp-ptrace.h" |
| 33 | 33 | |
| @@ -24,7 +24,7 @@ · 23 unmodified lines | ||
| 24 | 24 | #include <sys/wait.h> |
| 25 | 25 | #include <asm/hwcap.h> |
| 26 | 26 | |
| 27 | −#include "../../kselftest.h" | |
| 27 | +#include "kselftest.h" | |
| 28 | 28 | |
| 29 | 29 | #define MAX_VLS 16 |
| 30 | 30 | |
| @@ -12,7 +12,7 @@ · 11 unmodified lines | ||
| 12 | 12 | #include <sys/prctl.h> |
| 13 | 13 | #include <asm/sigcontext.h> |
| 14 | 14 | |
| 15 | −#include "../../kselftest.h" | |
| 15 | +#include "kselftest.h" | |
| 16 | 16 | #include "rdvl.h" |
| 17 | 17 | |
| 18 | 18 | int main(int argc, char **argv) |
| @@ -19,7 +19,7 @@ · 18 unmodified lines | ||
| 19 | 19 | #include <asm/sigcontext.h> |
| 20 | 20 | #include <asm/ptrace.h> |
| 21 | 21 | |
| 22 | −#include "../../kselftest.h" | |
| 22 | +#include "kselftest.h" | |
| 23 | 23 | |
| 24 | 24 | /* <linux/elf.h> and <sys/auxv.h> don't like each other, so: */ |
| 25 | 25 | #ifndef NT_ARM_SVE |
| @@ -19,7 +19,7 @@ · 18 unmodified lines | ||
| 19 | 19 | #include <asm/sigcontext.h> |
| 20 | 20 | #include <asm/hwcap.h> |
| 21 | 21 | |
| 22 | −#include "../../kselftest.h" | |
| 22 | +#include "kselftest.h" | |
| 23 | 23 | #include "rdvl.h" |
| 24 | 24 | |
| 25 | 25 | #define ARCH_MIN_VL SVE_VL_MIN |
| @@ -18,7 +18,7 @@ · 17 unmodified lines | ||
| 18 | 18 | #include <asm/sigcontext.h> |
| 19 | 19 | #include <asm/ptrace.h> |
| 20 | 20 | |
| 21 | −#include "../../kselftest.h" | |
| 21 | +#include "kselftest.h" | |
| 22 | 22 | |
| 23 | 23 | /* <linux/elf.h> and <sys/auxv.h> don't like each other, so: */ |
| 24 | 24 | #ifndef NT_ARM_ZA |
| @@ -18,7 +18,7 @@ · 17 unmodified lines | ||
| 18 | 18 | #include <asm/sigcontext.h> |
| 19 | 19 | #include <asm/ptrace.h> |
| 20 | 20 | |
| 21 | −#include "../../kselftest.h" | |
| 21 | +#include "kselftest.h" | |
| 22 | 22 | |
| 23 | 23 | /* <linux/elf.h> and <sys/auxv.h> don't like each other, so: */ |
| 24 | 24 | #ifndef NT_ARM_ZA |
| @@ -24,7 +24,7 @@ · 23 unmodified lines | ||
| 24 | 24 | #include <sys/wait.h> |
| 25 | 25 | #include <asm/hwcap.h> |
| 26 | 26 | |
| 27 | −#include "../../kselftest.h" | |
| 27 | +#include "kselftest.h" | |
| 28 | 28 | |
| 29 | 29 | struct child_data { |
| 30 | 30 | char *name, *output; |
| @@ -10,7 +10,7 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <setjmp.h> |
| 11 | 11 | #include <sched.h> |
| 12 | 12 | |
| 13 | −#include "../../kselftest_harness.h" | |
| 13 | +#include "kselftest_harness.h" | |
| 14 | 14 | #include "helper.h" |
| 15 | 15 | |
| 16 | 16 | #define PAC_COLLISION_ATTEMPTS 1000 |
| @@ -6,7 +6,7 @@ · 5 unmodified lines | ||
| 6 | 6 | #include <stdint.h> |
| 7 | 7 | #include <sys/prctl.h> |
| 8 | 8 | #include <sys/utsname.h> |
| 9 | −#include "../../kselftest.h" | |
| 9 | +#include "kselftest.h" | |
| 10 | 10 | |
| 11 | 11 | #define SHIFT_TAG(tag) ((uint64_t)(tag) << 56) |
| 12 | 12 | #define SET_TAG(ptr, tag) (((uint64_t)(ptr) & ~SHIFT_TAG(0xff)) | \ |
| @@ -96,7 +96,7 @@ · 95 unmodified lines | ||
| 96 | 96 | #include "xskxceiver.h" |
| 97 | 97 | #include <bpf/bpf.h> |
| 98 | 98 | #include <linux/filter.h> |
| 99 | −#include "../kselftest.h" | |
| 99 | +#include "kselftest.h" | |
| 100 | 100 | #include "xsk_xdp_common.h" |
| 101 | 101 | |
| 102 | 102 | #include <network_helpers.h> |
| @@ -18,7 +18,7 @@ · 17 unmodified lines | ||
| 18 | 18 | #include <errno.h> |
| 19 | 19 | #include <string.h> |
| 20 | 20 | |
| 21 | −#include "../kselftest.h" | |
| 21 | +#include "kselftest.h" | |
| 22 | 22 | |
| 23 | 23 | #define COUNT_ISN_BPS 4 |
| 24 | 24 | #define COUNT_WPS 4 |
| @@ -26,7 +26,7 @@ · 25 unmodified lines | ||
| 26 | 26 | #include <errno.h> |
| 27 | 27 | #include <signal.h> |
| 28 | 28 | |
| 29 | −#include "../kselftest.h" | |
| 29 | +#include "kselftest.h" | |
| 30 | 30 | |
| 31 | 31 | static volatile uint8_t var[96] __attribute__((__aligned__(32))); |
| 32 | 32 | |
| @@ -19,7 +19,7 @@ · 18 unmodified lines | ||
| 19 | 19 | #include <sys/types.h> |
| 20 | 20 | #include <sys/wait.h> |
| 21 | 21 | |
| 22 | −#include "../kselftest.h" | |
| 22 | +#include "kselftest.h" | |
| 23 | 23 | |
| 24 | 24 | void child(int cpu) |
| 25 | 25 | { |
| @@ -16,7 +16,7 @@ · 15 unmodified lines | ||
| 16 | 16 | #include <fcntl.h> |
| 17 | 17 | #include <errno.h> |
| 18 | 18 | |
| 19 | −#include "../kselftest.h" | |
| 19 | +#include "kselftest.h" | |
| 20 | 20 | |
| 21 | 21 | #define NR_TESTS 9 |
| 22 | 22 | |
| @@ -18,7 +18,7 @@ · 17 unmodified lines | ||
| 18 | 18 | #include <sys/prctl.h> |
| 19 | 19 | #include <sys/stat.h> |
| 20 | 20 | |
| 21 | −#include "../kselftest.h" | |
| 21 | +#include "kselftest.h" | |
| 22 | 22 | |
| 23 | 23 | static int nerrs; |
| 24 | 24 | static pid_t mpid; /* main() pid is used to avoid duplicate test counts */ |
| @@ -7,7 +7,7 @@ · 6 unmodified lines | ||
| 7 | 7 | #include <sys/prctl.h> |
| 8 | 8 | #include <sys/auxv.h> |
| 9 | 9 | |
| 10 | −#include "../kselftest.h" | |
| 10 | +#include "kselftest.h" | |
| 11 | 11 | |
| 12 | 12 | #if __GLIBC__ > 2 || (__GLIBC__ == 2 && __GLIBC_MINOR__ >= 19) |
| 13 | 13 | # define HAVE_GETAUXVAL |
| @@ -17,7 +17,7 @@ · 16 unmodified lines | ||
| 17 | 17 | #include <string.h> |
| 18 | 18 | #include <pthread.h> |
| 19 | 19 | |
| 20 | −#include "../kselftest.h" | |
| 20 | +#include "kselftest.h" | |
| 21 | 21 | #include "cgroup_util.h" |
| 22 | 22 | |
| 23 | 23 | static bool nsdelegate; |
| @@ -11,7 +11,7 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <time.h> |
| 12 | 12 | #include <unistd.h> |
| 13 | 13 | |
| 14 | −#include "../kselftest.h" | |
| 14 | +#include "kselftest.h" | |
| 15 | 15 | #include "cgroup_util.h" |
| 16 | 16 | |
| 17 | 17 | enum hog_clock_type { |
| @@ -3,7 +3,7 @@ · 2 unmodified lines | ||
| 3 | 3 | #include <linux/limits.h> |
| 4 | 4 | #include <signal.h> |
| 5 | 5 | |
| 6 | −#include "../kselftest.h" | |
| 6 | +#include "kselftest.h" | |
| 7 | 7 | #include "cgroup_util.h" |
| 8 | 8 | |
| 9 | 9 | static int idle_process_fn(const char *cgroup, void *arg) |
| @@ -11,7 +11,7 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <string.h> |
| 12 | 12 | #include <sys/wait.h> |
| 13 | 13 | |
| 14 | −#include "../kselftest.h" | |
| 14 | +#include "kselftest.h" | |
| 15 | 15 | #include "cgroup_util.h" |
| 16 | 16 | |
| 17 | 17 | #define DEBUG |
| @@ -7,7 +7,7 @@ · 6 unmodified lines | ||
| 7 | 7 | #include <stdlib.h> |
| 8 | 8 | #include <string.h> |
| 9 | 9 | #include <fcntl.h> |
| 10 | −#include "../kselftest.h" | |
| 10 | +#include "kselftest.h" | |
| 11 | 11 | #include "cgroup_util.h" |
| 12 | 12 | |
| 13 | 13 | #define ADDR ((void *)(0x0UL)) |
| @@ -9,7 +9,7 @@ · 8 unmodified lines | ||
| 9 | 9 | #include <sys/types.h> |
| 10 | 10 | #include <unistd.h> |
| 11 | 11 | |
| 12 | −#include "../kselftest.h" | |
| 12 | +#include "kselftest.h" | |
| 13 | 13 | #include "../pidfd/pidfd.h" |
| 14 | 14 | #include "cgroup_util.h" |
| 15 | 15 | |
| @@ -14,7 +14,7 @@ · 13 unmodified lines | ||
| 14 | 14 | #include <sys/sysinfo.h> |
| 15 | 15 | #include <pthread.h> |
| 16 | 16 | |
| 17 | −#include "../kselftest.h" | |
| 17 | +#include "kselftest.h" | |
| 18 | 18 | #include "cgroup_util.h" |
| 19 | 19 | |
| 20 | 20 | |
| @@ -18,7 +18,7 @@ · 17 unmodified lines | ||
| 18 | 18 | #include <errno.h> |
| 19 | 19 | #include <sys/mman.h> |
| 20 | 20 | |
| 21 | −#include "../kselftest.h" | |
| 21 | +#include "kselftest.h" | |
| 22 | 22 | #include "cgroup_util.h" |
| 23 | 23 | |
| 24 | 24 | static bool has_localevents; |
| @@ -9,7 +9,7 @@ · 8 unmodified lines | ||
| 9 | 9 | #include <sys/types.h> |
| 10 | 10 | #include <unistd.h> |
| 11 | 11 | |
| 12 | −#include "../kselftest.h" | |
| 12 | +#include "kselftest.h" | |
| 13 | 13 | #include "cgroup_util.h" |
| 14 | 14 | |
| 15 | 15 | static int run_success(const char *cgroup, void *arg) |
| @@ -10,7 +10,7 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <sys/wait.h> |
| 11 | 11 | #include <sys/mman.h> |
| 12 | 12 | |
| 13 | −#include "../kselftest.h" | |
| 13 | +#include "kselftest.h" | |
| 14 | 14 | #include "cgroup_util.h" |
| 15 | 15 | |
| 16 | 16 | static int read_int(const char *path, size_t *value) |
| @@ -18,7 +18,7 @@ · 17 unmodified lines | ||
| 18 | 18 | #include <unistd.h> |
| 19 | 19 | #include <sched.h> |
| 20 | 20 | |
| 21 | −#include "../kselftest.h" | |
| 21 | +#include "kselftest.h" | |
| 22 | 22 | #include "clone3_selftests.h" |
| 23 | 23 | |
| 24 | 24 | enum test_mode { |
| @@ -24,7 +24,7 @@ · 23 unmodified lines | ||
| 24 | 24 | #include <unistd.h> |
| 25 | 25 | #include <sched.h> |
| 26 | 26 | |
| 27 | −#include "../kselftest_harness.h" | |
| 27 | +#include "kselftest_harness.h" | |
| 28 | 28 | #include "clone3_selftests.h" |
| 29 | 29 | |
| 30 | 30 | static void child_exit(int ret) |
| @@ -13,7 +13,7 @@ · 12 unmodified lines | ||
| 13 | 13 | #include <sys/syscall.h> |
| 14 | 14 | #include <sys/wait.h> |
| 15 | 15 | |
| 16 | −#include "../kselftest.h" | |
| 16 | +#include "kselftest.h" | |
| 17 | 17 | #include "clone3_selftests.h" |
| 18 | 18 | |
| 19 | 19 | static void nop_handler(int signo) |
| @@ -11,7 +11,7 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <syscall.h> |
| 12 | 12 | #include <sys/wait.h> |
| 13 | 13 | |
| 14 | −#include "../kselftest.h" | |
| 14 | +#include "kselftest.h" | |
| 15 | 15 | |
| 16 | 16 | #define ptr_to_u64(ptr) ((__u64)((uintptr_t)(ptr))) |
| 17 | 17 | |
| @@ -20,7 +20,7 @@ · 19 unmodified lines | ||
| 20 | 20 | #include <unistd.h> |
| 21 | 21 | #include <sched.h> |
| 22 | 22 | |
| 23 | −#include "../kselftest.h" | |
| 23 | +#include "kselftest.h" | |
| 24 | 24 | #include "clone3_selftests.h" |
| 25 | 25 | |
| 26 | 26 | #define MAX_PID_NS_LEVEL 32 |
| @@ -16,7 +16,7 @@ · 15 unmodified lines | ||
| 16 | 16 | #include <signal.h> |
| 17 | 17 | #include <string.h> |
| 18 | 18 | |
| 19 | −#include "../kselftest.h" | |
| 19 | +#include "kselftest.h" | |
| 20 | 20 | |
| 21 | 21 | #define NL_MESSAGE_SIZE (sizeof(struct nlmsghdr) + sizeof(struct cn_msg) + \ |
| 22 | 22 | sizeof(struct proc_input)) |
| @@ -14,7 +14,7 @@ · 13 unmodified lines | ||
| 14 | 14 | #include <sys/resource.h> |
| 15 | 15 | #include <linux/close_range.h> |
| 16 | 16 | |
| 17 | −#include "../kselftest_harness.h" | |
| 17 | +#include "kselftest_harness.h" | |
| 18 | 18 | #include "../clone3/clone3_selftests.h" |
| 19 | 19 | |
| 20 | 20 | |
| @@ -14,7 +14,7 @@ · 13 unmodified lines | ||
| 14 | 14 | #include <sys/resource.h> |
| 15 | 15 | #include <linux/close_range.h> |
| 16 | 16 | |
| 17 | −#include "../kselftest_harness.h" | |
| 17 | +#include "kselftest_harness.h" | |
| 18 | 18 | #include "../clone3/clone3_selftests.h" |
| 19 | 19 | |
| 20 | 20 | TEST(unshare_EMFILE) |
| @@ -19,7 +19,7 @@ · 18 unmodified lines | ||
| 19 | 19 | #include <sys/un.h> |
| 20 | 20 | #include <unistd.h> |
| 21 | 21 | |
| 22 | −#include "../kselftest_harness.h" | |
| 22 | +#include "kselftest_harness.h" | |
| 23 | 23 | #include "../filesystems/wrappers.h" |
| 24 | 24 | #include "../pidfd/pidfd.h" |
| 25 | 25 | |
| @@ -15,7 +15,7 @@ · 14 unmodified lines | ||
| 15 | 15 | #include <linux/dma-buf.h> |
| 16 | 16 | #include <linux/dma-heap.h> |
| 17 | 17 | #include <drm/drm.h> |
| 18 | −#include "../kselftest.h" | |
| 18 | +#include "kselftest.h" | |
| 19 | 19 | |
| 20 | 20 | #define DEVPATH "/dev/dma_heap" |
| 21 | 21 | |
| @@ -16,7 +16,7 @@ · 15 unmodified lines | ||
| 16 | 16 | #include <sys/mman.h> |
| 17 | 17 | #include <linux/memfd.h> |
| 18 | 18 | #include <linux/udmabuf.h> |
| 19 | −#include "../../kselftest.h" | |
| 19 | +#include "kselftest.h" | |
| 20 | 20 | |
| 21 | 21 | #define TEST_PREFIX "drivers/dma-buf/udmabuf" |
| 22 | 22 | #define NUM_PAGES 4 |
| @@ -57,7 +57,7 @@ · 56 unmodified lines | ||
| 57 | 57 | #include <string.h> |
| 58 | 58 | #include <unistd.h> |
| 59 | 59 | |
| 60 | −#include "../../kselftest.h" | |
| 60 | +#include "kselftest.h" | |
| 61 | 61 | #include "../../net/lib/ksft.h" |
| 62 | 62 | |
| 63 | 63 | #define DPORT 8000 |
| @@ -55,7 +55,7 @@ · 54 unmodified lines | ||
| 55 | 55 | #include <ynl.h> |
| 56 | 56 | #include "ethtool-user.h" |
| 57 | 57 | |
| 58 | −#include "../../../kselftest.h" | |
| 58 | +#include "kselftest.h" | |
| 59 | 59 | #include "../../../net/lib/ksft.h" |
| 60 | 60 | |
| 61 | 61 | #define TOEPLITZ_KEY_MIN_LEN 40 |
| @@ -12,7 +12,7 @@ · 11 unmodified lines | ||
| 12 | 12 | #include <time.h> |
| 13 | 13 | #include <pthread.h> |
| 14 | 14 | #include <linux/ntsync.h> |
| 15 | −#include "../../kselftest_harness.h" | |
| 15 | +#include "kselftest_harness.h" | |
| 16 | 16 | |
| 17 | 17 | static int read_sem_state(int sem, __u32 *count, __u32 *max) |
| 18 | 18 | { |
| @@ -14,7 +14,7 @@ · 13 unmodified lines | ||
| 14 | 14 | |
| 15 | 15 | #include <asm/uvdevice.h> |
| 16 | 16 | |
| 17 | −#include "../../../kselftest_harness.h" | |
| 17 | +#include "kselftest_harness.h" | |
| 18 | 18 | |
| 19 | 19 | #define UV_PATH "/dev/uv" |
| 20 | 20 | #define BUFFER_SIZE 0x200 |
| @@ -30,7 +30,7 @@ · 29 unmodified lines | ||
| 30 | 30 | #define _ASM_GENERIC_FCNTL_H |
| 31 | 31 | #include <linux/fcntl.h> |
| 32 | 32 | |
| 33 | −#include "../kselftest_harness.h" | |
| 33 | +#include "kselftest_harness.h" | |
| 34 | 34 | |
| 35 | 35 | static int sys_execveat(int dirfd, const char *pathname, char *const argv[], |
| 36 | 36 | char *const envp[], int flags) |
| @@ -21,7 +21,7 @@ · 20 unmodified lines | ||
| 21 | 21 | #include <string.h> |
| 22 | 22 | #include <unistd.h> |
| 23 | 23 | |
| 24 | −#include "../kselftest.h" | |
| 24 | +#include "kselftest.h" | |
| 25 | 25 | |
| 26 | 26 | #define TESTS_EXPECTED 54 |
| 27 | 27 | #define TEST_NAME_LEN (PATH_MAX * 4) |
| @@ -6,7 +6,7 @@ · 5 unmodified lines | ||
| 6 | 6 | #include <stdio.h> |
| 7 | 7 | #include <stdlib.h> |
| 8 | 8 | #include <stdbool.h> |
| 9 | −#include "../kselftest.h" | |
| 9 | +#include "kselftest.h" | |
| 10 | 10 | |
| 11 | 11 | struct Statistics { |
| 12 | 12 | unsigned long long load_address; |
| @@ -9,7 +9,7 @@ · 8 unmodified lines | ||
| 9 | 9 | #include <sys/sysmacros.h> |
| 10 | 10 | #include <sys/types.h> |
| 11 | 11 | |
| 12 | −#include "../kselftest_harness.h" | |
| 12 | +#include "kselftest_harness.h" | |
| 13 | 13 | |
| 14 | 14 | /* Remove a file, ignoring the result if it didn't exist. */ |
| 15 | 15 | void rm(struct __test_metadata *_metadata, const char *pathname, |
| @@ -5,7 +5,7 @@ · 4 unmodified lines | ||
| 5 | 5 | #include <sys/types.h> |
| 6 | 6 | #include <sys/wait.h> |
| 7 | 7 | |
| 8 | −#include "../kselftest.h" | |
| 8 | +#include "kselftest.h" | |
| 9 | 9 | |
| 10 | 10 | #define FORK(exec) \ |
| 11 | 11 | do { \ |
| @@ -23,7 +23,7 @@ · 22 unmodified lines | ||
| 23 | 23 | #include <fcntl.h> |
| 24 | 24 | #include <sys/mount.h> |
| 25 | 25 | #include <unistd.h> |
| 26 | −#include "../kselftest.h" | |
| 26 | +#include "kselftest.h" | |
| 27 | 27 | |
| 28 | 28 | int main(void) |
| 29 | 29 | { |
| @@ -7,7 +7,7 @@ · 6 unmodified lines | ||
| 7 | 7 | #include <syscall.h> |
| 8 | 8 | #include <unistd.h> |
| 9 | 9 | |
| 10 | −#include "../kselftest.h" | |
| 10 | +#include "kselftest.h" | |
| 11 | 11 | |
| 12 | 12 | int sys_fchmodat2(int dfd, const char *filename, mode_t mode, int flags) |
| 13 | 13 | { |
| @@ -6,7 +6,7 @@ · 5 unmodified lines | ||
| 6 | 6 | #include <stdio.h> |
| 7 | 7 | #include <unistd.h> |
| 8 | 8 | #include <string.h> |
| 9 | −#include "../kselftest.h" | |
| 9 | +#include "kselftest.h" | |
| 10 | 10 | |
| 11 | 11 | static int lock_set(int fd, struct flock *fl) |
| 12 | 12 | { |
| @@ -6,7 +6,7 @@ · 5 unmodified lines | ||
| 6 | 6 | #include <stdio.h> |
| 7 | 7 | #include <sys/stat.h> |
| 8 | 8 | |
| 9 | −#include "../kselftest_harness.h" | |
| 9 | +#include "kselftest_harness.h" | |
| 10 | 10 | #include "wrappers.h" |
| 11 | 11 | |
| 12 | 12 | TEST(anon_inode_no_chown) |
| @@ -21,7 +21,7 @@ · 20 unmodified lines | ||
| 21 | 21 | #include <linux/android/binder.h> |
| 22 | 22 | #include <linux/android/binderfs.h> |
| 23 | 23 | |
| 24 | −#include "../../kselftest_harness.h" | |
| 24 | +#include "kselftest_harness.h" | |
| 25 | 25 | |
| 26 | 26 | #define DEFAULT_THREADS 4 |
| 27 | 27 | |
| @@ -11,7 +11,7 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <asm/ioctls.h> |
| 12 | 12 | #include <sys/mount.h> |
| 13 | 13 | #include <sys/wait.h> |
| 14 | −#include "../kselftest.h" | |
| 14 | +#include "kselftest.h" | |
| 15 | 15 | |
| 16 | 16 | static bool terminal_dup2(int duplicate, int original) |
| 17 | 17 | { |
| @@ -11,7 +11,7 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <sys/epoll.h> |
| 12 | 12 | #include <sys/socket.h> |
| 13 | 13 | #include <sys/eventfd.h> |
| 14 | −#include "../../kselftest_harness.h" | |
| 14 | +#include "kselftest_harness.h" | |
| 15 | 15 | |
| 16 | 16 | struct epoll_mtcontext |
| 17 | 17 | { |
| @@ -11,7 +11,7 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <pthread.h> |
| 12 | 12 | #include <sys/epoll.h> |
| 13 | 13 | #include <sys/eventfd.h> |
| 14 | −#include "../../kselftest_harness.h" | |
| 14 | +#include "kselftest_harness.h" | |
| 15 | 15 | |
| 16 | 16 | #define EVENTFD_TEST_ITERATIONS 100000UL |
| 17 | 17 | |
| @@ -13,7 +13,7 @@ · 12 unmodified lines | ||
| 13 | 13 | #include <unistd.h> |
| 14 | 14 | #include <sys/mount.h> |
| 15 | 15 | |
| 16 | −#include "../kselftest_harness.h" | |
| 16 | +#include "kselftest_harness.h" | |
| 17 | 17 | |
| 18 | 18 | #define ASSERT_ERRNO(expected, _t, seen) \ |
| 19 | 19 | __EXPECT(expected, #expected, \ |
| @@ -12,7 +12,7 @@ · 11 unmodified lines | ||
| 12 | 12 | #include <sys/mount.h> |
| 13 | 13 | #include <unistd.h> |
| 14 | 14 | |
| 15 | −#include "../kselftest_harness.h" | |
| 15 | +#include "kselftest_harness.h" | |
| 16 | 16 | |
| 17 | 17 | #include <linux/types.h> |
| 18 | 18 | #include <linux/mount.h> |
| @@ -17,7 +17,7 @@ · 16 unmodified lines | ||
| 17 | 17 | #include <sched.h> |
| 18 | 18 | #include <linux/limits.h> |
| 19 | 19 | |
| 20 | −#include "../../kselftest_harness.h" | |
| 20 | +#include "kselftest_harness.h" | |
| 21 | 21 | |
| 22 | 22 | #define FUSECTL_MOUNTPOINT "/sys/fs/fuse/connections" |
| 23 | 23 | #define FUSE_MOUNTPOINT "/tmp/fuse_mnt_XXXXXX" |
| @@ -7,7 +7,7 @@ · 6 unmodified lines | ||
| 7 | 7 | #include <sys/stat.h> |
| 8 | 8 | #include <sys/xattr.h> |
| 9 | 9 | |
| 10 | −#include "../kselftest_harness.h" | |
| 10 | +#include "kselftest_harness.h" | |
| 11 | 11 | #include "wrappers.h" |
| 12 | 12 | |
| 13 | 13 | TEST(kernfs_listxattr) |
| @@ -19,7 +19,7 @@ typedef struct { · 18 unmodified lines | ||
| 19 | 19 | #include <sys/syscall.h> |
| 20 | 20 | #include <sys/fanotify.h> |
| 21 | 21 | |
| 22 | −#include "../../kselftest_harness.h" | |
| 22 | +#include "kselftest_harness.h" | |
| 23 | 23 | #include "../statmount/statmount.h" |
| 24 | 24 | #include "../utils.h" |
| 25 | 25 | |
| @@ -19,7 +19,7 @@ typedef struct { · 18 unmodified lines | ||
| 19 | 19 | #include <sys/syscall.h> |
| 20 | 20 | #include <sys/fanotify.h> |
| 21 | 21 | |
| 22 | −#include "../../kselftest_harness.h" | |
| 22 | +#include "kselftest_harness.h" | |
| 23 | 23 | #include "../statmount/statmount.h" |
| 24 | 24 | #include "../utils.h" |
| 25 | 25 | |
| @@ -12,7 +12,7 @@ · 11 unmodified lines | ||
| 12 | 12 | #include <sys/mount.h> |
| 13 | 13 | #include <unistd.h> |
| 14 | 14 | |
| 15 | −#include "../../kselftest_harness.h" | |
| 15 | +#include "kselftest_harness.h" | |
| 16 | 16 | |
| 17 | 17 | #define MNT_NS_COUNT 11 |
| 18 | 18 | #define MNT_NS_LAST_INDEX 10 |
| @@ -15,7 +15,7 @@ · 14 unmodified lines | ||
| 15 | 15 | #include <sched.h> |
| 16 | 16 | #include <fcntl.h> |
| 17 | 17 | |
| 18 | −#include "../../kselftest.h" | |
| 18 | +#include "kselftest.h" | |
| 19 | 19 | #include "log.h" |
| 20 | 20 | #include "../wrappers.h" |
| 21 | 21 | |
| @@ -12,7 +12,7 @@ · 11 unmodified lines | ||
| 12 | 12 | #include <sys/mount.h> |
| 13 | 13 | #include <unistd.h> |
| 14 | 14 | |
| 15 | −#include "../../kselftest_harness.h" | |
| 15 | +#include "kselftest_harness.h" | |
| 16 | 16 | #include "../../pidfd/pidfd.h" |
| 17 | 17 | #include "log.h" |
| 18 | 18 | #include "../utils.h" |
| @@ -11,7 +11,7 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <unistd.h> |
| 12 | 12 | |
| 13 | 13 | #include "statmount.h" |
| 14 | −#include "../../kselftest_harness.h" | |
| 14 | +#include "kselftest_harness.h" | |
| 15 | 15 | |
| 16 | 16 | #ifndef LISTMOUNT_REVERSE |
| 17 | 17 | #define LISTMOUNT_REVERSE (1 << 0) /* List later mounts first */ |
| @@ -13,7 +13,7 @@ · 12 unmodified lines | ||
| 13 | 13 | #include <linux/stat.h> |
| 14 | 14 | |
| 15 | 15 | #include "statmount.h" |
| 16 | −#include "../../kselftest.h" | |
| 16 | +#include "kselftest.h" | |
| 17 | 17 | |
| 18 | 18 | static const char *const known_fs[] = { |
| 19 | 19 | "9p", "adfs", "affs", "afs", "aio", "anon_inodefs", "apparmorfs", |
| @@ -15,7 +15,7 @@ · 14 unmodified lines | ||
| 15 | 15 | |
| 16 | 16 | #include "statmount.h" |
| 17 | 17 | #include "../utils.h" |
| 18 | −#include "../../kselftest.h" | |
| 18 | +#include "kselftest.h" | |
| 19 | 19 | |
| 20 | 20 | #define NSID_PASS 0 |
| 21 | 21 | #define NSID_FAIL 1 |
| @@ -20,7 +20,7 @@ · 19 unmodified lines | ||
| 20 | 20 | #include <sys/xattr.h> |
| 21 | 21 | #include <sys/mount.h> |
| 22 | 22 | |
| 23 | −#include "../kselftest.h" | |
| 23 | +#include "kselftest.h" | |
| 24 | 24 | #include "wrappers.h" |
| 25 | 25 | #include "utils.h" |
| 26 | 26 | |
| @@ -18,7 +18,7 @@ · 17 unmodified lines | ||
| 18 | 18 | |
| 19 | 19 | #include "futextest.h" |
| 20 | 20 | #include "futex2test.h" |
| 21 | −#include "../../kselftest_harness.h" | |
| 21 | +#include "kselftest_harness.h" | |
| 22 | 22 | |
| 23 | 23 | #define MAX_THREADS 64 |
| 24 | 24 | |
| @@ -14,7 +14,7 @@ · 13 unmodified lines | ||
| 14 | 14 | #include <linux/prctl.h> |
| 15 | 15 | #include <sys/prctl.h> |
| 16 | 16 | |
| 17 | −#include "../../kselftest_harness.h" | |
| 17 | +#include "kselftest_harness.h" | |
| 18 | 18 | |
| 19 | 19 | #define MAX_THREADS 64 |
| 20 | 20 | |
| @@ -9,7 +9,7 @@ · 8 unmodified lines | ||
| 9 | 9 | #include <limits.h> |
| 10 | 10 | |
| 11 | 11 | #include "futextest.h" |
| 12 | −#include "../../kselftest_harness.h" | |
| 12 | +#include "kselftest_harness.h" | |
| 13 | 13 | |
| 14 | 14 | #define timeout_ns 30000000 |
| 15 | 15 | #define WAKE_WAIT_US 10000 |
| @@ -29,7 +29,7 @@ · 28 unmodified lines | ||
| 29 | 29 | |
| 30 | 30 | #include "atomic.h" |
| 31 | 31 | #include "futextest.h" |
| 32 | −#include "../../kselftest_harness.h" | |
| 32 | +#include "kselftest_harness.h" | |
| 33 | 33 | |
| 34 | 34 | #define MAX_WAKE_ITERS 1000 |
| 35 | 35 | #define THREAD_MAX 10 |
| @@ -25,7 +25,7 @@ · 24 unmodified lines | ||
| 25 | 25 | #include <time.h> |
| 26 | 26 | |
| 27 | 27 | #include "futextest.h" |
| 28 | −#include "../../kselftest_harness.h" | |
| 28 | +#include "kselftest_harness.h" | |
| 29 | 29 | |
| 30 | 30 | futex_t f1 = FUTEX_INITIALIZER; |
| 31 | 31 | futex_t f2 = FUTEX_INITIALIZER; |
| @@ -27,7 +27,7 @@ · 26 unmodified lines | ||
| 27 | 27 | |
| 28 | 28 | #include "atomic.h" |
| 29 | 29 | #include "futextest.h" |
| 30 | −#include "../../kselftest_harness.h" | |
| 30 | +#include "kselftest_harness.h" | |
| 31 | 31 | |
| 32 | 32 | #define DELAY_US 100 |
| 33 | 33 | |
| @@ -11,7 +11,7 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <fcntl.h> |
| 12 | 12 | |
| 13 | 13 | #include "futextest.h" |
| 14 | −#include "../../kselftest_harness.h" | |
| 14 | +#include "kselftest_harness.h" | |
| 15 | 15 | |
| 16 | 16 | #define timeout_ns 30000000 |
| 17 | 17 | #define WAKE_WAIT_US 10000 |
| @@ -28,7 +28,7 @@ · 27 unmodified lines | ||
| 28 | 28 | #include <signal.h> |
| 29 | 29 | |
| 30 | 30 | #include "futextest.h" |
| 31 | −#include "../../kselftest_harness.h" | |
| 31 | +#include "kselftest_harness.h" | |
| 32 | 32 | |
| 33 | 33 | #define PAGE_SZ 4096 |
| 34 | 34 | |
| @@ -19,7 +19,7 @@ · 18 unmodified lines | ||
| 19 | 19 | |
| 20 | 20 | #include "futextest.h" |
| 21 | 21 | #include "futex2test.h" |
| 22 | −#include "../../kselftest_harness.h" | |
| 22 | +#include "kselftest_harness.h" | |
| 23 | 23 | |
| 24 | 24 | static long timeout_ns = 100000; /* 100us default timeout */ |
| 25 | 25 | static futex_t futex_pi; |
| @@ -30,7 +30,7 @@ · 29 unmodified lines | ||
| 30 | 30 | #include <libgen.h> |
| 31 | 31 | |
| 32 | 32 | #include "futextest.h" |
| 33 | −#include "../../kselftest_harness.h" | |
| 33 | +#include "kselftest_harness.h" | |
| 34 | 34 | |
| 35 | 35 | #define WAIT_US 5000000 |
| 36 | 36 | |
| @@ -24,7 +24,7 @@ · 23 unmodified lines | ||
| 24 | 24 | |
| 25 | 25 | #include "futextest.h" |
| 26 | 26 | #include "futex2test.h" |
| 27 | −#include "../../kselftest_harness.h" | |
| 27 | +#include "kselftest_harness.h" | |
| 28 | 28 | |
| 29 | 29 | #define timeout_ns 100000 |
| 30 | 30 | |
| @@ -18,7 +18,7 @@ · 17 unmodified lines | ||
| 18 | 18 | |
| 19 | 19 | #include "futextest.h" |
| 20 | 20 | #include "futex2test.h" |
| 21 | −#include "../../kselftest_harness.h" | |
| 21 | +#include "kselftest_harness.h" | |
| 22 | 22 | |
| 23 | 23 | #define WAKE_WAIT_US 10000 |
| 24 | 24 | #define NR_FUTEXES 30 |
| @@ -1,7 +1,7 @@ | ||
| 1 | 1 | /* SPDX-License-Identifier: GPL-2.0 */ |
| 2 | 2 | /* Copyright (c) 2022-2024 Red Hat */ |
| 3 | 3 | |
| 4 | −#include "../kselftest_harness.h" | |
| 4 | +#include "kselftest_harness.h" | |
| 5 | 5 | |
| 6 | 6 | #include <fcntl.h> |
| 7 | 7 | #include <fnmatch.h> |
| @@ -11,7 +11,7 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <errno.h> |
| 12 | 12 | #include <string.h> |
| 13 | 13 | #include <time.h> |
| 14 | −#include "../kselftest.h" | |
| 14 | +#include "kselftest.h" | |
| 15 | 15 | |
| 16 | 16 | #define MSEC_PER_SEC 1000L |
| 17 | 17 | #define NSEC_PER_MSEC 1000000L |
| @@ -11,7 +11,7 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <assert.h> |
| 12 | 12 | #include <poll.h> |
| 13 | 13 | |
| 14 | −#include "../kselftest_harness.h" | |
| 14 | +#include "kselftest_harness.h" | |
| 15 | 15 | #include "../../../../drivers/iommu/iommufd/iommufd_test.h" |
| 16 | 16 | |
| 17 | 17 | /* Hack to make assertions more readable */ |
| @@ -7,7 +7,7 @@ · 6 unmodified lines | ||
| 7 | 7 | #include <sys/msg.h> |
| 8 | 8 | #include <fcntl.h> |
| 9 | 9 | |
| 10 | −#include "../kselftest.h" | |
| 10 | +#include "kselftest.h" | |
| 11 | 11 | |
| 12 | 12 | #define MAX_MSG_SIZE 32 |
| 13 | 13 | |
| @@ -23,7 +23,7 @@ · 22 unmodified lines | ||
| 23 | 23 | #include <dirent.h> |
| 24 | 24 | #include <sys/stat.h> |
| 25 | 25 | #include <fcntl.h> |
| 26 | −#include "../kselftest.h" | |
| 26 | +#include "kselftest.h" | |
| 27 | 27 | |
| 28 | 28 | #define TEST_SCANCODES 10 |
| 29 | 29 | #define SYSFS_PATH_MAX 256 |
| @@ -18,7 +18,7 @@ · 17 unmodified lines | ||
| 18 | 18 | #include <sys/wait.h> |
| 19 | 19 | #include <sys/epoll.h> |
| 20 | 20 | |
| 21 | −#include "../kselftest.h" | |
| 21 | +#include "kselftest.h" | |
| 22 | 22 | |
| 23 | 23 | static long sys_kcmp(int pid1, int pid2, int type, unsigned long fd1, unsigned long fd2) |
| 24 | 24 | { |
| @@ -59,6 +59,7 @@ function build_kernel() { · 58 unmodified lines | ||
| 59 | 59 | tee "$kconfig" > "$kho_config" <<EOF |
| 60 | 60 | CONFIG_BLK_DEV_INITRD=y |
| 61 | 61 | CONFIG_KEXEC_HANDOVER=y |
| 62 | +CONFIG_KEXEC_HANDOVER_DEBUGFS=y | |
| 62 | 63 | CONFIG_TEST_KEXEC_HANDOVER=y |
| 63 | 64 | CONFIG_DEBUG_KERNEL=y |
| 64 | 65 | CONFIG_DEBUG_VM=y |
| @@ -14,7 +14,7 @@ · 13 unmodified lines | ||
| 14 | 14 | * |
| 15 | 15 | * .. code-block:: c |
| 16 | 16 | * |
| 17 | − * #include "../kselftest_harness.h" | |
| 17 | + * #include "kselftest_harness.h" | |
| 18 | 18 | * |
| 19 | 19 | * TEST(standalone_test) { |
| 20 | 20 | * do_some_stuff; |
| @@ -8,7 +8,7 @@ · 7 unmodified lines | ||
| 8 | 8 | /* Avoid any inconsistencies */ |
| 9 | 9 | #define TH_LOG_STREAM stdout |
| 10 | 10 | |
| 11 | −#include "../kselftest_harness.h" | |
| 11 | +#include "kselftest_harness.h" | |
| 12 | 12 | |
| 13 | 13 | static void test_helper(struct __test_metadata *_metadata) |
| 14 | 14 | { |
| @@ -20,7 +20,7 @@ · 19 unmodified lines | ||
| 20 | 20 | #include <sys/time.h> |
| 21 | 21 | #include <unistd.h> |
| 22 | 22 | |
| 23 | −#include "../kselftest.h" | |
| 23 | +#include "kselftest.h" | |
| 24 | 24 | |
| 25 | 25 | #ifndef ARRAY_SIZE |
| 26 | 26 | #define ARRAY_SIZE(arr) (sizeof(arr) / sizeof((arr)[0])) |
| @@ -17,7 +17,7 @@ · 16 unmodified lines | ||
| 17 | 17 | #include <sys/wait.h> |
| 18 | 18 | #include <unistd.h> |
| 19 | 19 | |
| 20 | −#include "../kselftest_harness.h" | |
| 20 | +#include "kselftest_harness.h" | |
| 21 | 21 | #include "wrappers.h" |
| 22 | 22 | |
| 23 | 23 | #define TMP_DIR "tmp" |
| @@ -199,6 +199,9 @@ clean: $(if $(TEST_GEN_MODS_DIR),clean_mods_dir) · 198 unmodified lines | ||
| 199 | 199 | # Build with _GNU_SOURCE by default |
| 200 | 200 | CFLAGS += -D_GNU_SOURCE= |
| 201 | 201 | |
| 202 | +# Additional include paths needed by kselftest.h and local headers | |
| 203 | +CFLAGS += -I${top_srcdir}/tools/testing/selftests | |
| 204 | + | |
| 202 | 205 | # Enables to extend CFLAGS and LDFLAGS from command line, e.g. |
| 203 | 206 | # make USERCFLAGS=-Werror USERLDFLAGS=-static |
| 204 | 207 | CFLAGS += $(USERCFLAGS) |
| @@ -0,0 +1,9 @@ | ||
| 1 | +# SPDX-License-Identifier: GPL-2.0-only | |
| 2 | +* | |
| 3 | +!/**/ | |
| 4 | +!*.c | |
| 5 | +!*.h | |
| 6 | +!*.sh | |
| 7 | +!.gitignore | |
| 8 | +!config | |
| 9 | +!Makefile | |
| @@ -0,0 +1,34 @@ | ||
| 1 | +# SPDX-License-Identifier: GPL-2.0-only | |
| 2 | + | |
| 3 | +LIB_C += luo_test_utils.c | |
| 4 | + | |
| 5 | +TEST_GEN_PROGS += liveupdate | |
| 6 | + | |
| 7 | +TEST_GEN_PROGS_EXTENDED += luo_kexec_simple | |
| 8 | +TEST_GEN_PROGS_EXTENDED += luo_multi_session | |
| 9 | + | |
| 10 | +TEST_FILES += do_kexec.sh | |
| 11 | + | |
| 12 | +include ../lib.mk | |
| 13 | + | |
| 14 | +CFLAGS += $(KHDR_INCLUDES) | |
| 15 | +CFLAGS += -Wall -O2 -Wno-unused-function | |
| 16 | +CFLAGS += -MD | |
| 17 | + | |
| 18 | +LIB_O := $(patsubst %.c, $(OUTPUT)/%.o, $(LIB_C)) | |
| 19 | +TEST_O := $(patsubst %, %.o, $(TEST_GEN_PROGS)) | |
| 20 | +TEST_O += $(patsubst %, %.o, $(TEST_GEN_PROGS_EXTENDED)) | |
| 21 | + | |
| 22 | +TEST_DEP_FILES := $(patsubst %.o, %.d, $(LIB_O)) | |
| 23 | +TEST_DEP_FILES += $(patsubst %.o, %.d, $(TEST_O)) | |
| 24 | +-include $(TEST_DEP_FILES) | |
| 25 | + | |
| 26 | +$(LIB_O): $(OUTPUT)/%.o: %.c | |
| 27 | + $(CC) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c $< -o $@ | |
| 28 | + | |
| 29 | +$(TEST_GEN_PROGS) $(TEST_GEN_PROGS_EXTENDED): $(OUTPUT)/%: %.o $(LIB_O) | |
| 30 | + $(CC) $(CFLAGS) $(CPPFLAGS) $(LDFLAGS) $(TARGET_ARCH) $< $(LIB_O) $(LDLIBS) -o $@ | |
| 31 | + | |
| 32 | +EXTRA_CLEAN += $(LIB_O) | |
| 33 | +EXTRA_CLEAN += $(TEST_O) | |
| 34 | +EXTRA_CLEAN += $(TEST_DEP_FILES) | |
| @@ -0,0 +1,11 @@ | ||
| 1 | +CONFIG_BLK_DEV_INITRD=y | |
| 2 | +CONFIG_KEXEC_FILE=y | |
| 3 | +CONFIG_KEXEC_HANDOVER=y | |
| 4 | +CONFIG_KEXEC_HANDOVER_ENABLE_DEFAULT=y | |
| 5 | +CONFIG_KEXEC_HANDOVER_DEBUGFS=y | |
| 6 | +CONFIG_KEXEC_HANDOVER_DEBUG=y | |
| 7 | +CONFIG_LIVEUPDATE=y | |
| 8 | +CONFIG_LIVEUPDATE_TEST=y | |
| 9 | +CONFIG_MEMFD_CREATE=y | |
| 10 | +CONFIG_TMPFS=y | |
| 11 | +CONFIG_SHMEM=y | |
| @@ -0,0 +1,16 @@ | ||
| 1 | +#!/bin/sh | |
| 2 | +# SPDX-License-Identifier: GPL-2.0 | |
| 3 | +set -e | |
| 4 | + | |
| 5 | +# Use $KERNEL and $INITRAMFS to pass custom Kernel and optional initramfs | |
| 6 | + | |
| 7 | +KERNEL="${KERNEL:-/boot/bzImage}" | |
| 8 | +set -- -l -s --reuse-cmdline "$KERNEL" | |
| 9 | + | |
| 10 | +INITRAMFS="${INITRAMFS:-/boot/initramfs}" | |
| 11 | +if [ -f "$INITRAMFS" ]; then | |
| 12 | + set -- "$@" --initrd="$INITRAMFS" | |
| 13 | +fi | |
| 14 | + | |
| 15 | +kexec "$@" | |
| 16 | +kexec -e | |
| @@ -0,0 +1,348 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0 | |
| 2 | + | |
| 3 | +/* | |
| 4 | + * Copyright (c) 2025, Google LLC. | |
| 5 | + * Pasha Tatashin <pasha.tatashin@soleen.com> | |
| 6 | + */ | |
| 7 | + | |
| 8 | +/* | |
| 9 | + * Selftests for the Live Update Orchestrator. | |
| 10 | + * This test suite verifies the functionality and behavior of the | |
| 11 | + * /dev/liveupdate character device and its session management capabilities. | |
| 12 | + * | |
| 13 | + * Tests include: | |
| 14 | + * - Device access: basic open/close, and enforcement of exclusive access. | |
| 15 | + * - Session management: creation of unique sessions, and duplicate name detection. | |
| 16 | + * - Resource preservation: successfully preserving individual and multiple memfds, | |
| 17 | + * verifying contents remain accessible. | |
| 18 | + * - Complex multi-session scenarios involving mixed empty and populated files. | |
| 19 | + */ | |
| 20 | + | |
| 21 | +#include <errno.h> | |
| 22 | +#include <fcntl.h> | |
| 23 | +#include <string.h> | |
| 24 | +#include <sys/ioctl.h> | |
| 25 | +#include <unistd.h> | |
| 26 | + | |
| 27 | +#include <linux/liveupdate.h> | |
| 28 | + | |
| 29 | +#include "../kselftest.h" | |
| 30 | +#include "../kselftest_harness.h" | |
| 31 | + | |
| 32 | +#define LIVEUPDATE_DEV "/dev/liveupdate" | |
| 33 | + | |
| 34 | +FIXTURE(liveupdate_device) { | |
| 35 | + int fd1; | |
| 36 | + int fd2; | |
| 37 | +}; | |
| 38 | + | |
| 39 | +FIXTURE_SETUP(liveupdate_device) | |
| 40 | +{ | |
| 41 | + self->fd1 = -1; | |
| 42 | + self->fd2 = -1; | |
| 43 | +} | |
| 44 | + | |
| 45 | +FIXTURE_TEARDOWN(liveupdate_device) | |
| 46 | +{ | |
| 47 | + if (self->fd1 >= 0) | |
| 48 | + close(self->fd1); | |
| 49 | + if (self->fd2 >= 0) | |
| 50 | + close(self->fd2); | |
| 51 | +} | |
| 52 | + | |
| 53 | +/* | |
| 54 | + * Test Case: Basic Open and Close | |
| 55 | + * | |
| 56 | + * Verifies that the /dev/liveupdate device can be opened and subsequently | |
| 57 | + * closed without errors. Skips if the device does not exist. | |
| 58 | + */ | |
| 59 | +TEST_F(liveupdate_device, basic_open_close) | |
| 60 | +{ | |
| 61 | + self->fd1 = open(LIVEUPDATE_DEV, O_RDWR); | |
| 62 | + | |
| 63 | + if (self->fd1 < 0 && errno == ENOENT) | |
| 64 | + SKIP(return, "%s does not exist.", LIVEUPDATE_DEV); | |
| 65 | + | |
| 66 | + ASSERT_GE(self->fd1, 0); | |
| 67 | + ASSERT_EQ(close(self->fd1), 0); | |
| 68 | + self->fd1 = -1; | |
| 69 | +} | |
| 70 | + | |
| 71 | +/* | |
| 72 | + * Test Case: Exclusive Open Enforcement | |
| 73 | + * | |
| 74 | + * Verifies that the /dev/liveupdate device can only be opened by one process | |
| 75 | + * at a time. It checks that a second attempt to open the device fails with | |
| 76 | + * the EBUSY error code. | |
| 77 | + */ | |
| 78 | +TEST_F(liveupdate_device, exclusive_open) | |
| 79 | +{ | |
| 80 | + self->fd1 = open(LIVEUPDATE_DEV, O_RDWR); | |
| 81 | + | |
| 82 | + if (self->fd1 < 0 && errno == ENOENT) | |
| 83 | + SKIP(return, "%s does not exist.", LIVEUPDATE_DEV); | |
| 84 | + | |
| 85 | + ASSERT_GE(self->fd1, 0); | |
| 86 | + self->fd2 = open(LIVEUPDATE_DEV, O_RDWR); | |
| 87 | + EXPECT_LT(self->fd2, 0); | |
| 88 | + EXPECT_EQ(errno, EBUSY); | |
| 89 | +} | |
| 90 | + | |
| 91 | +/* Helper function to create a LUO session via ioctl. */ | |
| 92 | +static int create_session(int lu_fd, const char *name) | |
| 93 | +{ | |
| 94 | + struct liveupdate_ioctl_create_session args = {}; | |
| 95 | + | |
| 96 | + args.size = sizeof(args); | |
| 97 | + strncpy((char *)args.name, name, sizeof(args.name) - 1); | |
| 98 | + | |
| 99 | + if (ioctl(lu_fd, LIVEUPDATE_IOCTL_CREATE_SESSION, &args)) | |
| 100 | + return -errno; | |
| 101 | + | |
| 102 | + return args.fd; | |
| 103 | +} | |
| 104 | + | |
| 105 | +/* | |
| 106 | + * Test Case: Create Duplicate Session | |
| 107 | + * | |
| 108 | + * Verifies that attempting to create two sessions with the same name fails | |
| 109 | + * on the second attempt with EEXIST. | |
| 110 | + */ | |
| 111 | +TEST_F(liveupdate_device, create_duplicate_session) | |
| 112 | +{ | |
| 113 | + int session_fd1, session_fd2; | |
| 114 | + | |
| 115 | + self->fd1 = open(LIVEUPDATE_DEV, O_RDWR); | |
| 116 | + if (self->fd1 < 0 && errno == ENOENT) | |
| 117 | + SKIP(return, "%s does not exist", LIVEUPDATE_DEV); | |
| 118 | + | |
| 119 | + ASSERT_GE(self->fd1, 0); | |
| 120 | + | |
| 121 | + session_fd1 = create_session(self->fd1, "duplicate-session-test"); | |
| 122 | + ASSERT_GE(session_fd1, 0); | |
| 123 | + | |
| 124 | + session_fd2 = create_session(self->fd1, "duplicate-session-test"); | |
| 125 | + EXPECT_LT(session_fd2, 0); | |
| 126 | + EXPECT_EQ(-session_fd2, EEXIST); | |
| 127 | + | |
| 128 | + ASSERT_EQ(close(session_fd1), 0); | |
| 129 | +} | |
| 130 | + | |
| 131 | +/* | |
| 132 | + * Test Case: Create Distinct Sessions | |
| 133 | + * | |
| 134 | + * Verifies that creating two sessions with different names succeeds. | |
| 135 | + */ | |
| 136 | +TEST_F(liveupdate_device, create_distinct_sessions) | |
| 137 | +{ | |
| 138 | + int session_fd1, session_fd2; | |
| 139 | + | |
| 140 | + self->fd1 = open(LIVEUPDATE_DEV, O_RDWR); | |
| 141 | + if (self->fd1 < 0 && errno == ENOENT) | |
| 142 | + SKIP(return, "%s does not exist", LIVEUPDATE_DEV); | |
| 143 | + | |
| 144 | + ASSERT_GE(self->fd1, 0); | |
| 145 | + | |
| 146 | + session_fd1 = create_session(self->fd1, "distinct-session-1"); | |
| 147 | + ASSERT_GE(session_fd1, 0); | |
| 148 | + | |
| 149 | + session_fd2 = create_session(self->fd1, "distinct-session-2"); | |
| 150 | + ASSERT_GE(session_fd2, 0); | |
| 151 | + | |
| 152 | + ASSERT_EQ(close(session_fd1), 0); | |
| 153 | + ASSERT_EQ(close(session_fd2), 0); | |
| 154 | +} | |
| 155 | + | |
| 156 | +static int preserve_fd(int session_fd, int fd_to_preserve, __u64 token) | |
| 157 | +{ | |
| 158 | + struct liveupdate_session_preserve_fd args = {}; | |
| 159 | + | |
| 160 | + args.size = sizeof(args); | |
| 161 | + args.fd = fd_to_preserve; | |
| 162 | + args.token = token; | |
| 163 | + | |
| 164 | + if (ioctl(session_fd, LIVEUPDATE_SESSION_PRESERVE_FD, &args)) | |
| 165 | + return -errno; | |
| 166 | + | |
| 167 | + return 0; | |
| 168 | +} | |
| 169 | + | |
| 170 | +/* | |
| 171 | + * Test Case: Preserve MemFD | |
| 172 | + * | |
| 173 | + * Verifies that a valid memfd can be successfully preserved in a session and | |
| 174 | + * that its contents remain intact after the preservation call. | |
| 175 | + */ | |
| 176 | +TEST_F(liveupdate_device, preserve_memfd) | |
| 177 | +{ | |
| 178 | + const char *test_str = "hello liveupdate"; | |
| 179 | + char read_buf[64] = {}; | |
| 180 | + int session_fd, mem_fd; | |
| 181 | + | |
| 182 | + self->fd1 = open(LIVEUPDATE_DEV, O_RDWR); | |
| 183 | + if (self->fd1 < 0 && errno == ENOENT) | |
| 184 | + SKIP(return, "%s does not exist", LIVEUPDATE_DEV); | |
| 185 | + ASSERT_GE(self->fd1, 0); | |
| 186 | + | |
| 187 | + session_fd = create_session(self->fd1, "preserve-memfd-test"); | |
| 188 | + ASSERT_GE(session_fd, 0); | |
| 189 | + | |
| 190 | + mem_fd = memfd_create("test-memfd", 0); | |
| 191 | + ASSERT_GE(mem_fd, 0); | |
| 192 | + | |
| 193 | + ASSERT_EQ(write(mem_fd, test_str, strlen(test_str)), strlen(test_str)); | |
| 194 | + ASSERT_EQ(preserve_fd(session_fd, mem_fd, 0x1234), 0); | |
| 195 | + ASSERT_EQ(close(session_fd), 0); | |
| 196 | + | |
| 197 | + ASSERT_EQ(lseek(mem_fd, 0, SEEK_SET), 0); | |
| 198 | + ASSERT_EQ(read(mem_fd, read_buf, sizeof(read_buf)), strlen(test_str)); | |
| 199 | + ASSERT_STREQ(read_buf, test_str); | |
| 200 | + ASSERT_EQ(close(mem_fd), 0); | |
| 201 | +} | |
| 202 | + | |
| 203 | +/* | |
| 204 | + * Test Case: Preserve Multiple MemFDs | |
| 205 | + * | |
| 206 | + * Verifies that multiple memfds can be preserved in a single session, | |
| 207 | + * each with a unique token, and that their contents remain distinct and | |
| 208 | + * correct after preservation. | |
| 209 | + */ | |
| 210 | +TEST_F(liveupdate_device, preserve_multiple_memfds) | |
| 211 | +{ | |
| 212 | + const char *test_str1 = "data for memfd one"; | |
| 213 | + const char *test_str2 = "data for memfd two"; | |
| 214 | + char read_buf[64] = {}; | |
| 215 | + int session_fd, mem_fd1, mem_fd2; | |
| 216 | + | |
| 217 | + self->fd1 = open(LIVEUPDATE_DEV, O_RDWR); | |
| 218 | + if (self->fd1 < 0 && errno == ENOENT) | |
| 219 | + SKIP(return, "%s does not exist", LIVEUPDATE_DEV); | |
| 220 | + ASSERT_GE(self->fd1, 0); | |
| 221 | + | |
| 222 | + session_fd = create_session(self->fd1, "preserve-multi-memfd-test"); | |
| 223 | + ASSERT_GE(session_fd, 0); | |
| 224 | + | |
| 225 | + mem_fd1 = memfd_create("test-memfd-1", 0); | |
| 226 | + ASSERT_GE(mem_fd1, 0); | |
| 227 | + mem_fd2 = memfd_create("test-memfd-2", 0); | |
| 228 | + ASSERT_GE(mem_fd2, 0); | |
| 229 | + | |
| 230 | + ASSERT_EQ(write(mem_fd1, test_str1, strlen(test_str1)), strlen(test_str1)); | |
| 231 | + ASSERT_EQ(write(mem_fd2, test_str2, strlen(test_str2)), strlen(test_str2)); | |
| 232 | + | |
| 233 | + ASSERT_EQ(preserve_fd(session_fd, mem_fd1, 0xAAAA), 0); | |
| 234 | + ASSERT_EQ(preserve_fd(session_fd, mem_fd2, 0xBBBB), 0); | |
| 235 | + | |
| 236 | + memset(read_buf, 0, sizeof(read_buf)); | |
| 237 | + ASSERT_EQ(lseek(mem_fd1, 0, SEEK_SET), 0); | |
| 238 | + ASSERT_EQ(read(mem_fd1, read_buf, sizeof(read_buf)), strlen(test_str1)); | |
| 239 | + ASSERT_STREQ(read_buf, test_str1); | |
| 240 | + | |
| 241 | + memset(read_buf, 0, sizeof(read_buf)); | |
| 242 | + ASSERT_EQ(lseek(mem_fd2, 0, SEEK_SET), 0); | |
| 243 | + ASSERT_EQ(read(mem_fd2, read_buf, sizeof(read_buf)), strlen(test_str2)); | |
| 244 | + ASSERT_STREQ(read_buf, test_str2); | |
| 245 | + | |
| 246 | + ASSERT_EQ(close(mem_fd1), 0); | |
| 247 | + ASSERT_EQ(close(mem_fd2), 0); | |
| 248 | + ASSERT_EQ(close(session_fd), 0); | |
| 249 | +} | |
| 250 | + | |
| 251 | +/* | |
| 252 | + * Test Case: Preserve Complex Scenario | |
| 253 | + * | |
| 254 | + * Verifies a more complex scenario with multiple sessions and a mix of empty | |
| 255 | + * and non-empty memfds distributed across them. | |
| 256 | + */ | |
| 257 | +TEST_F(liveupdate_device, preserve_complex_scenario) | |
| 258 | +{ | |
| 259 | + const char *data1 = "data for session 1"; | |
| 260 | + const char *data2 = "data for session 2"; | |
| 261 | + char read_buf[64] = {}; | |
| 262 | + int session_fd1, session_fd2; | |
| 263 | + int mem_fd_data1, mem_fd_empty1, mem_fd_data2, mem_fd_empty2; | |
| 264 | + | |
| 265 | + self->fd1 = open(LIVEUPDATE_DEV, O_RDWR); | |
| 266 | + if (self->fd1 < 0 && errno == ENOENT) | |
| 267 | + SKIP(return, "%s does not exist", LIVEUPDATE_DEV); | |
| 268 | + ASSERT_GE(self->fd1, 0); | |
| 269 | + | |
| 270 | + session_fd1 = create_session(self->fd1, "complex-session-1"); | |
| 271 | + ASSERT_GE(session_fd1, 0); | |
| 272 | + session_fd2 = create_session(self->fd1, "complex-session-2"); | |
| 273 | + ASSERT_GE(session_fd2, 0); | |
| 274 | + | |
| 275 | + mem_fd_data1 = memfd_create("data1", 0); | |
| 276 | + ASSERT_GE(mem_fd_data1, 0); | |
| 277 | + ASSERT_EQ(write(mem_fd_data1, data1, strlen(data1)), strlen(data1)); | |
| 278 | + | |
| 279 | + mem_fd_empty1 = memfd_create("empty1", 0); | |
| 280 | + ASSERT_GE(mem_fd_empty1, 0); | |
| 281 | + | |
| 282 | + mem_fd_data2 = memfd_create("data2", 0); | |
| 283 | + ASSERT_GE(mem_fd_data2, 0); | |
| 284 | + ASSERT_EQ(write(mem_fd_data2, data2, strlen(data2)), strlen(data2)); | |
| 285 | + | |
| 286 | + mem_fd_empty2 = memfd_create("empty2", 0); | |
| 287 | + ASSERT_GE(mem_fd_empty2, 0); | |
| 288 | + | |
| 289 | + ASSERT_EQ(preserve_fd(session_fd1, mem_fd_data1, 0x1111), 0); | |
| 290 | + ASSERT_EQ(preserve_fd(session_fd1, mem_fd_empty1, 0x2222), 0); | |
| 291 | + ASSERT_EQ(preserve_fd(session_fd2, mem_fd_data2, 0x3333), 0); | |
| 292 | + ASSERT_EQ(preserve_fd(session_fd2, mem_fd_empty2, 0x4444), 0); | |
| 293 | + | |
| 294 | + ASSERT_EQ(lseek(mem_fd_data1, 0, SEEK_SET), 0); | |
| 295 | + ASSERT_EQ(read(mem_fd_data1, read_buf, sizeof(read_buf)), strlen(data1)); | |
| 296 | + ASSERT_STREQ(read_buf, data1); | |
| 297 | + | |
| 298 | + memset(read_buf, 0, sizeof(read_buf)); | |
| 299 | + ASSERT_EQ(lseek(mem_fd_data2, 0, SEEK_SET), 0); | |
| 300 | + ASSERT_EQ(read(mem_fd_data2, read_buf, sizeof(read_buf)), strlen(data2)); | |
| 301 | + ASSERT_STREQ(read_buf, data2); | |
| 302 | + | |
| 303 | + ASSERT_EQ(lseek(mem_fd_empty1, 0, SEEK_SET), 0); | |
| 304 | + ASSERT_EQ(read(mem_fd_empty1, read_buf, sizeof(read_buf)), 0); | |
| 305 | + | |
| 306 | + ASSERT_EQ(lseek(mem_fd_empty2, 0, SEEK_SET), 0); | |
| 307 | + ASSERT_EQ(read(mem_fd_empty2, read_buf, sizeof(read_buf)), 0); | |
| 308 | + | |
| 309 | + ASSERT_EQ(close(mem_fd_data1), 0); | |
| 310 | + ASSERT_EQ(close(mem_fd_empty1), 0); | |
| 311 | + ASSERT_EQ(close(mem_fd_data2), 0); | |
| 312 | + ASSERT_EQ(close(mem_fd_empty2), 0); | |
| 313 | + ASSERT_EQ(close(session_fd1), 0); | |
| 314 | + ASSERT_EQ(close(session_fd2), 0); | |
| 315 | +} | |
| 316 | + | |
| 317 | +/* | |
| 318 | + * Test Case: Preserve Unsupported File Descriptor | |
| 319 | + * | |
| 320 | + * Verifies that attempting to preserve a file descriptor that does not have | |
| 321 | + * a registered Live Update handler fails gracefully. | |
| 322 | + * Uses /dev/null as a representative of a file type (character device) | |
| 323 | + * that is not supported by the orchestrator. | |
| 324 | + */ | |
| 325 | +TEST_F(liveupdate_device, preserve_unsupported_fd) | |
| 326 | +{ | |
| 327 | + int session_fd, unsupported_fd; | |
| 328 | + int ret; | |
| 329 | + | |
| 330 | + self->fd1 = open(LIVEUPDATE_DEV, O_RDWR); | |
| 331 | + if (self->fd1 < 0 && errno == ENOENT) | |
| 332 | + SKIP(return, "%s does not exist", LIVEUPDATE_DEV); | |
| 333 | + ASSERT_GE(self->fd1, 0); | |
| 334 | + | |
| 335 | + session_fd = create_session(self->fd1, "unsupported-fd-test"); | |
| 336 | + ASSERT_GE(session_fd, 0); | |
| 337 | + | |
| 338 | + unsupported_fd = open("/dev/null", O_RDWR); | |
| 339 | + ASSERT_GE(unsupported_fd, 0); | |
| 340 | + | |
| 341 | + ret = preserve_fd(session_fd, unsupported_fd, 0xDEAD); | |
| 342 | + EXPECT_EQ(ret, -ENOENT); | |
| 343 | + | |
| 344 | + ASSERT_EQ(close(unsupported_fd), 0); | |
| 345 | + ASSERT_EQ(close(session_fd), 0); | |
| 346 | +} | |
| 347 | + | |
| 348 | +TEST_HARNESS_MAIN | |
| @@ -0,0 +1,89 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0-only | |
| 2 | + | |
| 3 | +/* | |
| 4 | + * Copyright (c) 2025, Google LLC. | |
| 5 | + * Pasha Tatashin <pasha.tatashin@soleen.com> | |
| 6 | + * | |
| 7 | + * A simple selftest to validate the end-to-end lifecycle of a LUO session | |
| 8 | + * across a single kexec reboot. | |
| 9 | + */ | |
| 10 | + | |
| 11 | +#include "luo_test_utils.h" | |
| 12 | + | |
| 13 | +#define TEST_SESSION_NAME "test-session" | |
| 14 | +#define TEST_MEMFD_TOKEN 0x1A | |
| 15 | +#define TEST_MEMFD_DATA "hello kexec world" | |
| 16 | + | |
| 17 | +/* Constants for the state-tracking mechanism, specific to this test file. */ | |
| 18 | +#define STATE_SESSION_NAME "kexec_simple_state" | |
| 19 | +#define STATE_MEMFD_TOKEN 999 | |
| 20 | + | |
| 21 | +/* Stage 1: Executed before the kexec reboot. */ | |
| 22 | +static void run_stage_1(int luo_fd) | |
| 23 | +{ | |
| 24 | + int session_fd; | |
| 25 | + | |
| 26 | + ksft_print_msg("[STAGE 1] Starting pre-kexec setup...\n"); | |
| 27 | + | |
| 28 | + ksft_print_msg("[STAGE 1] Creating state file for next stage (2)...\n"); | |
| 29 | + create_state_file(luo_fd, STATE_SESSION_NAME, STATE_MEMFD_TOKEN, 2); | |
| 30 | + | |
| 31 | + ksft_print_msg("[STAGE 1] Creating session '%s' and preserving memfd...\n", | |
| 32 | + TEST_SESSION_NAME); | |
| 33 | + session_fd = luo_create_session(luo_fd, TEST_SESSION_NAME); | |
| 34 | + if (session_fd < 0) | |
| 35 | + fail_exit("luo_create_session for '%s'", TEST_SESSION_NAME); | |
| 36 | + | |
| 37 | + if (create_and_preserve_memfd(session_fd, TEST_MEMFD_TOKEN, | |
| 38 | + TEST_MEMFD_DATA) < 0) { | |
| 39 | + fail_exit("create_and_preserve_memfd for token %#x", | |
| 40 | + TEST_MEMFD_TOKEN); | |
| 41 | + } | |
| 42 | + | |
| 43 | + close(luo_fd); | |
| 44 | + daemonize_and_wait(); | |
| 45 | +} | |
| 46 | + | |
| 47 | +/* Stage 2: Executed after the kexec reboot. */ | |
| 48 | +static void run_stage_2(int luo_fd, int state_session_fd) | |
| 49 | +{ | |
| 50 | + int session_fd, mfd, stage; | |
| 51 | + | |
| 52 | + ksft_print_msg("[STAGE 2] Starting post-kexec verification...\n"); | |
| 53 | + | |
| 54 | + restore_and_read_stage(state_session_fd, STATE_MEMFD_TOKEN, &stage); | |
| 55 | + if (stage != 2) | |
| 56 | + fail_exit("Expected stage 2, but state file contains %d", stage); | |
| 57 | + | |
| 58 | + ksft_print_msg("[STAGE 2] Retrieving session '%s'...\n", TEST_SESSION_NAME); | |
| 59 | + session_fd = luo_retrieve_session(luo_fd, TEST_SESSION_NAME); | |
| 60 | + if (session_fd < 0) | |
| 61 | + fail_exit("luo_retrieve_session for '%s'", TEST_SESSION_NAME); | |
| 62 | + | |
| 63 | + ksft_print_msg("[STAGE 2] Restoring and verifying memfd (token %#x)...\n", | |
| 64 | + TEST_MEMFD_TOKEN); | |
| 65 | + mfd = restore_and_verify_memfd(session_fd, TEST_MEMFD_TOKEN, | |
| 66 | + TEST_MEMFD_DATA); | |
| 67 | + if (mfd < 0) | |
| 68 | + fail_exit("restore_and_verify_memfd for token %#x", TEST_MEMFD_TOKEN); | |
| 69 | + close(mfd); | |
| 70 | + | |
| 71 | + ksft_print_msg("[STAGE 2] Test data verified successfully.\n"); | |
| 72 | + ksft_print_msg("[STAGE 2] Finalizing test session...\n"); | |
| 73 | + if (luo_session_finish(session_fd) < 0) | |
| 74 | + fail_exit("luo_session_finish for test session"); | |
| 75 | + close(session_fd); | |
| 76 | + | |
| 77 | + ksft_print_msg("[STAGE 2] Finalizing state session...\n"); | |
| 78 | + if (luo_session_finish(state_session_fd) < 0) | |
| 79 | + fail_exit("luo_session_finish for state session"); | |
| 80 | + close(state_session_fd); | |
| 81 | + | |
| 82 | + ksft_print_msg("\n--- SIMPLE KEXEC TEST PASSED ---\n"); | |
| 83 | +} | |
| 84 | + | |
| 85 | +int main(int argc, char *argv[]) | |
| 86 | +{ | |
| 87 | + return luo_test(argc, argv, STATE_SESSION_NAME, | |
| 88 | + run_stage_1, run_stage_2); | |
| 89 | +} | |
| @@ -0,0 +1,162 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0-only | |
| 2 | + | |
| 3 | +/* | |
| 4 | + * Copyright (c) 2025, Google LLC. | |
| 5 | + * Pasha Tatashin <pasha.tatashin@soleen.com> | |
| 6 | + * | |
| 7 | + * A selftest to validate the end-to-end lifecycle of multiple LUO sessions | |
| 8 | + * across a kexec reboot, including empty sessions and sessions with multiple | |
| 9 | + * files. | |
| 10 | + */ | |
| 11 | + | |
| 12 | +#include "luo_test_utils.h" | |
| 13 | + | |
| 14 | +#define SESSION_EMPTY_1 "multi-test-empty-1" | |
| 15 | +#define SESSION_EMPTY_2 "multi-test-empty-2" | |
| 16 | +#define SESSION_FILES_1 "multi-test-files-1" | |
| 17 | +#define SESSION_FILES_2 "multi-test-files-2" | |
| 18 | + | |
| 19 | +#define MFD1_TOKEN 0x1001 | |
| 20 | +#define MFD2_TOKEN 0x2002 | |
| 21 | +#define MFD3_TOKEN 0x3003 | |
| 22 | + | |
| 23 | +#define MFD1_DATA "Data for session files 1" | |
| 24 | +#define MFD2_DATA "First file for session files 2" | |
| 25 | +#define MFD3_DATA "Second file for session files 2" | |
| 26 | + | |
| 27 | +#define STATE_SESSION_NAME "kexec_multi_state" | |
| 28 | +#define STATE_MEMFD_TOKEN 998 | |
| 29 | + | |
| 30 | +/* Stage 1: Executed before the kexec reboot. */ | |
| 31 | +static void run_stage_1(int luo_fd) | |
| 32 | +{ | |
| 33 | + int s_empty1_fd, s_empty2_fd, s_files1_fd, s_files2_fd; | |
| 34 | + | |
| 35 | + ksft_print_msg("[STAGE 1] Starting pre-kexec setup for multi-session test...\n"); | |
| 36 | + | |
| 37 | + ksft_print_msg("[STAGE 1] Creating state file for next stage (2)...\n"); | |
| 38 | + create_state_file(luo_fd, STATE_SESSION_NAME, STATE_MEMFD_TOKEN, 2); | |
| 39 | + | |
| 40 | + ksft_print_msg("[STAGE 1] Creating empty sessions '%s' and '%s'...\n", | |
| 41 | + SESSION_EMPTY_1, SESSION_EMPTY_2); | |
| 42 | + s_empty1_fd = luo_create_session(luo_fd, SESSION_EMPTY_1); | |
| 43 | + if (s_empty1_fd < 0) | |
| 44 | + fail_exit("luo_create_session for '%s'", SESSION_EMPTY_1); | |
| 45 | + | |
| 46 | + s_empty2_fd = luo_create_session(luo_fd, SESSION_EMPTY_2); | |
| 47 | + if (s_empty2_fd < 0) | |
| 48 | + fail_exit("luo_create_session for '%s'", SESSION_EMPTY_2); | |
| 49 | + | |
| 50 | + ksft_print_msg("[STAGE 1] Creating session '%s' with one memfd...\n", | |
| 51 | + SESSION_FILES_1); | |
| 52 | + | |
| 53 | + s_files1_fd = luo_create_session(luo_fd, SESSION_FILES_1); | |
| 54 | + if (s_files1_fd < 0) | |
| 55 | + fail_exit("luo_create_session for '%s'", SESSION_FILES_1); | |
| 56 | + if (create_and_preserve_memfd(s_files1_fd, MFD1_TOKEN, MFD1_DATA) < 0) { | |
| 57 | + fail_exit("create_and_preserve_memfd for token %#x", | |
| 58 | + MFD1_TOKEN); | |
| 59 | + } | |
| 60 | + | |
| 61 | + ksft_print_msg("[STAGE 1] Creating session '%s' with two memfds...\n", | |
| 62 | + SESSION_FILES_2); | |
| 63 | + | |
| 64 | + s_files2_fd = luo_create_session(luo_fd, SESSION_FILES_2); | |
| 65 | + if (s_files2_fd < 0) | |
| 66 | + fail_exit("luo_create_session for '%s'", SESSION_FILES_2); | |
| 67 | + if (create_and_preserve_memfd(s_files2_fd, MFD2_TOKEN, MFD2_DATA) < 0) { | |
| 68 | + fail_exit("create_and_preserve_memfd for token %#x", | |
| 69 | + MFD2_TOKEN); | |
| 70 | + } | |
| 71 | + if (create_and_preserve_memfd(s_files2_fd, MFD3_TOKEN, MFD3_DATA) < 0) { | |
| 72 | + fail_exit("create_and_preserve_memfd for token %#x", | |
| 73 | + MFD3_TOKEN); | |
| 74 | + } | |
| 75 | + | |
| 76 | + close(luo_fd); | |
| 77 | + daemonize_and_wait(); | |
| 78 | +} | |
| 79 | + | |
| 80 | +/* Stage 2: Executed after the kexec reboot. */ | |
| 81 | +static void run_stage_2(int luo_fd, int state_session_fd) | |
| 82 | +{ | |
| 83 | + int s_empty1_fd, s_empty2_fd, s_files1_fd, s_files2_fd; | |
| 84 | + int mfd1, mfd2, mfd3, stage; | |
| 85 | + | |
| 86 | + ksft_print_msg("[STAGE 2] Starting post-kexec verification...\n"); | |
| 87 | + | |
| 88 | + restore_and_read_stage(state_session_fd, STATE_MEMFD_TOKEN, &stage); | |
| 89 | + if (stage != 2) { | |
| 90 | + fail_exit("Expected stage 2, but state file contains %d", | |
| 91 | + stage); | |
| 92 | + } | |
| 93 | + | |
| 94 | + ksft_print_msg("[STAGE 2] Retrieving all sessions...\n"); | |
| 95 | + s_empty1_fd = luo_retrieve_session(luo_fd, SESSION_EMPTY_1); | |
| 96 | + if (s_empty1_fd < 0) | |
| 97 | + fail_exit("luo_retrieve_session for '%s'", SESSION_EMPTY_1); | |
| 98 | + | |
| 99 | + s_empty2_fd = luo_retrieve_session(luo_fd, SESSION_EMPTY_2); | |
| 100 | + if (s_empty2_fd < 0) | |
| 101 | + fail_exit("luo_retrieve_session for '%s'", SESSION_EMPTY_2); | |
| 102 | + | |
| 103 | + s_files1_fd = luo_retrieve_session(luo_fd, SESSION_FILES_1); | |
| 104 | + if (s_files1_fd < 0) | |
| 105 | + fail_exit("luo_retrieve_session for '%s'", SESSION_FILES_1); | |
| 106 | + | |
| 107 | + s_files2_fd = luo_retrieve_session(luo_fd, SESSION_FILES_2); | |
| 108 | + if (s_files2_fd < 0) | |
| 109 | + fail_exit("luo_retrieve_session for '%s'", SESSION_FILES_2); | |
| 110 | + | |
| 111 | + ksft_print_msg("[STAGE 2] Verifying contents of session '%s'...\n", | |
| 112 | + SESSION_FILES_1); | |
| 113 | + mfd1 = restore_and_verify_memfd(s_files1_fd, MFD1_TOKEN, MFD1_DATA); | |
| 114 | + if (mfd1 < 0) | |
| 115 | + fail_exit("restore_and_verify_memfd for token %#x", MFD1_TOKEN); | |
| 116 | + close(mfd1); | |
| 117 | + | |
| 118 | + ksft_print_msg("[STAGE 2] Verifying contents of session '%s'...\n", | |
| 119 | + SESSION_FILES_2); | |
| 120 | + | |
| 121 | + mfd2 = restore_and_verify_memfd(s_files2_fd, MFD2_TOKEN, MFD2_DATA); | |
| 122 | + if (mfd2 < 0) | |
| 123 | + fail_exit("restore_and_verify_memfd for token %#x", MFD2_TOKEN); | |
| 124 | + close(mfd2); | |
| 125 | + | |
| 126 | + mfd3 = restore_and_verify_memfd(s_files2_fd, MFD3_TOKEN, MFD3_DATA); | |
| 127 | + if (mfd3 < 0) | |
| 128 | + fail_exit("restore_and_verify_memfd for token %#x", MFD3_TOKEN); | |
| 129 | + close(mfd3); | |
| 130 | + | |
| 131 | + ksft_print_msg("[STAGE 2] Test data verified successfully.\n"); | |
| 132 | + | |
| 133 | + ksft_print_msg("[STAGE 2] Finalizing all test sessions...\n"); | |
| 134 | + if (luo_session_finish(s_empty1_fd) < 0) | |
| 135 | + fail_exit("luo_session_finish for '%s'", SESSION_EMPTY_1); | |
| 136 | + close(s_empty1_fd); | |
| 137 | + | |
| 138 | + if (luo_session_finish(s_empty2_fd) < 0) | |
| 139 | + fail_exit("luo_session_finish for '%s'", SESSION_EMPTY_2); | |
| 140 | + close(s_empty2_fd); | |
| 141 | + | |
| 142 | + if (luo_session_finish(s_files1_fd) < 0) | |
| 143 | + fail_exit("luo_session_finish for '%s'", SESSION_FILES_1); | |
| 144 | + close(s_files1_fd); | |
| 145 | + | |
| 146 | + if (luo_session_finish(s_files2_fd) < 0) | |
| 147 | + fail_exit("luo_session_finish for '%s'", SESSION_FILES_2); | |
| 148 | + close(s_files2_fd); | |
| 149 | + | |
| 150 | + ksft_print_msg("[STAGE 2] Finalizing state session...\n"); | |
| 151 | + if (luo_session_finish(state_session_fd) < 0) | |
| 152 | + fail_exit("luo_session_finish for state session"); | |
| 153 | + close(state_session_fd); | |
| 154 | + | |
| 155 | + ksft_print_msg("\n--- MULTI-SESSION KEXEC TEST PASSED ---\n"); | |
| 156 | +} | |
| 157 | + | |
| 158 | +int main(int argc, char *argv[]) | |
| 159 | +{ | |
| 160 | + return luo_test(argc, argv, STATE_SESSION_NAME, | |
| 161 | + run_stage_1, run_stage_2); | |
| 162 | +} | |
| @@ -0,0 +1,266 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0-only | |
| 2 | + | |
| 3 | +/* | |
| 4 | + * Copyright (c) 2025, Google LLC. | |
| 5 | + * Pasha Tatashin <pasha.tatashin@soleen.com> | |
| 6 | + */ | |
| 7 | + | |
| 8 | +#define _GNU_SOURCE | |
| 9 | + | |
| 10 | +#include <stdio.h> | |
| 11 | +#include <stdlib.h> | |
| 12 | +#include <string.h> | |
| 13 | +#include <getopt.h> | |
| 14 | +#include <fcntl.h> | |
| 15 | +#include <unistd.h> | |
| 16 | +#include <sys/ioctl.h> | |
| 17 | +#include <sys/syscall.h> | |
| 18 | +#include <sys/mman.h> | |
| 19 | +#include <sys/types.h> | |
| 20 | +#include <sys/stat.h> | |
| 21 | +#include <errno.h> | |
| 22 | +#include <stdarg.h> | |
| 23 | + | |
| 24 | +#include "luo_test_utils.h" | |
| 25 | + | |
| 26 | +int luo_open_device(void) | |
| 27 | +{ | |
| 28 | + return open(LUO_DEVICE, O_RDWR); | |
| 29 | +} | |
| 30 | + | |
| 31 | +int luo_create_session(int luo_fd, const char *name) | |
| 32 | +{ | |
| 33 | + struct liveupdate_ioctl_create_session arg = { .size = sizeof(arg) }; | |
| 34 | + | |
| 35 | + snprintf((char *)arg.name, LIVEUPDATE_SESSION_NAME_LENGTH, "%.*s", | |
| 36 | + LIVEUPDATE_SESSION_NAME_LENGTH - 1, name); | |
| 37 | + | |
| 38 | + if (ioctl(luo_fd, LIVEUPDATE_IOCTL_CREATE_SESSION, &arg) < 0) | |
| 39 | + return -errno; | |
| 40 | + | |
| 41 | + return arg.fd; | |
| 42 | +} | |
| 43 | + | |
| 44 | +int luo_retrieve_session(int luo_fd, const char *name) | |
| 45 | +{ | |
| 46 | + struct liveupdate_ioctl_retrieve_session arg = { .size = sizeof(arg) }; | |
| 47 | + | |
| 48 | + snprintf((char *)arg.name, LIVEUPDATE_SESSION_NAME_LENGTH, "%.*s", | |
| 49 | + LIVEUPDATE_SESSION_NAME_LENGTH - 1, name); | |
| 50 | + | |
| 51 | + if (ioctl(luo_fd, LIVEUPDATE_IOCTL_RETRIEVE_SESSION, &arg) < 0) | |
| 52 | + return -errno; | |
| 53 | + | |
| 54 | + return arg.fd; | |
| 55 | +} | |
| 56 | + | |
| 57 | +int create_and_preserve_memfd(int session_fd, int token, const char *data) | |
| 58 | +{ | |
| 59 | + struct liveupdate_session_preserve_fd arg = { .size = sizeof(arg) }; | |
| 60 | + long page_size = sysconf(_SC_PAGE_SIZE); | |
| 61 | + void *map = MAP_FAILED; | |
| 62 | + int mfd = -1, ret = -1; | |
| 63 | + | |
| 64 | + mfd = memfd_create("test_mfd", 0); | |
| 65 | + if (mfd < 0) | |
| 66 | + return -errno; | |
| 67 | + | |
| 68 | + if (ftruncate(mfd, page_size) != 0) | |
| 69 | + goto out; | |
| 70 | + | |
| 71 | + map = mmap(NULL, page_size, PROT_WRITE, MAP_SHARED, mfd, 0); | |
| 72 | + if (map == MAP_FAILED) | |
| 73 | + goto out; | |
| 74 | + | |
| 75 | + snprintf(map, page_size, "%s", data); | |
| 76 | + munmap(map, page_size); | |
| 77 | + | |
| 78 | + arg.fd = mfd; | |
| 79 | + arg.token = token; | |
| 80 | + if (ioctl(session_fd, LIVEUPDATE_SESSION_PRESERVE_FD, &arg) < 0) | |
| 81 | + goto out; | |
| 82 | + | |
| 83 | + ret = 0; | |
| 84 | +out: | |
| 85 | + if (ret != 0 && errno != 0) | |
| 86 | + ret = -errno; | |
| 87 | + if (mfd >= 0) | |
| 88 | + close(mfd); | |
| 89 | + return ret; | |
| 90 | +} | |
| 91 | + | |
| 92 | +int restore_and_verify_memfd(int session_fd, int token, | |
| 93 | + const char *expected_data) | |
| 94 | +{ | |
| 95 | + struct liveupdate_session_retrieve_fd arg = { .size = sizeof(arg) }; | |
| 96 | + long page_size = sysconf(_SC_PAGE_SIZE); | |
| 97 | + void *map = MAP_FAILED; | |
| 98 | + int mfd = -1, ret = -1; | |
| 99 | + | |
| 100 | + arg.token = token; | |
| 101 | + if (ioctl(session_fd, LIVEUPDATE_SESSION_RETRIEVE_FD, &arg) < 0) | |
| 102 | + return -errno; | |
| 103 | + mfd = arg.fd; | |
| 104 | + | |
| 105 | + map = mmap(NULL, page_size, PROT_READ, MAP_SHARED, mfd, 0); | |
| 106 | + if (map == MAP_FAILED) | |
| 107 | + goto out; | |
| 108 | + | |
| 109 | + if (expected_data && strcmp(expected_data, map) != 0) { | |
| 110 | + ksft_print_msg("Data mismatch! Expected '%s', Got '%s'\n", | |
| 111 | + expected_data, (char *)map); | |
| 112 | + ret = -EINVAL; | |
| 113 | + goto out_munmap; | |
| 114 | + } | |
| 115 | + | |
| 116 | + ret = mfd; | |
| 117 | +out_munmap: | |
| 118 | + munmap(map, page_size); | |
| 119 | +out: | |
| 120 | + if (ret < 0 && errno != 0) | |
| 121 | + ret = -errno; | |
| 122 | + if (ret < 0 && mfd >= 0) | |
| 123 | + close(mfd); | |
| 124 | + return ret; | |
| 125 | +} | |
| 126 | + | |
| 127 | +int luo_session_finish(int session_fd) | |
| 128 | +{ | |
| 129 | + struct liveupdate_session_finish arg = { .size = sizeof(arg) }; | |
| 130 | + | |
| 131 | + if (ioctl(session_fd, LIVEUPDATE_SESSION_FINISH, &arg) < 0) | |
| 132 | + return -errno; | |
| 133 | + | |
| 134 | + return 0; | |
| 135 | +} | |
| 136 | + | |
| 137 | +void create_state_file(int luo_fd, const char *session_name, int token, | |
| 138 | + int next_stage) | |
| 139 | +{ | |
| 140 | + char buf[32]; | |
| 141 | + int state_session_fd; | |
| 142 | + | |
| 143 | + state_session_fd = luo_create_session(luo_fd, session_name); | |
| 144 | + if (state_session_fd < 0) | |
| 145 | + fail_exit("luo_create_session for state tracking"); | |
| 146 | + | |
| 147 | + snprintf(buf, sizeof(buf), "%d", next_stage); | |
| 148 | + if (create_and_preserve_memfd(state_session_fd, token, buf) < 0) | |
| 149 | + fail_exit("create_and_preserve_memfd for state tracking"); | |
| 150 | + | |
| 151 | + /* | |
| 152 | + * DO NOT close session FD, otherwise it is going to be unpreserved | |
| 153 | + */ | |
| 154 | +} | |
| 155 | + | |
| 156 | +void restore_and_read_stage(int state_session_fd, int token, int *stage) | |
| 157 | +{ | |
| 158 | + char buf[32] = {0}; | |
| 159 | + int mfd; | |
| 160 | + | |
| 161 | + mfd = restore_and_verify_memfd(state_session_fd, token, NULL); | |
| 162 | + if (mfd < 0) | |
| 163 | + fail_exit("failed to restore state memfd"); | |
| 164 | + | |
| 165 | + if (read(mfd, buf, sizeof(buf) - 1) < 0) | |
| 166 | + fail_exit("failed to read state mfd"); | |
| 167 | + | |
| 168 | + *stage = atoi(buf); | |
| 169 | + | |
| 170 | + close(mfd); | |
| 171 | +} | |
| 172 | + | |
| 173 | +void daemonize_and_wait(void) | |
| 174 | +{ | |
| 175 | + pid_t pid; | |
| 176 | + | |
| 177 | + ksft_print_msg("[STAGE 1] Forking persistent child to hold sessions...\n"); | |
| 178 | + | |
| 179 | + pid = fork(); | |
| 180 | + if (pid < 0) | |
| 181 | + fail_exit("fork failed"); | |
| 182 | + | |
| 183 | + if (pid > 0) { | |
| 184 | + ksft_print_msg("[STAGE 1] Child PID: %d. Resources are pinned.\n", pid); | |
| 185 | + ksft_print_msg("[STAGE 1] You may now perform kexec reboot.\n"); | |
| 186 | + exit(EXIT_SUCCESS); | |
| 187 | + } | |
| 188 | + | |
| 189 | + /* Detach from terminal so closing the window doesn't kill us */ | |
| 190 | + if (setsid() < 0) | |
| 191 | + fail_exit("setsid failed"); | |
| 192 | + | |
| 193 | + close(STDIN_FILENO); | |
| 194 | + close(STDOUT_FILENO); | |
| 195 | + close(STDERR_FILENO); | |
| 196 | + | |
| 197 | + /* Change dir to root to avoid locking filesystems */ | |
| 198 | + if (chdir("/") < 0) | |
| 199 | + exit(EXIT_FAILURE); | |
| 200 | + | |
| 201 | + while (1) | |
| 202 | + sleep(60); | |
| 203 | +} | |
| 204 | + | |
| 205 | +static int parse_stage_args(int argc, char *argv[]) | |
| 206 | +{ | |
| 207 | + static struct option long_options[] = { | |
| 208 | + {"stage", required_argument, 0, 's'}, | |
| 209 | + {0, 0, 0, 0} | |
| 210 | + }; | |
| 211 | + int option_index = 0; | |
| 212 | + int stage = 1; | |
| 213 | + int opt; | |
| 214 | + | |
| 215 | + optind = 1; | |
| 216 | + while ((opt = getopt_long(argc, argv, "s:", long_options, &option_index)) != -1) { | |
| 217 | + switch (opt) { | |
| 218 | + case 's': | |
| 219 | + stage = atoi(optarg); | |
| 220 | + if (stage != 1 && stage != 2) | |
| 221 | + fail_exit("Invalid stage argument"); | |
| 222 | + break; | |
| 223 | + default: | |
| 224 | + fail_exit("Unknown argument"); | |
| 225 | + } | |
| 226 | + } | |
| 227 | + return stage; | |
| 228 | +} | |
| 229 | + | |
| 230 | +int luo_test(int argc, char *argv[], | |
| 231 | + const char *state_session_name, | |
| 232 | + luo_test_stage1_fn stage1, | |
| 233 | + luo_test_stage2_fn stage2) | |
| 234 | +{ | |
| 235 | + int target_stage = parse_stage_args(argc, argv); | |
| 236 | + int luo_fd = luo_open_device(); | |
| 237 | + int state_session_fd; | |
| 238 | + int detected_stage; | |
| 239 | + | |
| 240 | + if (luo_fd < 0) { | |
| 241 | + ksft_exit_skip("Failed to open %s. Is the luo module loaded?\n", | |
| 242 | + LUO_DEVICE); | |
| 243 | + } | |
| 244 | + | |
| 245 | + state_session_fd = luo_retrieve_session(luo_fd, state_session_name); | |
| 246 | + if (state_session_fd == -ENOENT) | |
| 247 | + detected_stage = 1; | |
| 248 | + else if (state_session_fd >= 0) | |
| 249 | + detected_stage = 2; | |
| 250 | + else | |
| 251 | + fail_exit("Failed to check for state session"); | |
| 252 | + | |
| 253 | + if (target_stage != detected_stage) { | |
| 254 | + ksft_exit_fail_msg("Stage mismatch Requested --stage %d, but system is in stage %d.\n" | |
| 255 | + "(State session %s: %s)\n", | |
| 256 | + target_stage, detected_stage, state_session_name, | |
| 257 | + (detected_stage == 2) ? "EXISTS" : "MISSING"); | |
| 258 | + } | |
| 259 | + | |
| 260 | + if (target_stage == 1) | |
| 261 | + stage1(luo_fd); | |
| 262 | + else | |
| 263 | + stage2(luo_fd, state_session_fd); | |
| 264 | + | |
| 265 | + return 0; | |
| 266 | +} | |
| @@ -0,0 +1,44 @@ | ||
| 1 | +/* SPDX-License-Identifier: GPL-2.0 */ | |
| 2 | + | |
| 3 | +/* | |
| 4 | + * Copyright (c) 2025, Google LLC. | |
| 5 | + * Pasha Tatashin <pasha.tatashin@soleen.com> | |
| 6 | + * | |
| 7 | + * Utility functions for LUO kselftests. | |
| 8 | + */ | |
| 9 | + | |
| 10 | +#ifndef LUO_TEST_UTILS_H | |
| 11 | +#define LUO_TEST_UTILS_H | |
| 12 | + | |
| 13 | +#include <errno.h> | |
| 14 | +#include <string.h> | |
| 15 | +#include <linux/liveupdate.h> | |
| 16 | +#include "../kselftest.h" | |
| 17 | + | |
| 18 | +#define LUO_DEVICE "/dev/liveupdate" | |
| 19 | + | |
| 20 | +#define fail_exit(fmt, ...) \ | |
| 21 | + ksft_exit_fail_msg("[%s:%d] " fmt " (errno: %s)\n", \ | |
| 22 | + __func__, __LINE__, ##__VA_ARGS__, strerror(errno)) | |
| 23 | + | |
| 24 | +int luo_open_device(void); | |
| 25 | +int luo_create_session(int luo_fd, const char *name); | |
| 26 | +int luo_retrieve_session(int luo_fd, const char *name); | |
| 27 | +int luo_session_finish(int session_fd); | |
| 28 | + | |
| 29 | +int create_and_preserve_memfd(int session_fd, int token, const char *data); | |
| 30 | +int restore_and_verify_memfd(int session_fd, int token, const char *expected_data); | |
| 31 | + | |
| 32 | +void create_state_file(int luo_fd, const char *session_name, int token, | |
| 33 | + int next_stage); | |
| 34 | +void restore_and_read_stage(int state_session_fd, int token, int *stage); | |
| 35 | + | |
| 36 | +void daemonize_and_wait(void); | |
| 37 | + | |
| 38 | +typedef void (*luo_test_stage1_fn)(int luo_fd); | |
| 39 | +typedef void (*luo_test_stage2_fn)(int luo_fd, int state_session_fd); | |
| 40 | + | |
| 41 | +int luo_test(int argc, char *argv[], const char *state_session_name, | |
| 42 | + luo_test_stage1_fn stage1, luo_test_stage2_fn stage2); | |
| 43 | + | |
| 44 | +#endif /* LUO_TEST_UTILS_H */ | |
| @@ -13,7 +13,7 @@ · 12 unmodified lines | ||
| 13 | 13 | #include <stdio.h> |
| 14 | 14 | #include <unistd.h> |
| 15 | 15 | #include <sys/types.h> |
| 16 | −#include "../kselftest_harness.h" | |
| 16 | +#include "kselftest_harness.h" | |
| 17 | 17 | #include "common.h" |
| 18 | 18 | |
| 19 | 19 | static struct lsm_ctx *next_ctx(struct lsm_ctx *ctxp) |
| @@ -12,7 +12,7 @@ · 11 unmodified lines | ||
| 12 | 12 | #include <stdio.h> |
| 13 | 13 | #include <unistd.h> |
| 14 | 14 | #include <sys/types.h> |
| 15 | −#include "../kselftest_harness.h" | |
| 15 | +#include "kselftest_harness.h" | |
| 16 | 16 | #include "common.h" |
| 17 | 17 | |
| 18 | 18 | TEST(size_null_lsm_list_modules) |
| @@ -12,7 +12,7 @@ · 11 unmodified lines | ||
| 12 | 12 | #include <stdio.h> |
| 13 | 13 | #include <unistd.h> |
| 14 | 14 | #include <sys/types.h> |
| 15 | −#include "../kselftest_harness.h" | |
| 15 | +#include "kselftest_harness.h" | |
| 16 | 16 | #include "common.h" |
| 17 | 17 | |
| 18 | 18 | TEST(ctx_null_lsm_set_self_attr) |
| @@ -34,7 +34,7 @@ · 33 unmodified lines | ||
| 34 | 34 | #include <sys/stat.h> |
| 35 | 35 | #include <linux/media.h> |
| 36 | 36 | |
| 37 | −#include "../kselftest.h" | |
| 37 | +#include "kselftest.h" | |
| 38 | 38 | |
| 39 | 39 | int main(int argc, char **argv) |
| 40 | 40 | { |
| @@ -39,7 +39,7 @@ · 38 unmodified lines | ||
| 39 | 39 | #include <time.h> |
| 40 | 40 | #include <linux/media.h> |
| 41 | 41 | |
| 42 | −#include "../kselftest.h" | |
| 42 | +#include "kselftest.h" | |
| 43 | 43 | |
| 44 | 44 | int main(int argc, char **argv) |
| 45 | 45 | { |
| @@ -7,7 +7,7 @@ · 6 unmodified lines | ||
| 7 | 7 | #include <string.h> |
| 8 | 8 | #include <pthread.h> |
| 9 | 9 | |
| 10 | −#include "../kselftest.h" | |
| 10 | +#include "kselftest.h" | |
| 11 | 11 | |
| 12 | 12 | static int registrations; |
| 13 | 13 | |
| @@ -15,8 +15,8 @@ · 14 unmodified lines | ||
| 15 | 15 | #include <string.h> |
| 16 | 16 | #include <fcntl.h> |
| 17 | 17 | |
| 18 | −#include "../kselftest.h" | |
| 19 | −#include "../kselftest_harness.h" | |
| 18 | +#include "kselftest.h" | |
| 19 | +#include "kselftest_harness.h" | |
| 20 | 20 | |
| 21 | 21 | /* Default test file size: 4MB */ |
| 22 | 22 | #define MB (1UL << 20) |
| @@ -16,7 +16,7 @@ · 15 unmodified lines | ||
| 16 | 16 | #include <unistd.h> |
| 17 | 17 | #include <string.h> |
| 18 | 18 | |
| 19 | −#include "../kselftest.h" | |
| 19 | +#include "kselftest.h" | |
| 20 | 20 | |
| 21 | 21 | #define MAP_SIZE_MB 100 |
| 22 | 22 | #define MAP_SIZE (MAP_SIZE_MB * 1024 * 1024) |
| @@ -27,7 +27,7 @@ · 26 unmodified lines | ||
| 27 | 27 | #endif /* LOCAL_CONFIG_HAVE_LIBURING */ |
| 28 | 28 | |
| 29 | 29 | #include "../../../../mm/gup_test.h" |
| 30 | −#include "../kselftest.h" | |
| 30 | +#include "kselftest.h" | |
| 31 | 31 | #include "vm_util.h" |
| 32 | 32 | #include "thp_settings.h" |
| 33 | 33 | |
| @@ -13,7 +13,7 @@ · 12 unmodified lines | ||
| 13 | 13 | #include <sys/mman.h> |
| 14 | 14 | #include <linux/mman.h> |
| 15 | 15 | |
| 16 | −#include "../kselftest.h" | |
| 16 | +#include "kselftest.h" | |
| 17 | 17 | |
| 18 | 18 | int main(int argc, char *argv[]) |
| 19 | 19 | { |
| @@ -1,7 +1,7 @@ | ||
| 1 | 1 | // SPDX-License-Identifier: GPL-2.0-or-later |
| 2 | 2 | |
| 3 | 3 | #define _GNU_SOURCE |
| 4 | −#include "../kselftest_harness.h" | |
| 4 | +#include "kselftest_harness.h" | |
| 5 | 5 | #include <asm-generic/mman.h> /* Force the import of the tools version. */ |
| 6 | 6 | #include <assert.h> |
| 7 | 7 | #include <errno.h> |
| @@ -27,7 +27,7 @@ · 26 unmodified lines | ||
| 27 | 27 | #endif /* LOCAL_CONFIG_HAVE_LIBURING */ |
| 28 | 28 | |
| 29 | 29 | #include "../../../../mm/gup_test.h" |
| 30 | −#include "../kselftest.h" | |
| 30 | +#include "kselftest.h" | |
| 31 | 31 | #include "vm_util.h" |
| 32 | 32 | |
| 33 | 33 | static size_t pagesize; |
| @@ -12,7 +12,7 @@ · 11 unmodified lines | ||
| 12 | 12 | #include <pthread.h> |
| 13 | 13 | #include <assert.h> |
| 14 | 14 | #include <mm/gup_test.h> |
| 15 | −#include "../kselftest.h" | |
| 15 | +#include "kselftest.h" | |
| 16 | 16 | #include "vm_util.h" |
| 17 | 17 | |
| 18 | 18 | #define MB (1UL << 20) |
| @@ -10,7 +10,7 @@ · 9 unmodified lines | ||
| 10 | 10 | * bugs. |
| 11 | 11 | */ |
| 12 | 12 | |
| 13 | −#include "../kselftest_harness.h" | |
| 13 | +#include "kselftest_harness.h" | |
| 14 | 14 | |
| 15 | 15 | #include <errno.h> |
| 16 | 16 | #include <fcntl.h> |
| @@ -15,7 +15,7 @@ · 14 unmodified lines | ||
| 15 | 15 | #include <unistd.h> |
| 16 | 16 | #include <sys/mman.h> |
| 17 | 17 | #include <fcntl.h> |
| 18 | −#include "../kselftest.h" | |
| 18 | +#include "kselftest.h" | |
| 19 | 19 | |
| 20 | 20 | #define LENGTH (256UL*1024*1024) |
| 21 | 21 | #define PROTECTION (PROT_READ | PROT_WRITE) |
| @@ -24,7 +24,7 @@ · 23 unmodified lines | ||
| 24 | 24 | #include <sys/ioctl.h> |
| 25 | 25 | #include <string.h> |
| 26 | 26 | #include <stdbool.h> |
| 27 | −#include "../kselftest.h" | |
| 27 | +#include "kselftest.h" | |
| 28 | 28 | #include "vm_util.h" |
| 29 | 29 | |
| 30 | 30 | #define DEFAULT_LENGTH_MB 10UL |
| @@ -19,7 +19,7 @@ · 18 unmodified lines | ||
| 19 | 19 | #include <sys/mman.h> |
| 20 | 20 | #include <fcntl.h> |
| 21 | 21 | #include "vm_util.h" |
| 22 | −#include "../kselftest.h" | |
| 22 | +#include "kselftest.h" | |
| 23 | 23 | |
| 24 | 24 | #define MIN_FREE_PAGES 20 |
| 25 | 25 | #define NR_HUGE_PAGES 10 /* common number of pages to map/allocate */ |
| @@ -11,7 +11,7 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <errno.h> |
| 12 | 12 | #include <stdbool.h> |
| 13 | 13 | |
| 14 | −#include "../kselftest.h" | |
| 14 | +#include "kselftest.h" | |
| 15 | 15 | |
| 16 | 16 | #define PREFIX " ... " |
| 17 | 17 | #define ERROR_PREFIX " !!! " |
| @@ -24,7 +24,7 @@ · 23 unmodified lines | ||
| 24 | 24 | #include <sys/statfs.h> |
| 25 | 25 | #include <sys/types.h> |
| 26 | 26 | |
| 27 | −#include "../kselftest.h" | |
| 27 | +#include "kselftest.h" | |
| 28 | 28 | |
| 29 | 29 | #ifndef MADV_SOFT_OFFLINE |
| 30 | 30 | #define MADV_SOFT_OFFLINE 101 |
| @@ -18,7 +18,7 @@ · 17 unmodified lines | ||
| 18 | 18 | #include <string.h> |
| 19 | 19 | #include <sys/mman.h> |
| 20 | 20 | #include "vm_util.h" |
| 21 | −#include "../kselftest.h" | |
| 21 | +#include "kselftest.h" | |
| 22 | 22 | |
| 23 | 23 | void run_dio_using_hugetlb(unsigned int start_off, unsigned int end_off) |
| 24 | 24 | { |
| @@ -9,7 +9,7 @@ · 8 unmodified lines | ||
| 9 | 9 | #include <signal.h> |
| 10 | 10 | |
| 11 | 11 | #include "vm_util.h" |
| 12 | −#include "../kselftest.h" | |
| 12 | +#include "kselftest.h" | |
| 13 | 13 | |
| 14 | 14 | #define INLOOP_ITER 100 |
| 15 | 15 | |
| @@ -25,7 +25,7 @@ · 24 unmodified lines | ||
| 25 | 25 | #include <unistd.h> |
| 26 | 26 | |
| 27 | 27 | #include "vm_util.h" |
| 28 | −#include "../kselftest.h" | |
| 28 | +#include "kselftest.h" | |
| 29 | 29 | |
| 30 | 30 | #define INLOOP_ITER 100 |
| 31 | 31 | |
| @@ -21,7 +21,7 @@ · 20 unmodified lines | ||
| 21 | 21 | #include <sys/wait.h> |
| 22 | 22 | #include <linux/userfaultfd.h> |
| 23 | 23 | |
| 24 | −#include "../kselftest.h" | |
| 24 | +#include "kselftest.h" | |
| 25 | 25 | #include "vm_util.h" |
| 26 | 26 | |
| 27 | 27 | #define KiB 1024u |
| @@ -12,7 +12,7 @@ · 11 unmodified lines | ||
| 12 | 12 | #include <stdint.h> |
| 13 | 13 | #include <err.h> |
| 14 | 14 | |
| 15 | −#include "../kselftest.h" | |
| 15 | +#include "kselftest.h" | |
| 16 | 16 | #include <include/vdso/time64.h> |
| 17 | 17 | #include "vm_util.h" |
| 18 | 18 | #include "thp_settings.h" |
| @@ -17,7 +17,7 @@ · 16 unmodified lines | ||
| 17 | 17 | #include <linux/mman.h> |
| 18 | 18 | #include <sys/mman.h> |
| 19 | 19 | |
| 20 | −#include "../kselftest.h" | |
| 20 | +#include "kselftest.h" | |
| 21 | 21 | #include "vm_util.h" |
| 22 | 22 | |
| 23 | 23 | /* |
| @@ -12,7 +12,7 @@ · 11 unmodified lines | ||
| 12 | 12 | #include <stdio.h> |
| 13 | 13 | #include <stdlib.h> |
| 14 | 14 | #include <unistd.h> |
| 15 | −#include "../kselftest.h" | |
| 15 | +#include "kselftest.h" | |
| 16 | 16 | |
| 17 | 17 | static void dump_maps(void) |
| 18 | 18 | { |
| @@ -11,7 +11,7 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <sys/mman.h> |
| 12 | 12 | #include <fcntl.h> |
| 13 | 13 | #include "vm_util.h" |
| 14 | −#include "../kselftest.h" | |
| 14 | +#include "kselftest.h" | |
| 15 | 15 | |
| 16 | 16 | #define LENGTH (256UL*1024*1024) |
| 17 | 17 | #define PROTECTION (PROT_READ | PROT_WRITE) |
| @@ -16,7 +16,7 @@ · 15 unmodified lines | ||
| 16 | 16 | #include <stdlib.h> |
| 17 | 17 | #include <string.h> |
| 18 | 18 | #include <unistd.h> |
| 19 | −#include "../kselftest.h" | |
| 19 | +#include "kselftest.h" | |
| 20 | 20 | |
| 21 | 21 | #include "vm_util.h" |
| 22 | 22 | |
| @@ -14,7 +14,7 @@ · 13 unmodified lines | ||
| 14 | 14 | #include <sys/wait.h> |
| 15 | 15 | #include <unistd.h> |
| 16 | 16 | |
| 17 | −#include "../kselftest_harness.h" | |
| 17 | +#include "kselftest_harness.h" | |
| 18 | 18 | |
| 19 | 19 | #ifndef __aarch64__ |
| 20 | 20 | # define PROT_BTI 0 |
| @@ -22,7 +22,7 @@ · 21 unmodified lines | ||
| 22 | 22 | #include <stdio.h> |
| 23 | 23 | #include <fcntl.h> |
| 24 | 24 | |
| 25 | −#include "../kselftest.h" | |
| 25 | +#include "kselftest.h" | |
| 26 | 26 | |
| 27 | 27 | #define fail(fmt, ...) ksft_test_result_fail(fmt, ##__VA_ARGS__) |
| 28 | 28 | #define pass(fmt, ...) ksft_test_result_pass(fmt, ##__VA_ARGS__) |
| @@ -1,7 +1,7 @@ | ||
| 1 | 1 | // SPDX-License-Identifier: GPL-2.0-or-later |
| 2 | 2 | |
| 3 | 3 | #define _GNU_SOURCE |
| 4 | −#include "../kselftest_harness.h" | |
| 4 | +#include "kselftest_harness.h" | |
| 5 | 5 | #include <linux/prctl.h> |
| 6 | 6 | #include <fcntl.h> |
| 7 | 7 | #include <stdio.h> |
| @@ -4,7 +4,7 @@ · 3 unmodified lines | ||
| 4 | 4 | * paths in the kernel. |
| 5 | 5 | */ |
| 6 | 6 | |
| 7 | −#include "../kselftest_harness.h" | |
| 7 | +#include "kselftest_harness.h" | |
| 8 | 8 | #include "thp_settings.h" |
| 9 | 9 | |
| 10 | 10 | #include <strings.h> |
| @@ -22,7 +22,7 @@ · 21 unmodified lines | ||
| 22 | 22 | #include <linux/userfaultfd.h> |
| 23 | 23 | #include <linux/mempolicy.h> |
| 24 | 24 | |
| 25 | −#include "../kselftest.h" | |
| 25 | +#include "kselftest.h" | |
| 26 | 26 | #include "vm_util.h" |
| 27 | 27 | |
| 28 | 28 | static size_t pagesize; |
| @@ -13,7 +13,7 @@ · 12 unmodified lines | ||
| 13 | 13 | #include <sys/ipc.h> |
| 14 | 14 | #include <sys/shm.h> |
| 15 | 15 | #include <time.h> |
| 16 | −#include "../kselftest.h" | |
| 16 | +#include "kselftest.h" | |
| 17 | 17 | #include "mlock2.h" |
| 18 | 18 | |
| 19 | 19 | #define CHUNK_UNIT (128 * 1024) |
| @@ -7,7 +7,7 @@ · 6 unmodified lines | ||
| 7 | 7 | #include <sys/time.h> |
| 8 | 8 | #include <sys/resource.h> |
| 9 | 9 | #include <stdbool.h> |
| 10 | −#include "../kselftest.h" | |
| 10 | +#include "kselftest.h" | |
| 11 | 11 | #include "mlock2.h" |
| 12 | 12 | |
| 13 | 13 | struct vm_boundaries { |
| @@ -12,7 +12,7 @@ · 11 unmodified lines | ||
| 12 | 12 | #include <unistd.h> |
| 13 | 13 | #include <asm-generic/unistd.h> |
| 14 | 14 | #include "vm_util.h" |
| 15 | −#include "../kselftest.h" | |
| 15 | +#include "kselftest.h" | |
| 16 | 16 | |
| 17 | 17 | #define MB(x) (x << 20) |
| 18 | 18 | #define MAX_SIZE_MB 1024 |
| @@ -14,7 +14,7 @@ · 13 unmodified lines | ||
| 14 | 14 | #include <string.h> |
| 15 | 15 | #include <unistd.h> |
| 16 | 16 | |
| 17 | −#include "../kselftest.h" | |
| 17 | +#include "kselftest.h" | |
| 18 | 18 | |
| 19 | 19 | unsigned long page_size; |
| 20 | 20 | char *page_buffer; |
| @@ -16,7 +16,7 @@ · 15 unmodified lines | ||
| 16 | 16 | #include <time.h> |
| 17 | 17 | #include <stdbool.h> |
| 18 | 18 | |
| 19 | −#include "../kselftest.h" | |
| 19 | +#include "kselftest.h" | |
| 20 | 20 | |
| 21 | 21 | #define EXPECT_SUCCESS 0 |
| 22 | 22 | #define EXPECT_FAILURE 1 |
| @@ -8,7 +8,7 @@ · 7 unmodified lines | ||
| 8 | 8 | #include <sys/time.h> |
| 9 | 9 | #include <sys/resource.h> |
| 10 | 10 | #include <stdbool.h> |
| 11 | −#include "../kselftest.h" | |
| 11 | +#include "kselftest.h" | |
| 12 | 12 | #include <syscall.h> |
| 13 | 13 | #include <errno.h> |
| 14 | 14 | #include <stdio.h> |
| @@ -5,7 +5,7 @@ · 4 unmodified lines | ||
| 5 | 5 | #include <string.h> |
| 6 | 6 | #include <sys/time.h> |
| 7 | 7 | #include <sys/resource.h> |
| 8 | −#include "../kselftest.h" | |
| 8 | +#include "kselftest.h" | |
| 9 | 9 | |
| 10 | 10 | static void test_limit(void) |
| 11 | 11 | { |
| @@ -8,7 +8,7 @@ · 7 unmodified lines | ||
| 8 | 8 | #include <errno.h> |
| 9 | 9 | #include <malloc.h> |
| 10 | 10 | #include "vm_util.h" |
| 11 | −#include "../kselftest.h" | |
| 11 | +#include "kselftest.h" | |
| 12 | 12 | #include <linux/types.h> |
| 13 | 13 | #include <linux/memfd.h> |
| 14 | 14 | #include <linux/userfaultfd.h> |
| @@ -22,7 +22,7 @@ · 21 unmodified lines | ||
| 22 | 22 | #include <sys/mman.h> |
| 23 | 23 | #include <sys/wait.h> |
| 24 | 24 | |
| 25 | −#include "../kselftest_harness.h" | |
| 25 | +#include "kselftest_harness.h" | |
| 26 | 26 | #include "vm_util.h" |
| 27 | 27 | |
| 28 | 28 | static sigjmp_buf sigjmp_buf_env; |
| @@ -16,7 +16,7 @@ · 15 unmodified lines | ||
| 16 | 16 | #include <linux/mman.h> |
| 17 | 17 | #include <linux/types.h> |
| 18 | 18 | |
| 19 | −#include "../kselftest.h" | |
| 19 | +#include "kselftest.h" | |
| 20 | 20 | |
| 21 | 21 | /* Define some kernel-like types */ |
| 22 | 22 | typedef __u8 u8; |
| @@ -13,7 +13,7 @@ · 12 unmodified lines | ||
| 13 | 13 | #include <sys/prctl.h> |
| 14 | 14 | #include <sys/wait.h> |
| 15 | 15 | |
| 16 | −#include "../kselftest_harness.h" | |
| 16 | +#include "kselftest_harness.h" | |
| 17 | 17 | #include "thp_settings.h" |
| 18 | 18 | #include "vm_util.h" |
| 19 | 19 | |
| @@ -1,7 +1,7 @@ | ||
| 1 | 1 | // SPDX-License-Identifier: GPL-2.0-or-later |
| 2 | 2 | |
| 3 | 3 | #define _GNU_SOURCE |
| 4 | −#include "../kselftest_harness.h" | |
| 4 | +#include "kselftest_harness.h" | |
| 5 | 5 | #include <errno.h> |
| 6 | 6 | #include <setjmp.h> |
| 7 | 7 | #include <signal.h> |
| @@ -5,7 +5,7 @@ · 4 unmodified lines | ||
| 5 | 5 | * Author(s): Wei Yang <richard.weiyang@gmail.com> |
| 6 | 6 | */ |
| 7 | 7 | |
| 8 | −#include "../kselftest_harness.h" | |
| 8 | +#include "kselftest_harness.h" | |
| 9 | 9 | #include <strings.h> |
| 10 | 10 | #include <pthread.h> |
| 11 | 11 | #include <numa.h> |
| @@ -7,7 +7,7 @@ · 6 unmodified lines | ||
| 7 | 7 | #include <malloc.h> |
| 8 | 8 | #include <sys/mman.h> |
| 9 | 9 | |
| 10 | −#include "../kselftest.h" | |
| 10 | +#include "kselftest.h" | |
| 11 | 11 | #include "vm_util.h" |
| 12 | 12 | #include "thp_settings.h" |
| 13 | 13 | |
| @@ -20,7 +20,7 @@ · 19 unmodified lines | ||
| 20 | 20 | #include <stdbool.h> |
| 21 | 21 | #include <time.h> |
| 22 | 22 | #include "vm_util.h" |
| 23 | −#include "../kselftest.h" | |
| 23 | +#include "kselftest.h" | |
| 24 | 24 | |
| 25 | 25 | uint64_t pagesize; |
| 26 | 26 | unsigned int pageshift; |
| @@ -27,7 +27,7 @@ · 26 unmodified lines | ||
| 27 | 27 | #include <stdarg.h> |
| 28 | 28 | #include <string.h> |
| 29 | 29 | #include "vm_util.h" |
| 30 | −#include "../kselftest.h" | |
| 30 | +#include "kselftest.h" | |
| 31 | 31 | |
| 32 | 32 | #if !defined(MAP_HUGETLB) |
| 33 | 33 | #define MAP_HUGETLB 0x40000 |
| @@ -16,7 +16,7 @@ · 15 unmodified lines | ||
| 16 | 16 | #include <string.h> |
| 17 | 17 | #include <sys/mman.h> |
| 18 | 18 | #include "vm_util.h" |
| 19 | −#include "../kselftest.h" | |
| 19 | +#include "kselftest.h" | |
| 20 | 20 | |
| 21 | 21 | int backing_fd = -1; |
| 22 | 22 | int mmap_flags = MAP_ANONYMOUS | MAP_NORESERVE | MAP_PRIVATE; |
| @@ -35,7 +35,7 @@ · 34 unmodified lines | ||
| 35 | 35 | #include <sys/random.h> |
| 36 | 36 | #include <stdatomic.h> |
| 37 | 37 | |
| 38 | −#include "../kselftest.h" | |
| 38 | +#include "kselftest.h" | |
| 39 | 39 | #include "vm_util.h" |
| 40 | 40 | |
| 41 | 41 | #define UFFD_FLAGS (O_CLOEXEC | O_NONBLOCK | UFFD_USER_MODE_ONLY) |
| @@ -7,7 +7,7 @@ · 6 unmodified lines | ||
| 7 | 7 | #include <assert.h> |
| 8 | 8 | #include <linux/mman.h> |
| 9 | 9 | #include <sys/mman.h> |
| 10 | −#include "../kselftest.h" | |
| 10 | +#include "kselftest.h" | |
| 11 | 11 | #include "thp_settings.h" |
| 12 | 12 | #include "uffd-common.h" |
| 13 | 13 | |
| @@ -10,7 +10,7 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <string.h> |
| 11 | 11 | |
| 12 | 12 | #include "vm_util.h" |
| 13 | −#include "../kselftest.h" | |
| 13 | +#include "kselftest.h" | |
| 14 | 14 | |
| 15 | 15 | /* |
| 16 | 16 | * The hint addr value is used to allocate addresses |
| @@ -16,7 +16,7 @@ · 15 unmodified lines | ||
| 16 | 16 | #include <fcntl.h> |
| 17 | 17 | |
| 18 | 18 | #include "vm_util.h" |
| 19 | −#include "../kselftest.h" | |
| 19 | +#include "kselftest.h" | |
| 20 | 20 | |
| 21 | 21 | /* |
| 22 | 22 | * Maximum address range mapped with a single mmap() |
| @@ -9,7 +9,7 @@ · 8 unmodified lines | ||
| 9 | 9 | #include <linux/fs.h> |
| 10 | 10 | #include <sys/syscall.h> |
| 11 | 11 | #include <unistd.h> |
| 12 | −#include "../kselftest.h" | |
| 12 | +#include "kselftest.h" | |
| 13 | 13 | #include "vm_util.h" |
| 14 | 14 | |
| 15 | 15 | #define PMD_SIZE_FILE_PATH "/sys/kernel/mm/transparent_hugepage/hpage_pmd_size" |
| @@ -6,7 +6,7 @@ · 5 unmodified lines | ||
| 6 | 6 | #include <stdarg.h> |
| 7 | 7 | #include <strings.h> /* ffsl() */ |
| 8 | 8 | #include <unistd.h> /* _SC_PAGESIZE */ |
| 9 | −#include "../kselftest.h" | |
| 9 | +#include "kselftest.h" | |
| 10 | 10 | #include <linux/fs.h> |
| 11 | 11 | |
| 12 | 12 | #define BIT_ULL(nr) (1ULL << (nr)) |
| @@ -21,7 +21,7 @@ · 20 unmodified lines | ||
| 21 | 21 | #include <linux/mount.h> |
| 22 | 22 | |
| 23 | 23 | #include "../filesystems/wrappers.h" |
| 24 | −#include "../kselftest_harness.h" | |
| 24 | +#include "kselftest_harness.h" | |
| 25 | 25 | |
| 26 | 26 | #ifndef CLONE_NEWNS |
| 27 | 27 | #define CLONE_NEWNS 0x00020000 |
| @@ -15,7 +15,7 @@ · 14 unmodified lines | ||
| 15 | 15 | #include <stdarg.h> |
| 16 | 16 | #include <sys/syscall.h> |
| 17 | 17 | |
| 18 | −#include "../kselftest_harness.h" | |
| 18 | +#include "kselftest_harness.h" | |
| 19 | 19 | |
| 20 | 20 | #ifndef CLONE_NEWNS |
| 21 | 21 | #define CLONE_NEWNS 0x00020000 |
| @@ -33,7 +33,7 @@ · 32 unmodified lines | ||
| 33 | 33 | #include <mqueue.h> |
| 34 | 34 | #include <error.h> |
| 35 | 35 | |
| 36 | −#include "../kselftest.h" | |
| 36 | +#include "kselftest.h" | |
| 37 | 37 | |
| 38 | 38 | static char *usage = |
| 39 | 39 | "Usage:\n" |
| @@ -40,7 +40,7 @@ · 39 unmodified lines | ||
| 40 | 40 | #include <popt.h> |
| 41 | 41 | #include <error.h> |
| 42 | 42 | |
| 43 | −#include "../kselftest.h" | |
| 43 | +#include "kselftest.h" | |
| 44 | 44 | |
| 45 | 45 | static char *usage = |
| 46 | 46 | "Usage:\n" |
| @@ -11,8 +11,8 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <string.h> |
| 12 | 12 | #include <stdbool.h> |
| 13 | 13 | |
| 14 | −#include "../kselftest.h" | |
| 15 | −#include "../kselftest_harness.h" | |
| 14 | +#include "kselftest.h" | |
| 15 | +#include "kselftest_harness.h" | |
| 16 | 16 | |
| 17 | 17 | #define VMFLAGS "VmFlags:" |
| 18 | 18 | #define MSEAL_FLAGS "sl" |
| @@ -14,7 +14,7 @@ · 13 unmodified lines | ||
| 14 | 14 | #include <sys/wait.h> |
| 15 | 15 | #include <unistd.h> |
| 16 | 16 | #include <linux/unistd.h> |
| 17 | −#include "../kselftest_harness.h" | |
| 17 | +#include "kselftest_harness.h" | |
| 18 | 18 | |
| 19 | 19 | #ifndef FD_NSFS_ROOT |
| 20 | 20 | #define FD_NSFS_ROOT -10003 /* Root of the nsfs filesystem */ |
| @@ -11,7 +11,7 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <string.h> |
| 12 | 12 | #include <linux/nsfs.h> |
| 13 | 13 | |
| 14 | −#include "../kselftest_harness.h" | |
| 14 | +#include "kselftest_harness.h" | |
| 15 | 15 | |
| 16 | 16 | struct ns_info { |
| 17 | 17 | const char *name; |
| @@ -20,7 +20,7 @@ · 19 unmodified lines | ||
| 20 | 20 | #include <linux/fs.h> |
| 21 | 21 | #include <linux/limits.h> |
| 22 | 22 | #include <linux/nsfs.h> |
| 23 | −#include "../kselftest_harness.h" | |
| 23 | +#include "kselftest_harness.h" | |
| 24 | 24 | |
| 25 | 25 | /* Fixture for tests that create child processes */ |
| 26 | 26 | FIXTURE(nsid) { |
| @@ -16,7 +16,7 @@ · 15 unmodified lines | ||
| 16 | 16 | #include <sys/socket.h> |
| 17 | 17 | #include <linux/nfc.h> |
| 18 | 18 | |
| 19 | −#include "../kselftest_harness.h" | |
| 19 | +#include "kselftest_harness.h" | |
| 20 | 20 | |
| 21 | 21 | #define GENLMSG_DATA(glh) ((void *)(NLMSG_DATA(glh) + GENL_HDRLEN)) |
| 22 | 22 | #define GENLMSG_PAYLOAD(glh) (NLMSG_PAYLOAD(glh, 0) - GENL_HDRLEN) |
| @@ -14,7 +14,7 @@ · 13 unmodified lines | ||
| 14 | 14 | #include <sys/types.h> |
| 15 | 15 | #include <sys/un.h> |
| 16 | 16 | |
| 17 | −#include "../../kselftest_harness.h" | |
| 17 | +#include "kselftest_harness.h" | |
| 18 | 18 | |
| 19 | 19 | FIXTURE(diag_uid) |
| 20 | 20 | { |
| @@ -11,7 +11,7 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <sys/signalfd.h> |
| 12 | 12 | #include <sys/socket.h> |
| 13 | 13 | |
| 14 | −#include "../../kselftest_harness.h" | |
| 14 | +#include "kselftest_harness.h" | |
| 15 | 15 | |
| 16 | 16 | #define BUF_SZ 32 |
| 17 | 17 | |
| @@ -6,7 +6,7 @@ · 5 unmodified lines | ||
| 6 | 6 | #include <sys/socket.h> |
| 7 | 7 | #include <sys/types.h> |
| 8 | 8 | |
| 9 | −#include "../../kselftest_harness.h" | |
| 9 | +#include "kselftest_harness.h" | |
| 10 | 10 | |
| 11 | 11 | #define NR_CHUNKS 100 |
| 12 | 12 | #define MSG_LEN 256 |
| @@ -16,7 +16,7 @@ · 15 unmodified lines | ||
| 16 | 16 | #include <sys/wait.h> |
| 17 | 17 | |
| 18 | 18 | #include "../../pidfd/pidfd.h" |
| 19 | −#include "../../kselftest_harness.h" | |
| 19 | +#include "kselftest_harness.h" | |
| 20 | 20 | |
| 21 | 21 | #define clean_errno() (errno == 0 ? "None" : strerror(errno)) |
| 22 | 22 | #define log_err(MSG, ...) \ |
| @@ -10,7 +10,7 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <sys/socket.h> |
| 11 | 11 | #include <sys/un.h> |
| 12 | 12 | |
| 13 | −#include "../../kselftest_harness.h" | |
| 13 | +#include "kselftest_harness.h" | |
| 14 | 14 | |
| 15 | 15 | FIXTURE(scm_rights) |
| 16 | 16 | { |
| @@ -10,7 +10,7 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <sys/socket.h> |
| 11 | 11 | #include <sys/un.h> |
| 12 | 12 | |
| 13 | −#include "../../kselftest_harness.h" | |
| 13 | +#include "kselftest_harness.h" | |
| 14 | 14 | |
| 15 | 15 | FIXTURE(unix_connect) |
| 16 | 16 | { |
| @@ -4,7 +4,7 @@ · 3 unmodified lines | ||
| 4 | 4 | #include <sys/socket.h> |
| 5 | 5 | #include <netinet/in.h> |
| 6 | 6 | |
| 7 | −#include "../kselftest_harness.h" | |
| 7 | +#include "kselftest_harness.h" | |
| 8 | 8 | |
| 9 | 9 | FIXTURE(bind_timewait) |
| 10 | 10 | { |
| @@ -4,7 +4,7 @@ · 3 unmodified lines | ||
| 4 | 4 | #include <sys/socket.h> |
| 5 | 5 | #include <netinet/in.h> |
| 6 | 6 | |
| 7 | −#include "../kselftest_harness.h" | |
| 7 | +#include "kselftest_harness.h" | |
| 8 | 8 | |
| 9 | 9 | static const __u32 in4addr_any = INADDR_ANY; |
| 10 | 10 | static const __u32 in4addr_loopback = INADDR_LOOPBACK; |
| @@ -19,7 +19,7 @@ · 18 unmodified lines | ||
| 19 | 19 | #include <linux/can.h> |
| 20 | 20 | #include <linux/can/raw.h> |
| 21 | 21 | |
| 22 | −#include "../../kselftest_harness.h" | |
| 22 | +#include "kselftest_harness.h" | |
| 23 | 23 | |
| 24 | 24 | #define ID 0x123 |
| 25 | 25 | |
| @@ -16,7 +16,7 @@ · 15 unmodified lines | ||
| 16 | 16 | #include <linux/udp.h> |
| 17 | 17 | #include <sys/socket.h> |
| 18 | 18 | |
| 19 | −#include "../kselftest.h" | |
| 19 | +#include "kselftest.h" | |
| 20 | 20 | |
| 21 | 21 | enum { |
| 22 | 22 | ERN_SUCCESS = 0, |
| @@ -23,7 +23,7 @@ · 22 unmodified lines | ||
| 23 | 23 | #include <sys/ioctl.h> |
| 24 | 24 | #include <sys/socket.h> |
| 25 | 25 | |
| 26 | −#include "../kselftest_harness.h" | |
| 26 | +#include "kselftest_harness.h" | |
| 27 | 27 | |
| 28 | 28 | /* if the headers haven't been updated, we need to define some things */ |
| 29 | 29 | #if !defined(EPOLL_IOC_TYPE) |
| @@ -10,7 +10,7 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <fcntl.h> |
| 11 | 11 | #include <netinet/ip.h> |
| 12 | 12 | |
| 13 | −#include "../kselftest_harness.h" | |
| 13 | +#include "kselftest_harness.h" | |
| 14 | 14 | |
| 15 | 15 | #ifndef IP_LOCAL_PORT_RANGE |
| 16 | 16 | #define IP_LOCAL_PORT_RANGE 51 |
| @@ -34,7 +34,7 @@ · 33 unmodified lines | ||
| 34 | 34 | #include <time.h> |
| 35 | 35 | #include <unistd.h> |
| 36 | 36 | |
| 37 | −#include "../kselftest.h" | |
| 37 | +#include "kselftest.h" | |
| 38 | 38 | |
| 39 | 39 | #define printk(fmt, ...) \ |
| 40 | 40 | ksft_print_msg("%d[%u] " fmt "\n", getpid(), __LINE__, ##__VA_ARGS__) |
| @@ -34,7 +34,7 @@ · 33 unmodified lines | ||
| 34 | 34 | #include <sys/ioctl.h> |
| 35 | 35 | #include <sys/socket.h> |
| 36 | 36 | #include <unistd.h> |
| 37 | −#include "../kselftest.h" | |
| 37 | +#include "kselftest.h" | |
| 38 | 38 | |
| 39 | 39 | #define MTU 1500 |
| 40 | 40 | #define LARGER_THAN_MTU 8192 |
| @@ -10,7 +10,7 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <linux/netfilter/nfnetlink.h> |
| 11 | 11 | #include <linux/netfilter/nfnetlink_conntrack.h> |
| 12 | 12 | #include <linux/netfilter/nf_conntrack_tcp.h> |
| 13 | −#include "../../kselftest_harness.h" | |
| 13 | +#include "kselftest_harness.h" | |
| 14 | 14 | |
| 15 | 15 | #define TEST_ZONE_ID 123 |
| 16 | 16 | #define NF_CT_DEFAULT_ZONE_ID 0 |
| @@ -18,7 +18,7 @@ · 17 unmodified lines | ||
| 18 | 18 | #include <linux/mqueue.h> |
| 19 | 19 | #include <linux/rtnetlink.h> |
| 20 | 20 | |
| 21 | −#include "../kselftest_harness.h" | |
| 21 | +#include "kselftest_harness.h" | |
| 22 | 22 | |
| 23 | 23 | #include <ynl.h> |
| 24 | 24 | |
| @@ -32,7 +32,7 @@ · 31 unmodified lines | ||
| 32 | 32 | |
| 33 | 33 | #include <sys/socket.h> |
| 34 | 34 | |
| 35 | −#include "../../kselftest.h" | |
| 35 | +#include "kselftest.h" | |
| 36 | 36 | |
| 37 | 37 | /* defines to make checkpatch happy */ |
| 38 | 38 | #define strscpy strncpy |
| @@ -10,7 +10,7 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <stdlib.h> |
| 11 | 11 | #include <unistd.h> |
| 12 | 12 | |
| 13 | −#include "../kselftest_harness.h" | |
| 13 | +#include "kselftest_harness.h" | |
| 14 | 14 | |
| 15 | 15 | static const char ctrl_cmd_stop[] = "stop"; |
| 16 | 16 | static const char ctrl_cmd_start[] = "start"; |
| @@ -54,7 +54,7 @@ · 53 unmodified lines | ||
| 54 | 54 | #include <unistd.h> |
| 55 | 55 | |
| 56 | 56 | #include "psock_lib.h" |
| 57 | −#include "../kselftest.h" | |
| 57 | +#include "kselftest.h" | |
| 58 | 58 | |
| 59 | 59 | #define RING_NUM_FRAMES 20 |
| 60 | 60 | |
| @@ -46,7 +46,7 @@ · 45 unmodified lines | ||
| 46 | 46 | |
| 47 | 47 | #include "psock_lib.h" |
| 48 | 48 | |
| 49 | −#include "../kselftest.h" | |
| 49 | +#include "kselftest.h" | |
| 50 | 50 | |
| 51 | 51 | #ifndef bug_on |
| 52 | 52 | # define bug_on(cond) assert(!(cond)) |
| @@ -22,7 +22,7 @@ · 21 unmodified lines | ||
| 22 | 22 | #include <sys/socket.h> |
| 23 | 23 | #include <sys/types.h> |
| 24 | 24 | #include <unistd.h> |
| 25 | −#include "../kselftest_harness.h" | |
| 25 | +#include "kselftest_harness.h" | |
| 26 | 26 | |
| 27 | 27 | struct reuse_opts { |
| 28 | 28 | int reuseaddr[2]; |
| @@ -24,7 +24,7 @@ · 23 unmodified lines | ||
| 24 | 24 | #include <sys/resource.h> |
| 25 | 25 | #include <unistd.h> |
| 26 | 26 | |
| 27 | −#include "../kselftest.h" | |
| 27 | +#include "kselftest.h" | |
| 28 | 28 | |
| 29 | 29 | struct test_params { |
| 30 | 30 | int recv_family; |
| @@ -23,7 +23,7 @@ · 22 unmodified lines | ||
| 23 | 23 | #include <unistd.h> |
| 24 | 24 | #include <numa.h> |
| 25 | 25 | |
| 26 | −#include "../kselftest.h" | |
| 26 | +#include "kselftest.h" | |
| 27 | 27 | |
| 28 | 28 | static const int PORT = 8888; |
| 29 | 29 | |
| @@ -18,7 +18,7 @@ · 17 unmodified lines | ||
| 18 | 18 | #include <linux/net_tstamp.h> |
| 19 | 19 | #include <linux/errqueue.h> |
| 20 | 20 | |
| 21 | −#include "../kselftest.h" | |
| 21 | +#include "kselftest.h" | |
| 22 | 22 | |
| 23 | 23 | struct options { |
| 24 | 24 | int so_timestamp; |
| @@ -8,7 +8,7 @@ · 7 unmodified lines | ||
| 8 | 8 | #include <sys/types.h> |
| 9 | 9 | #include <netinet/in.h> |
| 10 | 10 | #include <arpa/inet.h> |
| 11 | −#include "../kselftest.h" | |
| 11 | +#include "kselftest.h" | |
| 12 | 12 | |
| 13 | 13 | static char *afstr(int af, int proto) |
| 14 | 14 | { |
| @@ -9,7 +9,7 @@ · 8 unmodified lines | ||
| 9 | 9 | #include <sys/socket.h> |
| 10 | 10 | #include <sys/sysinfo.h> |
| 11 | 11 | |
| 12 | −#include "../kselftest_harness.h" | |
| 12 | +#include "kselftest_harness.h" | |
| 13 | 13 | |
| 14 | 14 | FIXTURE(so_incoming_cpu) |
| 15 | 15 | { |
| @@ -7,7 +7,7 @@ · 6 unmodified lines | ||
| 7 | 7 | #include <sys/socket.h> |
| 8 | 8 | #include <netinet/in.h> |
| 9 | 9 | |
| 10 | −#include "../kselftest.h" | |
| 10 | +#include "kselftest.h" | |
| 11 | 11 | |
| 12 | 12 | struct socket_testcase { |
| 13 | 13 | int domain; |
| @@ -17,7 +17,7 @@ · 16 unmodified lines | ||
| 17 | 17 | #include <linux/virtio_net.h> |
| 18 | 18 | #include <netinet/ip.h> |
| 19 | 19 | #include <netinet/udp.h> |
| 20 | −#include "../kselftest_harness.h" | |
| 20 | +#include "kselftest_harness.h" | |
| 21 | 21 | |
| 22 | 22 | static const char param_dev_tap_name[] = "xmacvtap0"; |
| 23 | 23 | static const char param_dev_dummy_name[] = "xdummy0"; |
| @@ -9,7 +9,7 @@ · 8 unmodified lines | ||
| 9 | 9 | * Can't be included in the header: it defines static variables which |
| 10 | 10 | * will be unique to every object. Let's include it only once here. |
| 11 | 11 | */ |
| 12 | −#include "../../../kselftest.h" | |
| 12 | +#include "kselftest.h" | |
| 13 | 13 | |
| 14 | 14 | /* Prevent overriding of one thread's output by another */ |
| 15 | 15 | static pthread_mutex_t ksft_print_lock = PTHREAD_MUTEX_INITIALIZER; |
| @@ -26,7 +26,7 @@ · 25 unmodified lines | ||
| 26 | 26 | #include <fcntl.h> |
| 27 | 27 | #include <time.h> |
| 28 | 28 | |
| 29 | −#include "../kselftest.h" | |
| 29 | +#include "kselftest.h" | |
| 30 | 30 | |
| 31 | 31 | #ifndef TCP_FASTOPEN_KEY |
| 32 | 32 | #define TCP_FASTOPEN_KEY 33 |
| @@ -10,7 +10,7 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <sched.h> |
| 11 | 11 | #include <stdlib.h> |
| 12 | 12 | |
| 13 | −#include "../kselftest_harness.h" | |
| 13 | +#include "kselftest_harness.h" | |
| 14 | 14 | |
| 15 | 15 | #define DST_PORT 30000 |
| 16 | 16 | #define SRC_PORT 40000 |
| @@ -21,7 +21,7 @@ · 20 unmodified lines | ||
| 21 | 21 | #include <sys/socket.h> |
| 22 | 22 | #include <sys/stat.h> |
| 23 | 23 | |
| 24 | −#include "../kselftest_harness.h" | |
| 24 | +#include "kselftest_harness.h" | |
| 25 | 25 | |
| 26 | 26 | #define TLS_PAYLOAD_MAX_LEN 16384 |
| 27 | 27 | #define SOL_TLS 282 |
| @@ -15,7 +15,7 @@ · 14 unmodified lines | ||
| 15 | 15 | #include <sys/ioctl.h> |
| 16 | 16 | #include <sys/socket.h> |
| 17 | 17 | |
| 18 | −#include "../kselftest_harness.h" | |
| 18 | +#include "kselftest_harness.h" | |
| 19 | 19 | |
| 20 | 20 | static int tun_attach(int fd, char *dev) |
| 21 | 21 | { |
| @@ -25,7 +25,7 @@ · 24 unmodified lines | ||
| 25 | 25 | #include <sys/types.h> |
| 26 | 26 | #include <unistd.h> |
| 27 | 27 | |
| 28 | −#include "../kselftest.h" | |
| 28 | +#include "kselftest.h" | |
| 29 | 29 | |
| 30 | 30 | #ifndef ETH_MAX_MTU |
| 31 | 31 | #define ETH_MAX_MTU 0xFFFFU |
| @@ -12,7 +12,7 @@ · 11 unmodified lines | ||
| 12 | 12 | #include <stdbool.h> |
| 13 | 13 | #include <errno.h> |
| 14 | 14 | #include <linux/types.h> |
| 15 | −#include "../kselftest.h" | |
| 15 | +#include "kselftest.h" | |
| 16 | 16 | |
| 17 | 17 | #define ARRAY_LEN(X) (sizeof (X) / sizeof (*(X))) |
| 18 | 18 | #define BUILD_BUG_ON(e) ((void)(sizeof(struct { int:(-!!(e)); }))) |
| @@ -15,7 +15,7 @@ · 14 unmodified lines | ||
| 15 | 15 | #include <stdbool.h> |
| 16 | 16 | #include <string.h> |
| 17 | 17 | |
| 18 | −#include "../kselftest.h" | |
| 18 | +#include "kselftest.h" | |
| 19 | 19 | #include "helpers.h" |
| 20 | 20 | |
| 21 | 21 | /* |
| @@ -22,7 +22,7 @@ · 21 unmodified lines | ||
| 22 | 22 | #include <limits.h> |
| 23 | 23 | #include <unistd.h> |
| 24 | 24 | |
| 25 | −#include "../kselftest.h" | |
| 25 | +#include "kselftest.h" | |
| 26 | 26 | #include "helpers.h" |
| 27 | 27 | |
| 28 | 28 | /* Construct a test directory with the following structure: |
| @@ -14,7 +14,7 @@ · 13 unmodified lines | ||
| 14 | 14 | #include <stdbool.h> |
| 15 | 15 | #include <string.h> |
| 16 | 16 | |
| 17 | −#include "../kselftest.h" | |
| 17 | +#include "kselftest.h" | |
| 18 | 18 | #include "helpers.h" |
| 19 | 19 | |
| 20 | 20 | /* |
| @@ -20,7 +20,7 @@ · 19 unmodified lines | ||
| 20 | 20 | |
| 21 | 21 | #include "../../../../include/uapi/linux/pcitest.h" |
| 22 | 22 | |
| 23 | −#include "../kselftest_harness.h" | |
| 23 | +#include "kselftest_harness.h" | |
| 24 | 24 | |
| 25 | 25 | #define pci_ep_ioctl(cmd, arg) \ |
| 26 | 26 | ({ \ |
| @@ -14,7 +14,7 @@ · 13 unmodified lines | ||
| 14 | 14 | |
| 15 | 15 | #include <linux/perf_event.h> |
| 16 | 16 | |
| 17 | −#include "../kselftest_harness.h" | |
| 17 | +#include "kselftest_harness.h" | |
| 18 | 18 | |
| 19 | 19 | #define RB_SIZE 0x3000 |
| 20 | 20 | #define AUX_SIZE 0x10000 |
| @@ -30,7 +30,7 @@ · 29 unmodified lines | ||
| 30 | 30 | #include <sys/syscall.h> |
| 31 | 31 | #include <unistd.h> |
| 32 | 32 | |
| 33 | −#include "../kselftest_harness.h" | |
| 33 | +#include "kselftest_harness.h" | |
| 34 | 34 | |
| 35 | 35 | static volatile int signal_count; |
| 36 | 36 | |
| @@ -31,7 +31,7 @@ · 30 unmodified lines | ||
| 31 | 31 | #include <sys/syscall.h> |
| 32 | 32 | #include <unistd.h> |
| 33 | 33 | |
| 34 | −#include "../kselftest_harness.h" | |
| 34 | +#include "kselftest_harness.h" | |
| 35 | 35 | |
| 36 | 36 | #define NUM_THREADS 5 |
| 37 | 37 | |
| @@ -15,7 +15,7 @@ · 14 unmodified lines | ||
| 15 | 15 | #include <sys/wait.h> |
| 16 | 16 | #include <unistd.h> |
| 17 | 17 | |
| 18 | −#include "../kselftest_harness.h" | |
| 18 | +#include "kselftest_harness.h" | |
| 19 | 19 | |
| 20 | 20 | static int sigio_count; |
| 21 | 21 | |
| @@ -13,7 +13,7 @@ · 12 unmodified lines | ||
| 13 | 13 | #include <sys/mount.h> |
| 14 | 14 | #include <sys/wait.h> |
| 15 | 15 | |
| 16 | −#include "../kselftest_harness.h" | |
| 16 | +#include "kselftest_harness.h" | |
| 17 | 17 | #include "../pidfd/pidfd.h" |
| 18 | 18 | |
| 19 | 19 | #define __STACK_SIZE (8 * 1024 * 1024) |
| @@ -11,7 +11,7 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <syscall.h> |
| 12 | 12 | #include <sys/wait.h> |
| 13 | 13 | |
| 14 | −#include "../kselftest_harness.h" | |
| 14 | +#include "kselftest_harness.h" | |
| 15 | 15 | #include "../pidfd/pidfd.h" |
| 16 | 16 | |
| 17 | 17 | /* |
| @@ -25,7 +25,7 @@ · 24 unmodified lines | ||
| 25 | 25 | #undef SCHED_FLAG_KEEP_ALL |
| 26 | 26 | #undef SCHED_FLAG_UTIL_CLAMP |
| 27 | 27 | |
| 28 | −#include "../kselftest.h" | |
| 28 | +#include "kselftest.h" | |
| 29 | 29 | #include "../clone3/clone3_selftests.h" |
| 30 | 30 | |
| 31 | 31 | #ifndef FD_PIDFS_ROOT |
| @@ -14,7 +14,7 @@ · 13 unmodified lines | ||
| 14 | 14 | #include <unistd.h> |
| 15 | 15 | |
| 16 | 16 | #include "pidfd.h" |
| 17 | −#include "../kselftest_harness.h" | |
| 17 | +#include "kselftest_harness.h" | |
| 18 | 18 | #include "../filesystems/wrappers.h" |
| 19 | 19 | |
| 20 | 20 | FIXTURE(pidfd_bind_mount) { |
| @@ -16,7 +16,7 @@ · 15 unmodified lines | ||
| 16 | 16 | #include <sys/mount.h> |
| 17 | 17 | |
| 18 | 18 | #include "pidfd.h" |
| 19 | −#include "../kselftest.h" | |
| 19 | +#include "kselftest.h" | |
| 20 | 20 | |
| 21 | 21 | struct error { |
| 22 | 22 | int code; |
| @@ -20,7 +20,7 @@ · 19 unmodified lines | ||
| 20 | 20 | #include <sys/stat.h> |
| 21 | 21 | |
| 22 | 22 | #include "pidfd.h" |
| 23 | −#include "../kselftest_harness.h" | |
| 23 | +#include "kselftest_harness.h" | |
| 24 | 24 | |
| 25 | 25 | FIXTURE(file_handle) |
| 26 | 26 | { |
| @@ -19,7 +19,7 @@ · 18 unmodified lines | ||
| 19 | 19 | #include <linux/kcmp.h> |
| 20 | 20 | |
| 21 | 21 | #include "pidfd.h" |
| 22 | −#include "../kselftest_harness.h" | |
| 22 | +#include "kselftest_harness.h" | |
| 23 | 23 | |
| 24 | 24 | /* |
| 25 | 25 | * UNKNOWN_FD is an fd number that should never exist in the child, as it is |
| @@ -21,7 +21,7 @@ · 20 unmodified lines | ||
| 21 | 21 | #include <sys/stat.h> |
| 22 | 22 | |
| 23 | 23 | #include "pidfd.h" |
| 24 | −#include "../kselftest_harness.h" | |
| 24 | +#include "kselftest_harness.h" | |
| 25 | 25 | |
| 26 | 26 | FIXTURE(pidfd_info) |
| 27 | 27 | { |
| @@ -20,7 +20,7 @@ · 19 unmodified lines | ||
| 20 | 20 | #include <unistd.h> |
| 21 | 21 | |
| 22 | 22 | #include "pidfd.h" |
| 23 | −#include "../kselftest.h" | |
| 23 | +#include "kselftest.h" | |
| 24 | 24 | |
| 25 | 25 | static int safe_int(const char *numstr, int *converted) |
| 26 | 26 | { |
| @@ -14,7 +14,7 @@ · 13 unmodified lines | ||
| 14 | 14 | #include <unistd.h> |
| 15 | 15 | |
| 16 | 16 | #include "pidfd.h" |
| 17 | −#include "../kselftest.h" | |
| 17 | +#include "kselftest.h" | |
| 18 | 18 | |
| 19 | 19 | static bool timeout; |
| 20 | 20 | |
| @@ -22,7 +22,7 @@ · 21 unmodified lines | ||
| 22 | 22 | #include <sys/xattr.h> |
| 23 | 23 | |
| 24 | 24 | #include "pidfd.h" |
| 25 | −#include "../kselftest_harness.h" | |
| 25 | +#include "kselftest_harness.h" | |
| 26 | 26 | |
| 27 | 27 | FIXTURE(pidfs_setattr) |
| 28 | 28 | { |
| @@ -18,7 +18,7 @@ · 17 unmodified lines | ||
| 18 | 18 | #include <sys/stat.h> |
| 19 | 19 | |
| 20 | 20 | #include "pidfd.h" |
| 21 | −#include "../kselftest_harness.h" | |
| 21 | +#include "kselftest_harness.h" | |
| 22 | 22 | |
| 23 | 23 | enum { |
| 24 | 24 | PIDFD_NS_USER, |
| @@ -20,7 +20,7 @@ · 19 unmodified lines | ||
| 20 | 20 | #include <unistd.h> |
| 21 | 21 | |
| 22 | 22 | #include "pidfd.h" |
| 23 | −#include "../kselftest.h" | |
| 23 | +#include "kselftest.h" | |
| 24 | 24 | |
| 25 | 25 | #define str(s) _str(s) |
| 26 | 26 | #define _str(s) #s |
| @@ -17,7 +17,7 @@ · 16 unmodified lines | ||
| 17 | 17 | #include <unistd.h> |
| 18 | 18 | |
| 19 | 19 | #include "pidfd.h" |
| 20 | −#include "../kselftest_harness.h" | |
| 20 | +#include "kselftest_harness.h" | |
| 21 | 21 | |
| 22 | 22 | #define ptr_to_u64(ptr) ((__u64)((uintptr_t)(ptr))) |
| 23 | 23 | |
| @@ -22,7 +22,7 @@ · 21 unmodified lines | ||
| 22 | 22 | #include <sys/xattr.h> |
| 23 | 23 | |
| 24 | 24 | #include "pidfd.h" |
| 25 | −#include "../kselftest_harness.h" | |
| 25 | +#include "kselftest_harness.h" | |
| 26 | 26 | |
| 27 | 27 | FIXTURE(pidfs_xattr) |
| 28 | 28 | { |
| @@ -10,7 +10,7 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <sys/mman.h> |
| 11 | 11 | #include <string.h> |
| 12 | 12 | |
| 13 | −#include "../kselftest_harness.h" | |
| 13 | +#include "kselftest_harness.h" | |
| 14 | 14 | |
| 15 | 15 | #define AREA_SIZE 1024 |
| 16 | 16 | |
| @@ -7,7 +7,7 @@ · 6 unmodified lines | ||
| 7 | 7 | #include <sys/prctl.h> |
| 8 | 8 | #include <string.h> |
| 9 | 9 | |
| 10 | −#include "../kselftest_harness.h" | |
| 10 | +#include "kselftest_harness.h" | |
| 11 | 11 | |
| 12 | 12 | #define CHANGE_NAME "changename" |
| 13 | 13 | #define EMPTY_NAME "" |
| @@ -23,7 +23,7 @@ · 22 unmodified lines | ||
| 23 | 23 | * |
| 24 | 24 | */ |
| 25 | 25 | #define _GNU_SOURCE |
| 26 | −#include "../kselftest_harness.h" | |
| 26 | +#include "kselftest_harness.h" | |
| 27 | 27 | #include <errno.h> |
| 28 | 28 | #include <fcntl.h> |
| 29 | 29 | #include <pthread.h> |
| @@ -51,7 +51,7 @@ · 50 unmodified lines | ||
| 51 | 51 | #define __maybe_unused __attribute__((__unused__)) |
| 52 | 52 | #endif |
| 53 | 53 | |
| 54 | −#include "../kselftest.h" | |
| 54 | +#include "kselftest.h" | |
| 55 | 55 | |
| 56 | 56 | static inline long sys_execveat(int dirfd, const char *pathname, char **argv, char **envp, int flags) |
| 57 | 57 | { |
| @@ -16,7 +16,7 @@ · 15 unmodified lines | ||
| 16 | 16 | #include <sys/stat.h> |
| 17 | 17 | #include <sys/prctl.h> |
| 18 | 18 | |
| 19 | −#include "../kselftest_harness.h" | |
| 19 | +#include "kselftest_harness.h" | |
| 20 | 20 | |
| 21 | 21 | #define ASSERT_ERRNO(expected, _t, seen) \ |
| 22 | 22 | __EXPECT(expected, #expected, \ |
| @@ -1,6 +1,6 @@ | ||
| 1 | 1 | // SPDX-License-Identifier: GPL-2.0 |
| 2 | 2 | #define _GNU_SOURCE |
| 3 | −#include "../kselftest_harness.h" | |
| 3 | +#include "kselftest_harness.h" | |
| 4 | 4 | #include <stdio.h> |
| 5 | 5 | #include <string.h> |
| 6 | 6 | #include <errno.h> |
| @@ -7,7 +7,7 @@ · 6 unmodified lines | ||
| 7 | 7 | * matches userspace expectations. |
| 8 | 8 | */ |
| 9 | 9 | |
| 10 | −#include "../kselftest_harness.h" | |
| 10 | +#include "kselftest_harness.h" | |
| 11 | 11 | #include <err.h> |
| 12 | 12 | #include <signal.h> |
| 13 | 13 | #include <asm/unistd.h> |
| @@ -7,7 +7,7 @@ · 6 unmodified lines | ||
| 7 | 7 | * matches userspace expectations. |
| 8 | 8 | */ |
| 9 | 9 | |
| 10 | −#include "../kselftest_harness.h" | |
| 10 | +#include "kselftest_harness.h" | |
| 11 | 11 | #include <err.h> |
| 12 | 12 | #include <fcntl.h> |
| 13 | 13 | #include <signal.h> |
| @@ -7,7 +7,7 @@ · 6 unmodified lines | ||
| 7 | 7 | * when de_thread is blocked with ->cred_guard_mutex held. |
| 8 | 8 | */ |
| 9 | 9 | |
| 10 | −#include "../kselftest_harness.h" | |
| 10 | +#include "kselftest_harness.h" | |
| 11 | 11 | #include <stdio.h> |
| 12 | 12 | #include <fcntl.h> |
| 13 | 13 | #include <pthread.h> |
| @@ -23,7 +23,7 @@ · 22 unmodified lines | ||
| 23 | 23 | #include <asm/unistd.h> |
| 24 | 24 | #include <linux/perf_event.h> |
| 25 | 25 | #include <linux/compiler.h> |
| 26 | −#include "../kselftest.h" | |
| 26 | +#include "kselftest.h" | |
| 27 | 27 | |
| 28 | 28 | #define MB (1024 * 1024) |
| 29 | 29 | #define RESCTRL_PATH "/sys/fs/resctrl" |
| @@ -17,7 +17,7 @@ · 16 unmodified lines | ||
| 17 | 17 | #include <sys/ioctl.h> |
| 18 | 18 | |
| 19 | 19 | #include "../user_events/user_events_selftests.h" /* share tracefs setup */ |
| 20 | −#include "../kselftest_harness.h" | |
| 20 | +#include "kselftest_harness.h" | |
| 21 | 21 | |
| 22 | 22 | #define TRACEFS_ROOT "/sys/kernel/tracing" |
| 23 | 23 | |
| @@ -9,7 +9,7 @@ · 8 unmodified lines | ||
| 9 | 9 | #include <sys/wait.h> |
| 10 | 10 | #include <unistd.h> |
| 11 | 11 | |
| 12 | −#include "../../kselftest.h" | |
| 12 | +#include "kselftest.h" | |
| 13 | 13 | |
| 14 | 14 | #ifndef PR_PMLEN_SHIFT |
| 15 | 15 | #define PR_PMLEN_SHIFT 24 |
| @@ -18,7 +18,7 @@ · 17 unmodified lines | ||
| 18 | 18 | #include <getopt.h> |
| 19 | 19 | |
| 20 | 20 | #include "hwprobe.h" |
| 21 | −#include "../../kselftest.h" | |
| 21 | +#include "kselftest.h" | |
| 22 | 22 | |
| 23 | 23 | #define MK_CBO(fn) le32_bswap((uint32_t)(fn) << 20 | 10 << 15 | 2 << 12 | 0 << 7 | 15) |
| 24 | 24 | #define MK_PREFETCH(fn) \ |
| @@ -1,6 +1,6 @@ | ||
| 1 | 1 | // SPDX-License-Identifier: GPL-2.0-only |
| 2 | 2 | #include "hwprobe.h" |
| 3 | −#include "../../kselftest.h" | |
| 3 | +#include "kselftest.h" | |
| 4 | 4 | |
| 5 | 5 | int main(int argc, char **argv) |
| 6 | 6 | { |
| @@ -14,7 +14,7 @@ · 13 unmodified lines | ||
| 14 | 14 | #include <assert.h> |
| 15 | 15 | |
| 16 | 16 | #include "hwprobe.h" |
| 17 | −#include "../../kselftest.h" | |
| 17 | +#include "kselftest.h" | |
| 18 | 18 | |
| 19 | 19 | static void help(void) |
| 20 | 20 | { |
| @@ -2,7 +2,7 @@ · 1 unmodified line | ||
| 2 | 2 | #include <sys/mman.h> |
| 3 | 3 | #include <mmap_test.h> |
| 4 | 4 | |
| 5 | −#include "../../kselftest_harness.h" | |
| 5 | +#include "kselftest_harness.h" | |
| 6 | 6 | |
| 7 | 7 | TEST(infinite_rlimit) |
| 8 | 8 | { |
| @@ -2,7 +2,7 @@ · 1 unmodified line | ||
| 2 | 2 | #include <sys/mman.h> |
| 3 | 3 | #include <mmap_test.h> |
| 4 | 4 | |
| 5 | −#include "../../kselftest_harness.h" | |
| 5 | +#include "kselftest_harness.h" | |
| 6 | 6 | |
| 7 | 7 | TEST(default_rlimit) |
| 8 | 8 | { |
| @@ -5,7 +5,7 @@ · 4 unmodified lines | ||
| 5 | 5 | #include <sys/resource.h> |
| 6 | 6 | #include <stddef.h> |
| 7 | 7 | #include <strings.h> |
| 8 | −#include "../../kselftest_harness.h" | |
| 8 | +#include "kselftest_harness.h" | |
| 9 | 9 | |
| 10 | 10 | #define TOP_DOWN 0 |
| 11 | 11 | #define BOTTOM_UP 1 |
| @@ -4,7 +4,7 @@ · 3 unmodified lines | ||
| 4 | 4 | #include <stdlib.h> |
| 5 | 5 | #include <ucontext.h> |
| 6 | 6 | #include <linux/ptrace.h> |
| 7 | −#include "../../kselftest_harness.h" | |
| 7 | +#include "kselftest_harness.h" | |
| 8 | 8 | |
| 9 | 9 | #define RISCV_V_MAGIC 0x53465457 |
| 10 | 10 | #define DEFAULT_VALUE 2 |
| @@ -1,6 +1,6 @@ | ||
| 1 | 1 | // SPDX-License-Identifier: GPL-2.0-only |
| 2 | 2 | |
| 3 | −#include "../../kselftest_harness.h" | |
| 3 | +#include "kselftest_harness.h" | |
| 4 | 4 | #include "v_helpers.h" |
| 5 | 5 | |
| 6 | 6 | #define NEXT_PROGRAM "./v_exec_initval_nolibc" |
| @@ -6,7 +6,7 @@ · 5 unmodified lines | ||
| 6 | 6 | #include <sys/types.h> |
| 7 | 7 | #include <stdlib.h> |
| 8 | 8 | |
| 9 | −#include "../../kselftest_harness.h" | |
| 9 | +#include "kselftest_harness.h" | |
| 10 | 10 | #include "v_helpers.h" |
| 11 | 11 | |
| 12 | 12 | #define NEXT_PROGRAM "./vstate_exec_nolibc" |
| @@ -9,7 +9,7 @@ · 8 unmodified lines | ||
| 9 | 9 | #include <string.h> |
| 10 | 10 | #include <stddef.h> |
| 11 | 11 | |
| 12 | −#include "../kselftest.h" | |
| 12 | +#include "kselftest.h" | |
| 13 | 13 | #include "rseq.h" |
| 14 | 14 | |
| 15 | 15 | #ifdef BUILDOPT_RSEQ_PERCPU_MM_CID |
| @@ -33,7 +33,7 @@ · 32 unmodified lines | ||
| 33 | 33 | |
| 34 | 34 | #include <linux/compiler.h> |
| 35 | 35 | |
| 36 | −#include "../kselftest.h" | |
| 36 | +#include "kselftest.h" | |
| 37 | 37 | #include "rseq.h" |
| 38 | 38 | |
| 39 | 39 | /* |
| @@ -16,7 +16,7 @@ · 15 unmodified lines | ||
| 16 | 16 | #include <time.h> |
| 17 | 17 | #include <unistd.h> |
| 18 | 18 | |
| 19 | −#include "../kselftest_harness.h" | |
| 19 | +#include "kselftest_harness.h" | |
| 20 | 20 | |
| 21 | 21 | #define NUM_UIE 3 |
| 22 | 22 | #define ALARM_DELTA 3 |
| @@ -20,7 +20,7 @@ · 19 unmodified lines | ||
| 20 | 20 | #include <sys/syscall.h> |
| 21 | 21 | #include <sys/types.h> |
| 22 | 22 | |
| 23 | −#include "../kselftest.h" | |
| 23 | +#include "kselftest.h" | |
| 24 | 24 | |
| 25 | 25 | unsigned long long timing(clockid_t clk_id, unsigned long long samples) |
| 26 | 26 | { |
| @@ -54,7 +54,7 @@ · 53 unmodified lines | ||
| 54 | 54 | #include <sys/syscall.h> |
| 55 | 55 | #include <poll.h> |
| 56 | 56 | |
| 57 | −#include "../kselftest_harness.h" | |
| 57 | +#include "kselftest_harness.h" | |
| 58 | 58 | #include "../clone3/clone3_selftests.h" |
| 59 | 59 | |
| 60 | 60 | /* Attempt to de-conflict with the selftests tree. */ |
| @@ -18,7 +18,7 @@ · 17 unmodified lines | ||
| 18 | 18 | #include <sys/types.h> |
| 19 | 19 | #include <sys/auxv.h> |
| 20 | 20 | #include "defines.h" |
| 21 | −#include "../kselftest_harness.h" | |
| 21 | +#include "kselftest_harness.h" | |
| 22 | 22 | #include "main.h" |
| 23 | 23 | |
| 24 | 24 | static const uint64_t MAGIC = 0x1122334455667788ULL; |
| @@ -39,7 +39,7 @@ · 38 unmodified lines | ||
| 39 | 39 | #include <signal.h> |
| 40 | 40 | #include <ucontext.h> |
| 41 | 41 | |
| 42 | −#include "../kselftest.h" | |
| 42 | +#include "kselftest.h" | |
| 43 | 43 | |
| 44 | 44 | void handler_verify_ucontext(int signo, siginfo_t *info, void *uc) |
| 45 | 45 | { |
| @@ -19,7 +19,7 @@ · 18 unmodified lines | ||
| 19 | 19 | #include <errno.h> |
| 20 | 20 | #include <sys/auxv.h> |
| 21 | 21 | |
| 22 | −#include "../kselftest.h" | |
| 22 | +#include "kselftest.h" | |
| 23 | 23 | #include "current_stack_pointer.h" |
| 24 | 24 | |
| 25 | 25 | #ifndef SS_AUTODISARM |
| @@ -16,7 +16,7 @@ · 15 unmodified lines | ||
| 16 | 16 | #include <sys/stat.h> |
| 17 | 17 | #include <unistd.h> |
| 18 | 18 | |
| 19 | −#include "../../kselftest.h" | |
| 19 | +#include "kselftest.h" | |
| 20 | 20 | |
| 21 | 21 | #define DEBUG_LEVEL_1_BIT (0x0001) |
| 22 | 22 | #define DEBUG_LEVEL_2_BIT (0x0002) |
| @@ -34,7 +34,7 @@ · 33 unmodified lines | ||
| 34 | 34 | #include <errno.h> |
| 35 | 35 | #include <string.h> |
| 36 | 36 | |
| 37 | −#include "../kselftest.h" | |
| 37 | +#include "kselftest.h" | |
| 38 | 38 | #include "synctest.h" |
| 39 | 39 | |
| 40 | 40 | static int run_test(int (*test)(void), char *name) |
| @@ -14,7 +14,7 @@ · 13 unmodified lines | ||
| 14 | 14 | #include <stdlib.h> |
| 15 | 15 | |
| 16 | 16 | #include <asm/unistd.h> |
| 17 | −#include "../kselftest_harness.h" | |
| 17 | +#include "kselftest_harness.h" | |
| 18 | 18 | |
| 19 | 19 | #ifndef PR_SET_SYSCALL_USER_DISPATCH |
| 20 | 20 | # define PR_SET_SYSCALL_USER_DISPATCH 59 |
| @@ -13,7 +13,7 @@ · 12 unmodified lines | ||
| 13 | 13 | #include <fcntl.h> |
| 14 | 14 | |
| 15 | 15 | #include <linux/tdx-guest.h> |
| 16 | −#include "../kselftest_harness.h" | |
| 16 | +#include "kselftest_harness.h" | |
| 17 | 17 | |
| 18 | 18 | #define TDX_GUEST_DEVNAME "/dev/tdx_guest" |
| 19 | 19 | #define HEX_DUMP_SIZE 8 |
| @@ -7,7 +7,7 @@ · 6 unmodified lines | ||
| 7 | 7 | #include <stdlib.h> |
| 8 | 8 | #include <stdbool.h> |
| 9 | 9 | |
| 10 | −#include "../kselftest.h" | |
| 10 | +#include "kselftest.h" | |
| 11 | 11 | |
| 12 | 12 | #ifndef CLONE_NEWTIME |
| 13 | 13 | # define CLONE_NEWTIME 0x00000080 |
| @@ -24,7 +24,7 @@ · 23 unmodified lines | ||
| 24 | 24 | #include <time.h> |
| 25 | 25 | #include <include/vdso/time64.h> |
| 26 | 26 | |
| 27 | −#include "../kselftest.h" | |
| 27 | +#include "kselftest.h" | |
| 28 | 28 | |
| 29 | 29 | #define MILLION 1000000 |
| 30 | 30 | |
| @@ -30,7 +30,7 @@ · 29 unmodified lines | ||
| 30 | 30 | #include <pthread.h> |
| 31 | 31 | #include <include/vdso/time64.h> |
| 32 | 32 | #include <errno.h> |
| 33 | −#include "../kselftest.h" | |
| 33 | +#include "kselftest.h" | |
| 34 | 34 | |
| 35 | 35 | #define UNREASONABLE_LAT (NSEC_PER_SEC * 5) /* hopefully we resume in 5 secs */ |
| 36 | 36 | |
| @@ -28,7 +28,7 @@ · 27 unmodified lines | ||
| 28 | 28 | #include <sys/time.h> |
| 29 | 29 | #include <sys/timex.h> |
| 30 | 30 | #include <time.h> |
| 31 | −#include "../kselftest.h" | |
| 31 | +#include "kselftest.h" | |
| 32 | 32 | |
| 33 | 33 | int change_skew_test(int ppm) |
| 34 | 34 | { |
| @@ -34,7 +34,7 @@ · 33 unmodified lines | ||
| 34 | 34 | #include <sys/wait.h> |
| 35 | 35 | #include <time.h> |
| 36 | 36 | #include <unistd.h> |
| 37 | −#include "../kselftest.h" | |
| 37 | +#include "kselftest.h" | |
| 38 | 38 | |
| 39 | 39 | |
| 40 | 40 | int get_clocksources(char list[][30]) |
| @@ -15,7 +15,7 @@ · 14 unmodified lines | ||
| 15 | 15 | #include <time.h> |
| 16 | 16 | #include <unistd.h> |
| 17 | 17 | |
| 18 | −#include "../kselftest.h" | |
| 18 | +#include "kselftest.h" | |
| 19 | 19 | |
| 20 | 20 | #define SAMPLES 100 |
| 21 | 21 | #define SAMPLE_READINGS 10 |
| @@ -29,7 +29,7 @@ · 28 unmodified lines | ||
| 29 | 29 | #include <string.h> |
| 30 | 30 | #include <signal.h> |
| 31 | 31 | #include <include/vdso/time64.h> |
| 32 | −#include "../kselftest.h" | |
| 32 | +#include "kselftest.h" | |
| 33 | 33 | |
| 34 | 34 | /* CLOCK_HWSPECIFIC == CLOCK_SGI_CYCLE (Deprecated) */ |
| 35 | 35 | #define CLOCK_HWSPECIFIC 10 |
| @@ -49,7 +49,7 @@ · 48 unmodified lines | ||
| 49 | 49 | #include <signal.h> |
| 50 | 50 | #include <unistd.h> |
| 51 | 51 | #include <include/vdso/time64.h> |
| 52 | −#include "../kselftest.h" | |
| 52 | +#include "kselftest.h" | |
| 53 | 53 | |
| 54 | 54 | #define CLOCK_TAI 11 |
| 55 | 55 | |
| @@ -22,7 +22,7 @@ · 21 unmodified lines | ||
| 22 | 22 | #include <sys/timex.h> |
| 23 | 23 | #include <string.h> |
| 24 | 24 | #include <signal.h> |
| 25 | −#include "../kselftest.h" | |
| 25 | +#include "kselftest.h" | |
| 26 | 26 | |
| 27 | 27 | /* clear NTP time_status & time_state */ |
| 28 | 28 | int clear_time_state(void) |
| @@ -30,7 +30,7 @@ · 29 unmodified lines | ||
| 30 | 30 | #include <errno.h> |
| 31 | 31 | #include <mqueue.h> |
| 32 | 32 | #include <include/vdso/time64.h> |
| 33 | −#include "../kselftest.h" | |
| 33 | +#include "kselftest.h" | |
| 34 | 34 | |
| 35 | 35 | |
| 36 | 36 | #define TARGET_TIMEOUT 100000000 /* 100ms in nanoseconds */ |
| @@ -28,7 +28,7 @@ · 27 unmodified lines | ||
| 28 | 28 | #include <string.h> |
| 29 | 29 | #include <signal.h> |
| 30 | 30 | #include <include/vdso/time64.h> |
| 31 | −#include "../kselftest.h" | |
| 31 | +#include "kselftest.h" | |
| 32 | 32 | |
| 33 | 33 | /* CLOCK_HWSPECIFIC == CLOCK_SGI_CYCLE (Deprecated) */ |
| 34 | 34 | #define CLOCK_HWSPECIFIC 10 |
| @@ -25,7 +25,7 @@ · 24 unmodified lines | ||
| 25 | 25 | #include <string.h> |
| 26 | 26 | #include <signal.h> |
| 27 | 27 | #include <include/vdso/time64.h> |
| 28 | −#include "../kselftest.h" | |
| 28 | +#include "kselftest.h" | |
| 29 | 29 | |
| 30 | 30 | #define UNRESONABLE_LATENCY 40000000 /* 40ms in nanosecs */ |
| 31 | 31 | |
| @@ -20,7 +20,7 @@ · 19 unmodified lines | ||
| 20 | 20 | #include <pthread.h> |
| 21 | 21 | #include <stdbool.h> |
| 22 | 22 | |
| 23 | −#include "../kselftest.h" | |
| 23 | +#include "kselftest.h" | |
| 24 | 24 | |
| 25 | 25 | #define DELAY 2 |
| 26 | 26 | |
| @@ -26,7 +26,7 @@ · 25 unmodified lines | ||
| 26 | 26 | #include <sys/timex.h> |
| 27 | 27 | #include <time.h> |
| 28 | 28 | #include <include/vdso/time64.h> |
| 29 | −#include "../kselftest.h" | |
| 29 | +#include "kselftest.h" | |
| 30 | 30 | |
| 31 | 31 | #define shift_right(x, s) ({ \ |
| 32 | 32 | __typeof__(x) __x = (x); \ |
| @@ -18,7 +18,7 @@ · 17 unmodified lines | ||
| 18 | 18 | #include <stdlib.h> |
| 19 | 19 | #include <errno.h> |
| 20 | 20 | |
| 21 | −#include "../kselftest.h" | |
| 21 | +#include "kselftest.h" | |
| 22 | 22 | |
| 23 | 23 | /* |
| 24 | 24 | * This expects the new RTC class driver framework, working with |
| @@ -28,7 +28,7 @@ · 27 unmodified lines | ||
| 28 | 28 | #include <time.h> |
| 29 | 29 | #include <sys/time.h> |
| 30 | 30 | #include <include/vdso/time64.h> |
| 31 | −#include "../kselftest.h" | |
| 31 | +#include "kselftest.h" | |
| 32 | 32 | |
| 33 | 33 | #define KTIME_MAX ((long long)~((unsigned long long)1 << 63)) |
| 34 | 34 | #define KTIME_SEC_MAX (KTIME_MAX / NSEC_PER_SEC) |
| @@ -23,7 +23,7 @@ · 22 unmodified lines | ||
| 23 | 23 | #include <string.h> |
| 24 | 24 | #include <signal.h> |
| 25 | 25 | #include <unistd.h> |
| 26 | −#include "../kselftest.h" | |
| 26 | +#include "kselftest.h" | |
| 27 | 27 | |
| 28 | 28 | int set_tai(int offset) |
| 29 | 29 | { |
| @@ -29,7 +29,7 @@ · 28 unmodified lines | ||
| 29 | 29 | #include <stdlib.h> |
| 30 | 30 | #include <pthread.h> |
| 31 | 31 | #include <include/vdso/time64.h> |
| 32 | −#include "../kselftest.h" | |
| 32 | +#include "kselftest.h" | |
| 33 | 33 | |
| 34 | 34 | /* CLOCK_HWSPECIFIC == CLOCK_SGI_CYCLE (Deprecated) */ |
| 35 | 35 | #define CLOCK_HWSPECIFIC 10 |
| @@ -23,7 +23,7 @@ · 22 unmodified lines | ||
| 23 | 23 | #include <string.h> |
| 24 | 24 | #include <signal.h> |
| 25 | 25 | #include <unistd.h> |
| 26 | −#include "../kselftest.h" | |
| 26 | +#include "kselftest.h" | |
| 27 | 27 | |
| 28 | 28 | int set_tz(int min, int dst) |
| 29 | 29 | { |
| @@ -34,7 +34,7 @@ · 33 unmodified lines | ||
| 34 | 34 | #include <fcntl.h> |
| 35 | 35 | #include <string.h> |
| 36 | 36 | #include <sys/wait.h> |
| 37 | −#include "../kselftest.h" | |
| 37 | +#include "kselftest.h" | |
| 38 | 38 | |
| 39 | 39 | int main(int argc, char **argv) |
| 40 | 40 | { |
| @@ -21,7 +21,7 @@ · 20 unmodified lines | ||
| 21 | 21 | #include <stdlib.h> |
| 22 | 22 | #include <sys/time.h> |
| 23 | 23 | #include <pthread.h> |
| 24 | −#include "../kselftest.h" | |
| 24 | +#include "kselftest.h" | |
| 25 | 25 | |
| 26 | 26 | /* serializes shared list access */ |
| 27 | 27 | pthread_mutex_t list_lock = PTHREAD_MUTEX_INITIALIZER; |
| @@ -30,7 +30,7 @@ · 29 unmodified lines | ||
| 30 | 30 | #include <signal.h> |
| 31 | 31 | #include <unistd.h> |
| 32 | 32 | #include <include/vdso/time64.h> |
| 33 | −#include "../kselftest.h" | |
| 33 | +#include "kselftest.h" | |
| 34 | 34 | |
| 35 | 35 | #define ADJ_SETOFFSET 0x0100 |
| 36 | 36 | |
| @@ -23,7 +23,7 @@ · 22 unmodified lines | ||
| 23 | 23 | #include <sys/mount.h> |
| 24 | 24 | #include <unistd.h> |
| 25 | 25 | |
| 26 | −#include "../kselftest.h" | |
| 26 | +#include "kselftest.h" | |
| 27 | 27 | |
| 28 | 28 | int main(void) |
| 29 | 29 | { |
| @@ -9,7 +9,7 @@ · 8 unmodified lines | ||
| 9 | 9 | #include <unistd.h> |
| 10 | 10 | #include <linux/limits.h> |
| 11 | 11 | |
| 12 | −#include "../kselftest.h" | |
| 12 | +#include "kselftest.h" | |
| 13 | 13 | |
| 14 | 14 | #define MIN_TTY_PATH_LEN 8 |
| 15 | 15 | |
| @@ -19,7 +19,7 @@ · 18 unmodified lines | ||
| 19 | 19 | #include <sys/wait.h> |
| 20 | 20 | #include <unistd.h> |
| 21 | 21 | |
| 22 | −#include "../kselftest_harness.h" | |
| 22 | +#include "kselftest_harness.h" | |
| 23 | 23 | |
| 24 | 24 | #define __DEV_FULL "/sys/devices/virtual/mem/full/uevent" |
| 25 | 25 | #define __UEVENT_BUFFER_SIZE (2048 * 2) |
| @@ -20,7 +20,7 @@ · 19 unmodified lines | ||
| 20 | 20 | #include <string.h> |
| 21 | 21 | #include <asm/unistd.h> |
| 22 | 22 | |
| 23 | −#include "../kselftest_harness.h" | |
| 23 | +#include "kselftest_harness.h" | |
| 24 | 24 | #include "user_events_selftests.h" |
| 25 | 25 | |
| 26 | 26 | const char *data_file = "/sys/kernel/tracing/user_events_data"; |
| @@ -14,7 +14,7 @@ · 13 unmodified lines | ||
| 14 | 14 | #include <sys/stat.h> |
| 15 | 15 | #include <unistd.h> |
| 16 | 16 | |
| 17 | −#include "../kselftest_harness.h" | |
| 17 | +#include "kselftest_harness.h" | |
| 18 | 18 | #include "user_events_selftests.h" |
| 19 | 19 | |
| 20 | 20 | const char *dyn_file = "/sys/kernel/tracing/dynamic_events"; |
| @@ -15,7 +15,7 @@ · 14 unmodified lines | ||
| 15 | 15 | #include <sys/uio.h> |
| 16 | 16 | #include <unistd.h> |
| 17 | 17 | |
| 18 | −#include "../kselftest_harness.h" | |
| 18 | +#include "kselftest_harness.h" | |
| 19 | 19 | #include "user_events_selftests.h" |
| 20 | 20 | |
| 21 | 21 | const char *data_file = "/sys/kernel/tracing/user_events_data"; |
| @@ -16,7 +16,7 @@ · 15 unmodified lines | ||
| 16 | 16 | #include <unistd.h> |
| 17 | 17 | #include <asm/unistd.h> |
| 18 | 18 | |
| 19 | −#include "../kselftest_harness.h" | |
| 19 | +#include "kselftest_harness.h" | |
| 20 | 20 | #include "user_events_selftests.h" |
| 21 | 21 | |
| 22 | 22 | const char *data_file = "/sys/kernel/tracing/user_events_data"; |
| @@ -9,7 +9,7 @@ · 8 unmodified lines | ||
| 9 | 9 | #include <unistd.h> |
| 10 | 10 | #include <errno.h> |
| 11 | 11 | |
| 12 | −#include "../kselftest.h" | |
| 12 | +#include "kselftest.h" | |
| 13 | 13 | |
| 14 | 14 | static inline void tracefs_unmount(void) |
| 15 | 15 | { |
| @@ -18,7 +18,7 @@ · 17 unmodified lines | ||
| 18 | 18 | #include <unistd.h> |
| 19 | 19 | #include <sys/syscall.h> |
| 20 | 20 | |
| 21 | −#include "../kselftest.h" | |
| 21 | +#include "kselftest.h" | |
| 22 | 22 | #include "vdso_config.h" |
| 23 | 23 | #include "vdso_call.h" |
| 24 | 24 | #include "parse_vdso.h" |
| @@ -10,7 +10,7 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <string.h> |
| 11 | 11 | #include <stdint.h> |
| 12 | 12 | #include <stdbool.h> |
| 13 | −#include "../kselftest.h" | |
| 13 | +#include "kselftest.h" | |
| 14 | 14 | |
| 15 | 15 | #if defined(__aarch64__) |
| 16 | 16 | static bool cpu_has_capabilities(void) |
| @@ -21,7 +21,7 @@ · 20 unmodified lines | ||
| 21 | 21 | |
| 22 | 22 | #include "vdso_config.h" |
| 23 | 23 | #include "vdso_call.h" |
| 24 | −#include "../kselftest.h" | |
| 24 | +#include "kselftest.h" | |
| 25 | 25 | |
| 26 | 26 | static const char **name; |
| 27 | 27 | |
| @@ -11,7 +11,7 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <sys/auxv.h> |
| 12 | 12 | #include <sys/time.h> |
| 13 | 13 | |
| 14 | −#include "../kselftest.h" | |
| 14 | +#include "kselftest.h" | |
| 15 | 15 | #include "parse_vdso.h" |
| 16 | 16 | #include "vdso_config.h" |
| 17 | 17 | #include "vdso_call.h" |
| @@ -23,7 +23,7 @@ · 22 unmodified lines | ||
| 23 | 23 | #include <linux/random.h> |
| 24 | 24 | #include <linux/ptrace.h> |
| 25 | 25 | |
| 26 | −#include "../kselftest.h" | |
| 26 | +#include "kselftest.h" | |
| 27 | 27 | #include "parse_vdso.h" |
| 28 | 28 | #include "vdso_config.h" |
| 29 | 29 | #include "vdso_call.h" |
| @@ -16,7 +16,7 @@ · 15 unmodified lines | ||
| 16 | 16 | #include <sys/time.h> |
| 17 | 17 | #endif |
| 18 | 18 | |
| 19 | −#include "../kselftest.h" | |
| 19 | +#include "kselftest.h" | |
| 20 | 20 | #include "parse_vdso.h" |
| 21 | 21 | #include "vdso_config.h" |
| 22 | 22 | #include "vdso_call.h" |
| @@ -19,7 +19,7 @@ · 18 unmodified lines | ||
| 19 | 19 | #include <linux/types.h> |
| 20 | 20 | #include <linux/vfio.h> |
| 21 | 21 | |
| 22 | −#include "../../../kselftest.h" | |
| 22 | +#include "kselftest.h" | |
| 23 | 23 | #include <libvfio.h> |
| 24 | 24 | |
| 25 | 25 | #define PCI_SYSFS_PATH "/sys/bus/pci/devices" |
| @@ -1,5 +1,5 @@ | ||
| 1 | 1 | // SPDX-License-Identifier: GPL-2.0-only |
| 2 | −#include "../../../kselftest.h" | |
| 2 | +#include "kselftest.h" | |
| 3 | 3 | #include <libvfio.h> |
| 4 | 4 | |
| 5 | 5 | #ifdef __x86_64__ |
| @@ -12,7 +12,7 @@ · 11 unmodified lines | ||
| 12 | 12 | |
| 13 | 13 | #include <libvfio.h> |
| 14 | 14 | |
| 15 | −#include "../kselftest_harness.h" | |
| 15 | +#include "kselftest_harness.h" | |
| 16 | 16 | |
| 17 | 17 | static const char *device_bdf; |
| 18 | 18 | |
| @@ -11,7 +11,7 @@ · 10 unmodified lines | ||
| 11 | 11 | #include <unistd.h> |
| 12 | 12 | |
| 13 | 13 | #include <libvfio.h> |
| 14 | −#include "../kselftest_harness.h" | |
| 14 | +#include "kselftest_harness.h" | |
| 15 | 15 | |
| 16 | 16 | static const char iommu_dev_path[] = "/dev/iommu"; |
| 17 | 17 | static const char *cdev_path; |
| @@ -12,7 +12,7 @@ · 11 unmodified lines | ||
| 12 | 12 | |
| 13 | 13 | #include <libvfio.h> |
| 14 | 14 | |
| 15 | −#include "../kselftest_harness.h" | |
| 15 | +#include "kselftest_harness.h" | |
| 16 | 16 | |
| 17 | 17 | static const char *device_bdf; |
| 18 | 18 | |
| @@ -7,7 +7,7 @@ · 6 unmodified lines | ||
| 7 | 7 | |
| 8 | 8 | #include <libvfio.h> |
| 9 | 9 | |
| 10 | −#include "../kselftest_harness.h" | |
| 10 | +#include "kselftest_harness.h" | |
| 11 | 11 | |
| 12 | 12 | static const char *device_bdf; |
| 13 | 13 | |
| @@ -81,7 +81,7 @@ CONFIG_WQ_WATCHDOG=y · 80 unmodified lines | ||
| 81 | 81 | CONFIG_DETECT_HUNG_TASK=y |
| 82 | 82 | CONFIG_BOOTPARAM_HARDLOCKUP_PANIC=y |
| 83 | 83 | CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC=y |
| 84 | −CONFIG_BOOTPARAM_HUNG_TASK_PANIC=y | |
| 84 | +CONFIG_BOOTPARAM_HUNG_TASK_PANIC=1 | |
| 85 | 85 | CONFIG_PANIC_TIMEOUT=-1 |
| 86 | 86 | CONFIG_STACKTRACE=y |
| 87 | 87 | CONFIG_EARLY_PRINTK=y |
| @@ -17,7 +17,7 @@ · 16 unmodified lines | ||
| 17 | 17 | #include <stdint.h> |
| 18 | 18 | #include <sys/wait.h> |
| 19 | 19 | |
| 20 | −#include "../kselftest.h" /* For __cpuid_count() */ | |
| 20 | +#include "kselftest.h" /* For __cpuid_count() */ | |
| 21 | 21 | #include "helpers.h" |
| 22 | 22 | |
| 23 | 23 | static inline int xsave_enabled(void) |
| @@ -7,7 +7,7 @@ · 6 unmodified lines | ||
| 7 | 7 | |
| 8 | 8 | #include <asm/processor-flags.h> |
| 9 | 9 | |
| 10 | −#include "../kselftest.h" | |
| 10 | +#include "kselftest.h" | |
| 11 | 11 | |
| 12 | 12 | static inline unsigned long get_eflags(void) |
| 13 | 13 | { |
| @@ -18,7 +18,7 @@ · 17 unmodified lines | ||
| 18 | 18 | |
| 19 | 19 | #include <sys/uio.h> |
| 20 | 20 | #include <linux/io_uring.h> |
| 21 | −#include "../kselftest.h" | |
| 21 | +#include "kselftest.h" | |
| 22 | 22 | |
| 23 | 23 | #ifndef __x86_64__ |
| 24 | 24 | # error This test is 64-bit only |
| @@ -25,7 +25,7 @@ · 24 unmodified lines | ||
| 25 | 25 | #include <sys/mman.h> |
| 26 | 26 | |
| 27 | 27 | #include <linux/ptrace.h> |
| 28 | −#include "../kselftest.h" | |
| 28 | +#include "kselftest.h" | |
| 29 | 29 | |
| 30 | 30 | /* Common system call numbers */ |
| 31 | 31 | #define SYS_READ 0 |
| @@ -20,7 +20,7 @@ · 19 unmodified lines | ||
| 20 | 20 | #include <sys/auxv.h> |
| 21 | 21 | #include <sys/syscall.h> |
| 22 | 22 | #include <sys/wait.h> |
| 23 | −#include "../kselftest.h" | |
| 23 | +#include "kselftest.h" | |
| 24 | 24 | |
| 25 | 25 | #define PAGE_SIZE 4096 |
| 26 | 26 | |
| @@ -21,7 +21,7 @@ · 20 unmodified lines | ||
| 21 | 21 | #include <sys/uio.h> |
| 22 | 22 | |
| 23 | 23 | #include "helpers.h" |
| 24 | −#include "../kselftest.h" | |
| 24 | +#include "kselftest.h" | |
| 25 | 25 | |
| 26 | 26 | #ifdef __x86_64__ |
| 27 | 27 | #define TOTAL_TESTS 13 |
| @@ -4,7 +4,7 @@ · 3 unmodified lines | ||
| 4 | 4 | |
| 5 | 5 | #include <stdint.h> |
| 6 | 6 | |
| 7 | −#include "../kselftest.h" | |
| 7 | +#include "kselftest.h" | |
| 8 | 8 | |
| 9 | 9 | #define XSAVE_HDR_OFFSET 512 |
| 10 | 10 | #define XSAVE_HDR_SIZE 64 |