.. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0).. See the bottom of this file for additional redistribution information. Reporting issues++++++++++++++++ The short guide (aka TL;DR)=========================== Are you facing a regression with vanilla kernels from the same stable orlongterm series? One still supported? Then search the `LKML<https://lore.kernel.org/lkml/>`_ and the `Linux stable mailing list<https://lore.kernel.org/stable/>`_ archives for matching reports to join. Ifyou don't find any, install `the latest release from that series<https://kernel.org/>`_. If it still shows the issue, report it to the stablemailing list (stable@vger.kernel.org) and CC the regressions list(regressions@lists.linux.dev); ideally also CC the maintainer and the mailinglist for the subsystem in question. In all other cases try your best guess which kernel part might be causing theissue. Check the :ref:`MAINTAINERS <maintainers>` file for how its developersexpect to be told about problems, which most of the time will be by email with amailing list in CC. Check the destination's archives for matching reports;search the `LKML <https://lore.kernel.org/lkml/>`_ and the web, too. If youdon't find any to join, install `the latest mainline kernel<https://kernel.org/>`_. If the issue is present there, send a report. The issue was fixed there, but you would like to see it resolved in a stillsupported stable or longterm series as well? Then install its latest release.If it shows the problem, search for the change that fixed it in mainline andcheck if backporting is in the works or was discarded; if it's neither, askthose who handled the change for it. **General remarks**: When installing and testing a kernel as outlined above,ensure it's vanilla (IOW: not patched and not using add-on modules). Also makesure it's built and running in a healthy environment and not already taintedbefore the issue occurs. If you are facing multiple issues with the Linux kernel at once, report eachseparately. While writing your report, include all information relevant to theissue, like the kernel and the distro used. In case of a regression, CC theregressions mailing list (regressions@lists.linux.dev) to your report. Also tryto pinpoint the culprit with a bisection; if you succeed, include itscommit-id and CC everyone in the sign-off-by chain. Once the report is out, answer any questions that come up and help where youcan. That includes keeping the ball rolling by occasionally retesting with newerreleases and sending a status update afterwards. .. Note: If you see this note, you are reading the text's source file. You might want to switch to a rendered version: It makes it a lot easier to read and navigate this document -- especially when you want to look something up in the reference section, then jump back to where you left off... Find the latest rendered version of this text here: https://docs.kernel.org/admin-guide/reporting-issues.html Step-by-step guide how to report issues to the kernel maintainers================================================================= The above TL;DR outlines roughly how to report issues to the Linux kerneldevelopers. It might be all that's needed for people already familiar withreporting issues to Free/Libre & Open Source Software (FLOSS) projects. Foreveryone else there is this section. It is more detailed and uses astep-by-step approach. It still tries to be brief for readability and leavesout a lot of details; those are described below the step-by-step guide in areference section, which explains each of the steps in more detail. Note: this section covers a few more aspects than the TL;DR and does things ina slightly different order. That's in your interest, to make sure you noticeearly if an issue that looks like a Linux kernel problem is actually caused bysomething else. These steps thus help to ensure the time you invest in thisprocess won't feel wasted in the end: * Are you facing an issue with a Linux kernel a hardware or software vendor provided? Then in almost all cases you are better off to stop reading this document and reporting the issue to your vendor instead, unless you are willing to install the latest Linux version yourself. Be aware the latter will often be needed anyway to hunt down and fix issues. * Perform a rough search for existing reports with your favorite internet search engine; additionally, check the archives of the `Linux Kernel Mailing List (LKML) <https://lore.kernel.org/lkml/>`_. If you find matching reports, join the discussion instead of sending a new one. * See if the issue you are dealing with qualifies as regression, security issue, or a really severe problem: those are 'issues of high priority' that need special handling in some steps that are about to follow. * Make sure it's not the kernel's surroundings that are causing the issue you face. * Create a fresh backup and put system repair and restore tools at hand. * Ensure your system does not enhance its kernels by building additional kernel modules on-the-fly, which solutions like DKMS might be doing locally without your knowledge. * Check if your kernel was 'tainted' when the issue occurred, as the event that made the kernel set this flag might be causing the issue you face. * Write down coarsely how to reproduce the issue. If you deal with multiple issues at once, create separate notes for each of them and make sure they work independently on a freshly booted system. That's needed, as each issue needs to get reported to the kernel developers separately, unless they are strongly entangled. * If you are facing a regression within a stable or longterm version line (say something broke when updating from 5.10.4 to 5.10.5), scroll down to 'Dealing with regressions within a stable and longterm kernel line'. * Locate the driver or kernel subsystem that seems to be causing the issue. Find out how and where its developers expect reports. Note: most of the time this won't be bugzilla.kernel.org, as issues typically need to be sent by mail to a maintainer and a public mailing list. * Search the archives of the bug tracker or mailing list in question thoroughly for reports that might match your issue. If you find anything, join the discussion instead of sending a new report. After these preparations you'll now enter the main part: * Unless you are already running the latest 'mainline' Linux kernel, better go and install it for the reporting process. Testing and reporting with the latest 'stable' Linux can be an acceptable alternative in some situations; during the merge window that actually might be even the best approach, but in that development phase it can be an even better idea to suspend your efforts for a few days anyway. Whatever version you choose, ideally use a 'vanilla' build. Ignoring all of this advice will dramatically increase the risk your report will be rejected or ignored. * Ensure the kernel you just installed does not 'taint' itself when running. * Reproduce the issue with the kernel you just installed. If it doesn't show up there, scroll down to the instructions for issues only happening with stable and longterm kernels. * Optimize your notes: try to find and write the most straightforward way to reproduce your issue. Make sure the end result has all the important details, and at the same time is easy to read and understand for others that hear about it for the first time. And if you learned something in this process, consider searching again for existing reports about the issue. * If your failure involves a 'panic', 'Oops', 'warning', or 'BUG', consider decoding the kernel log to find the line of code that triggered the error. * If your problem is a regression, try to narrow down when the issue was introduced as much as possible. * Start to compile the report by writing a detailed description about the issue. Always mention a few things: the latest kernel version you installed for reproducing, the Linux Distribution used, and your notes on how to reproduce the issue. Ideally, make the kernel's build configuration (.config) and the output from ``dmesg`` available somewhere on the net and link to it. Include or upload all other information that might be relevant, like the output/screenshot of an Oops or the output from ``lspci``. Once you wrote this main part, insert a normal length paragraph on top of it outlining the issue and the impact quickly. On top of this add one sentence that briefly describes the problem and gets people to read on. Now give the thing a descriptive title or subject that yet again is shorter. Then you're ready to send or file the report like the MAINTAINERS file told you, unless you are dealing with one of those 'issues of high priority': they need special care which is explained in 'Special handling for high priority issues' below. * Wait for reactions and keep the thing rolling until you can accept the outcome in one way or the other. Thus react publicly and in a timely manner to any inquiries. Test proposed fixes. Do proactive testing: retest with at least every first release candidate (RC) of a new mainline version and report your results. Send friendly reminders if things stall. And try to help yourself, if you don't get any help or if it's unsatisfying. Reporting regressions within a stable and longterm kernel line-------------------------------------------------------------- This subsection is for you, if you followed above process and got sent here atthe point about regression within a stable or longterm kernel version line. Youface one of those if something breaks when updating from 5.10.4 to 5.10.5 (aswitch from 5.9.15 to 5.10.5 does not qualify). The developers want to fix suchregressions as quickly as possible, hence there is a streamlined process toreport them: * Check if the kernel developers still maintain the Linux kernel version line you care about: go to the `front page of kernel.org <https://kernel.org/>`_ and make sure it mentions the latest release of the particular version line without an '[EOL]' tag. * Check the archives of the `Linux stable mailing list <https://lore.kernel.org/stable/>`_ for existing reports. * Install the latest release from the particular version line as a vanilla kernel. Ensure this kernel is not tainted and still shows the problem, as the issue might have already been fixed there. If you first noticed the problem with a vendor kernel, check a vanilla build of the last version known to work performs fine as well. * Send a short problem report to the Linux stable mailing list (stable@vger.kernel.org) and CC the Linux regressions mailing list (regressions@lists.linux.dev); if you suspect the cause in a particular subsystem, CC its maintainer and its mailing list. Roughly describe the issue and ideally explain how to reproduce it. Mention the first version that shows the problem and the last version that's working fine. Then wait for further instructions. The reference section below explains each of these steps in more detail. Reporting issues only occurring in older kernel version lines------------------------------------------------------------- This subsection is for you, if you tried the latest mainline kernel as outlinedabove, but failed to reproduce your issue there; at the same time you want tosee the issue fixed in a still supported stable or longterm series or vendorkernels regularly rebased on those. If that is the case, follow these steps: * Prepare yourself for the possibility that going through the next few steps might not get the issue solved in older releases: the fix might be too big or risky to get backported there. * Perform the first three steps in the section "Dealing with regressions within a stable and longterm kernel line" above. * Search the Linux kernel version control system for the change that fixed the issue in mainline, as its commit message might tell you if the fix is scheduled for backporting already. If you don't find anything that way, search the appropriate mailing lists for posts that discuss such an issue or peer-review possible fixes; then check the discussions if the fix was deemed unsuitable for backporting. If backporting was not considered at all, join the newest discussion, asking if it's in the cards. * One of the former steps should lead to a solution. If that doesn't work out, ask the maintainers for the subsystem that seems to be causing the issue for advice; CC the mailing list for the particular subsystem as well as the stable mailing list. The reference section below explains each of these steps in more detail. Conclusion of the step-by-step guide------------------------------------ Did you run into trouble following the step-by-step guide not cleared up by thereference section below? Did you spot errors? Or do you have ideas on how toimprove the guide? If any of that applies, please let the developers know by sending a short noteor a patch to Thorsten Leemhuis <linux@leemhuis.info> while ideally CCing thepublic Linux docs mailing list <linux-doc@vger.kernel.org>. Such feedback isvital to improve this text further, which is in everybody's interest, as it willenable more people to master the task described here. Reference section: Reporting issues to the kernel maintainers============================================================= The step-by-step guide above outlines all the major steps in brief fashion,which usually covers everything required. But even experienced users willsometimes wonder how to actually realize some of those steps or why they areneeded; there are also corner cases the guide ignores for readability. That iswhat the entries in this reference section are for, which provide additionalinformation for each of the steps in the guide. A few words of general advice: * The Linux developers are well aware that reporting bugs to them is more complicated and demanding than in other FLOSS projects. Some of it is because the kernel is different, among others due to its mail-driven development process and because it consists mostly of drivers. Some of it is because improving things would require work in several technical areas and people triaging bugs –– and nobody has stepped up to do or fund that work. * A warranty or support contract with some vendor doesn't entitle you to request fixes from the upstream Linux developers: Such contracts are completely outside the scope of the upstream Linux kernel, its development community, and this document -- even if those handling the issue work for the vendor who issued the contract. If you want to claim your rights, use the vendor's support channel. * If you never reported an issue to a FLOSS project before, consider skimming guides like `How to ask good questions <https://jvns.ca/blog/good-questions/>`_, `How To Ask Questions The Smart Way <http://www.catb.org/esr/faqs/smart-questions.html>`_, and `How to Report Bugs Effectively <https://www.chiark.greenend.org.uk/~sgtatham/bugs.html>`_,. With that off the table, find below details for the steps from the detailedguide on reporting issues to the Linux kernel developers. Make sure you're using the upstream Linux kernel------------------------------------------------ *Are you facing an issue with a Linux kernel a hardware or software vendor provided? Then in almost all cases you are better off to stop reading this document and reporting the issue to your vendor instead, unless you are willing to install the latest Linux version yourself. Be aware the latter will often be needed anyway to hunt down and fix issues.* Like most programmers, Linux kernel developers don't like to spend time dealingwith reports for issues that don't even happen with their current code. It'sjust a waste everybody's time, especially yours. Unfortunately such situationseasily happen when it comes to the kernel and often leads to frustration on bothsides. That's because almost all Linux-based kernels pre-installed on devices(Computers, Laptops, Smartphones, Routers, …) and most shipped by Linuxdistributors are quite distant from the official Linux kernel as distributed bykernel.org: these kernels from these vendors are often ancient from the point ofLinux development or heavily modified, often both. Most of these vendor kernels are quite unsuitable for reporting issues to theLinux kernel developers: an issue you face with one of them might have beenfixed by the Linux kernel developers months or years ago already; additionally,the modifications and enhancements by the vendor might be causing the issue youface, even if they look small or totally unrelated. That's why you should reportissues with these kernels to the vendor. Its developers should look into thereport and, in case it turns out to be an upstream issue, fix it directlyupstream or forward the report there. In practice that often does not work outor might not what you want. You thus might want to consider circumventing thevendor by installing the very latest Linux kernel core yourself. If that's anoption for you move ahead in this process, as a later step in this guide willexplain how to do that once it rules out other potential causes for your issue. Note, the previous paragraph is starting with the word 'most', as sometimesdevelopers in fact are willing to handle reports about issues occurring withvendor kernels. If they do in the end highly depends on the developers and theissue in question. Your chances are quite good if the distributor applied onlysmall modifications to a kernel based on a recent Linux version; that forexample often holds true for the mainline kernels shipped by Debian GNU/LinuxSid or Fedora Rawhide. Some developers will also accept reports about issueswith kernels from distributions shipping the latest stable kernel, as long asit's only slightly modified; that for example is often the case for Arch Linux,regular Fedora releases, and openSUSE Tumbleweed. But keep in mind, you betterwant to use a mainline Linux and avoid using a stable kernel for thisprocess, as outlined in the section 'Install a fresh kernel for testing' in moredetail. Obviously you are free to ignore all this advice and report problems with an oldor heavily modified vendor kernel to the upstream Linux developers. But note,those often get rejected or ignored, so consider yourself warned. But it's stillbetter than not reporting the issue at all: sometimes such reports directly orindirectly will help to get the issue fixed over time. Search for existing reports, first run-------------------------------------- *Perform a rough search for existing reports with your favorite internet search engine; additionally, check the archives of the Linux Kernel Mailing List (LKML). If you find matching reports, join the discussion instead of sending a new one.* Reporting an issue that someone else already brought forward is often a waste oftime for everyone involved, especially you as the reporter. So it's in your owninterest to thoroughly check if somebody reported the issue already. At thisstep of the process it's okay to just perform a rough search: a later step willtell you to perform a more detailed search once you know where your issue needsto be reported to. Nevertheless, do not hurry with this step of the reportingprocess, it can save you time and trouble. Simply search the internet with your favorite search engine first. Afterwards,search the `Linux Kernel Mailing List (LKML) archives<https://lore.kernel.org/lkml/>`_. If you get flooded with results consider telling your search engine to limitsearch timeframe to the past month or year. And wherever you search, make sureto use good search terms; vary them a few times, too. While doing so try tolook at the issue from the perspective of someone else: that will help you tocome up with other words to use as search terms. Also make sure not to use toomany search terms at once. Remember to search with and without information likethe name of the kernel driver or the name of the affected hardware component.But its exact brand name (say 'ASUS Red Devil Radeon RX 5700 XT Gaming OC')often is not much helpful, as it is too specific. Instead try search terms likethe model line (Radeon 5700 or Radeon 5000) and the code name of the main chip('Navi' or 'Navi10') with and without its manufacturer ('AMD'). In case you find an existing report about your issue, join the discussion, asyou might be able to provide valuable additional information. That can beimportant even when a fix is prepared or in its final stages already, asdevelopers might look for people that can provide additional information ortest a proposed fix. Jump to the section 'Duties after the report went out' fordetails on how to get properly involved. Note, searching `bugzilla.kernel.org <https://bugzilla.kernel.org/>`_ might alsobe a good idea, as that might provide valuable insights or turn up matchingreports. If you find the latter, just keep in mind: most subsystems expectreports in different places, as described below in the section "Check where youneed to report your issue". The developers that should take care of the issuethus might not even be aware of the bugzilla ticket. Hence, check the ticket ifthe issue already got reported as outlined in this document and if not considerdoing so. Issue of high priority?----------------------- *See if the issue you are dealing with qualifies as regression, security issue, or a really severe problem: those are 'issues of high priority' that need special handling in some steps that are about to follow.* Linus Torvalds and the leading Linux kernel developers want to see some issuesfixed as soon as possible, hence there are 'issues of high priority' that gethandled slightly differently in the reporting process. Three type of casesqualify: regressions, security issues, and really severe problems. You deal with a regression if some application or practical use case runningfine with one Linux kernel works worse or not at all with a newer versioncompiled using a similar configuration. The documentDocumentation/admin-guide/reporting-regressions.rst explains this in moredetail. It also provides a good deal of other information about regressions youmight want to be aware of; it for example explains how to add your issue to thelist of tracked regressions, to ensure it won't fall through the cracks. What qualifies as security issue is left to your judgment. Consider readingDocumentation/process/security-bugs.rst before proceeding, as itprovides additional details how to best handle security issues. An issue is a 'really severe problem' when something totally unacceptably badhappens. That's for example the case when a Linux kernel corrupts the data it'shandling or damages hardware it's running on. You're also dealing with a severeissue when the kernel suddenly stops working with an error message ('kernelpanic') or without any farewell note at all. Note: do not confuse a 'panic' (afatal error where the kernel stop itself) with a 'Oops' (a recoverable error),as the kernel remains running after the latter. Ensure a healthy environment---------------------------- *Make sure it's not the kernel's surroundings that are causing the issue you face.* Problems that look a lot like a kernel issue are sometimes caused by build orruntime environment. It's hard to rule out that problem completely, but youshould minimize it: * Use proven tools when building your kernel, as bugs in the compiler or the binutils can cause the resulting kernel to misbehave. * Ensure your computer components run within their design specifications; that's especially important for the main processor, the main memory, and the motherboard. Therefore, stop undervolting or overclocking when facing a potential kernel issue. * Try to make sure it's not faulty hardware that is causing your issue. Bad main memory for example can result in a multitude of issues that will manifest itself in problems looking like kernel issues. * If you're dealing with a filesystem issue, you might want to check the file system in question with ``fsck``, as it might be damaged in a way that leads to unexpected kernel behavior. * When dealing with a regression, make sure it's not something else that changed in parallel to updating the kernel. The problem for example might be caused by other software that was updated at the same time. It can also happen that a hardware component coincidentally just broke when you rebooted into a new kernel for the first time. Updating the systems BIOS or changing something in the BIOS Setup can also lead to problems that on look a lot like a kernel regression. Prepare for emergencies----------------------- *Create a fresh backup and put system repair and restore tools at hand.* Reminder, you are dealing with computers, which sometimes do unexpected things,especially if you fiddle with crucial parts like the kernel of its operatingsystem. That's what you are about to do in this process. Thus, make sure tocreate a fresh backup; also ensure you have all tools at hand to repair orreinstall the operating system as well as everything you need to restore thebackup. Make sure your kernel doesn't get enhanced------------------------------------------ *Ensure your system does not enhance its kernels by building additional kernel modules on-the-fly, which solutions like DKMS might be doing locally without your knowledge.* The risk your issue report gets ignored or rejected dramatically increases ifyour kernel gets enhanced in any way. That's why you should remove or disablemechanisms like akmods and DKMS: those build add-on kernel modulesautomatically, for example when you install a new Linux kernel or boot it forthe first time. Also remove any modules they might have installed. Then rebootbefore proceeding. Note, you might not be aware that your system is using one of these solutions:they often get set up silently when you install Nvidia's proprietary graphicsdriver, VirtualBox, or other software that requires a some support from amodule not part of the Linux kernel. That why your might need to uninstall thepackages with such software to get rid of any 3rd party kernel module. Check 'taint' flag------------------ Lines 1–500 of 1770. Load more
.. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0) .. See the bottom of this file for additional redistribution information. Reporting issues ++++++++++++++++ The short guide (aka TL;DR) =========================== Are you facing a regression with vanilla kernels from the same stable or longterm series? One still supported? Then search the `LKML <https://lore.kernel.org/lkml/>`_ and the `Linux stable mailing list <https://lore.kernel.org/stable/>`_ archives for matching reports to join. If you don't find any, install `the latest release from that series <https://kernel.org/>`_. If it still shows the issue, report it to the stable mailing list (stable@vger.kernel.org) and CC the regressions list (regressions@lists.linux.dev); ideally also CC the maintainer and the mailing list for the subsystem in question. In all other cases try your best guess which kernel part might be causing the issue. Check the :ref:`MAINTAINERS <maintainers>` file for how its developers expect to be told about problems, which most of the time will be by email with a mailing list in CC. Check the destination's archives for matching reports; search the `LKML <https://lore.kernel.org/lkml/>`_ and the web, too. If you don't find any to join, install `the latest mainline kernel <https://kernel.org/>`_. If the issue is present there, send a report. The issue was fixed there, but you would like to see it resolved in a still supported stable or longterm series as well? Then install its latest release. If it shows the problem, search for the change that fixed it in mainline and check if backporting is in the works or was discarded; if it's neither, ask those who handled the change for it. **General remarks**: When installing and testing a kernel as outlined above, ensure it's vanilla (IOW: not patched and not using add-on modules). Also make sure it's built and running in a healthy environment and not already tainted before the issue occurs. If you are facing multiple issues with the Linux kernel at once, report each separately. While writing your report, include all information relevant to the issue, like the kernel and the distro used. In case of a regression, CC the regressions mailing list (regressions@lists.linux.dev) to your report. Also try to pinpoint the culprit with a bisection; if you succeed, include its commit-id and CC everyone in the sign-off-by chain. Once the report is out, answer any questions that come up and help where you can. That includes keeping the ball rolling by occasionally retesting with newer releases and sending a status update afterwards. .. Note: If you see this note, you are reading the text's source file. You might want to switch to a rendered version: It makes it a lot easier to read and navigate this document -- especially when you want to look something up in the reference section, then jump back to where you left off. .. Find the latest rendered version of this text here: https://docs.kernel.org/admin-guide/reporting-issues.html Step-by-step guide how to report issues to the kernel maintainers ================================================================= The above TL;DR outlines roughly how to report issues to the Linux kernel developers. It might be all that's needed for people already familiar with reporting issues to Free/Libre & Open Source Software (FLOSS) projects. For everyone else there is this section. It is more detailed and uses a step-by-step approach. It still tries to be brief for readability and leaves out a lot of details; those are described below the step-by-step guide in a reference section, which explains each of the steps in more detail. Note: this section covers a few more aspects than the TL;DR and does things in a slightly different order. That's in your interest, to make sure you notice early if an issue that looks like a Linux kernel problem is actually caused by something else. These steps thus help to ensure the time you invest in this process won't feel wasted in the end: * Are you facing an issue with a Linux kernel a hardware or software vendor provided? Then in almost all cases you are better off to stop reading this document and reporting the issue to your vendor instead, unless you are willing to install the latest Linux version yourself. Be aware the latter will often be needed anyway to hunt down and fix issues. * Perform a rough search for existing reports with your favorite internet search engine; additionally, check the archives of the `Linux Kernel Mailing List (LKML) <https://lore.kernel.org/lkml/>`_. If you find matching reports, join the discussion instead of sending a new one. * See if the issue you are dealing with qualifies as regression, security issue, or a really severe problem: those are 'issues of high priority' that need special handling in some steps that are about to follow. * Make sure it's not the kernel's surroundings that are causing the issue you face. * Create a fresh backup and put system repair and restore tools at hand. * Ensure your system does not enhance its kernels by building additional kernel modules on-the-fly, which solutions like DKMS might be doing locally without your knowledge. * Check if your kernel was 'tainted' when the issue occurred, as the event that made the kernel set this flag might be causing the issue you face. * Write down coarsely how to reproduce the issue. If you deal with multiple issues at once, create separate notes for each of them and make sure they work independently on a freshly booted system. That's needed, as each issue needs to get reported to the kernel developers separately, unless they are strongly entangled. * If you are facing a regression within a stable or longterm version line (say something broke when updating from 5.10.4 to 5.10.5), scroll down to 'Dealing with regressions within a stable and longterm kernel line'. * Locate the driver or kernel subsystem that seems to be causing the issue. Find out how and where its developers expect reports. Note: most of the time this won't be bugzilla.kernel.org, as issues typically need to be sent by mail to a maintainer and a public mailing list. * Search the archives of the bug tracker or mailing list in question thoroughly for reports that might match your issue. If you find anything, join the discussion instead of sending a new report. After these preparations you'll now enter the main part: * Unless you are already running the latest 'mainline' Linux kernel, better go and install it for the reporting process. Testing and reporting with the latest 'stable' Linux can be an acceptable alternative in some situations; during the merge window that actually might be even the best approach, but in that development phase it can be an even better idea to suspend your efforts for a few days anyway. Whatever version you choose, ideally use a 'vanilla' build. Ignoring all of this advice will dramatically increase the risk your report will be rejected or ignored. * Ensure the kernel you just installed does not 'taint' itself when running. * Reproduce the issue with the kernel you just installed. If it doesn't show up there, scroll down to the instructions for issues only happening with stable and longterm kernels. * Optimize your notes: try to find and write the most straightforward way to reproduce your issue. Make sure the end result has all the important details, and at the same time is easy to read and understand for others that hear about it for the first time. And if you learned something in this process, consider searching again for existing reports about the issue. * If your failure involves a 'panic', 'Oops', 'warning', or 'BUG', consider decoding the kernel log to find the line of code that triggered the error. * If your problem is a regression, try to narrow down when the issue was introduced as much as possible. * Start to compile the report by writing a detailed description about the issue. Always mention a few things: the latest kernel version you installed for reproducing, the Linux Distribution used, and your notes on how to reproduce the issue. Ideally, make the kernel's build configuration (.config) and the output from ``dmesg`` available somewhere on the net and link to it. Include or upload all other information that might be relevant, like the output/screenshot of an Oops or the output from ``lspci``. Once you wrote this main part, insert a normal length paragraph on top of it outlining the issue and the impact quickly. On top of this add one sentence that briefly describes the problem and gets people to read on. Now give the thing a descriptive title or subject that yet again is shorter. Then you're ready to send or file the report like the MAINTAINERS file told you, unless you are dealing with one of those 'issues of high priority': they need special care which is explained in 'Special handling for high priority issues' below. * Wait for reactions and keep the thing rolling until you can accept the outcome in one way or the other. Thus react publicly and in a timely manner to any inquiries. Test proposed fixes. Do proactive testing: retest with at least every first release candidate (RC) of a new mainline version and report your results. Send friendly reminders if things stall. And try to help yourself, if you don't get any help or if it's unsatisfying. Reporting regressions within a stable and longterm kernel line -------------------------------------------------------------- This subsection is for you, if you followed above process and got sent here at the point about regression within a stable or longterm kernel version line. You face one of those if something breaks when updating from 5.10.4 to 5.10.5 (a switch from 5.9.15 to 5.10.5 does not qualify). The developers want to fix such regressions as quickly as possible, hence there is a streamlined process to report them: * Check if the kernel developers still maintain the Linux kernel version line you care about: go to the `front page of kernel.org <https://kernel.org/>`_ and make sure it mentions the latest release of the particular version line without an '[EOL]' tag. * Check the archives of the `Linux stable mailing list <https://lore.kernel.org/stable/>`_ for existing reports. * Install the latest release from the particular version line as a vanilla kernel. Ensure this kernel is not tainted and still shows the problem, as the issue might have already been fixed there. If you first noticed the problem with a vendor kernel, check a vanilla build of the last version known to work performs fine as well. * Send a short problem report to the Linux stable mailing list (stable@vger.kernel.org) and CC the Linux regressions mailing list (regressions@lists.linux.dev); if you suspect the cause in a particular subsystem, CC its maintainer and its mailing list. Roughly describe the issue and ideally explain how to reproduce it. Mention the first version that shows the problem and the last version that's working fine. Then wait for further instructions. The reference section below explains each of these steps in more detail. Reporting issues only occurring in older kernel version lines ------------------------------------------------------------- This subsection is for you, if you tried the latest mainline kernel as outlined above, but failed to reproduce your issue there; at the same time you want to see the issue fixed in a still supported stable or longterm series or vendor kernels regularly rebased on those. If that is the case, follow these steps: * Prepare yourself for the possibility that going through the next few steps might not get the issue solved in older releases: the fix might be too big or risky to get backported there. * Perform the first three steps in the section "Dealing with regressions within a stable and longterm kernel line" above. * Search the Linux kernel version control system for the change that fixed the issue in mainline, as its commit message might tell you if the fix is scheduled for backporting already. If you don't find anything that way, search the appropriate mailing lists for posts that discuss such an issue or peer-review possible fixes; then check the discussions if the fix was deemed unsuitable for backporting. If backporting was not considered at all, join the newest discussion, asking if it's in the cards. * One of the former steps should lead to a solution. If that doesn't work out, ask the maintainers for the subsystem that seems to be causing the issue for advice; CC the mailing list for the particular subsystem as well as the stable mailing list. The reference section below explains each of these steps in more detail. Conclusion of the step-by-step guide ------------------------------------ Did you run into trouble following the step-by-step guide not cleared up by the reference section below? Did you spot errors? Or do you have ideas on how to improve the guide? If any of that applies, please let the developers know by sending a short note or a patch to Thorsten Leemhuis <linux@leemhuis.info> while ideally CCing the public Linux docs mailing list <linux-doc@vger.kernel.org>. Such feedback is vital to improve this text further, which is in everybody's interest, as it will enable more people to master the task described here. Reference section: Reporting issues to the kernel maintainers ============================================================= The step-by-step guide above outlines all the major steps in brief fashion, which usually covers everything required. But even experienced users will sometimes wonder how to actually realize some of those steps or why they are needed; there are also corner cases the guide ignores for readability. That is what the entries in this reference section are for, which provide additional information for each of the steps in the guide. A few words of general advice: * The Linux developers are well aware that reporting bugs to them is more complicated and demanding than in other FLOSS projects. Some of it is because the kernel is different, among others due to its mail-driven development process and because it consists mostly of drivers. Some of it is because improving things would require work in several technical areas and people triaging bugs –– and nobody has stepped up to do or fund that work. * A warranty or support contract with some vendor doesn't entitle you to request fixes from the upstream Linux developers: Such contracts are completely outside the scope of the upstream Linux kernel, its development community, and this document -- even if those handling the issue work for the vendor who issued the contract. If you want to claim your rights, use the vendor's support channel. * If you never reported an issue to a FLOSS project before, consider skimming guides like `How to ask good questions <https://jvns.ca/blog/good-questions/>`_, `How To Ask Questions The Smart Way <http://www.catb.org/esr/faqs/smart-questions.html>`_, and `How to Report Bugs Effectively <https://www.chiark.greenend.org.uk/~sgtatham/bugs.html>`_,. With that off the table, find below details for the steps from the detailed guide on reporting issues to the Linux kernel developers. Make sure you're using the upstream Linux kernel ------------------------------------------------ *Are you facing an issue with a Linux kernel a hardware or software vendor provided? Then in almost all cases you are better off to stop reading this document and reporting the issue to your vendor instead, unless you are willing to install the latest Linux version yourself. Be aware the latter will often be needed anyway to hunt down and fix issues.* Like most programmers, Linux kernel developers don't like to spend time dealing with reports for issues that don't even happen with their current code. It's just a waste everybody's time, especially yours. Unfortunately such situations easily happen when it comes to the kernel and often leads to frustration on both sides. That's because almost all Linux-based kernels pre-installed on devices (Computers, Laptops, Smartphones, Routers, …) and most shipped by Linux distributors are quite distant from the official Linux kernel as distributed by kernel.org: these kernels from these vendors are often ancient from the point of Linux development or heavily modified, often both. Most of these vendor kernels are quite unsuitable for reporting issues to the Linux kernel developers: an issue you face with one of them might have been fixed by the Linux kernel developers months or years ago already; additionally, the modifications and enhancements by the vendor might be causing the issue you face, even if they look small or totally unrelated. That's why you should report issues with these kernels to the vendor. Its developers should look into the report and, in case it turns out to be an upstream issue, fix it directly upstream or forward the report there. In practice that often does not work out or might not what you want. You thus might want to consider circumventing the vendor by installing the very latest Linux kernel core yourself. If that's an option for you move ahead in this process, as a later step in this guide will explain how to do that once it rules out other potential causes for your issue. Note, the previous paragraph is starting with the word 'most', as sometimes developers in fact are willing to handle reports about issues occurring with vendor kernels. If they do in the end highly depends on the developers and the issue in question. Your chances are quite good if the distributor applied only small modifications to a kernel based on a recent Linux version; that for example often holds true for the mainline kernels shipped by Debian GNU/Linux Sid or Fedora Rawhide. Some developers will also accept reports about issues with kernels from distributions shipping the latest stable kernel, as long as it's only slightly modified; that for example is often the case for Arch Linux, regular Fedora releases, and openSUSE Tumbleweed. But keep in mind, you better want to use a mainline Linux and avoid using a stable kernel for this process, as outlined in the section 'Install a fresh kernel for testing' in more detail. Obviously you are free to ignore all this advice and report problems with an old or heavily modified vendor kernel to the upstream Linux developers. But note, those often get rejected or ignored, so consider yourself warned. But it's still better than not reporting the issue at all: sometimes such reports directly or indirectly will help to get the issue fixed over time. Search for existing reports, first run -------------------------------------- *Perform a rough search for existing reports with your favorite internet search engine; additionally, check the archives of the Linux Kernel Mailing List (LKML). If you find matching reports, join the discussion instead of sending a new one.* Reporting an issue that someone else already brought forward is often a waste of time for everyone involved, especially you as the reporter. So it's in your own interest to thoroughly check if somebody reported the issue already. At this step of the process it's okay to just perform a rough search: a later step will tell you to perform a more detailed search once you know where your issue needs to be reported to. Nevertheless, do not hurry with this step of the reporting process, it can save you time and trouble. Simply search the internet with your favorite search engine first. Afterwards, search the `Linux Kernel Mailing List (LKML) archives <https://lore.kernel.org/lkml/>`_. If you get flooded with results consider telling your search engine to limit search timeframe to the past month or year. And wherever you search, make sure to use good search terms; vary them a few times, too. While doing so try to look at the issue from the perspective of someone else: that will help you to come up with other words to use as search terms. Also make sure not to use too many search terms at once. Remember to search with and without information like the name of the kernel driver or the name of the affected hardware component. But its exact brand name (say 'ASUS Red Devil Radeon RX 5700 XT Gaming OC') often is not much helpful, as it is too specific. Instead try search terms like the model line (Radeon 5700 or Radeon 5000) and the code name of the main chip ('Navi' or 'Navi10') with and without its manufacturer ('AMD'). In case you find an existing report about your issue, join the discussion, as you might be able to provide valuable additional information. That can be important even when a fix is prepared or in its final stages already, as developers might look for people that can provide additional information or test a proposed fix. Jump to the section 'Duties after the report went out' for details on how to get properly involved. Note, searching `bugzilla.kernel.org <https://bugzilla.kernel.org/>`_ might also be a good idea, as that might provide valuable insights or turn up matching reports. If you find the latter, just keep in mind: most subsystems expect reports in different places, as described below in the section "Check where you need to report your issue". The developers that should take care of the issue thus might not even be aware of the bugzilla ticket. Hence, check the ticket if the issue already got reported as outlined in this document and if not consider doing so. Issue of high priority? ----------------------- *See if the issue you are dealing with qualifies as regression, security issue, or a really severe problem: those are 'issues of high priority' that need special handling in some steps that are about to follow.* Linus Torvalds and the leading Linux kernel developers want to see some issues fixed as soon as possible, hence there are 'issues of high priority' that get handled slightly differently in the reporting process. Three type of cases qualify: regressions, security issues, and really severe problems. You deal with a regression if some application or practical use case running fine with one Linux kernel works worse or not at all with a newer version compiled using a similar configuration. The document Documentation/admin-guide/reporting-regressions.rst explains this in more detail. It also provides a good deal of other information about regressions you might want to be aware of; it for example explains how to add your issue to the list of tracked regressions, to ensure it won't fall through the cracks. What qualifies as security issue is left to your judgment. Consider reading Documentation/process/security-bugs.rst before proceeding, as it provides additional details how to best handle security issues. An issue is a 'really severe problem' when something totally unacceptably bad happens. That's for example the case when a Linux kernel corrupts the data it's handling or damages hardware it's running on. You're also dealing with a severe issue when the kernel suddenly stops working with an error message ('kernel panic') or without any farewell note at all. Note: do not confuse a 'panic' (a fatal error where the kernel stop itself) with a 'Oops' (a recoverable error), as the kernel remains running after the latter. Ensure a healthy environment ---------------------------- *Make sure it's not the kernel's surroundings that are causing the issue you face.* Problems that look a lot like a kernel issue are sometimes caused by build or runtime environment. It's hard to rule out that problem completely, but you should minimize it: * Use proven tools when building your kernel, as bugs in the compiler or the binutils can cause the resulting kernel to misbehave. * Ensure your computer components run within their design specifications; that's especially important for the main processor, the main memory, and the motherboard. Therefore, stop undervolting or overclocking when facing a potential kernel issue. * Try to make sure it's not faulty hardware that is causing your issue. Bad main memory for example can result in a multitude of issues that will manifest itself in problems looking like kernel issues. * If you're dealing with a filesystem issue, you might want to check the file system in question with ``fsck``, as it might be damaged in a way that leads to unexpected kernel behavior. * When dealing with a regression, make sure it's not something else that changed in parallel to updating the kernel. The problem for example might be caused by other software that was updated at the same time. It can also happen that a hardware component coincidentally just broke when you rebooted into a new kernel for the first time. Updating the systems BIOS or changing something in the BIOS Setup can also lead to problems that on look a lot like a kernel regression. Prepare for emergencies ----------------------- *Create a fresh backup and put system repair and restore tools at hand.* Reminder, you are dealing with computers, which sometimes do unexpected things, especially if you fiddle with crucial parts like the kernel of its operating system. That's what you are about to do in this process. Thus, make sure to create a fresh backup; also ensure you have all tools at hand to repair or reinstall the operating system as well as everything you need to restore the backup. Make sure your kernel doesn't get enhanced ------------------------------------------ *Ensure your system does not enhance its kernels by building additional kernel modules on-the-fly, which solutions like DKMS might be doing locally without your knowledge.* The risk your issue report gets ignored or rejected dramatically increases if your kernel gets enhanced in any way. That's why you should remove or disable mechanisms like akmods and DKMS: those build add-on kernel modules automatically, for example when you install a new Linux kernel or boot it for the first time. Also remove any modules they might have installed. Then reboot before proceeding. Note, you might not be aware that your system is using one of these solutions: they often get set up silently when you install Nvidia's proprietary graphics driver, VirtualBox, or other software that requires a some support from a module not part of the Linux kernel. That why your might need to uninstall the packages with such software to get rid of any 3rd party kernel module. Check 'taint' flag ------------------