2: HOW THE DEVELOPMENT PROCESS WORKS Linux kernel development in the early 1990's was a pretty loose affair,with relatively small numbers of users and developers involved.  With auser base in the millions and with some 2,000 developers involved over thecourse of one year, the kernel has since had to evolve a number ofprocesses to keep development happening smoothly.  A solid understanding ofhow the process works is required in order to be an effective part of it.  2.1: THE BIG PICTURE The kernel developers use a loosely time-based release process, with a newmajor kernel release happening every two or three months.  The recentrelease history looks like this: 	2.6.26	July 13, 2008	2.6.25	April 16, 2008	2.6.24	January 24, 2008	2.6.23	October 9, 2007	2.6.22	July 8, 2007	2.6.21	April 25, 2007	2.6.20	February 4, 2007 Every 2.6.x release is a major kernel release with new features, internalAPI changes, and more.  A typical 2.6 release can contain over 10,000changesets with changes to several hundred thousand lines of code.  2.6 isthus the leading edge of Linux kernel development; the kernel uses arolling development model which is continually integrating major changes. A relatively straightforward discipline is followed with regard to themerging of patches for each release.  At the beginning of each developmentcycle, the "merge window" is said to be open.  At that time, code which isdeemed to be sufficiently stable (and which is accepted by the developmentcommunity) is merged into the mainline kernel.  The bulk of changes for anew development cycle (and all of the major changes) will be merged duringthis time, at a rate approaching 1,000 changes ("patches," or "changesets")per day. (As an aside, it is worth noting that the changes integrated during themerge window do not come out of thin air; they have been collected, tested,and staged ahead of time.  How that process works will be described indetail later on). The merge window lasts for two weeks.  At the end of this time, LinusTorvalds will declare that the window is closed and release the first ofthe "rc" kernels.  For the kernel which is destined to be 2.6.26, forexample, the release which happens at the end of the merge window will becalled 2.6.26-rc1.  The -rc1 release is the signal that the time to mergenew features has passed, and that the time to stabilize the next kernel hasbegun. Over the next six to ten weeks, only patches which fix problems should besubmitted to the mainline.  On occasion a more significant change will beallowed, but such occasions are rare; developers who try to merge newfeatures outside of the merge window tend to get an unfriendly reception.As a general rule, if you miss the merge window for a given feature, thebest thing to do is to wait for the next development cycle.  (An occasionalexception is made for drivers for previously-unsupported hardware; if theytouch no in-tree code, they cannot cause regressions and should be safe toadd at any time). As fixes make their way into the mainline, the patch rate will slow overtime.  Linus releases new -rc kernels about once a week; a normal serieswill get up to somewhere between -rc6 and -rc9 before the kernel isconsidered to be sufficiently stable and the final 2.6.x release is made.At that point the whole process starts over again. As an example, here is how the 2.6.25 development cycle went (all dates in2008):  	January 24	2.6.24 stable release	February 10	2.6.25-rc1, merge window closes	February 15	2.6.25-rc2	February 24	2.6.25-rc3	March 4	 	2.6.25-rc4	March 9		2.6.25-rc5	March 16	2.6.25-rc6	March 25	2.6.25-rc7	April 1		2.6.25-rc8	April 11	2.6.25-rc9	April 16	2.6.25 stable release How do the developers decide when to close the development cycle and createthe stable release?  The most significant metric used is the list ofregressions from previous releases.  No bugs are welcome, but those whichbreak systems which worked in the past are considered to be especiallyserious.  For this reason, patches which cause regressions are looked uponunfavorably and are quite likely to be reverted during the stabilizationperiod.  The developers' goal is to fix all known regressions before the stablerelease is made.  In the real world, this kind of perfection is hard toachieve; there are just too many variables in a project of this size.There comes a point where delaying the final release just makes the problemworse; the pile of changes waiting for the next merge window will growlarger, creating even more regressions the next time around.  So most 2.6.xkernels go out with a handful of known regressions though, hopefully, noneof them are serious. Once a stable release is made, its ongoing maintenance is passed off to the"stable team," currently comprised of Greg Kroah-Hartman and Chris Wright.The stable team will release occasional updates to the stable release usingthe 2.6.x.y numbering scheme.  To be considered for an update release, apatch must (1) fix a significant bug, and (2) already be merged into themainline for the next development kernel.  Continuing our 2.6.25 example,the history (as of this writing) is: 	May 1		2.6.25.1	May 6		2.6.25.2 	May 9		2.6.25.3 	May 15		2.6.25.4	June 7		2.6.25.5	June 9		2.6.25.6	June 16		2.6.25.7	June 21		2.6.25.8	June 24		2.6.25.9 Stable updates for a given kernel are made for approximately six months;after that, the maintenance of stable releases is solely the responsibilityof the distributors which have shipped that particular kernel.  2.2: THE LIFECYCLE OF A PATCH Patches do not go directly from the developer's keyboard into the mainlinekernel.  There is, instead, a somewhat involved (if somewhat informal)process designed to ensure that each patch is reviewed for quality and thateach patch implements a change which is desirable to have in the mainline.This process can happen quickly for minor fixes, or, in the case of largeand controversial changes, go on for years.  Much developer frustrationcomes from a lack of understanding of this process or from attempts tocircumvent it.   In the hopes of reducing that frustration, this document will describe howa patch gets into the kernel.  What follows below is an introduction whichdescribes the process in a somewhat idealized way.  A much more detailedtreatment will come in later sections. The stages that a patch goes through are, generally:  - Design.  This is where the real requirements for the patch - and the way   those requirements will be met - are laid out.  Design work is often   done without involving the community, but it is better to do this work   in the open if at all possible; it can save a lot of time redesigning   things later.  - Early review.  Patches are posted to the relevant mailing list, and   developers on that list reply with any comments they may have.  This   process should turn up any major problems with a patch if all goes   well.  - Wider review.  When the patch is getting close to ready for mainline   inclusion, it will be accepted by a relevant subsystem maintainer -   though this acceptance is not a guarantee that the patch will make it   all the way to the mainline.  The patch will show up in the maintainer's   subsystem tree and into the staging trees (described below).  When the   process works, this step leads to more extensive review of the patch and   the discovery of any problems resulting from the integration of this   patch with work being done by others.  - Merging into the mainline.  Eventually, a successful patch will be   merged into the mainline repository managed by Linus Torvalds.  More   comments and/or problems may surface at this time; it is important that   the developer be responsive to these and fix any issues which arise.  - Stable release.  The number of users potentially affected by the patch   is now large, so, once again, new problems may arise.  - Long-term maintenance.  While it is certainly possible for a developer   to forget about code after merging it, that sort of behavior tends to   leave a poor impression in the development community.  Merging code   eliminates some of the maintenance burden, in that others will fix   problems caused by API changes.  But the original developer should   continue to take responsibility for the code if it is to remain useful   in the longer term. One of the largest mistakes made by kernel developers (or their employers)is to try to cut the process down to a single "merging into the mainline"step.  This approach invariably leads to frustration for everybodyinvolved.  2.3: HOW PATCHES GET INTO THE KERNEL There is exactly one person who can merge patches into the mainline kernelrepository: Linus Torvalds.  But, of the over 12,000 patches which wentinto the 2.6.25 kernel, only 250 (around 2%) were directly chosen by Linushimself.  The kernel project has long since grown to a size where no singledeveloper could possibly inspect and select every patch unassisted.  Theway the kernel developers have addressed this growth is through the use ofa lieutenant system built around a chain of trust. The kernel code base is logically broken down into a set of subsystems:networking, specific architecture support, memory management, videodevices, etc.  Most subsystems have a designated maintainer, a developerwho has overall responsibility for the code within that subsystem.  Thesesubsystem maintainers are the gatekeepers (in a loose way) for the portionof the kernel they manage; they are the ones who will (usually) accept apatch for inclusion into the mainline kernel. Subsystem maintainers each manage their own version of the kernel sourcetree, usually (but certainly not always) using the git source managementtool.  Tools like git (and related tools like quilt or mercurial) allowmaintainers to track a list of patches, including authorship informationand other metadata.  At any given time, the maintainer can identify whichpatches in his or her repository are not found in the mainline. When the merge window opens, top-level maintainers will ask Linus to "pull"the patches they have selected for merging from their repositories.  IfLinus agrees, the stream of patches will flow up into his repository,becoming part of the mainline kernel.  The amount of attention that Linuspays to specific patches received in a pull operation varies.  It is clearthat, sometimes, he looks quite closely.  But, as a general rule, Linustrusts the subsystem maintainers to not send bad patches upstream. Subsystem maintainers, in turn, can pull patches from other maintainers.For example, the networking tree is built from patches which accumulatedfirst in trees dedicated to network device drivers, wireless networking,etc.  This chain of repositories can be arbitrarily long, though it rarelyexceeds two or three links.  Since each maintainer in the chain truststhose managing lower-level trees, this process is known as the "chain oftrust."  Clearly, in a system like this, getting patches into the kernel depends onfinding the right maintainer.  Sending patches directly to Linus is notnormally the right way to go.  2.4: STAGING TREES The chain of subsystem trees guides the flow of patches into the kernel,but it also raises an interesting question: what if somebody wants to lookat all of the patches which are being prepared for the next merge window?Developers will be interested in what other changes are pending to seewhether there are any conflicts to worry about; a patch which changes acore kernel function prototype, for example, will conflict with any otherpatches which use the older form of that function.  Reviewers and testerswant access to the changes in their integrated form before all of thosechanges land in the mainline kernel.  One could pull changes from all ofthe interesting subsystem trees, but that would be a big and error-pronejob. The answer comes in the form of staging trees, where subsystem trees arecollected for testing and review.  The older of these trees, maintained byAndrew Morton, is called "-mm" (for memory management, which is how it gotstarted).  The -mm tree integrates patches from a long list of subsystemtrees; it also has some patches aimed at helping with debugging.   Beyond that, -mm contains a significant collection of patches which havebeen selected by Andrew directly.  These patches may have been posted on amailing list, or they may apply to a part of the kernel for which there isno designated subsystem tree.  As a result, -mm operates as a sort ofsubsystem tree of last resort; if there is no other obvious path for apatch into the mainline, it is likely to end up in -mm.  Miscellaneouspatches which accumulate in -mm will eventually either be forwarded on toan appropriate subsystem tree or be sent directly to Linus.  In a typicaldevelopment cycle, approximately 10% of the patches going into the mainlineget there via -mm. The current -mm patch can always be found from the front page of 	http://kernel.org/ Those who want to see the current state of -mm can get the "-mm of themoment" tree, found at: 	http://userweb.kernel.org/~akpm/mmotm/ Use of the MMOTM tree is likely to be a frustrating experience, though;there is a definite chance that it will not even compile. The other staging tree, started more recently, is linux-next, maintained byStephen Rothwell.  The linux-next tree is, by design, a snapshot of whatthe mainline is expected to look like after the next merge window closes.Linux-next trees are announced on the linux-kernel and linux-next mailinglists when they are assembled; they can be downloaded from: 	http://www.kernel.org/pub/linux/kernel/people/sfr/linux-next/ Some information about linux-next has been gathered at: 	http://linux.f-seidel.de/linux-next/pmwiki/ How the linux-next tree will fit into the development process is stillchanging.  As of this writing, the first full development cycle involvinglinux-next (2.6.26) is coming to an end; thus far, it has proved to be avaluable resource for finding and fixing integration problems before thebeginning of the merge window.  See http://lwn.net/Articles/287155/ formore information on how linux-next has worked to set up the 2.6.27 mergewindow. Some developers have begun to suggest that linux-next should be used as thetarget for future development as well.  The linux-next tree does tend to befar ahead of the mainline and is more representative of the tree into whichany new work will be merged.  The downside to this idea is that thevolatility of linux-next tends to make it a difficult development target.See http://lwn.net/Articles/289013/ for more information on this topic, andstay tuned; much is still in flux where linux-next is involved.  2.5: TOOLS As can be seen from the above text, the kernel development process dependsheavily on the ability to herd collections of patches in variousdirections.  The whole thing would not work anywhere near as well as itdoes without suitably powerful tools.  Tutorials on how to use these toolsare well beyond the scope of this document, but there is space for a fewpointers. By far the dominant source code management system used by the kernelcommunity is git.  Git is one of a number of distributed version controlsystems being developed in the free software community.  It is well tunedfor kernel development, in that it performs quite well when dealing withlarge repositories and large numbers of patches.  It also has a reputationfor being difficult to learn and use, though it has gotten better overtime.  Some sort of familiarity with git is almost a requirement for kerneldevelopers; even if they do not use it for their own work, they'll need gitto keep up with what other developers (and the mainline) are doing. Git is now packaged by almost all Linux distributions.  There is a homepage at  	http://git.or.cz/ That page has pointers to documentation and tutorials.  One should beaware, in particular, of the Kernel Hacker's Guide to git, which hasinformation specific to kernel development: 	http://linux.yyz.us/git-howto.html Among the kernel developers who do not use git, the most popular choice isalmost certainly Mercurial: 	http://www.selenic.com/mercurial/ Mercurial shares many features with git, but it provides an interface whichmany find easier to use. The other tool worth knowing about is Quilt: 	http://savannah.nongnu.org/projects/quilt/ Quilt is a patch management system, rather than a source code managementsystem.  It does not track history over time; it is, instead, orientedtoward tracking a specific set of changes against an evolving code base.Some major subsystem maintainers use quilt to manage patches intended to goupstream.  For the management of certain kinds of trees (-mm, for example),quilt is the best tool for the job.  2.6: MAILING LISTS A great deal of Linux kernel development work is done by way of mailinglists.  It is hard to be a fully-functioning member of the communitywithout joining at least one list somewhere.  But Linux mailing lists alsorepresent a potential hazard to developers, who risk getting buried under aload of electronic mail, running afoul of the conventions used on the Linuxlists, or both. Most kernel mailing lists are run on vger.kernel.org; the master list canbe found at: 	http://vger.kernel.org/vger-lists.html There are lists hosted elsewhere, though; a number of them are atlists.redhat.com. The core mailing list for kernel development is, of course, linux-kernel.This list is an intimidating place to be; volume can reach 500 messages perday, the amount of noise is high, the conversation can be severelytechnical, and participants are not always concerned with showing a highdegree of politeness.  But there is no other place where the kerneldevelopment community comes together as a whole; developers who avoid thislist will miss important information. There are a few hints which can help with linux-kernel survival: - Have the list delivered to a separate folder, rather than your main  mailbox.  One must be able to ignore the stream for sustained periods of  time. - Do not try to follow every conversation - nobody else does.  It is  important to filter on both the topic of interest (though note that  long-running conversations can drift away from the original subject  without changing the email subject line) and the people who are  participating.   - Do not feed the trolls.  If somebody is trying to stir up an angry  response, ignore them. - When responding to linux-kernel email (or that on other lists) preserve  the Cc: header for all involved.  In the absence of a strong reason (such  as an explicit request), you should never remove recipients.  Always make  sure that the person you are responding to is in the Cc: list.  This  convention also makes it unnecessary to explicitly ask to be copied on  replies to your postings. - Search the list archives (and the net as a whole) before asking  questions.  Some developers can get impatient with people who clearly  have not done their homework. - Avoid top-posting (the practice of putting your answer above the quoted  text you are responding to).  It makes your response harder to read and  makes a poor impression. - Ask on the correct mailing list.  Linux-kernel may be the general meeting  point, but it is not the best place to find developers from all  subsystems. The last point - finding the correct mailing list - is a common place forbeginning developers to go wrong.  Somebody who asks a networking-relatedquestion on linux-kernel will almost certainly receive a polite suggestionto ask on the netdev list instead, as that is the list frequented by mostnetworking developers.  Other lists exist for the SCSI, video4linux, IDE,filesystem, etc. subsystems.  The best place to look for mailing lists isin the MAINTAINERS file packaged with the kernel source.  2.7: GETTING STARTED WITH KERNEL DEVELOPMENT Questions about how to get started with the kernel development process arecommon - from both individuals and companies.  Equally common are misstepswhich make the beginning of the relationship harder than it has to be. Companies often look to hire well-known developers to get a developmentgroup started.  This can, in fact, be an effective technique.  But it alsotends to be expensive and does not do much to grow the pool of experiencedkernel developers.  It is possible to bring in-house developers up to speedon Linux kernel development, given the investment of a bit of time.  Takingthis time can endow an employer with a group of developers who understandthe kernel and the company both, and who can help to train others as well.Over the medium term, this is often the more profitable approach. Individual developers are often, understandably, at a loss for a place tostart.  Beginning with a large project can be intimidating; one often wantsto test the waters with something smaller first.  This is the point wheresome developers jump into the creation of patches fixing spelling errors orminor coding style issues.  Unfortunately, such patches create a level ofnoise which is distracting for the development community as a whole, so,increasingly, they are looked down upon.  New developers wishing tointroduce themselves to the community will not get the sort of receptionthey wish for by these means. Andrew Morton gives this advice for aspiring kernel developers 	The #1 project for all kernel beginners should surely be "make sure	that the kernel runs perfectly at all times on all machines which	you can lay your hands on".  Usually the way to do this is to work	with others on getting things fixed up (this can require	persistence!) but that's fine - it's a part of kernel development. (http://lwn.net/Articles/283982/). In the absence of obvious problems to fix, developers are advised to lookat the current lists of regressions and open bugs in general.  There isnever any shortage of issues in need of fixing; by addressing these issues,developers will gain experience with the process while, at the same time,building respect with the rest of the development community.