Has anyone else noticed a sharp decline in the quality and stability of the 2.6 kernels recently? I know that they’ve supposedly done away with “stable” releases and changes are being merged from the development trees straight into the 2.6 mainline, but surely we can still expect some level of stability. I’ve run into some fairly major problems lately.

In early December I tried to upgrade our Biscuit PC distributions kernel package from 2.6.13.1 to 2.6.14 but was stymied by the fact that 2.6.14 seems to have completely removed support for building out of kernel modules against the kernel headers only, the full kernel source is now required to be present. This does not play nicely with Debian kernel packages. I haven’t had time to try 2.6.15 yet to see if this change has been reverted, but its a fairly major thing to drop support for and completely removes the ability for me to build packages for external drivers like madwifi.

Today I was upgrading a machine with software raid from Woody -> Sarge and proceeded to upgrade the kernel too. I pulled down a copy of 2.6.15.4, ran make oldconfig and then verified that all the crucial options were still selected (raid, promise controller, e100, etc). make-kpkg, dpkg -i, reboot. BANG.

VFS: Cannot open root device "md0" or unknown-block(9,0)
Please append a correct "root=" boot option
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(9,0)

A quick investigation revealed the promise controller as the source of the problem

PDC20267: IDE controller at PCI slot 0000:02:0e.0
PDC20267: chipset revision 2
PDC20267: ROM enabled at 0xf4640000
PDC20267: 100% native mode on irq 22
PDC20267: neither IDE port enabled (BIOS)

Now, the Promise controllor is certainly not disabled in BIOS, and 2.4.21 manages to boot off it just fine so what is wrong with 2.6.15!? Google suggested that I needed to enable the PDC202xx_FORCE option, (which was enabled in 2.4.21 but in 2.6.15 is only available if you happen to select the completely unrelated driver for the next model up promise card to what I have - a bug that has apparently been acknowledged since 2.6.9 but still not fixed!!), however that made no difference whatsoever.

After a couple of hours of fiddling around with various kernel settings and boot options (did I mention this machine is 200km away from me…) I gave up and started trying early kernel versions starting with the existing 2.6.13 kernel I was using for the Biscuit PC distribution, which worked first time. So far I’ve not been able to find any other reports of problems with 2.6.15 and the Promise drivers, but from what I saw today they are completely non functional.

The common argument I seem to hear when I usually mention this subject (2.6 suckiness) is “but you should just use your distributions kernel”, to which my reply is “but I do…”, for my desktop. The standard Ubuntu kernels work great and I have had no reason to meddle with them.

However my desktop needs are a far cry from what I need to be able to do for the servers I maintain at work, or the kernel packages that I need to build for our BiscuitPC distribution. I’m certainly not one of those gentoo type people that need the latest and greatest the instant that it comes out, and I don’t try and run the latest kernel just for the sake of it. But the fact is, new kernels bring new features (not to mention bugfixes and security patches) that are frequently needed, especially in the networking and wireless areas. When I have to spend several days debugging and testing every time we need a new kernel only to find some critical flaw that prevents us from using it I have to stop and wonder whether the current kernel development process is working.

Have others had similar experiences with 2.6 kernels recently, is the fact that the latest kernel-image I can find in sid is 2.6.8 confirmation that it’s not just me, or am I really just overly angry due to the hours I’ve spent battling 2.6 lately?

Update: Several people have pointed out that I failed to look for linux-image instead of kernel-image and that 2.6.15 is already in sid. My mistake. And yes, I do plan to file bugs, not just rant, but sometimes you need to vent first