2.6 Suckiness
Has anyone else noticed a sharp decline in the quality and stability of the 2.6 kernels recently? I know that they’ve supposedly done away with “stable” releases and changes are being merged from the development trees straight into the 2.6 mainline, but surely we can still expect some level of stability. I’ve run into some fairly major problems lately.
In early December I tried to upgrade our Biscuit PC distributions kernel package from 2.6.13.1 to 2.6.14 but was stymied by the fact that 2.6.14 seems to have completely removed support for building out of kernel modules against the kernel headers only, the full kernel source is now required to be present. This does not play nicely with Debian kernel packages. I haven’t had time to try 2.6.15 yet to see if this change has been reverted, but its a fairly major thing to drop support for and completely removes the ability for me to build packages for external drivers like madwifi.
Today I was upgrading a machine with software raid from Woody -> Sarge and proceeded to upgrade the kernel too. I pulled down a copy of 2.6.15.4, ran make oldconfig and then verified that all the crucial options were still selected (raid, promise controller, e100, etc). make-kpkg, dpkg -i, reboot. BANG.
VFS: Cannot open root device "md0" or unknown-block(9,0)
Please append a correct "root=" boot option
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(9,0)
A quick investigation revealed the promise controller as the source of the problem
PDC20267: IDE controller at PCI slot 0000:02:0e.0
PDC20267: chipset revision 2
PDC20267: ROM enabled at 0xf4640000
PDC20267: 100% native mode on irq 22
PDC20267: neither IDE port enabled (BIOS)
Now, the Promise controllor is certainly not disabled in BIOS, and 2.4.21 manages to boot off it just fine so what is wrong with 2.6.15!? Google suggested that I needed to enable the PDC202xx_FORCE option, (which was enabled in 2.4.21 but in 2.6.15 is only available if you happen to select the completely unrelated driver for the next model up promise card to what I have – a bug that has apparently been acknowledged since 2.6.9 but still not fixed!!), however that made no difference whatsoever.
After a couple of hours of fiddling around with various kernel settings and boot options (did I mention this machine is 200km away from me…) I gave up and started trying early kernel versions starting with the existing 2.6.13 kernel I was using for the Biscuit PC distribution, which worked first time. So far I’ve not been able to find any other reports of problems with 2.6.15 and the Promise drivers, but from what I saw today they are completely non functional.
The common argument I seem to hear when I usually mention this subject (2.6 suckiness) is “but you should just use your distributions kernel”, to which my reply is “but I do…”, for my desktop. The standard Ubuntu kernels work great and I have had no reason to meddle with them.
However my desktop needs are a far cry from what I need to be able to do for the servers I maintain at work, or the kernel packages that I need to build for our BiscuitPC distribution. I’m certainly not one of those gentoo type people that need the latest and greatest the instant that it comes out, and I don’t try and run the latest kernel just for the sake of it. But the fact is, new kernels bring new features (not to mention bugfixes and security patches) that are frequently needed, especially in the networking and wireless areas. When I have to spend several days debugging and testing every time we need a new kernel only to find some critical flaw that prevents us from using it I have to stop and wonder whether the current kernel development process is working.
Have others had similar experiences with 2.6 kernels recently, is the fact that the latest kernel-image I can find in sid is 2.6.8 confirmation that it’s not just me, or am I really just overly angry due to the hours I’ve spent battling 2.6 lately?
Update: Several people have pointed out that I failed to look for linux-image instead of kernel-image and that 2.6.15 is already in sid. My mistake. And yes, I do plan to file bugs, not just rant, but sometimes you need to vent first 
I have 2.6.15 on my x86 and powerpc machines, installed from sid binary packages.. eg, linux-image-2.6.15-1-686. Perhaps you missed the change from kernel-image to linux-image?
Comment by fuzzie — February 18, 2006 @ 12:17 am
The removal of devfs was really a pain with debian, since it’s needed to use backported versions of yaird. For me the 2.6.11.12 and 2.6.14.7 work fairly well. 2.6.14.7 does have the ICMP remote DOS in it, so a custom patch should be applied on it.
2.6.15 has reports on lkml that there are still a fair bit of bugs in it, but these might be fixed in 2.6.15.4. I currently run or plan on running 2.6.14.7 on my servers with the ICMP DOS patch and the backported udev/yaird packages.
Comment by Geert — February 18, 2006 @ 1:00 am
The kernel-image packages were renamed to linux-image. There is already version 2.6.15 in sid.
StefanB
Comment by StefanB — February 18, 2006 @ 1:11 am
The latest kernel image in sid is 2.6.15-1.
I’m going to venture a guess that you haven’t noticed the kernel naming policy change. Instead of kernel-image-xxx, the format of all kernels is now linux-kernel-xxx.
And, just to help ease some of your concerns, I’ve yet to experience any significant problems with the Debian-packaged 2.6 kernels.
Comment by Stephen Touset — February 18, 2006 @ 4:08 am
I’ve had that issue before. It’s a bug (I forget the number) in the tools that generate the initrd images. You need to add a line to /etc/mkinitramfs/modules (or something similar) with your IDE controller and rerun mkinitram-fs.
Comment by Benjamin Seidenberg — February 18, 2006 @ 5:18 am
they are called linux-image in sid.
Comment by Justin — February 18, 2006 @ 5:43 am
better search for linux-image if you want something up2date in etch, unstable or experimental
and they are working great.
bugreport against latest 2.6.15 or newer are welcomed at bugzilla.kernel.org
Comment by anonym — February 18, 2006 @ 5:58 am
Run this is your headers directory to use them for module building:
make prepare scripts
That should solve all your problems.
Comment by Jason Clinton — February 18, 2006 @ 6:19 am
Kernel packages in sid have been renamed to use linux- as a stem. Look for linux-image, the current version in sid is pretty much the latest 2.6.15.x. And please report all the problems you encounter by filing a bug against linux-2.6, or contacting debian-kernel mailing list.
Jurij.
Comment by Jurij Smakov — February 18, 2006 @ 7:01 am
“Have others had similar experiences with 2.6 kernels recently, is the fact that the latest kernel-image I can find in sid is 2.6.8″
Maybe you should “aptitude search linux-source”
2.6.15 in Etch
As linux is not the only kernel Debian runs on (FreeBSD, Hurd e.g.), it was renamed from ‘kernel-source”
Well, I didn’t experience such problems, but you must admit that 2.4 to 2.6 is a big jump.
Comment by cf — February 18, 2006 @ 9:18 am
The last time 2.6 was truely usable for me was around 2.6.4. Since then, a succession of breakages of one kind or another have made 2.6 a real pain in the ass. As such, while ALSA is finally fixed for some soundchips on recent kernels, others things like the ext3 driver have seen regular breakage. This whole idea of not having a proper stable branch at all seems to be the cause.
Another issue I noticed is how the kernel cyclically slows to a crawl, which makes the mouse cursor jump to a different random corner of the screen and a few keystrokes go into bit heaven, resulting in sentences I type missing a few words or having heavily misspelt words. Both issues were brought up by others to Linus, in response to his desktop rant a couple of months ago, so I know I’m not dreaming this.
Also, the hard-disk driver, which was pleasantly unobstructive in early 2.6 kernels, is once again back to its annoying 2.4 behavior of stealing all resources and slowing other processes down, including affecting keyboard input and text scrolling, during intensive accesses. Again, this was not the case up until about 2.6.5, so someone must have regressed that in some way. Why?
Comment by Martin-Éric — February 18, 2006 @ 9:51 am
Linux 2.6 SUX!
First, I know that I should STFU because I’m not going to write a patch to fix it but I can’t as I’m not a kernel developer and the knowledge I have about the kernel development is the same as the knowledge I have about T.V internals
Trackback by Foolab.org - Mohammed Sameer's — February 18, 2006 @ 2:18 pm
Matt: the kernel people apparently feel that it’s not really their responsibility to stabilise the kernels users run, it’s the distributors’. Only the “sucker trees” AKA 2.6.x.y get any stabilisation love.
It blows, if you ask me.
Comment by Aristotle Pagaltzis — February 18, 2006 @ 10:19 pm
Well this is certainly an interesting one and something I think about a bit too. Some random points:
- from 2.6.12 I think you need udev really and what’s worse if you try and go back to an earlier kernel you can’t even boot and you often need to use a rescue disk to fix your filesystem
- Linus has recognised that kernel process sux and now the first two weeks of a new kernel are for new features and after that it is supposed to be bug fixes only. This in itself should help substantially in my opinion as you don’t get a major change days before a kernel release
- The stable tree 2.6.x.y is getting a lot of fixes in it. This means either that 1) We are catching up with past bugs or 2) When people fix bugs in development kernels they are backporting it. Probably a mixture of both
- It is hard to get good kernels out there when you don’t have testers testing development code. However I think Linux could really do with some better automated testing.
- I think you should be able to use old include headers. Linux takes great pains to preseve this compatiability
- Part of the problem (a large part IMHO) is that Linux is struggling with growth on a monolithic kernel. As a kernel developer (UML, TCP, DCCP) I wish it was a microkernel design so that my bad code doesn’t kill everybody elses code
- It is bloody hard to debug in the kernel so hard to test things and fix them at times. I have to resort to printk’s as I don’t have a serial console, UML is too unstable and involves invasive changes. qemu looks OK in theory but hard work getting a filesystem going for it…. Any pointers from anybody I would appreciate on this…
Comment by Ian McDonald — February 20, 2006 @ 4:03 pm
Hi,
I had the same problem with 2.6.15.4 (standard Debian Sarge kernel 2.6.8-2 works fine), compiling in the other Promise controller aswell with the FORCE option solved it:
CONFIG_BLK_DEV_PDC202XX_OLD=y
CONFIG_PDC202XX_BURST=y
CONFIG_BLK_DEV_PDC202XX_NEW=y
CONFIG_PDC202XX_FORCE=y
Hth,
Guus
Comment by Guus Houtzager — February 28, 2006 @ 3:47 am
If you want to help track down the bug, try git-bisect on the kernel. Install the package “git-core”, then just “git clone http://www.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git“, build/test, “git bisect bad” if it doesn’t work, then “git bisect good v2.6.13″, then it will continue to hand you kernels to build and test (with you saying “git bisect good” or “git bisect bad”) until it tells you exactly the commit at fault. You can also use “git bisect visualize” to see your progress.
Comment by Anonymous — April 13, 2006 @ 6:42 pm