Update – 21 Jun 2006: This post just featured in DWN even though its hardly fresh news! I’ve since updated the scripts to use reprepro, so I would highly recommend you read the updated post. The remainder of this page exists only for historical interest!

Hello Planet Debian ,

Steve Kemp wants a solution to magically build uploaded packages. I maintain several small Debian archives and recently needed a solution like this too. The solution needs to provide semi-automated package builds and installation. The full Debian Archive Kit (DAK) was far too heavy for my needs.

I ended up creating a couple of wrapper scripts around sbuild and debarchiver to achieve this. The basic flow of packages through my archives goes like this:

1. Packages uploaded to an incoming directory (/home/sbuild/incoming/<dist_name>)
2. process_packages script scans incoming directory for each dist every minute and spawns builds (via sbuild) as necessary
3. Once packages are built site admin is emailed build logs
4. Admin checks packages and runs add-packages to add the packages to the archive
5. add-packages invokes debarchiver to do the heavy lifting and then generates the release files.

There is no support for GPG signed packages at this stage, hence the manual check before packages are added to the archive. The scripts were a quick hack a few months ago, and I’ve just hacked them again to pull all the site specific stuff out into easily modifiable variables at the top of the scripts, so hopefully I haven’t introduced any syntax errors! You will need correctly configured sbuild and debarchiver setups prior to running these scripts.

If you want to use them, feel free, I’d love to know if they are useful to anyone but me. If there is some interest I’d be happy to tidy them up, package them properly and maintain them.

Get them from:
http://www.mattb.net.nz/debian/misc/process_packages
http://www.mattb.net.nz/debian/misc/add-packages

I run process_packages from cron with an entry like

*/1 * * * * sbuild /home/sbuild/bin/process_packages &>/dev/null


My /home/sbuild layout looks like

drwxr-xr-x 2 root sbuild 4.0K Jul 31 11:16 bin
drwxr-xr-x 2 sbuild sbuild 12K Oct 26 16:32 build
drwxr-xr-x 6 root root 4.0K Sep 14 12:38 chroots
drwxr-xr-x 4 sbuild sbuild 4.0K Jul 13 09:59 incoming
drwxr-xr-x 2 sbuild sbuild 24K Oct 26 16:31 logs

bin contains the scripts, build contains symlinks to the chroots, and is used as the directory to invoke sbuild from. The resulting .debs are also placed in build. chroots contains the actual distribution chroots, incoming holds a directory for each distribution to process and is scanned by process_packages. logs contains a dump of all the sbuild logs. Debarchiver should be configured to pick up packages out of the build directory.

Let me know if I need to provide more details.

I came one step closer to becoming a Debian Developer today when Alexander Wirt was assigned as my Application Manager. Now I will spend several months working through the NM process with him.

You can keep track of my progress at the follwing URL: https://nm.debian.org/nmstatus.php?email=debian%40mattb.net.nz

I’ve been testing Breezy Badger for the last few weeks to do my bit to help out with the latest release. I have been very dissapointed with Ubuntu’s method of bug reporting and handling. To put it bluntly it absolutely sucks. Debian (which Ubuntu is derived from) has an absolutely kick ass bug tracking system and is one of the best projects that I have ever worked on in terms of being able to report, search, categorise and generally manage bugs. Ubuntu seems to make it hard to do all of these things.

Attempt 1 to report a bug
I upgraded to breezy and was having a problem with Gnome freezing on login. I tracked this down to the fact that I had specified programs to run at startup in the session widget. Removing these prevented Gnome from freezing at the splash screen for two minutes. I wanted to report this bug. I fired up reportbug (standard Debian way of reporting bugs) and typed out the bug report. Went to send it, reportbug dies. It reported that the bug report had been sent to ubuntu-users@lists.ubuntu.com but as of today (a week later) it still hasn’t appeared in the archives there. I tried running report bug several times, and repeatedly got the same error message. It was 1am at night so I gave up on reporting the bug and went to bed.

Looking at my outgoing mail logs seemed to indicate that no outgoing email messages were generated, despite reportbug printing a message telling me that the report had been sent to ubuntu-users@lists.ubuntu.com and copied to my email address.

Attempt 2
Several days after bug attempt to file a bug about Gnome freezing on startup I ran into a problem with the network manager applet. Specifically its right click menu lacks the standard applet options such as Move, Lock to Panel and Remove from Panel. This makes it incredibly difficult to customise your panel to your liking. Having given up on being able to use reportbug to file a bug, I resorted to loading up mozilla and browsing to the Ubuntu website, where they recommend on the bug reporting webpage that you to file bugs in the Ubuntu Bugzilla. I dislike bugzilla a fair amount, it’s clunky, hard to use and I have to register to be able to submit bugs. I hate to think how a new Linux user would fare trying to report a bug this way. Anyway, I resolved to ignore my prejudice against bugzilla and sign up anyway. I trolled through Bugzilla for a while, carefully checking as best I could to ensure that I wasn’t going to file a duplicate bug. In the process of all this searching I continually stumbled across references to bugs filed in Malone and Launchpad that were not present in Bugzilla but were referenced.

From what I understand of Malone and Launchpad they are meant to be all-knowing, all-seeing “applications” that draw together bug reports, patches and releases from every distribution on the planet to make the package maintainers job easier. From what I can see at the moment, all they are achieving is to fragment bug reports and increase the amount of work it takes for maintainers and bug submitters to make useful contributions.

Eventually I decided that the bug wasn’t already reported in Bugzilla or Malone and I filed my report. 7 days later I haven’t had a response and I’m not getting my hopes up that I’ll get one soon.

Attempt 3
I ran a dist-upgrade this morning and pulled down the latest batch of updates. This upgraded (among other things) Xorg and Gstreamer. After the upgrade I attempted to start rhythmbox, which failed to start, displaying an error message telling me that I needed to run gst-register. Other gnome audio applications such as CD Player also reported that my plugin registry was corrupt and I needed to run gst-register. This seemed like a relatively simply bug to me, gstreamer just needs to run gst-register in its postinst script and all will be well. How should I go about reporting this bug? reportbug fails, bugzilla is hard to use, clunky and appears to be ignored (I base this comment on looking at other bug reports, not just on the lack of response to the bug above). I decided to give reportbug another try.

Searching through bugzilla I found that the bug I am encountering had already been reported, a month ago. As of today there has been no response from the maintainer, although there are several “me too” comments and confirmation that the bug is caused by a broken print statement when reportbug tries to display an informational message. I fired up vi and hacked the reportbug code (written in python) to disable the print statement which is entirely informational and then tried to submit the bug again.

Whee, reportbug now runs without errors and completes successfully, displaying nice messages telling me my bug has been submitted and copied to my email address. Great, all is happy in Ubuntu land. NO. Looking at my outgoing mail logs again seems to indicate that despite appearances reportbug still failed to generate any email at all, or if it did, it was submitted directly to Ubuntu’s SMTP server rather than using my local MTA.

Conclusion
Reporting bugs in Ubuntu sucks. There is much confusion over what method to use, Bugzilla appears to be recommended on the website, however cursory examination of the Bugzilla database doesn’t show much activity happening. The reportbug script is buggy and completely fails to work by default, when fixed it doesn’t appear to actually send any email at all, and even if it did send email it appears to be sending it to a completely different location than the Ubuntu Bugzilla which is supposedly the recommended location for bug reports. To complicate things even further, bugs reported from within Canonical seem to be reported in Malone and Launchpad rather than Bugzilla.

To Ubuntu’s defense the website does mention that bug reporting is “currently in a state of flux” and to please use the Bugzilla in the meantime. However that page has not been updated since it was created in April this year, so bug reporting has been in a state of flux for a long time.

What makes this situation all the more painful for me is that Debian’s Bug Tracking System is so excellent. Why Ubuntu haven’t chosen to use this is a mystery to me! The Debian BTS supports:

• Email bug submission and manipulation of existing bug reports
• Bug assignment and categorisation via system defined tags
• Tagging of bugs into user definable categories
• Version support to track where and when a bug is fixed
• Pretty web interface
• Excellent integration with reportbug script
• Easy to use searching
• Integration with the entire Debian package management system

Just about all of these features are missing in Ubuntu’s current setup.

As part of my effort to become more involved in Debian, I’ve recently adopted the PHPwiki packages which have been orphaned for quite a while and are in desperate need of some TLC.

David Moreno Garza is co-maintaining the package with me and sponsoring my uploads into the archive. We uploaded PHPwiki 1.3.10-1 today and it is slowly making it’s way through the buildd network, and hopefully into unstable sometime in the next day or two.

The next step is to package PHPwiki 1.3.11 to bring us in line with the latest upstream release. Once we have done that we need to tidy up the outstanding bugs on the package so that it can be considered for progression into testing.

Alongside all this Debian specific work, I’m also still trying to work on merging WLUG’s patches into PHPwiki 1.3.11 so that we can contribute to the PHPwiki community and hopefully revitalise it!

If you want to help out with this process please read the instructions on WlugWikiPatches, pick a patch and starting merging. Feel free to let me know if you need help.

I’m currently in the process of adopting the Debian PHPwiki Package, which involves fixing a whole lot of bugs and upgrading to PHPwiki 1.3.10 (latest stable release).

1.3.10 changed some internal logic relating to URL handling so the old rewriting scheme no longer works. I’m trying to come up with a new all-singing, all-dancing configuration setup that will allow the package to work “out of the box” in the following 4 setups with as little modification to the default configuration files as possible (ie. as little use of sed/awk to insert config values as possible).

1. mod_rewrite enabled, installed in subdir, eg (http://www.example.org/phpwiki/HomePage)
2. mod_rewrite disabled, installed in subdir, eg (http://www.example.org/phpwiki/index.php/HomePage
3. mod_rewrite enabled, installed in root, eg (http://www.example.org/HomePage)
4. mod_rewrite disabled, installed in root, eg (http://www.example.org/index.php/HomePage

The PHPwiki source code is always installed in /usr/share/phpwiki/. The basic configuration that I’ve come up with is as follows:
apache.conf

<ifmodule mod_rewrite.c>
        RewriteEngine on
        RewriteRule /phpwikidata/(.*)$  /usr/share/phpwiki/$1   [L]
        RewriteRule <url>(.*)       /usr/share/phpwiki/index.php/$1  [L]
</ifmodule>

<ifmodule !mod_rewrite.c>
        Alias  <url>                 /usr/share/phpwiki/
        Alias /phpwikidata/         /usr/share/phpwiki/
</ifmodule>

<directory /usr/share/phpwiki/>
        DirectoryIndex index.php
        Options +FollowSymLinks
        AllowOverride None
        order allow,deny
        allow from all
</Directory>

config.ini

DATA_PATH = /phpwikidata
; Note following two are commented out
;USE_PATH_INFO = false
;VIRTUAL_PATH = /phpwiki

This configuration works perfectly for cases 1 and 2 listed above. It requires the apache config snippet to be generated by the postinst script (<url> is replaced with either ‘/phpwiki/’ or ‘/’), but the actual config.ini file does not need to be processed at all.

However this configuration does not work for cases 3 and 4 as PHPwiki seems to get confused as to where it is installed and starts linking to http://www.example.com//HomePage (double slash) which PHPwiki then interprets as the page ‘/HomePage’ which doesn’t exist. To combat this problem we can modify config.ini to specify the virtual path (or URL) that phpwiki appears at. For example:

config.ini

DATA_PATH = /phpwikidata
USE_PATH_INFO = true
VIRTUAL_PATH = <urlpath>

This configuration now works perfectly for cases 1 and 3 listed above, however it breaks cases 2 and 4. Additionally it requires us to process config.ini to replace the <urlpath> variable. This is bearable however as it is a simple subsitution.

So the summary of the problem is: using my current configuration, the USE_PATH_INFO and VIRTUAL_PATH directives must be present if mod_rewrite is enabled in apache, but must be absent otherwise. This is hard to detect in a postinst script, makes generating the config.ini file much more complex and will no doubt cause havoc with future upgrades.

Does anyone know of a better way that I can make all four scenarios above work with minimal (preferably none) modifications to config.ini?

If not, given that I’d prefer not to put logic into the postinst to detect mod_rewrite state I think I’ll make the package only support installing in a subdirectory and add a notice to the user (README.Debian or similar) telling them how to manually configure it for usage in the webroot. Would this be an acceptable fall back position?