Skip to main content.

Sun, 22 Jun 2014

Interactive semi-automated package review (by abusing Travis-CI)

I just did some Debian package review in a somewhat unusual way, and I wanted to share that. I'm hoping other Debian developers (and other free software contributors) that need to review others' contributions can learn something from this, and that I can use this blog post as a way to find out if other people are doing something similar.

It was pretty exciting! At the end of it, I joined #debian-mentors to talk about how my cool process. Someone summarized it very accurately: 

<sney> it almost sounds like you're working to replace yourself with automation

Context about alpine in Debian

(Skip to "Package review, with automation" if you're familiar with Debian.)

I'm the maintainer of alpine in Debian. There are quite a few problems with the alpine package in Debian right now, the biggest of which are:

This doesn't change my deep love for alpine -- I've had that for about half my life now, and so far, I don't see it going away.

A month or so ago, I got a friendly private message from Unit193, saying he had converted the package to the dh style, and also packaged the newer version. They wanted to know if they should clean this up into something high-enough quality to land in Debian.

(In Debian, we have a common situation where enthusiastic users update or create a new package, and aren't yet Debian Developers, so they don't have permission to upload that directly to the "Debian archive", which is the Debian equivalent of git master. Package "sponsorship" is how we handle that -- a Debian Developer reviews the package, makes sure it is of high quality, and uploads it to the Debian archive along with the Debian Developer's OpenPGP signature, so the archive processing tools know to trust it.)

On Friday evening, I had a spare moment, so I sent a private message to Unit193 apologizing for not getting back to them in a reasonable amount of time. Having another person help maintain is a pretty exciting prospect, and I wanted to treat that enthusiasm with the respect it deserves, or at least apologize when I haven't. I was surprised to see a reply within a few minutes. At that point, I thought: I wasn't planning on doing any package review this weekend, but if they're online and I'm online... might as well!

Package review, with automation

Unit193 and I popped into ##alpine on irc.freenode.net, and I started reading through their packaging changes, asking questions. As I asked questions, I wondered -- how will I know if they are going to fix the issues I'm raising?

Luckily, Unit193 wanted to use git to track the packaging, and we settled on using git-buildpackage, a tool that was fairly new to both of us. I thought, I might as well have some executable documentation so I don't forget how to use it. ("Executable documentation" is Asheesh-speak for a shell script.)

One thing I knew was that I'd have to test the package in a pbuilder, or other pristine build environment. But all I had on me at the moment was my work laptop, which didn't have one set up. Then I had a bright idea: I could use Travis-CI, a public continuous integration service, to check Unit193's packaging. If I had any concerns, I could add them to the shell script and then point at the build log and say, "This needs to be fixed." Then there would be great clarity about the problems.

Some wonderful things about Travis-CI:

Since Unit193's packaging was in git (but not on github), I created a git repo containing the same contents, where I would experiment with fixes for packaging problems I found. It'd be up to Unit193 to fix the problems in the Alioth packaging. This way, I would be providing advice, and Unit193 would have an opportunity to ask questions, so it would be more like mentorship and less like me fixing things.

We did a few rounds of feedback this way, and got the packaging to higher and higher quality. Every time Unit193 made a fix and pushed it, I would re-run the auto-build, and see if the problems I spotted had gone away.

While the auto-build runs, I can focus on conversing with my mentee about problems or just generally chatting. Chatting is valuable community-building! It's extremely nice that I can do that while waiting on the build, knowing that I don't have to read it carefully -- I can just wait a few minutes, then see if it's done, and see if it's red or green. Having the mentee around while I'm reviewing it means that I can use the time I'm waiting on builds as fun free software social time. (Contrast this with asynchronous review, where, all alone, I would wait for a build to finish, then write up an email at the end of it all.)

This kind of mentorship + chatting was spread out over Friday night, Saturday night, and Sunday morning. By the end of it, we had a superb package that I'm excited to sign and push into Debian when I'm next near my OpenPGP key.

Implementation details

You can see the final shell script here:

And you can see the various builds here:

The shell script:

I also had a local shell script that would run, effectively:

This was needed since I was basically using Travis-CI as remote shell service -- moreover, the scripts Travis-CI runs are in a different repo (travis-debcheck) than the software I'm actually testing (collab-maint/alpine.git).

Unit193 and I had a technical disagreement at one point, and I realized that rather than discuss it, I could just ask Travis-CI to test which one of us was right. At one point in the revisions, the binary package build failed to build on Precise Pangolin (the Ubuntu release that the Travis-CI worker is running), and Unit193 said that it was probably due to a problem with building on Ubuntu. So I added a configuration option to build just the source package in Ubuntu, keeping the binary package test-build within the Debian sid pbuilder, although I believed that there was actually a problem with the packaging. This way, I could modify the script so that I could demonstrate the problem could be reproduced in a sid pbuilder. Of course, by the time I got that far, Unit193 had figured out that it was indeed a packaging bug.

I also created an option to SKIP_PBUILDER; initially, I wanted to get quick automated feedback on the quality of the source package without waiting for pbuilder to create the chroot and for the test build to happen.

You might notice that the script is not very secure -- Niels Thykier already did! That's fine by me; it's only Travis-CI's machines that could be worsened by that insecurity, and really, they already gave me a root shell with no password. (This might sound dismissive of Travis-CI -- I don't mean it to be! I just mean that their security model already presumably involves throwing away the environment in which my code is executing, and I enjoy taking advantage of that.)

Since the Travis virtual machine is Ubuntu, and we want to run the latest version of lintian (a Debian packaging "lint" checker), we run lintian within the Debian sid pbuilder. To do that, I use the glorious "B90lintian" sample pbuilder hook script, which comes bundled with pbuilder in /usr/share/doc/pbuilder/.

The full build, which includes creating a sid pbuilder from scratch, takes merely 7-10 minutes. I personally find this kind of shockingly speedy -- in 2005, when I first got involved, doing a pbuilder build seemed like it would take forever. Now, a random free shell service on the Internet will create a pbuilder, and do a test build within it, in about 5 minutes.

Package review, without automation

I've done package review for other mentees in the past. I tend to do it in a very bursty fashion -- one weekend day or one weeknight I decide sure, it's a good day to read Debian packages and provide feedback.

Usually we do it asynchronously on the following protocol:

  1. I dig up an email from someone who needed review.
  2. I read through the packaging files, doing a variety of checks as they occur to me.
  3. If I find problems, I write an email about them to the mentee. If not, success! I sign and upload the package.

There are some problems with the above:

I did this for Simon Fondrie-Teitler's python-pypump package recently. We followed the above protocol. I wrote a long email to Simon, where I remarked on various good and bad points of the packaging. It was part commentary, part terminal transcript -- I use the terminal transcripts to explain what I mean. This is part of the email I sent: 

I got an error in the man page generation phase -- because at 
build-time, I don't have requests-oauthlib:

make[2]: Leaving directory `/tmp/python-pypump-0.5-1+dfsg/docs'
help2man --no-info \
	-n 'sets up an environment and oauth tokens and allows for interactive testing' \
        --version-string=0.5.1 /tmp/python-pypump-0.5-1+dfsg/pypump-shell > /tmp/python-pypump-0.5-1+dfsg/debian/pypump-shell.1
help2man: can't get `--help' info from /tmp/python-pypump-0.5-1+dfsg/pypump-shell
Try `--no-discard-stderr' if option outputs to stderr
make[1]: *** [override_dh_auto_build] Error 1

This seems to be because:

➜  python-pypump-0.5-1+dfsg  ./pypump-shell 
Traceback (most recent call last):
  File "./pypump-shell", line 26, in <module>
    from pypump import PyPump, Client
  File "/tmp/python-pypump-0.5-1+dfsg/pypump/__init__.py", line 19, in <module>
    from pypump.pypump import PyPump, WebPump
  File "/tmp/python-pypump-0.5-1+dfsg/pypump/pypump.py", line 28, in <module>
    from six.moves.urllib import parse
ImportError: No module named urllib

$ ./pypump-shell 
Traceback (most recent call last):
  File "./pypump-shell", line 26, in <module>
    from pypump import PyPump, Client
  File "/tmp/python-pypump-0.5-1+dfsg/pypump/__init__.py", line 19, in <module>
    from pypump.pypump import PyPump, WebPump
  File "/tmp/python-pypump-0.5-1+dfsg/pypump/pypump.py", line 29, in <module>
    from requests_oauthlib import OAuth1
ImportError: No module named requests_oauthlib

The deeper problem was a missing build-dependency, and I explained that in my email. But the meta problem is that Simon didn't try building this in a pbuilder, or otherwise clean build environment.

Simon fixed these problems, and submitted a fresh package to Paul Tagliamonte and myself. It happened to have some typos in the names of the new build dependencies. Paul reviewed the fixed package, noticed the typos, fixed them, and uploaded it. Simon had forgotten to do a test build the second time, too, which is an understandable human failure. There was a two-day delay between Simon's fixed resubmission, and Paul signing+uploading the fixed result.

In a pedagogical sense, there's something disappointing about that exchange for me: Paul fixed an error Simon introduced, so we're not teaching Simon to take total responsibility for his packages in Debian, nor to understand the Debian system as well as he could. (Luckily, I think Simon already understands the importance of taking responsibility! In this case, it's just a hypothetical in this case.)

For the future

The next time I review a package, I'm going to try to do something similar to my Travis-CI hack. It would be nice to have the do.sh script be a little more abstract; I imagine that as I try to use it for a different package, I'll discover the right abstractions.

I'd love it if Travis-CI did not require the git repositories to be on GitHub. I'd also like if the .travis.yml file could be in a different path. If so, we could create debian/travis-configuration (or something) and keep the packaging files nice and separate from the upstream source.

I'd also love to hear about other people's feedback. Are you doing something similar? Do you want to be? Would you have done some of this differently? Leave a comment here, or ping me (paulproteus) on #debian-mentors on irc.debian.org (aka irc.oftc.net).

I'll leave you with some conversation from #debian-mentors: 

<paulproteus> The automation here, I think, is really interesting.
<paulproteus> What I really want is for mentees to show up to me and say "Here is my package + build log with pbuilder, can you sponsor it please?"
<Unit193> Oooooh!
-*- Unit193 gets ideas.
<paulproteus> Although the irony is that I actually like the community-building and relationship-building nature of having these things be conversations.
<bremner> how will this brave new world cope with licensing issues?
<paulproteus> bremner: It's not a replacement for actual review, just a tool-assist.
<paulproteus> bremner: You might be relieved to know that much of Unit193's and my back and forth related to get-orig-source and licensing. (-:
<bremner> I didn't doubt you ;).
<paulproteus> If necessary I can just be a highly productive reviewer, but I would prefer to figure out some way that I can get other non-paulproteus people to get a similar assist.
<paulproteus> I think the current blocker is "omg travis why are you bound to githubbbbbbbb" which is a reasonable concern.

[] permanent link and comments

Comment form

  • The following HTML is supported: <a href>, <em>, <i>, <b>, <blockquote>, <br/>, <p>, <abbr>, <acronym>, <big>, <cite>, <code>, <dfn>, <kbd>, <pre>, <small> <strong>, <sub>, <sup>, <tt>, <var>
  • I do not display your email address. It is for my personal use only.

Name: 
Your email address: 
Your website: 
 
Comment: