Fri, 22 Aug 2008
dd, dd_rescue, and ddrescue
The short answer: "Use GNU ddrescue. GNU stands for Quality."
dd is a classic UNIX utility to read from and write to files (often devices). Typically, one uses it to copy a hard disk to a file, or to image a hard drive by copying a backup onto it.
One hits a problem when the hard disk has errors. In this case, dd abruptly stops working in the middle, reporting an "Input/output error." But when the hard disk has errors, usually what you want is to get an image of all the blocks on the hard disk that are readable - not just the first few before the first error!
(Note for the pedantic: Yes, I know about dd conv=notrunc,noerror. They're so easy to misuse (mostly by forgetting one of those two options) that they're worth avoiding.)
Two tools are available for this particular purpose. Confusingly, one is called ddrescue, and the other is called dd_rescue.
Around 2001, Kurt Garloff wrote dd_rescue. It does what dd does if you pass it some options, but it comes with instructions on how to use it to recover data from drivers, like by running it multiple times or bakcwards. A wrapper script called dd_rhelp automates that process.
When you're running dd_rescue on an obscure OS like Mac OS X 10.3 because you dropped your laptop in Uganda and the Linux partition grew bad blocks and you still want your data, you will find that dd_rhelp is written as a complicated shell script that relies on GNU versions of core system utilities. OS X provides non-GNU versions, and you will waste hours fiddling with compiling those utilities just so you can run some dumb shell script.
In the summer of 2004, the same summer as I dropped my laptop, Antonio Diaz Diaz wrote "ddrescue," a stand-alone C++ tool that does the same things as dd_rhelp, but more sanely and therefore more efficiently. It became an official GNU project. GNU ddrescue, like dd_rhelp, can keep a log file to let itself gracefully pick up after interrputions.
When your hard disk fails, you should turn to your backups. But if you need a tool like these, just remember: "GNU ddrescue."
$ sudo apt-get install gddrescue
Sun, 03 Aug 2008
Real DOS on a virtual disk
Sometimes you need to run DOS programs, like to flash BIOSs on your laptop. Sometimes, if you're Kragen, that lets you fix ACPI on your BIOS, giving you a hope that X will boot up more often than 1 in 3, sound will skip less, and the first PC card you insert will be assigned a valid IRQ. (The last one is particularly interesting: to get a working PC card before the promised joy of the BIOS update, you have to plug in one card, watch it get assigned the mostly broken IRQ 3, plug in a second card, watch it get assigned the useful IRQ 4, and then you can remove the first one. This is a good way to get a wifi card working.)
Here's a simple HOWTO for getting that going on a Linux machine without repartitioning or booting off external media.
I'll refer to aptitude; I'm assuming you're using a Debian/Ubuntu machine so that makes sense.
Step 1: Install syslinux
$ sudo aptitude install syslinux
Now memdisk is in /usr/lib/syslinux/memdisk .
You should copy it to /boot/ in case your root filesystem is encrypted:
$ sudo cp /usr/lib/syslinux/memdisk /boot/
Step 2: Get your DOS floppy in /boot
Debian packages FreeDOS in dosemu-freedos. Unfortunately that doesn't include a floppy image. Instead:
$ cd /boot $ sudo wget http://www.ibiblio.org/pub/micro/pc-stuff/freedos/files/distributions/1.0/fdboot.img
Step 3: Configure GRUB
Put this in your /boot/grub/menu.list and smoke it:
title FreeDOS kernel /memdisk initrd /fdboot.img boot
Step 4: Reboot, and choose FreeDOS!
Ta-da, you're done.
More options
For bonus points, you can customize the floppy disk image. The easiest way to modify is to mount it loopback:
$ sudo mount -o loop,mode=777 /boot/fdboot.img /mnt/
Then you can copy files into /mnt/, and then when you're done:
$ sudo umount /mnt/
Ta-da, the image has been changed! (Thanks to Kragen for confirming that this actually works.)
The lame old way to customize the image is to use "mtools."
P.S. Thanks to Albert Lee for explaining this trick to me in the first place!
Wed, 16 Jul 2008
IP over Avian Carrier: Security implications
BBC News has a story on Brazil's pigeon drug mules.
Quoth Bruce Schneier:
I think this is the first security vulnerability found in RFC 1149: "Standard for the transmission of IP datagrams on avian carriers." Deep packet inspection seems to be the only way to prevent this attack, although adequate fencing will prevent the protocol from running in the first place.
Mon, 26 May 2008
IPv6
Kartik pointed me to this post about "The Future without IPv6". IPv6 is the future addressing scheme that the Internet will hopefully be transitioning to in the next decade or so; IPv4 is the current setup. The reason we need a new addressing scheme is simple - we've run out of addresses. The dream of the Internet was "end to end connectivity," but you can't do that if you don't have enough addresses to give everyone on the network an address.
You've seen this every time you open up a laptop and it gets an IP address from a "wireless router" - the IP address created for you by that router actually can't be reached from the broader Internet. Network Address Translation (NAT) is a trick the router plays where it changes the headers on your messages destined for the Internet so everyone else on the network thinks the box sent the message. But this means if you want to do something not allowed by that box in the middle, or allowed but misunderstood, it is in an incontrovertible position to screw that up.
The article writes:
Ubiquitous multilevel NAT means the Internet becomes a system for making TCP connections.
Using the Internet only for TCP connections to me spells the end of decades of Internet innovations like Voice over IP that rely on the flexibility of the Internet. And the fact that these connections must always go to the few servers able to have their own dedicated IP addresses creates a separate class of connection in the Internet world: "consumer" vs. "distributor". That class distinction is what IP was designed to erase.
I don't agree with the author that we will never move to IPv6, but I also know we won't do it fast enough to satisfy me. Luckily, thanks to the "end to end" nature of the Internet, especially IPv6, I can do my own migration now and give my computers both IPv4 and IPv6 addresses. I'll be here in the future, waiting for the rest of you guys.
P.S. I'm already "multiplexing multiple transports over a single TCP connection" with my always-on SSH tunnel. I am aware of the drawbacks he lists.
Mon, 03 Mar 2008
Interactive ext3 performance
In 2001, drobbins published an article on IBM DeveloperWorks remarking that the data=journal mount option improved interactive performance on one test from ca. 70 seconds to 7 seconds.
Even today, the openSUSE wiki echoes this advice. I wonder if it still holds.
Tue, 08 Jan 2008
Passwordless alpine with Dovecot
Do you run a server that people SSH into for email?
Do you like to offer them (al)pine?
Do you use the Dovecot IMAP server?
Do you think your users would like to enjoy IMAP benefits from the comfort of alpine without entering their password?
I do! Here's how: Add this to /etc/pine.conf:
rsh-command=/usr/local/bin/alpine-rimapd %s %s %s %s rsh-path=/bin/sh
And store this in /usr/local/alpine-rimapd:
#!/bin/sh exec /usr/sbin/dovecot --exec-mail imap
chmod +x that sucker and you're off to the races!
Tue, 25 Dec 2007
Sending mail from a laptop
I often find myself on what I would call "hostile" networks: They allow only very limited Internet access, like by blocking port 25 so I can't connect to my mail server. Maybe for you, you're never on filtered Internet access, but your home ISP doesn't let you send mail out when you're not at home, but you want to send email directly from your laptop anyway.
Just do what I do! Let me explain.
Summary
- inetd listens on port 125
- Connections to it go through an SSH tunnel that executes "nc localhost 25" on some mail server
- (Optional) A real MTA runs on the laptop, so that I can send mail when offline; when mail delivery fails temporarily, Postfix queues the message until I get back online.
Justification
- Easy. Apps can be configured to use localhost port 25 (or port 125) with no password.
- Correct: Postfix (when using 25) handles sending mail when offline, and reattempts delivery for me.
- Secure: Encryption all the way through the network, with the icing on the cake that this all looks like SSH, so nosy networkers near your laptop can't even see that's what you're doing.
Implementation in Three Steps
Step 1: ssh tunnel
This is the hardest part. To make things simple, I create a dedicated user on each end.On the remote server (server)
[me@laptop] $ ssh me@server [me@server] $ sudo adduser tunnelendpoint [me@server] $ sudo su - tunnelendpoint [tunnelendpoint@server] $ mkdir .ssh
On the local machine (laptop)
[me@laptop] $ sudo adduser tunnelclient [me@laptop] $ sudo su - tunnelclient [tunnelclient@laptop] $ ssh-keygen -t rsa # make it passwordless [tunnelclient@laptop] $ cat .ssh/id_rsa.pub | ssh tunnelendpoint@server 'mkdir -p .ssh ; chmod 0700 .ssh ; cat >> .ssh/authorized_keys'
On the remote server
[me@server] $ sudo su - tunnelendpoint [tunnelendpoint@server] $ nano -w .ssh/authorized_keysYou'll see a key that starts with "ssh-dss". Before that, add this string and leave a space before "ssh-dss":
command="nc localhost 25",no-X11-forwarding,no-agent-forwarding,no-port-forwarding
(Note: "nc" is in the netcat package.)
On the local machine (laptop)
[tunnelclient@laptop] $ ssh tunnelendpoint@server 220 rose.makesad.us ESMTP Postfix (Debian/GNU): "every tragedy is a beauty that has passed"
Hooray! If you see a reply like mine that starts with "220", then all is well.
You're done with the hard part. Now the easy parts.
Step 2: inetd
[me@laptop] $ sudo aptitude install openbsd-inetd
Now edit /etc/inetd.conf to have this line:
127.0.0.1:125 stream tcp nowait tunnelclient /usr/bin/ssh -q -T tunnelendpoint@server
Now restart the inetd (sudo /etc/init.d/openbsd-inetd restart) and test it:
[me@laptop] $ telnet localhost 125 220 rose.makesad.us ESMTP Postfix (Debian/GNU): "every tragedy is a beauty that has passed"
Step 3: Postfix (optional)
This is my favorite part, but it's only necessary if you plan to send email when you're not connected to the Internet.
Just install Postfix, and add this to /etc/postfix/main.cf:
relayhost = 127.0.0.1:125
Restart Postfix and you should be set. Try sending some mail!
Closing
I was inspired by a Debian Administration post, except I had my own ideas about the best way to do it. I still like my way best.
One problem with the above approach is that it requires root on "server". It would be possible to do the ssh tunnel thing without using a separate "tunnelendpoint" account, but instead to add that key to your regular username.
Tue, 23 Oct 2007
Announce and discuss lists
I have a habit of entring a community and leaving both an announce and a discuss list wherever I go. The wisdom of this is still unresolved. I thought I'd share one thing I do beyond that: set the reply-to header on the announce list to go to the discuss list.
That way, when there's an announcement and the peanut gallery wants to add something, they'll reply and the people interested in hearing more will hear it.
I remembered this upon reading that the BALUG lists have the same sort of split, and that in particular that they were considering (on an opt-out basis) auto-adding people from discuss to announce.
Update 2007-10-24: It actually works now. I worked around what appears to be a bug in the Python email module. Report pending. I'll post a screenshot in a day or two and call this project a success, except for all the other non-Mailman services to index.
