The Substantially Similar Weblog

Before I forget: I have a bunch of files I mirror between Windows/NTFS and Linux/ext4 filesystems that include not only accented characters but curly quotes in the filenames. (I know: the easiest solution would be to just get rid of the extended characters). The curly quotes were created in Windows, so don’t render properly in standard Linux character sets (UTF-8, iso8859-1, iso8859-15, etc.).

This all came up because iTunes under Windows couldn’t find curly-quote files when it was reading from the exported Samba share filesystem rather than an attached NTFS drive. The files showed up as missing because they had different filenames.

The solution was not easily google-able, so for the record, in brief, add this to the [Global] section of /etc/samba/smb.conf:

unix charset = cp1252
display charset = cp1252

And reload Samba.

Also, to make the characters render properly from a terminal on the Linux box, first create the relevant character set:

sudo localedef -f CP1252 -i en_US en_US.CP1252

Now you can use this charset on your Linux box, and, like magic, the curly characters will be back:

export LC_ALL='en_US.cp1252'

I recently upgraded my home router box to Debian Lenny. Everything went fairly smoothly, with a few exceptions. My NFS mounts no longer worked because apparently wildcards are no longer allowed in IP addresses in /etc/exports; the export addresses needed to be translated to subnet format (e.g., 192.168.98.* becomes 192.168.98.0/824).

But after a power failure last night, the router box rebooted and I was no longer able to access the Internet from any clients on my LAN. Strangely, I could ping or traceroute external hosts and perform DNS lookups, but web surfing and ssh timed out after an initial handshake. I noticed by telnetting to port 80 of an external host, I got an error back from an invalid HTTP request (e.g. “oeunthioues”), but if I sent a standard valid request (GET /index.html HTTP/1.0), the connection just hung with no response.

I won’t recount all the false leads I had in diagnosing this problem. It turned out that the Internet-facing NIC on my router box had been reset to a low MTU. By setting the MTU on the LAN clients to that low number, or raising the MTU on the Internet-facing NIC back to 1500, the problem was solved:

# ifconfig eth2 mtu 1500

After restarting networking on the router box, the MTU was again set back down to 576, which is apparently the default MTU for an X.25 network. I have no idea why the interface is getting that value by default (where it wasn’t before), so I just added a hack to /etc/network/interfaces to fix it:

iface eth2 inet dhcp
  post-up /sbin/ifconfig eth2 mtu 1500

Interestingly, pre-up didn’t work.

Hopefully I’ve included enough relevant terms in this entry that others with this problem will find it. It was hard to diagnose because no errors appeared in any log file, and I had partial but not complete connectivity from internal clients to the Internet. My first guess was that it was due to the iptables upgrade, but in fact it was entirely unrelated.

[tags]Debian, Lenny, iptables, firewall, router, MTU[/tags]

There is a widespread problem with the Linux driver for the Realtek 8168/8169 cards where the modules load properly and the card is visible but no link is detected. E.g.:

Jun 21 18:28:41 localhost kernel: r8169: eth0: link down
Jun 21 18:28:41 localhost kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready

There are lots of details and suggested solutions from the Ubuntu people. None of the suggestions worked for me, however. Several of them suggest configuring the card under Windows, but the box containing this device is a single-boot linux fileserver. The “wake-on-LAN” functionality seems to be implicated, but not in a way I can see how to fix.

After much head-banging (in the bad sense), I found a simple solution:

(1) Install ethtool
(2) Modify /etc/network/interfaces as follows (substitute your r8169 interface for ‘eth1’ and other settings accordingly):

iface eth1 inet static
pre-up /usr/sbin/ethtool -s eth1 autoneg off
address 192.168.98.1
netmask 255.255.255.0

This did the trick for me where no other solution would work. Of course, link autodetection no longer occurs, but that’s a small price to pay for connectivity.

This is a Debian etch installation using a slightly more recent kernel (2.6.25-2-686).

As an interesting side note, on this new box, the interface appears as eth0 in the kernel logs, but is actually mapped as eth1. Similarly, a second Ethernet interface appears in the log as a different device number than that by which it is referenced. Any ideas why?

Update 6/22/08: Still not getting 1000BaseT (Gigabit), however. If I force 1000BaseT with ethtool -s eth1 speed 1000, the link goes down again (even with autoneg off). The same card in another box, however, detects the link and goes to 1000BaseT automatically. So I’m stuck at 100BaseT.

Update 6/24/08: Linux 2.6.26-rc5 fixes the problem 100% for me.

[Tags]realtek, r8169, linux, drivers, hacking[/Tags]

Quick link to ClearStationery.bas

I don’t have much (any?) history of posting tips for the Windows platform, but I’m currently stuck with it for daily work use, so I figured I might as well share some tips that my readers who happen to be in the same predicament will find useful. (Planet Debian readers please have mercy.)

One of the worst things you that Microsoft Outlook allows a user to do is select a “stationery” for email. Stationery goes beyond regular old HTML mail (e.g., fonts, colors, and bullet lists) to add a patterned background, invariably rendering the content much less readable than it would be with a white (or even any other color) background. What’s worse is every reply to an email with stationery also adopts the original sender’s stationery!

I searched quite a bit for a solution that does not involve sending a nastygram to the original sender. Of course you can convert the email to plain text (or set Outlook to only display the plain text version) and then convert back to HTML or Rich Text, but you’ll also lose other formatting that you might want to retain. You could cut and paste the text into a new email, but what is really needed is a simple VBA macro that will strip the stationery but not other formatting.

Strangely, I don’t think that macro already exists. So I wrote one, to some extent cribbing from related code snippets (mostly from here). I now present to the world ClearStationery.bas, my best contribution to date to the Outlook ecosystem. Simply paste it into your Outlook Visual Basic Editor (ALT-F11) and then map the macro ClearStationeryFormatting() onto a toolbar with a hotkey, and you can instantly remove stationery from any email, whether it is in the “preview” pane or the full message view.

Comments, bug reports, and improvements are welcome:

Sub ClearStationeryFormatting()
On Error GoTo ClearStationeryFormatting_Error
    Dim strEmbeddedImageTag As String
    Dim strStyle As String
    Dim strReplaceThis As String
    Dim intX As Integer, intY As Integer
    Dim myMessage As Outlook.MailItem

    ' First, check to see if we are in preview-pane mode or message-view mode
    ' If neither, quit out
    Select Case TypeName(Outlook.Application.ActiveWindow)
        Case "Explorer"
            Set myMessage = ActiveExplorer.Selection.Item(1)
        Case "Inspector"
            Set myMessage = ActiveInspector.CurrentItem
        Case Else
            MsgBox ("No message selected.")
            Exit Sub
    End Select

    ' Sanity check to make sure selected message is actually a mail item
    If TypeName(myMessage) <> "MailItem" Then
       MsgBox ("No message selected.")
       Exit Sub
    End If

    ' Remove attributes from <BODY> tag
    intX = InStr(1, myMessage.HTMLBody, "<BODY", vbTextCompare)
    If intX > 0 Then
        intY = InStr(intX, myMessage.HTMLBody, ">", vbTextCompare)
        strReplaceThis = Mid(myMessage.HTMLBody, intX, intY - intX + 1)
    End If

    If strReplaceThis <> "" Then
        myMessage.HTMLBody = Replace(myMessage.HTMLBody, strReplaceThis, "<BODY>")
        strReplaceThis = ""
    Else
        Err.Raise vbObjectError + 7, , "An unexpected error occurred searching for the BODY tag in the e-mail message."
        Exit Sub
    End If

    ' Find and replace <STYLE> tag
    intX = InStr(1, myMessage.HTMLBody, "<STYLE>", vbTextCompare)
    If intX > 0 Then
        intY = InStr(8, myMessage.HTMLBody, "</STYLE>", vbTextCompare)
        strReplaceThis = Mid(myMessage.HTMLBody, intX, ((intY + 8) - intX))
    End If

    If strReplaceThis <> "" Then
        myMessage.HTMLBody = Replace(myMessage.HTMLBody, strReplaceThis, "")
    End If

    If InStr(1, myMessage.HTMLBody, "<center><img id=", vbTextCompare) > 0 Then
        strEmbeddedImageTag = "<center><img id="
        '"<center><img id=""ridImg"" src="citbannA.gif align=bottom></center>"
        intX = InStr(1, myMessage.HTMLBody, strEmbeddedImageTag, vbTextCompare)
        If intX = 0 Then
            Err.Raise vbObjectError + 8, , "An unexpected error occurred searching for the embedded image file name start tag in the e-mail message."
            Exit Sub
        End If
        intY = InStr(intX + Len(strEmbeddedImageTag), myMessage.HTMLBody, " align=bottom></center>", vbTextCompare)
        If intY = 0 Then
            Err.Raise vbObjectError + 9, , "An unexpected error occurred searching for the embedded image file name end tag in the e-mail message."
            Exit Sub
        End If
        strEmbeddedImageTag = Mid(myMessage.HTMLBody, intX, intY - intX)
        intX = InStr(1, myMessage.HTMLBody, "<CENTER>", vbTextCompare)
        intY = InStr(intX, myMessage.HTMLBody, "</CENTER>", vbTextCompare)
        strReplaceThis = Mid(myMessage.HTMLBody, intX, intY - intX) & "</CENTER>"
        myMessage.HTMLBody = Replace(myMessage.HTMLBody, strReplaceThis, "", , , vbTextCompare)
    End If

    ' Finally, saved modified message
    myMessage.Save

    On Error GoTo 0
    Exit Sub

ClearStationeryFormatting_Error:

    MsgBox "Error " & Err.Number & " (" & Err.Description & ")"
    Resume Next
End Sub

Stationery-B-Gone!

I’ve been playing around with Gallery and wpg2. I’m still a bit puzzled attempting to integrate Gallery and WordPress. I’ve resolved most issues; the main remaining issue is to display images in the Ajaxian theme without running over the borders in the Ajax/slideshow views. Also, the embedded image apparently doesn’t render in the RSS feed. Update: I’ve given up on the G2 tinymce plugin and the WPG2 tag for now and just hardcoded the image and album URL. Update 2: now the embedded image is working again for no good reason. Suggestions on the entire configuration are welcome.

In any case, I took some pretty photos today in our back yard (use left and right arrow keys to scroll through images after clicking on the one below — I still can’t get the navigation icons to appear):

Before: Proof of Spring 2007.

[Tags]Autumn, Foliage, Trees, WordPress, WPG2, Gallery[/Tags]

There are a lot of pages out there explaining how to convert your mail from mbox to maildir. I’m going to add one more, resurrecting the “tips” section of my blog. (All the tips from my blosxom days are grouped together on one page — someday I’ll get around to breaking those up properly.)

If nothing else, I’ll never have to write these instructions in an email to someone again, instead I’ll just send them the link to this entry. I hope these steps are specific enough so anyone savvy enough to even have this problem in the first place will be able to follow them.

One thing I find consistently concerns people when converting to maildir is how to make sure no mail is lost in the transition. The steps below address that concern. It’s really not hard at all once you understand how maildir works.

First, I’m assuming you have mb2md installed and procmail support. Also, if you plan to access your mail through IMAP, your IMAP server needs to know to use ~/Maildir if ~/Mail does not exist. This is often default behavior (at least it is for Dovecot).

Finally, I assume that your inbox is in /var/mail/username, and your other mail folders are in ~/mail.

So, here goes:

1. Edit your .procmailrc. The first two lines should read:
   MAILDIR=$HOME/Maildir
DEFAULT=$MAILDIR/
   This will cause all new mail to be delivered (after you save your .procmailrc) to be stored in maildir format under ~/Maildir. The key for procmail to know to use the maildir format is the trailing slash in the “DEFAULT” line. Note that under the new system, your inbox and all your folders are kept in ~/Maildir, rather than having an inbox in /var/mail and your folders under your home directory. (This is nice because it makes it easier to back up all your mail together, among other things).
2. If you have any specific procmail rules in your .procmailrc, you’ll also need to change the target folders for that. For example, if you have a spam filtering rule like this:
   :0 H
* ^X-Spam-Level: *****
spam
It needs to be changed to this:
   
:0 H
* ^X-Spam-Level: *****
.spam/
   The trailing slash, again, tells procmail to use maildir rather mbox format for delivery. The leading dot will make the folder appear properly for your IMAP client (there’s a trick I suggest below for making “dot” folders easy to access with a command line client like mutt).

   If you have mail being delivered to a nested mail folder, an additional change is needed because maildir has a flat hierarchy. So if your mail was being delivered to “misc/other/spam” the new target should be “.misc.other.spam/”.

3. Save your .procmailrc. From this point forward, all mail will be delivered to your maildir folders. You can now import all your old mail to the new folders without concern. Since maildir stores each individual message as a separate unique file, you don’t need to worry about any overwriting issues while you are making the transition. New mail will be delivered to newly named files; old mail will be imported to other, unique, newly named files.
4. Import your mail folders:
   mb2md -s ~/mail -R
   The -R just means to import recursively through all your folders; if your mail folder is Mail rather than mail, use that instead. No destination is needed because mb2md uses ~/Maildir by default.
5. Import your inbox:
   mb2md -m
6. Move your old inbox out of the way. Most people won’t actually be able to move or delete files in /var/mail (since they don’t have directory write privileges), so the easiest thing is probably to just open the inbox in your IMAP client or with mutt and save or delete all the messages.
7. Move your old mail folder out of the way:mv ~/mail{,.bak}Once you’ve confirmed that everything is okay, you should be able to delete ~/mail.bak (or compress it and move it elsewhere.) If you saved your inbox to mail folder in step 6, that will get moved out of the way here as well.
8. That’s it, you’re done.
9. Okay, so you’re not quite done if you access your mail with mutt rather than an IMAP client. (You could just point your mutt at your local IMAP server, of course). I’ve found the most convenient way to access the “dot” folders with mutt is to create a parallel symlinked folder structure without the dots. Here’s a hacked-together shell script that will do that. This could obviously be rewritten to be much more elegant (probably in perl). But we all have our messy scripts, right?
   
   #!/bin/bash

   cd ~/Maildir

   for x in .*
   do
   y=${x##.}
   if ( echo $y | grep -q '.' )
   then
   mkdir -p ~/mymail/${y%.*}
   fi
   ln -s ~/Maildir/$x ~/mymail/`echo $y | sed "s/.///g"`
   done

   for x in *
do
ln -s ~/Maildir/$x ~/mymail
done
   Then, just add these lines to your .muttrc file:
   set folder="~/mymail"
set spoolfile="~/mymail"

Any questions?

As a result of the Verizon v710 Bluetooth Class Action Lawsuit Settlement, I just got the LG VX9800, aka “LG the V.” It’s actually not as mediocre as I thought—more on that later. In keeping with my tradition of posting solutions to problems that are not easily found on google (an omnibus entry I plan to separate out into separate entries some day), I’m posting this tip about getting on to Verizon’s high speed (“3G”) network from Linux over a Bluetooth connection.

I was able to get on with no problem with my old phone — the v710. I was surprised when wvdial kept failing with the new phone. The key, it turns out, was adding the following to /etc/ppp/options:

 -chap -mschap -mschap-v2 

(I’m not even sure all of those are necessary, since -mschap and -mschap-v2 are not documented, but they can’t hurt.)

Here’s the whole process:

1. Establish a Bluetooth connection with the phone.
   • Make the phone “discoverable” — go into Bluetooth Options and set it to be visible to all devices
   • hcitool scan to find the phone’s Bluetooth ID (henceforth ##:##:##:##:##:##)
   • hcitool cc ##:##:##:##:##:##
   • hcitool auth ##:##:##:##:##:##
2. Add a stanza to /etc/bluetooth/rfcomm.conf:
   

    rfcomm { bind yes; device ##:##:##:##:##:##; channel 8; } 

3. Add a stanza to /etc/wvdial.conf:
   

    [Dialer Defaults] Modem = /dev/rfcomm0 Carrier Check = No Baud = 115200 Init1 = ATZ Init2 = AT$QCMDR=3 Phone = #777 Username = [Your Verizon 10 Digit Phone Number]@vzw3g.com Password = vzw 

4. As dicussed above, add to /etc/ppp/options:
   

    -chap -mschap -mschap-v2 

You’ll of course have to deal with permissions issues (on /dev/rfcomm0), have the packages installed (bluetooth, wvdial, ppp), etc., but that sort of information is available elsewhere.

It should be obvious that this method costs you airtime. But in my experience Verizon imposes no other charges. I get about 16 kilobytes/second out here in Vermont, which isn’t bad compared to dial-up.

Finally, I haven’t nailed this down yet, but in some situations (maybe where you are outside EV-DO range), you may need to do the following:

• Open the phone
• Press OK – 0 – 000000
• Press 3, Network Select
• Press 1, Mode Preference
• Select 1X only.

(Remember to change it back to Digital Only Hybrid when done)

Finally, there is already an excellent page describing how to use the VX9800 (“LG the V”) on Linux. I would not want to displace its PageRank.