Sunday, May 24, 2009

Share Files Over BitTorrent Using a Local Tracker and Client in Linux

Everyone has heard of the BitTorrent protocol. It's widely used to quickly distribute large files with a low-bandwidth connection. There are also a lot of other features in BitTorrent that make it superior to either HTTP or FTP. For example, pseudo archives, built-in checksums, partial-download resumes, and send/receive bandwidth capping.

This post will show you how to share your personal files using BitTorrent tools that come standard with Ubuntu Linux (Jaunty 9.04). This article will cover the following steps:

  1. Configuring and running a local "tracker" server.
  2. Building torrent files from your personal file(s) you wish to share.
  3. Auto-seed and share your active torrent files.

Configure a Local BitTorrent Tracker Server

Before creating a torrent file, you should have an active tracker server configured and running. It is perfectly acceptable to use a public tracker, but running a personal tracker lets your clients connect over a private network (192.168.*). Also, if your torrent files are for commercial use, you likely do not want trust the reliability of your service to a third party.

The BitTorrent tracker server must be accessible to all your clients. Also, it must have either a static DNS or IP address so your clients can always contact it.

From the command line, install Bittornado, start the tracker, and verify it is running:

$ sudo apt-get install bittornado
$ bttrack --port 6969 --dfile ~/.bttrack/dstate --logfile ~/.bttrack/tracker.log --nat_check 0 --scrape_allowed full
$ firefox http://localhost:6969

The bttrack command will start a tracker that allows clients behind NAT connections. Connecting to the tracker port in FireFox will display a page with known torrents and status of each one.

Finally, be aware that with this configuration, the server will register all torrent files it is notified of. So, potentially someone could use this tracker for their own torrents. If you are on private network this may not be an issue, or you can use additional options to restrict the torrents allowed on the tracker:

--allowed_dir $HOME/btfiles --parse_dir_interval 10 --allow_get 1
These additional arguments to bttrack will force the tracker to only use files located the the ~HOME/btfiles directory. Remember, if you use these options, your torrents must be manually copied into this directory before the tracker will accept seeding requests.

Advanced Bttrack Config and Settings

Simplify the tracker startup with a simple shell script:

$ cat < bin/bttrack.sh 
#!/bin/bash

PORT=6969
DFILE="$HOME/.bttrack/dstate"
LFILE="$HOME/.bttrack/tracker.log"
# GET="--allowed_dir $HOME/btfiles --allow_get 1 --parse_dir_interval 10"
OPTS="--nat_check 0 --scrape_allowed full"

mkdir -p $HOME/.bttrack
bttrack --port $PORT --dfile $DFILE --logfile $LFILE $GET $OPTS
EOF

$ chmod 755 bin/bttrack.sh

You may want to add bttrack to your systems init.d startup scripts. Start with a copy of the default service skeleton file from /etc/init.d/skeleton

$ sudo cp /etc/init.d/skeleton /etc/init.d/bttrack
$ sudo vi /etc/init.d/bttrack

Replace the top section of the /etc/init.d/bttrack file using the example code below. Notice that the --dstate and --logfile values have been changed to a more standard location.

PATH=/usr/sbin:/usr/bin:/sbin:/bin
DESC="BitTorrent protocol tracker"
NAME=bttrack
DAEMON=/usr/bin/$NAME
DAEMON_ARGS="--dfile /var/run/$NAME/dstate --logfile /var/log/$NAME.log
   --port 6969 --nat_check 0 --scrape_allowed full"
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME

do_start()
{
 start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --test > /dev/null \
  || return 1
 start-stop-daemon --start --background --make-pidfile --quiet --pidfile $PIDFILE \
      --exec $DAEMON -- $DAEMON_ARGS \
  || return 2
}

Create a Torrent File

The next step is to create one or more torrents files to provide the necessary information to begin seeding files inside of a BitTorrent client. The following command is not the only method you can use. The other options for creating torrents is using a graphical client, such as Deluge or Transmission. I prefer the command line, since it is more concise:

$ sudo apt-get install bittornado
$ btmakemetafile --piece_size_pow2 SIZE http://URL.COM:PORT/announce SOURCE --target OUTPUT.torrent
SIZE [OPTIONAL]
The torrent's "piece" size as a power of 2, where a "piece" is the smallest transferable unit of data. For example, 512KB (19) and 1MB (20). If not defined, this value is selected automatically. For fast connections, use a larger value such as 4MB (22).
http://URL.COM:PORT/announce
The address of the tracker and port number. The address can either be a DNS name or IP number.
SOURCE
The location of source directory or list of file(s) to add to the torrent.
OUTPUT.torrent [OPTIONAL]
The output name of the torrent file. Changing the name after creating the torrent file will break some BitTorrent clients, so set the name hear. If you leave this value empty, the torrent file will be named using the SOURCE name.
--announce_list LIST [OPTIONAL]
If your tracker has more than one DNS or IP, use this optional argument to set all tracker addresses. Make sure to duplicate the primary announce address, http://URL.COM:PORT/announce, in this list as well. The list can be "," separated for equal tiered trackers, or use "|" separator to indicate secondary trackers. See the man page for more details.

Seeding and Sharing Torrent Files

After you've created the torrent files for seeding, the final step is to start them from any BitTorrent client. If you need to seed files for an extended period of time, I recommend using rTorrent inside of a screen session. This allows you to run the client in the background, and pull-up the text-based GUI at any time.

The following steps will create an rTorrent configuration file that will automatically load torrents from a target directory and begin seeding:

$ cat < ~/.rtorrent.rc
check_hash = no
session = ~/.rtorrent
directory = /data/torrents
schedule = watch_directory,5,5,load_start=/data/torrents/autostart/*.torrent
schedule = untied_directory,5,5,stop_untied=
EOF
$ sudo apt-get install rtorrent screen
$ screen rtorrent

rTorrent should be running in the current terminal. If you are not familiar with screen, here are a few keyboard shortcuts for navigation:

<Ctrl-a> ?
Show screen's help
<Ctrl-a> d
Detach the screen session
<Ctrl-a> k
Kill current screen session
<Ctrl-a> c
Create a new shell window in screen
<Ctrl-a> n
Show next window
<Ctrl-a> p
Show previous window

If you close the screen session, you can reconnect with the command "screen -r".

After launching rTorrent, it is now possible to start seeding the torrents.

  1. Move (or create a symbolic link to) the torrent data into rTorrent's "download" directory. The default download directory is defined in the rtorrent.rc file.
  2. Notice that in the rtorrent.rc file, it also specified the "watch" directory. Place the new torrent files in the "watch" directory, and they will be picked-up by rTorrent.
  3. Go back to the rTorrent GUI, and verify that the torrent is being re-hashed and seeded properly.
  4. Copy or e-mail the *.torrent files you have created to anyone that you want to share your files with

Advanced Torrent File Sharing

Instead of manually sending *.torrent files to your family and friends, a better option is to run a public web server where the torrents can be made available 24/7:

$ sudo apt install lighttpd
$ cd /etc/lighttpd
$ sudo bash -c 'cat < conf-available/40-site-torrent.conf
SERVER["socket"] == ":88" {
   server.document-root    = "/data/torrents/autostart"
}
EOF'
$ sudo /etc/init.d/lighttpd restart

This command creates a local web server running on port 88. The source directory can be mapped directly to the "watch" directory of your BitTorrent client. This is where all the active torrent files are stored while the torrents are active. Once enabled, the torrent files can be accessed from http://localhost:88.

.... Read more!

Saturday, May 2, 2009

Mounting Unknown RAID+LVM Volumes Step-by-Step

Every time your system boots, startup scripts will handle all disk mounting tasks. This is good, yet unfortunate if you ever run in to problems, because you're going to need to know how to get things up without the help of the scripts. On my systems, I run a combination of LVM volumes on top of multiple RAID arrays. This provides data integrity and future flexibility to add more space as needed, and makes recovery very complicated.

There are two cases where you will eventually need to know how this all works. The most likely case is recovering from a boot failure, in which you are relatively familiar with the configuration. An even more difficult case is mounting a set of unknown drives with a RAID/LVM configuration. Keep reading and I'll explain how you can handle even the hardest case like a pro.

If there is a chance you will be mounting two or more LVM Volume Groups with identical names, then read the warning at end before continuing.

  1. After installing all the drives into the system, verify they are detected in /dev/sd*:
    $ ls /dev/sd*
    /dev/sda   /dev/sdb   /dev/sdc   /dev/sdd   /dev/sde   /dev/sdf   /dev/sdf3
    /dev/sda1  /dev/sdb1  /dev/sdc1  /dev/sdd1  /dev/sde1  /dev/sdf1  /dev/sdf4
    /dev/sda2  /dev/sdb2  /dev/sdc2  /dev/sdd2  /dev/sde2  /dev/sdf2
    
  2. Find out which partitions of those are mounted:
    $ df -lhx tmpfs 
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/mapper/lvm_vg0-root
                           10G  3.7G  6.4G  37% /
    /dev/mapper/lvm_vg0-home
                          160G   83G   78G  52% /home
    /dev/md0               99M   16M   79M  17% /boot
    
    This output doesn't directly indicate what partitions are being used. Instead it indicates two LVM volumes; /dev/mapper/lvm_vg0-root, /dev/mapper/lvm_vg0-home and one RAID array; /dev/md0. The next steps will reduce this info to the physical partitions.
    1. Lookup the two LVM volumes lvm_vg0-root and lvm_vg0-home to determine the LVM volume group (VG):
      $ sudo lvdisplay | grep 'VG Name'
        VG Name                lvm_vg0
        VG Name                lvm_vg0
      
    2. The LVM VG, lvm_vg0, maps to one ore more physical volumes:
      $ sudo pvdisplay
        --- Physical volume ---
        PV Name               /dev/md1
        VG Name               lvm_vg0
        PV Size               297.99 GB / not usable 3.25 MB
        Allocatable           yes 
        PE Size (KByte)       4096
        Total PE              76285
        Free PE               32765
        Allocated PE          43520
        PV UUID               mD6jat-SvPv-dOop-P3qZ-T7zC-r8ke-xSYRSj
      
      This output shows us that the LVM pysical volume (PV) is using /dev/md1.
    3. Now, lookup md0 (from the top step) and md1 devices:
      $ cat /proc/mdstat
      Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
      md1 : active raid10 sde2[0] sdd2[1]
            312466688 blocks 64K chunks 2 far-copies [2/2] [UU]
      
      md0 : active raid1 sde1[0] sdd1[1]
            104320 blocks [2/2] [UU]
      
      OK, so the final answer is that md1 is using partitions: sde2, sdd2 and array md0 is using partitions: sde1, sdd1.
  3. Now, create list of unmounted partitions, by filtering out the mounted partitions:
    $ ls --color=none /dev/sd* | grep [0-9] | egrep -v 'sde2|sdd2|sde1|sdd1'
    /dev/sda1
    /dev/sda2
    /dev/sdb1
    /dev/sdb2
    /dev/sdc1
    /dev/sdc2
    /dev/sdf1
    /dev/sdf2
    /dev/sdf3
    /dev/sdf4
    
  4. To find RAID partitions, every device can be scanned for volume UUID value:
    $ mdadm --examine /dev/sda1 | grep UUID
               UUID : ae9a1594:27fc3cec:ceff8456:d50a26a6
    
    With a little more work, all the unmounted devices can be scanned in a single pass:
    $ ls --color=none /dev/sd* | grep [0-9] | egrep -v 'sde2|sdd2|sde1|sdd1' | xargs -i sh -c 'sudo mdadm --examine {} | grep -H --label={} UUID'
    /dev/sda1:           UUID : ae9a1594:27fc3cec:ceff8456:d50a26a6
    /dev/sda2:           UUID : cd72d060:e1139a47:8d379926:979de13e
    /dev/sdb1:           UUID : ae9a1594:27fc3cec:ceff8456:d50a26a6
    /dev/sdb2:           UUID : 2eb0a293:e7705a2f:8d379926:979de13e
    mdadm: No md superblock detected on /dev/sdc1.
    /dev/sdc2:           UUID : 29d74ba3:2dff5d1d:e1635e97:4ded1971
    mdadm: No md superblock detected on /dev/sdf1.
    /dev/sdf2:           UUID : 29d74ba3:2dff5d1d:e1635e97:4ded1971
    /dev/sdf3:           UUID : 2eb0a293:e7705a2f:8d379926:979de13e
    /dev/sdf4:           UUID : cd72d060:e1139a47:8d379926:979de13e
    
    It looks like every device is part of a RAID array except /dev/sdc1 and /dev/sdf1.
  5. Manually sort the list in the last step by the UUID values. Every matching UUID value indicates a single RAID array. Here is what your list should look like:
    /dev/sda1:           UUID : ae9a1594:27fc3cec:ceff8456:d50a26a6
    /dev/sdb1:           UUID : ae9a1594:27fc3cec:ceff8456:d50a26a6
    -----
    /dev/sda2:           UUID : cd72d060:e1139a47:8d379926:979de13e
    /dev/sdf4:           UUID : cd72d060:e1139a47:8d379926:979de13e
    -----
    /dev/sdb2:           UUID : 2eb0a293:e7705a2f:8d379926:979de13e
    /dev/sdf3:           UUID : 2eb0a293:e7705a2f:8d379926:979de13e
    -----
    /dev/sdc2:           UUID : 29d74ba3:2dff5d1d:e1635e97:4ded1971
    /dev/sdf2:           UUID : 29d74ba3:2dff5d1d:e1635e97:4ded1971
    
    Using these value, assemble each RAID array. The /dev/md* is not important as long as you choose a value that does not currently exists in the system.
    $ sudo mdadm --assemble /dev/md2 /dev/sda1 /dev/sdb1 
    mdadm: /dev/md2 has been started with 2 drives.
    
    $ sudo mdadm --assemble /dev/md3 /dev/sda2 /dev/sdf4 
    mdadm: /dev/md3 has been started with 2 drives.
    
    $ sudo mdadm --assemble /dev/md4 /dev/sdb2 /dev/sdf3 
    mdadm: /dev/md4 has been started with 2 drives.
    
    $ sudo mdadm --assemble /dev/md5 /dev/sdc2 /dev/sdf2 
    mdadm: /dev/md5 has been started with 2 drives.
    
    Now check that all the previous steps by printing the mdstat file:
    $ cat /proc/mdstat 
    Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
    md5 : active raid10 sdc2[0] sdf2[1]
          488281984 blocks 64K chunks 2 far-copies [2/2] [UU]
    
    md4 : active raid10 sdb2[0] sdf3[1]
          244094080 blocks 64K chunks 2 far-copies [2/2] [UU]
    
    md3 : active raid10 sda2[0] sdf4[1]
          244094080 blocks 64K chunks 2 far-copies [2/2] [UU]
    
    md2 : active raid1 sdb1[0] sda1[1]
          104320 blocks [2/2] [UU]
    
    md1 : active raid10 sde2[0] sdd2[1]
          312466688 blocks 64K chunks 2 far-copies [2/2] [UU]
    
    md0 : active raid1 sde1[0] sdd1[1]
          104320 blocks [2/2] [UU]
    
    unused devices: 
    
  6. When you are happy with the device mappings, update the /etc/mdadm/mdadm.conf file:
    $ sudo sh -c 'mdadm --detail --scan >> /etc/mdadm/mdadm.conf'
    
    After this step, edit the file to remove any old entries. Because of a bug in mdadm, change the "metadata" version from "00.90" to "0.90".
  7. Now, with the RAID arrays active, scan for LVM volumes. First the physical volumes (PV):
    $ sudo pvscan
      PV /dev/md4   VG lvm_vg0   lvm2 [232.79 GB / 0    free]
      PV /dev/md3   VG lvm_vg0   lvm2 [232.79 GB / 956.00 MB free]
      PV /dev/md5   VG lvm_vg0   lvm2 [465.66 GB / 274.60 GB free]
      PV /dev/md1   VG lvm_vg0   lvm2 [297.99 GB / 127.99 GB free]
      Total: 4 [1.20 TB] / in use: 4 [1.20 TB] / in no VG: 0 [0   ]
    
    Next the volume groups (VG):
    $ sudo vgdisplay 
      --- Volume group ---
      VG Name               lvm_vg0
      System ID             
      Format                lvm2
      Metadata Areas        3
      Metadata Sequence No  58
      VG Access             read/write
      VG Status             resizable
      MAX LV                0
      Cur LV                3
      Open LV               1
      Max PV                0
      Cur PV                3
      Act PV                3
      VG Size               931.23 GB
      PE Size               4.00 MB
      Total PE              238395
      Alloc PE / Size       167859 / 655.70 GB
      Free  PE / Size       70536 / 275.53 GB
      VG UUID               yLjVr3-BKkp-BW33-QIsh-rpZX-byL7-bzF2Xg
    
      --- Volume group ---
      VG Name               lvm_vg0
      System ID             
      Format                lvm2
      Metadata Areas        1
      Metadata Sequence No  3
      VG Access             read/write
      VG Status             resizable
      MAX LV                0
      Cur LV                2
      Open LV               2
      Max PV                0
      Cur PV                1
      Act PV                1
      VG Size               297.99 GB
      PE Size               4.00 MB
      Total PE              76285
      Alloc PE / Size       43520 / 170.00 GB
      Free  PE / Size       32765 / 127.99 GB
      VG UUID               138ikB-v1Z0-381g-Qvi8-dWN2-pgJ7-namuLh
    

    Be very careful if your system indicates identically named LVM Volume Groups (shown above). LVM will NOT let you mount the volumes from one or all of the groups. You must resolve the name conflict before continuing.

    Additionally, you will not be able to rename LVM VGs if either the old or new VG has mounted logical volumes! If one of your VG's LV's is the root volume, you will have to resort to a live-boot CD.

    A VG can be renamed with the following command:
    $vgrename vg0-old vg0-new
    
  8. Finally, mount the new volumes and update your /etc/fstab like normal.
.... Read more!

Thursday, March 5, 2009

Better Google Keyboard Shortcuts using Greasemonkey

I used to be one of those "simple" folk using Google the normal way. This has got to be one of the most aggravating experiences. Even more so when looking for answers on an obscure topic. This is because Google is tricky to use, and getting results always takes multiple attempts. Even with a good search, it's likely the best match is buried a few pages in. The action of typing, scrolling, mousing, then clicking a few links, only to "rinse and repeat" 15 seconds later is enough to inflame my wrist with an RSI in under 20 minutes. After a full day of work, this becomes hazardous to your health.

You might be thinking that this kind of thing doesn't bother you. Well, just wait until you try Google with optimized keyboard shortcuts. You will never want to go back, I promise!

Since sometime back in 2008, I have been using the Google Experimental Keyboard Search feature to the point that it is almost second nature. With it, I can bring up a search and get 10 result pages loading in the background, in under 5 seconds. The speed comes from the fact that you can search and open selected results without ever having to leave the keyboard. Try it out for yourself.

You can easily enable this feature, but there are a few minor annoyances you have to deal with:

  1. The "stickiness" of the feature is tricky and I frequently find that the Google search page eventually resets back to the default mode. The only way to solve this it seems is to go back and re-join the experiment.
  2. There is no way to skip forward or back in the search results. Essentially you are stuck to the first page of results. This actually used to not be an issue, but sometime around late 2008 the J/K keys were disabled from jumping to the next and previous result pages.

Greasemonkey Script

Ok, so what is the solution to the bugs? The answer is Greasemonkey. If you haven't heard of Greasemonkey, it is an add-on for Firefox users (you do use Firefox right?) to tweak any web page you download with custom JavaScript code.

I wrote a custom Greasemonkey script that will always enable the Google Experimental Keyboard Search feature. This script also solves the other two bugs listed above.

Setup

Using Greasemonkey and user scripts is easy.

  1. If you do not have Greasemonkey, install it from the Mozilla Add-on page. Restart Firefox.
  2. Install my "Google Search Enhanced Keyboard Navigation" user script with the install button:

Usage

After the script is installed, the Google search result page will have a selection arrow on the left-hand side. The arrow indicates the active link that can be opened.

The following keyboard commands can control the page navigation:

KeyAction
H Opens the previous results page.
J Selects the next result.
K Selects the previous result.
L Opens the next results page.
O Opens the selected result.
<Enter> Opens the selected result.
/ Puts the cursor in the search box.
<Esc> Removes the cursor from the search box.

Happy Googling!

.... Read more!

Sunday, February 8, 2009

Automated File Synchronization with Rsync, Expect, and Cron

- Easy Backup / Synchronize / Share Files Between Multiple PCs

If you're like me, you probably have two or more Linux system that you frequently switch between throughout the day. It's common to have multiple desk/lab systems at work, while at home manage a server, laptop, and media center.

In a multi-machine, multi-network environment, each system gets customized with unique settings, scripts, documents, and code that need to be manually duplicated across every system. With no easy way to do this, most people resort to manually copying files or making gradual ad-hoc changes on every system, and eventually making typos that linger undetected.

If this sounds familiar to you, then this post is the solution.

After going through this tedium for a few weeks, I implemented a simple way to easily keep multiple Linux machines in sync, and it can also be used to quickly bootstrap a new machine with all of your customized settings. An added benefit is that no 3rd-party software or background process are required, such as the case with DropBox (however if you feel DropBox will work in your situation, it might be the best option).

Overview

The automated file synchronization works as follows:

  1. All files to sync are moved into a new directory (e.g. ~/rsync/) under the home path.
  2. Original file locations are replaced with symbolic links to the new location.
  3. A script synchronizes files in the sync sub-directory to an online server using rsync and SSH.
  4. Any changes to the local files, are eventually uploaded to the server. Also any newer files on the server will replace the out-of-date files on the local systems.
  5. The rsync command does not handle merging multiple changes, so some care may be needed to make sure any changes are made to freshly updated files. This can be accomplished by manually running the sync script.
  6. Lastly, to simplify "syncing" a fresh machine, a setup script can be used to quickly create all custom symlinks.

Online Server Storage

A file server is used as the repository to store and retrieve all data. Ideally the server must be accessible over the Internet and available 24/7. One could designate a local server, if every machine resides on the same network. The only requirements are that the machine allows for SSH logins and has enough storage space to meet your needs.

In my case, I use an online web-hosting company called NearlyFreeSpeech.net. NFS is very unique in that there are no sign-up fees, and you are only billed for the amount of services you use, such as storage space and transfer amounts. For my setup, the cost is somewhere around $0.02 to $0.05 per month. Assuming you have only a few megabytes of data you wish to synchronize, their prices are extremely low.

Synchronization Script

Now for the interesting part. The synchronization script requires rsync and Expect to login to the remote server and perform any file updates. Expect automates the SSH login process so it works without user interaction. Another method to automate SSH logins is to use shared keys, but some web-hosts do not enable this feature. Since this method is very common, I will not cover it here. If you are using NFS hosting, then you should stick with the default method.

Ensure you have the required dependencies for the script.

sudo apt-get install rsync expect

Save the script to the ~/bin or /usr/local/bin directory.

File: sync.sh
#!/bin/bash

ROOT="/home/private/"
SITE="ssh.phx.nearlyfreespeech.net"
USER="my_user_name"
PASSWORD="my_password"
RSYNC_OPTS="-e \\\"ssh -o StrictHostKeyChecking=no\\\" -azuv"

auto_rsync() {
   expect -c "eval spawn -noecho rsync --exclude .*.swp $RSYNC_OPTS $1 $2
      match_max 10000k
      expect \"*?assword:*\"
      send -- \"$PASSWORD\r\"
      expect eof"
}

sync() {
   FILE=$(basename $1)
   DEST=$(dirname $1)
   # download remote site file to current location
   auto_rsync $USER@$SITE:$ROOT$FILE $DEST
   # update remote site file if newer than backup
   auto_rsync $1 $USER@$SITE:$ROOT
}

# backup specific files
sync "$HOME/rsync"

Note: Modify the following variables in the script before using:

ROOTremote directory location to store all files
SITEremote SSH server hostname
USERremote SSH user name
PASSWORDremote SSH password

How To Use the Sync.sh Script

The last line of the script defines the local directory/file to backup to the remote SSH site. The command sync "$HOME/rsync" tells the script that all files in the ~/rsync/ directory should be copied to the remote site. If a newer version exists on the remote site it will not be updated, but instead copied to the local machine.

I find that it is easiest to move all synchronized files under the ~/rsync/ directory so that the script will automatically manage them. Another method would be to add additional directives to specific individual file names. Some users might find this method to be better suited for their needs. Keep in mind that you can not have duplicate filenames on the server in the ROOT directory, and that each directive line initiates a new SSH login (i.e. connection setup overhead).

Symlink Generator Script

If you wish to synchronize common environment files, then a symbolic link must be used in the path to link back to a file synchronized in ~/rsync/. To simplify the management of system files stored in the ~/rsync/ directory, there is a second script to auto-generate symbolic links. First the script:

File: ~/rsync/install.sh
#!/bin/bash

RSYNC="$HOME/rsync"

ln -sf $RSYNC/.bashrc            ~/
ln -sf $RSYNC/.bash_aliases      ~/
ln -sf $RSYNC/.vim               ~/
ln -sf $RSYNC/.gvimrc            ~/
ln -sf $RSYNC/.vimrc             ~/
ln -sf $RSYNC/.forward           ~/
ln -sf $RSYNC/passwordmaker.rdf  ~/

mkdir -p ~/bin
ln -sf $RSYNC/bin/sync.sh  ~/bin/

Warning: This script will replace any destination files that already exist. Be sure that you have saved copies of any files you do not wish to erase.

This script lists many of the files in my home directory that I synchronize across machines. I store the script in the ~/rsync/ directory, so that it is synchronized across each machine for easy execution. Before running the script for the first time on a new machine, make sure you have a backup of any files with localized changes you want to keep.

Notice that .vim is actually a directory. All the files and sub-directories stored in .vim will work correctly with this single symlink.

Another interesting feature is that the sync.sh script is actually stored in $RSYNC/bin/sync.sh. By doing this, the script can be quickly updated across all of the systems.

Cron Setup

To automate the synchronization process, a cron command can be added to each machine. The example defines a bi-hourly update of your files. You can increase or decrease the frequency as needed. Edit your crontab file with the command below:

crontab -e

Then add the following line to the end of the file:

36 */2 * * *     ~/bin/sync.sh > /dev/null

The first number should be a random value between 0 and 59. This defines the minute of each even hour to connect to the remote SSH server. Using a random value prevents multiple systems from trying to update the SSH server at the same time.

.... Read more!

Saturday, January 24, 2009

Solving Digital Audio Playback Issues in Linux

Today I spent hours debugging the digital audio output on my XBMC system running Ubuntu 8.10 (Intrepid Ibex). It turned out the fix was a simple one line command to set the mode of the IEC958 port. In this post I wanted to document a simple set of steps to debug audio output problems in Linux for anyone in a similar situation.

The symptoms of the issue began when all XBMC audio from the digital coax became disabled, with the exception of DVD movie audio (that is AC3/DTS streams only). Getting at least some audio was a helpful start, and it meant the audio kernel module was still loading correctly. Knowing this, I assumed a system setting (either in ALSA or XBMC) had been misconfigured, even though I hadn't made any system changes before the problems.

Since not everyone will be able to test DVD audio output right away, the process below shows you how to first check the detected audio hardware (if any) in your system. Then it steps through other issues that may be blocking a digital audio output device.

Step 1: Setup

First off, make sure you have the ALSA tools installed to help debug the sound hardware.

sudo apt-get install alsa-utils

Step 2: Hardware

Test the state of your current sound hardware. A problem in this area indicates either that your ALSA driver is not working, or not installed. Run the command:

aplay -l

You should see output as shown:

**** List of PLAYBACK Hardware Devices ****
card 0: VT82xx [HDA VIA VT82xx], device 0: ALC861 Analog [ALC861 Analog]
  Subdevices: 0/1
  Subdevice #0: subdevice #0
card 0: VT82xx [HDA VIA VT82xx], device 1: ALC861 Digital [ALC861 Digital]
  Subdevices: 1/1
  Subdevice #0: subdevice #0

The output above tells me that I have two(2) devices, labeled "ALC861 Analog" and "ALC861 Digital". We are only interested in the digital device for now. If you have no devices, or no digital device, than there is something wrong with your ALSA driver.

One item of note, are the "card" and "device" values. If you would like to explicitly access the device then use these values in the form, hw:<CARD,DEVICE>. For example, the digital device on my system would be hw:0,1

Problem: No device listed

In this case your best bet is to find out what audio hardware you have and either search Google or the ALSA website. Your audio hardware can be found with the command:

lspci -v | grep Audio

Problem: Only analog device listed

If you have an analog device, and no digital device. Then its possible that adding an argument when loading the audio driver module will resolve this. In the case of my ALC861 device driver module, it accepts the argument "model=3stack-dig". More information on this step can be found in the ALSA Configuration Guide

Step 3: Mixer Setup / IEC958 Setup

Assuming the last step worked, now make sure the mixer settings are correctly set. The easiest way to do this is to run:

alsamixer

Make sure that the output devices are unmuted including the IEC958 device. A picture of my settings are shown:

Second, you must verify the state of the IEC958 (digital out) device using the command:

iecset

which will produce the output:

Mode: consumer
Data: audio
Rate: 48000 Hz
Copyright: permitted
Emphasis: none
Category: PCM coder
Original: original
Clock: 1000 ppm

Verify that the Data value is set to audio. If it is not set, then run the command:

iecset audio on

My ultimate problem was that this value was not set correctly. Somehow it had switched to non-audio, make sure you check this!

Step 4: Asound.conf

ALSA allows for a set of configuration files to create/modify plugins to the audio hardware layer. These plugins can perform rate conversion and add multi-source mixing capabilities. For now, I will recommend disabling then, as they could introduce unknown problems for now. The ALSA configuration files should either be removed or renamed on your system, until you can verify that the audio hardware is working. The locations of these files are:

  • /etc/asound.conf
  • ~/.asoundrc

Step 5: Test Audio Playback

This is the final step to verify your IEC958/spdif/Digital audio output is working. Before running these commands, it would help (though not required) to stop any processes that could be using the sound hardware. If you hear sound after this point, then you are likely done (or very close). You can use any of these commands to test for sound output:

aplay -D default /usr/share/sounds/alsa/Front_Center.wav
aplay -D plug:spdif /usr/share/sounds/alsa/Front_Center.wav
aplay -D plug:iec958 /usr/share/sounds/alsa/Front_Center.wav
aplay -D hw:0,1 /usr/share/sounds/alsa/Front_Center.wav

Step 6: Playback Software Setup

I am including this section to cover the settings I am using for Digtal Audio in XBMC. Simply navigate to Settings->Systems->Audio Hardware:

  • Audio Ougput -> Digital
  • Dolby Digital AC3 Capable Receiver -> ON
  • DTS Capable Receiver -> ON
  • Audio Output Device -> default
  • Passthrough Output Device -> iec958
.... Read more!

Thursday, December 11, 2008

Vim ReStructureText Macros

This post is about a great new document generator call Sphinx. It was created with the goal of simplifying document creation for Python projects, but can be used for much more. Using a simple text mark-up called reStructuredText, it will output beautiful HTML and PDF documents such as books, manuals, journals, etc.

Writing reStructuredText is extremely easy, however there are a few tweaks we can add to Vim to make adding reStructureText mark-up easier than using MS Word.

An example of reStructuredText from docs.python.org is shown below:

**********************************
Brief Tour of the Standard Library
**********************************

.. _tut-os-interface:

Operating System Interface
==========================

The :mod:`os` module provides dozens of functions for interacting with the
operating system::

>>> import os
>>> os.system('time 0:02')
0
>>> os.getcwd()      # Return the current working directory
'C:\\Python26'
>>> os.chdir('/server/accesslogs')

Be sure to use the ``import os`` style instead of ``from os import *``.  This
will keep :func:`os.open` from shadowing the builtin :func:`open` function which
operates much differently.

Section Macros

You will notice there are a few simple ways to denote sections of text. The most basic method is to underline text to indicate the start of a new section or sub-section. Because this is such a common operation when writing reStructureText, we can easily simplify these steps using macros in the Vim text editor.

Simply add the following macros to your ~/.vimrc file to easily turn any line of text into a reStructuredText section heading.

" Restructured Text
" #########################
   " Ctrl-u 1:    underline Parts w/ #'s
   noremap  <C-u>1 yyPVr#yyjp
   inoremap <C-u>1 <esc>yyPVr#yyjpA
   " Ctrl-u 2:    underline Chapters w/ *'s
   noremap  <C-u>2 yyPVr*yyjp
   inoremap <C-u>2 <esc>yyPVr*yyjpA
   " Ctrl-u 3:    underline Section Level 1 w/ ='s
   noremap  <C-u>3 yypVr=
   inoremap <C-u>3 <esc>yypVr=A
   " Ctrl-u 4:    underline Section Level 2 w/ -'s
   noremap  <C-u>4 yypVr-
   inoremap <C-u>4 <esc>yypVr-A
   " Ctrl-u 5:    underline Section Level 3 w/ ^'s
   noremap  <C-u>5 yypVr^
   inoremap <C-u>5 <esc>yypVr^A

The macro's can be enabled with the following keypress: Control-U, #. '#' must be a number from 1-5. The numbering order of the macro's follow the Sphinx Section Heading mark-up definitions. For example the text 'MY SECTION HEADING', will be modified in the following manor for each macro.

Macro ShortcutFinal Output
Ctrl-U 1
##################
MY SECTION HEADING
##################
Ctrl-U 2
******************
MY SECTION HEADING
******************
Ctrl-U 3
MY SECTION HEADING
==================
Ctrl-U 4
MY SECTION HEADING
------------------
Ctrl-U 5
MY SECTION HEADING
^^^^^^^^^^^^^^^^^^

Stay Tuned ...

In the future, I plan on adding any other useful Vim macros for reStructuredText at the end of this post.

.... Read more!

Thursday, June 5, 2008

Configure TFTPD Server in Ubuntu

I recently needed a TFTP server in Ubuntu for PXE booting my MythTV system. Up until now I was using the built-in DnsMasq server, which was great. When I moved my DHCP service to my Linksys router I needed stand-alone TFTP service. Hopefully this step-by-step HowTo will help someone get up an running quickly.

  1. The first step is to install the TFTP client and server packages on your system. The client, tftp-hpa, is not necessary, but helps to debug your system if anything goes wrong.
    sudo apt-get install tftp-hpa tftpd-hpa

  2. The default directory location to store your TFTP files is /var/lib/tftpboot. You can chose your own location if you don't want to use the default. For now, setup the directory permissions of the TFTP server root directory:
    sudo mkdir /var/lib/tftpboot
    sudo chown nobody.nogroup /var/lib/tftpboot
    sudo chmod 777 /var/lib/tftpboot

  3. Edit the TFTP server configuration file, /etc/default/tftpd-hpa, to put the service in daemon mode. Also, if you are using a custom TFTP directory, change /var/lib/tftpboot to the correct directory location.

    /etc/default/tftpd-hpa
    #Defaults for tftpd-hpa
    RUN_DAEMON="yes"
    OPTIONS="-l -s /var/lib/tftpboot"

  4. Start the TFTP init script.
    sudo /etc/init.d/tftpd-hpa start

  5. There should be an indication that the service started. Also, use netstat to check the port has been opened:
    netstat -a |grep tftp

  6. After adding a file the TFTP server directory, you should be able to access a file using the TFTP client. Assuming the file /var/lib/tftpboot/pxelinux.0 exists, try:
    tftp localhost -c get pxelinux.0
    You will know that everyhing worked if the file pxelinux.0 now exists in your current directory.
.... Read more!