GNO Lab

Do you have a lot of downloaded material, coming from bittorrent downloads, or from a FTP-session, where the content is encapsulated in big multi-volume archives like RAR and ZIP files (a.k.a. scene releases). Unpacking those archives can take a lot of manual effort before the actual content can be consumed.

This Python script tries to ease the burden of processing this kind of material, by investigating the directory and categorize its files as one of

• archive, i.e. the files belongs to an archive, for example file.rar and file.001, etc. The archive also has a base file, that is, the first file in the archive, usually the file having the extension .rar in case of RAR-archives.
• file, the file is an ordinary file, for example any *.nfo , and *.jpg file.
• ignored, extra files, not needed for consumption, like those hash files (*.diz, etc.) used only to verify the downloaded content, and also sample, i.e a path matching *sample*.

The tool automatically unpacks the archives, and copies the files, to a given destination (or the current directory if you so like). Further, the destination directory can be derived  by removing a incoming base from the folder examined. The part left is appended to the destination given (or the current directory). Using this feature you can mimic  your content structure  within your download structure, without needing to create folders before moving the downloaded content into its final destination – all is taken care of within  btpostprocess.

Examples:

$ btpostprocess --process --incoming-base=/downloads/ready/pictures \
--base-path=/downloads/ready/pictures/cats \
/torrents/cats.torrent MyPictures

Finds archives and files in /downloads/ready/pictures/cats and a sub-folder indicated by the cats.torrent (or directly under /downloads/ready/pictures/cats) and unpack archives and copy files
into a cats sub-folder under MyPictures. The incoming-base is
used to construct a derived sub-folder under the destination folder.

Integration

The script was made with rtorrent in mind. By using  rtorrent’s  on finished capabilities the post processing can be set up using something like (version 0.8.2):

on_finished = process,"execute=btpostprocess,--incoming-base,/srv/incoming,--base-path,$d.get_base_path=,$d.get_tied_to_file=,/srv/depot"

Indicating downloads are to be found under /srv/incoming, and a folder given by the tied torrent file. Any files downloaded are unpacked (in case of archives) and copied into a sub-folder of /srv/depot.

In case you need to run some logic before spawning btpostprocess, for example, deciding if the download path is outside the normal download path, and in that case skip the post processing, all you need is to wrap the btprocess in a more specific on_finished script.

It’s all about automation – DOWNLOAD

New apartment purchased. Ok my idea for it to cost no more than two month salaries didn’t hold… I still earn to little money… Or maybe I live in a too costly area Well. The apartment is though really nice.

Infrastructrure is setup,  with a Samsung A656 in the center, delivering crisp video from my HTPC (running MediaPortal), and from my H/K AVR5500 I can terror my poor neighbours

What to come? Well more damping material in the living room, to stop some echo from the H/K. Hiding of some cables, some painting to to…. and ehh, there are things to do.

Old gateway’s disks crashed, after some eight years of operation; an 450Mh Compaq Pressario. I’ve had disk crashes before, but have always had problems reinserting new ones.  Number one — it takes me a whole day to open the cabinet.

A new one was set up. This time with redundancy in mind, that is RADI1 and RADI5 (and LVM). The computer as it was assembled:

• Gigabyte GA-MA770-UD3
• Dual core AMD 8850
• 4GB ram memory
• Three 640 GB disks

Had some problems with my DFS setup, not making it possible for my Windows XP machine to utilize the DFS share on my local machine. Som googling and the answer:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Mup\Parameters]
"EnableDfsLoopbackTargets"=dword:00000001

My old computer had problem decoding HD-material. I could decode 720, but 1080 was not fun. One day I decided to upgrade my existing system, so here it is:

• GA-MA770-UD3 (rev. 1.0)
  I wanted a simple, relatively new, single PCI-e 16x slot, and supporting optical S/PDIF connection.
• AMD 9850 Quad Core
  OK performance to decode 1080p material; actually I can without lack of performance deocde at least 7 streams simultaneously.
• Head sink, NOCTUA NH-U9B.
• Reuse of my previous ATI 3870 GPU.
• Power Supply, Corsair PowerSupply (PSU) 550W VX ATX.
• Resue of my old Antec chassi.

Main operating system is Windows XP. Media Portal as HTPC software. Video output to a Samung A656 LCD panel.  Sound reproduced by H /K 5500.

A HD3780, silent. Heatpipe. Need a new mother board to be able to taka advantage of all its features.

Things planned to be done in home net.

1. Replace my locally compiled packages to those from the distribution:
   Rationale: Using the package systems makes it easier to keep software up to date, and makes it easier to handle dependencies. All in all, making it easier for the system to grow.

• Fix (correct) startup scripts for
  • lirc
  • firewall
• Bind (√ 2008-02-20)
• DHCP (√ 2008-02-20)
• Apache2 (√ 2008-02-20)
• OpenLDAP (√ 2008-02-20)
• PHP5 (√ 2008-02-20)
• MySQL 5(√ 2009-05-05)
• Samba
• Subversion (√ 2008-02-20)
• Use new local IP subnet (√ 2008-??-??)
  • 192.168.56.0/24
  • Servers: 192.168.56.2-24
  • Workstations: 192.168.56.56-156
  • Reserved:
    • Gateway: 192.168.56.5
    • Wireless Access point: 192.168.56.4
    • WTC Server: 192.168.56.3

Install the distribution’s package and remove the locally compiled files from /usr/local/bin, /usr/local/bin, usr/local/man, etc. That is, I keep my locally handcrafted configuration files, usually stored in /etc, or /usr/local/etc, which is copied/merged to the distribution package’s configuration area.

To make sure I wipe out all my locally installed files, under /usr/local/, the following script was used:

[sniplet dpkg-clean-local-dir]

1. Installation of VPN solution, with capability on hosting LAN reachable from Internet.

There goes another drive, a Maxtor 6B200P0 (203 GB) ATA.

Just bought a new spare… a Samsung HD501LJ (500 GB). Too bad I never buy enough drives to set up a RAID-system

I used R-Studio NTFS to save a whole lot from the old disk. My apps folder (unzipped tools and programs), my Cygwin installation, and a number of media files. I unfortunately lost a number of good DVD concerts, there among Pink Floyd – Pulse

I actually was able to reinititate the faulty drive. A chkdsk may indicate the actual health.

Ok, a chkdsk and the disk is actually up…

Some year ago I built my own Lirc device, a home brew transceiver, which I attached to my Linux machine in my hall. Using a somewhat long telephone cable I could place the receiver and transmitter in the living room. The cable is somewhat hidden, so I’m kind of pleased with the installation.

With the infrastructure in place it was time to utilize software. I first just used the terminal to easily send IR-codes, via irsend. No automation there! I realized I needed an OK remote to be able to control the whole HTPC environment. By using a remote controller having ordinary navigational operations and also the play and record buttons was enough to control all devices accordingly. This is my setup:

+-------+                   .--------|(
|       |                  /     IR Receiver                   +-------------
|       |RS232            /                                \   |
|       +----------------+                                -- ( | IR Remote
|       |                 \                                /   |
|       |LAN               \     IR Transmitter                +-------------
|       +------.            `--------|)
+-------+       \
Linux Box        \
                  \                                   +---------------------+
                   \                    H/K 5500      | .---------------. o |
                  +--------+          +----------+    | |               | o |
                  |        |TV-out    |  ||||    |    | |               | o |
                  |        +--------->|  - - - O |--->| |               |   |
                  |        |          +----------+    | `---------------´   |
                  +--------+                   |      +---------------------+
                  HTPC Computer                |             Television
                                               |
                                               V
                                           5+1 Audio System

Linux Box

The Linux box is where the LIRC device is attached. This box must be configured to receive and transmit IR signals. With a correct LIRC configuration, see below, this can be accomplished.

HTPC Computer

I usually use Zoom Player for playing media files, because of its nifty full screen file navigator. So my primary target is to control this application

Zoom Player needs to be started when when a certain button on the remote controller is pressed. To achieve this a special agent is used–EventGhost. EventGhost can actually be used to connect to a LIRC server, and controlling Zoom Player, using a downloadable plug-in. Though, I’ve chosen to not use EventGhost as my logic controller. Instead the Linux box will be used as orchestrian, sending commands to desired places, i.e. a push approach, when certain input stimulus (IR signals) is received.

The Television

A Phillips PW3818 acts as video display, recieiving the video signal via S-Video.

H/K 5500

A The AVR Receiver, H/K 5500, on which the volume and surround emulation must be chosen. Sound is delivered to a 5+1 loudspeaker system.

The Remote Controller

A Logitech Harmony 525 is used as my only remote. I added the XBox360 device to the Harmony. This device will act as the HTPC controller. It has functions for playing as well as recording, and also navigation buttons. More than needed. And it won’t interfere with my existing devices, that is, I don’t own a XBox!

LIRC Configuration

The Xbox360 configuration was added to my lircd.conf. The controller was included in the big remotes.tar.gz file. I did have some problom using the the signals sent from the virtual XBox360. It was solved by adding

toggle_bit 22

to lircd.conf and the XBox360 device.

My lirc.conf also includes the configurations for my television and my AVR. With this configuration I should be able to recieive signals as well as react and trigger transmitting of signals.

The solution is quite elegant. A simple bash script is used to send commands to Zoom Player, via its remote interface. This happens via a FIFO and a a netcat process, which is run in the background. In case netcat needs to be started/respawned, it does this automatically. Netcat running as a background process is needed, to let Zoom Player send back responses, on a not closed connection. (If not Zoom Players generates errors.)

Now I only needed to map the XBox commands to call the script with relevant commands, which is done via lircrc, for example:

begin
remote = Microsoft_Xbox360
button = UpArrow
repeat = 2
prog = irexec
config = /usr/local/sbin/zprpc VK_UP
end


(More to come)

This weekend I put myself into more work than desired. For a long time I’ve wanted to upgrade my existing 2.4 kernel to a more recent 2.6 ditto. So this weekend was set to to it

I started by downloading the sources, reading the “The Debian Way”, configured, configured, compiled. Run dpkg -i .... “This must work”, I said to myself. Rebooted…. Hm, where did the nice bootloader menu go? I only saw a simple MBR (Master Boot Record) stated in the upper left corner of the display. This is an old Compaq 450Mhz PC, and have parts of the BIOS stored on the first hard drive. OK, something didn’t go as expected. That’s for sure. The system does not even have a CD ROM reader. There is a Zip-drive, and a floppy. Ah, find a boot floppy, so I can boot into the system and make changes to the MBR.

To be noted is that this system is my gateway to Internet. Fortunately, by activating a second NIC on my other server, that machine can act as gateway in situations like this. So, Internet up and going, at least.

I found some files; the Debian floppy installation suite (boot.img, root.img, etc.). Digging in my old shelf I also found some diskettes. Probably nothing important on those . A rawwrite and then reboot into floppy land Error reading track x.… the screen instead stated. Old diskettes! Just rawwrite again, and throw the corrupted diskettes. Every second diskette or so worked. You can imagine the time it took to get a working boot suite!

Rescue boot. Finally I got a prompt and mounted my system partitions (/ and /boot from /dev/hda1 resp. /dev/hda2 under /mnt), and then issue a $ chroot /mnt/root sbin/lilo -P fix to fix the MBR.

Reboot, and I was i heaven! Actually I got back to my previous kernel. A working system! But… where is my network? While I was switching input computer via my keyboard/mouse/screen switcher, deep inside my “cabinet” I almost teared it all down… It all began to fall… (must set up some stopper there). Could my network switch have been broken in the fall? Well, test, test… For some reason I figured out that my NICs had swapped places, i.e. eth0 was now eth1 and vice verse. It only took me around two hours to figure that out! By digging into configuration files I could alter my NIC’s purpose and got it working.

I now had a system anyway. I decided to replace lilo with grub. Easy task; just aptitude etc… etc… and run install-grub. A modern bootloader. And it worked! Some reboots later and I’m confirmed it works.

In my system I could again go and compile the kernel. Configure, tweak, and actually be able to install, and reboot There are some tweaks yet to settle. Only one of the NIC’s is getting up during boot. Some module issue, I guess. I have an ancient serial board (with two extra RS232 ports) which I need to do setpci on to work. It must be included in the boot.

So, that was my weekend. Hope you had one more relaxed.