Why pigz freakin' rock(s)

I find myself quite often in the need to copy large textfiles over the network. Usually one would go with gzip, either transparently by using the compression switch on scp -C or by archiving a file before pushing it over the wire.

Turns out gzip can compress quite well, but it won't saturate your 100mbit line if you do something like this:

cat bigfile.txt | gzip -c | ssh me@other.side 'cat | gunzip -d > bigfile.txt'

While this has the same effect as scp bigfile.txt me@other.side: it will be helpful to understand the alternatives coming up next.

On a pretty decent machine (Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz) I could get about 60% saturation of the network link. So, what can we do? We could lower the compression level to take load of the CPU and shift it towards the network. We could also use an alternative compression algorithm such as LZO. "lzop" is a free implementation available in most common linux distributions, so this might be the easiest way to go:

cat bigfile.txt | lzop -c | ssh me@other.side 'cat | lzop -cd > bigfile.txt'

My initial tests shown that LZO compression level is about 20% lower than gzip's with default settings. Transfer time was almost cut in half on the other hand. So, how do we get 100% network saturation*AND* great compression? Pigz is a parallel gzip implementation, so instead maxing out only one thread as gzip does, it will use all the available cores and threads your fancy server provides. Downside? No debian stable repository packages available yet. But on the other hand: it does not even require a configure script, how about one header file and 2 "c" files. Most probably your remote connection to the server will take longer to refresh the console output than the compilation process itself.

So, emerge, apt-get install, port install or whatever "pv" and have some fun like this:

  1. me@host:/mnt/data1/import$ cat bigfile.txt | pv | pigz -c | ssh me@otherhost 'cat | unpigz > /mnt/data1/bigfile.txt'
  2. 1.83GB 0:00:18 [95.3MB/s] [ <=> ]

Some more numbers: gzip gives me 45 mb/s and lzop 60mb/s

I think "pv" stands for pipe view, it is responsible for the nice stats during the transfer. And yes, this *is* a 100mbit network connection pushing a textfile at ~100mb/s. Nice, isn't it?

You can find pigz over here

PC Keyboard Layout for a Mac (German)

While switching my main workstation over to a Mac I ran into an issue with the keyboard shipped with those new iMac. To be honest, I would not even consider calling it a keyboard, must be some torture tool straight from hell. Luckily I have a couple of those ergonomic Microsoft keyboard to my disposal. Now the tricky part: the physical keyboard layout does not correspond with the results on the screen. A problem definitely worth fixing, so here we go: a German custom keyboard Layout. Fixed Keys: "{ } [ ] | ~ @ \"

Keyboard Layout InstallationKeyboard Layout Installation

Install it by dropping the bundle file into "~/Library/Keyboard Layouts/" - you will probably have to create that directory.

A great companion to that layout is the Microsoft driver for that Keyboard which will allow you to remap the ALT, WIN and / APP Key:

Microsoft Keyboard PrefsMicrosoft Keyboard Prefs

Download the bundle file right here

Sharepoint 2007 Variations Woes

While fighting some more problems with Sharepoint 2007 Variations "System" I had to write this comment in my code today:

  1. // note: there is a bug in Sharepoint 2007 when using ", " within a page name - "foo, bar.aspx" for instance.
  2. // note: This will render the relationship list entry useless. A healthy object id attribute in a
  3. // note: relationship list's entry looks like this:
  4. // note: ows_ObjectID='http://mucskysp04:4711/DE/Seiten/Seite mit Sönderzeichen.aspx, /DE/Seiten/Seite mit Sönderzeichen.aspx'
  5. // note: once "broken" using the naming-"exploit", the relationship will be represented as:
  6. // note: ows_ObjectID='http://mucskysp04:4711/DE/Seiten/variationsseite, mitkommaundleerzeichen.aspx'
  7. // note: note the missing relative path. There is no reason to work around this bug in this tool since the original
  8. // note: entry in Sharepoint won't be functional anyway (no entries will be shown in the variations dropdown)

Cry? Laugh? I think I'll stick with pulling my hair.

Visual Studio and Remote Debugging: The easy way

If you find yourself like me struggling with remote debugging using visual studio: here is a stupid easy way to get things running: READ THE FREAKIN' DIALOG MESSAGES!

So, here is the recipe:

  • Download a copy of the Remote-Debugger from Microsoft (I got mine over here: http://www.microsoft.com/downloads/details.aspx?FamilyID=440ec902-3260-4... )
  • Install it on the target machine
  • Don't bother yourself with setting it up as a service, we'll be running the remote debugging application manually.
  • Make sure the the username and password of your remote machine and the local machine are the same. Otherwise you might run into authentication problems when connecting to the remote host. Easiest way to achive that is being a part of a NT Domain and using the same account on both machines.
  • Now, listen very closely: Remote debugging needs the debug symbols to be available to the remote machine. Here is a rather stupid but dead simple way to get this working: In visual studio head over to Tools->Options->Debugging->Symbols and set it to C:\temp
  • Deploy your "to be debugged application" onto the remote host into C:\temp - include those funny .pdb files. Those are the debug symbols we'll need.
  • See a pattern here? If not re-read the last 2 list items
  • run the app on the remote host
  • Debug-> Attach Process: Enter the hostname of the remote machine (labeled Qualifier), select managed code from "Attach to:", hit refresh and pick your process.
  • Done.

And yes, I am pissed, mostly at myself because I did not pay attention to the debug symbols and was wondering why I could attach to a process but not set any breakpoints. Still a spoiled java coder I guess. No such headaches over there.

XBMC and IPTV / T-Entertain

Almost there... Almost....


You are missing some Flash content that should appear here! Perhaps your browser cannot display it, or maybe it did not initialise correctly.

Is It Dark Outside

So, here it is. Thanks to http://isitdarkoutside.com you will never have to trouble yourself with the question "is it dark outside?" anymore. This super-advanced Android application will let you focus on more important questions such as "which shoe goes on which foot?" from now on. Thank us later.

Is It Dark Outside QR Code
download APK file here

You are missing some Flash content that should appear here! Perhaps your browser cannot display it, or maybe it did not initialise correctly.

Version 0.1 - 2010-02-07,

  • New icon
  • Public release

Version 0.0.1 - 2010-02-05,

  • better errorhandling (missing location status, no network)
  • Added sunrise and sundset hours to the information dialog

Version 0.0.0 - 2010-02-05
N/A - Dev-Snapshot, link removed

Using a Canvas Element in XUL / Mozilla-Extensions

It's actually pretty simple, but many small things can go wrong. So, here is your small boilerplate code for a XUL overlay which renders a small canvas area in the status bar:

  1. <?xml version="1.0"?>
  3. <overlay id="myOverlay"
  4. xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul"
  5. xmlns:html="http://www.w3.org/1999/xhtml">
  7. <statusbar id="status-bar">
  8. <statusbarpanel>
  9. <box>
  10. <html:canvas id="myCanvas" width="15" height="15" style="border:1px solid black;"/>
  11. </box>
  12. </statusbarpanel>
  13. </statusbar>
  14. </overlay>

and some simple painting in JS:

  1. var canvas = window.document.getElementById("myCanvas");
  2. var ctx = canvas.getContext("2d");
  3. ctx.fillStyle = "red";
  4. ctx.fillRect(5,5,5,5);

If done right, you will get a result like this


Using full-blown SQL Server instead of SQL Server Express Edition in Visual Studio / IIS

Dear Diary

Today kicked off rather troublesome. After installing the developer edition of SQL Server 2008 I've decided to get rid of the SQL Server Express Edition. Why would I want to run multiple versions of SQL Server on the same machine? This turned out not to be a very good idea, especially when you are about developing ASP.NET applications using Visual Studio. Once you try to run an ASP.NET application which requires some kind of role management and/or personalization (Web Parts do), you will run into this pretty error:


Yes, the local IIS is still referencing the SQL Server Express datasource. One would assume that you could change those settings somewhere within Visual Studio as you can change the Port of the ASP.NET application. Well… Don’t waste time as I did. It is not that easy, here is what you need to do:

Create a new Database dedicated to the IIS, for the sake of simplicity we’ll call it “IIS”:

Create a new IIS DatabaseCreate a new IIS Database

Locate and run “aspnet_regsql.exe”. In my case it is "C:\Windows\Microsoft.NET\Framework\v2.0.50727\aspnet_regsql.exe". Here you want to stick with the first option (Configure SQL Server for application services) and point it to the freshly created database on the next and let the configuration tool initialize the empty db.

Initialize the new DatabaseInitialize the new Database

Almost done, you still need to point the LocalSqlServer configuration not to use the SQLEXPRESS Server but the dedicated Database. Locate the machine.config and provide the connection string to the IIS Database:

Modify machine.configModify machine.config

Exchange Specs

Dear Diary:

http://msdn.microsoft.com/en-us/library/cc425499.aspx - specifications for the exchange protocol and other related "stuff" (cab-file format for instance) #specs #api #exchange

So, seems there is a HTTP REST-Like interface to Active Sync aaaaand a WEBDAVy way to access your Mailbox... Sounds cool, might be a handy solution to a problem we ran into at the office recently. Here is a small PHP Example: http://www.troywolf.com/articles/php/exchange_webdav_examples.php

Blurry & Shaky Cam version of XBMC SVN runing on Nvidia ION

Woohoo! SVN builds of XBMC are getting pretty smooooooooth and stable. I think that Acer Revo box I purchased some time ago will be able to replace the rockstabe Popcorn Hour A-100 in no time.

Oh, in case you have been wondering what's so special about this: What you see is an Intel Atom based SFF PC powered by the NVidia ION chipset. Yes, 1080p decoding on the GPU. And yes, It's running on Linux, the Windows version of XBMC is not doing a great job regarding offloading video decoding onto the GPU yet.

So far I have found 2 issues with the current build. The decoder uses 16-235 colorspace instead of 0-255, so you will be missing some contrast. Does not apply to all videos though, old DIVX files decode correctly. The other issue is really a minor one, XBMC won't run when you use compiz at the same time.

There is also a new feature in the current dev snapshot I could not give a try, XBMC does support some fancy SD upscaling on the GPU. Unfortunately the ION Chipset does not support the "VDPAU Featureset C" - only newer cards like the GT2XX series do.

And here is the updated video. A bit less blurry / shaky but still crappy.

You are missing some Flash content that should appear here! Perhaps your browser cannot display it, or maybe it did not initialise correctly.

You are missing some Flash content that should appear here! Perhaps your browser cannot display it, or maybe it did not initialise correctly.

tested version: xbmc_9.04.3+svn23321-jaunty1

Syndicate content