codewut

warning: Creating default object from empty value in /home/pulsar/webroot/htdocs/modules/taxonomy/taxonomy.pages.inc on line 33.

Load-Sensitive ThreadPool / Queue in Java

Here is a little code-snippet (actually a ready to run jUnit TestCase) which might come handy if you need a fairly open ThreadPool not primarily limited by the number of active threads but rather by a predicted load factor. Latter one might be pretty much everything such as CPU load or a total number of "items" allowed to be processed by the whole ThreadPool at a given time.

If the predicted load is not dynamic enough for you, you might want to add another monitoring thread looking at some indicators (CPU, RAM, I/O) and adjust the LoadTracker's currentLoad value accordingly. Another path would be to skip the monitoring thread and extend the canHandle(load) method of the LoadTracker to respect the current indicator states.

Oh, and please let me know if I am reinventing the wheel, sometimes it is difficult not to.

In retrospect, same pattern could be applied to the Queue beneath the ThreadPool by coupling a LoadTrackableJob with a specific BlockingQueue. I guess you can always make the code / architecture prettier.

  1. public class TestThreadPool extends TestCase
  2. {
  3. private static Log log = LogFactory.getLog(TestThreadPool.class);
  4.  
  5. public void testLoadTracker() throws InterruptedException
  6. {
  7. int maxRunningThreads = 128;
  8. int maxLoad = 500;
  9.  
  10. LoadTracker load = new LoadTracker(maxLoad);
  11. ExecutorService pool = Executors.newFixedThreadPool(maxRunningThreads);
  12.  
  13. for (int i = 0; i < 500; i++)
  14. {
  15. // here you would create your real job and *predict* its impact on the load factor.
  16. // we choose the load to be random.
  17. int predictedJobLoad = (int) Math.round(Math.random() * 10l);
  18. MyJob aJob = new MyJob(load, predictedJobLoad,"job-"+i,this);
  19.  
  20. while (!load.canHandle(predictedJobLoad))
  21. {
  22. log.debug(String.format("WAIT: current load %d and new job is about to be %d", load.get(), predictedJobLoad));
  23. synchronized (this) { this.wait(1000); }
  24. }
  25.  
  26. log.debug(String.format("QUEUE: current load is %d and new job is about to be %d", load.get(), predictedJobLoad));
  27. pool.execute(aJob);
  28. }
  29. pool.shutdown();
  30. pool.awaitTermination(42,TimeUnit.DAYS);
  31. assertEquals(0, load.get());
  32. }
  33.  
  34. private class MyJob implements Runnable
  35. {
  36. private LoadTracker loadTracker;
  37. private int load;
  38. private String jobId;
  39. private Object monitor;
  40.  
  41. public MyJob(LoadTracker loadTracker, int load, String jobId, Object monitor)
  42. {
  43. this.jobId = jobId;
  44. this.loadTracker = loadTracker;
  45. this.load = load;
  46. this.monitor = monitor;
  47. loadTracker.add(load);
  48. }
  49.  
  50. @Override
  51. public void run()
  52. {
  53. log.debug(String.format("RUN: %s with a load of %d", jobId, load));
  54. try
  55. {
  56. Thread.sleep((int) Math.round(Math.random() * 5000l));
  57. }
  58. {
  59. e.printStackTrace();
  60. }
  61. log.debug(String.format("FIN: %s with a load of %d", jobId, load));
  62. loadTracker.remove(load);
  63. if (monitor != null) synchronized (monitor) { monitor.notify(); }
  64. }
  65. }
  66.  
  67. private class LoadTracker
  68. {
  69. private int currentLoad = 0;
  70. private int maxLoad = 0;
  71.  
  72. public LoadTracker(int maxLoad)
  73. {
  74. this.maxLoad = maxLoad;
  75. }
  76.  
  77. private synchronized void add(int load)
  78. {
  79. this.currentLoad += load;
  80. }
  81.  
  82. private synchronized void remove(int load)
  83. {
  84. this.currentLoad -= load;
  85. }
  86.  
  87. public synchronized int get()
  88. {
  89. return currentLoad;
  90. }
  91.  
  92. public synchronized boolean canHandle(int additionalLoad)
  93. {
  94. return ((this.get() + additionalLoad) < maxLoad);
  95. }
  96. }
  97. }

plenty of videos and slides straight from hadoop world

plenty of videos and slides straight from hadoop world: http://is.gd/h1YeD #base #hadoop #bigdata

Why pigz freakin' rock(s)

I find myself quite often in the need to copy large textfiles over the network. Usually one would go with gzip, either transparently by using the compression switch on scp -C or by archiving a file before pushing it over the wire.

Turns out gzip can compress quite well, but it won't saturate your 100mbit line if you do something like this:

cat bigfile.txt | gzip -c | ssh me@other.side 'cat | gunzip -d > bigfile.txt'

While this has the same effect as scp bigfile.txt me@other.side: it will be helpful to understand the alternatives coming up next.

On a pretty decent machine (Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz) I could get about 60% saturation of the network link. So, what can we do? We could lower the compression level to take load of the CPU and shift it towards the network. We could also use an alternative compression algorithm such as LZO. "lzop" is a free implementation available in most common linux distributions, so this might be the easiest way to go:

cat bigfile.txt | lzop -c | ssh me@other.side 'cat | lzop -cd > bigfile.txt'

My initial tests shown that LZO compression level is about 20% lower than gzip's with default settings. Transfer time was almost cut in half on the other hand. So, how do we get 100% network saturation*AND* great compression? Pigz is a parallel gzip implementation, so instead maxing out only one thread as gzip does, it will use all the available cores and threads your fancy server provides. Downside? No debian stable repository packages available yet. But on the other hand: it does not even require a configure script, how about one header file and 2 "c" files. Most probably your remote connection to the server will take longer to refresh the console output than the compilation process itself.

So, emerge, apt-get install, port install or whatever "pv" and have some fun like this:

  1. me@host:/mnt/data1/import$ cat bigfile.txt | pv | pigz -c | ssh me@otherhost 'cat | unpigz > /mnt/data1/bigfile.txt'
  2. 1.83GB 0:00:18 [95.3MB/s] [ <=> ]

Some more numbers: gzip gives me 45 mb/s and lzop 60mb/s

I think "pv" stands for pipe view, it is responsible for the nice stats during the transfer. And yes, this *is* a 100mbit network connection pushing a textfile at ~100mb/s. Nice, isn't it?

You can find pigz over here

PC Keyboard Layout for a Mac (German)

While switching my main workstation over to a Mac I ran into an issue with the keyboard shipped with those new iMac. To be honest, I would not even consider calling it a keyboard, must be some torture tool straight from hell. Luckily I have a couple of those ergonomic Microsoft keyboard to my disposal. Now the tricky part: the physical keyboard layout does not correspond with the results on the screen. A problem definitely worth fixing, so here we go: a German custom keyboard Layout. Fixed Keys: "{ } [ ] | ~ @ \"

Keyboard Layout InstallationKeyboard Layout Installation

Install it by dropping the bundle file into "~/Library/Keyboard Layouts/" - you will probably have to create that directory.

A great companion to that layout is the Microsoft driver for that Keyboard which will allow you to remap the ALT, WIN and / APP Key:

Microsoft Keyboard PrefsMicrosoft Keyboard Prefs

Download the bundle file right here

Sharepoint 2007 Variations Woes

While fighting some more problems with Sharepoint 2007 Variations "System" I had to write this comment in my code today:

  1. // note: there is a bug in Sharepoint 2007 when using ", " within a page name - "foo, bar.aspx" for instance.
  2. // note: This will render the relationship list entry useless. A healthy object id attribute in a
  3. // note: relationship list's entry looks like this:
  4. // note: ows_ObjectID='http://mucskysp04:4711/DE/Seiten/Seite mit Sönderzeichen.aspx, /DE/Seiten/Seite mit Sönderzeichen.aspx'
  5. // note: once "broken" using the naming-"exploit", the relationship will be represented as:
  6. // note: ows_ObjectID='http://mucskysp04:4711/DE/Seiten/variationsseite, mitkommaundleerzeichen.aspx'
  7. // note: note the missing relative path. There is no reason to work around this bug in this tool since the original
  8. // note: entry in Sharepoint won't be functional anyway (no entries will be shown in the variations dropdown)

Cry? Laugh? I think I'll stick with pulling my hair.

Is It Dark Outside

So, here it is. Thanks to http://isitdarkoutside.com you will never have to trouble yourself with the question "is it dark outside?" anymore. This super-advanced Android application will let you focus on more important questions such as "which shoe goes on which foot?" from now on. Thank us later.

Is It Dark Outside QR Code
download APK file here

You are missing some Flash content that should appear here! Perhaps your browser cannot display it, or maybe it did not initialise correctly.

Version 0.1 - 2010-02-07,

  • New icon
  • Public release

Version 0.0.1 - 2010-02-05,

  • better errorhandling (missing location status, no network)
  • Added sunrise and sundset hours to the information dialog

Version 0.0.0 - 2010-02-05
N/A - Dev-Snapshot, link removed

Using a Canvas Element in XUL / Mozilla-Extensions

It's actually pretty simple, but many small things can go wrong. So, here is your small boilerplate code for a XUL overlay which renders a small canvas area in the status bar:

  1. <?xml version="1.0"?>
  2.  
  3. <overlay id="myOverlay"
  4. xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul"
  5. xmlns:html="http://www.w3.org/1999/xhtml">
  6.  
  7. <statusbar id="status-bar">
  8. <statusbarpanel>
  9. <box>
  10. <html:canvas id="myCanvas" width="15" height="15" style="border:1px solid black;"/>
  11. </box>
  12. </statusbarpanel>
  13. </statusbar>
  14. </overlay>

and some simple painting in JS:

  1. var canvas = window.document.getElementById("myCanvas");
  2. var ctx = canvas.getContext("2d");
  3. ctx.fillStyle = "red";
  4. ctx.fillRect(5,5,5,5);

If done right, you will get a result like this

CANVAS in XULCANVAS in XUL

Disable the close button for a dojo / dijit.Dialog

  1. dojo.provide("myWidgets.Dialog");
  2.  
  3. dojo.declare
  4. (
  5. "myWidgets.Dialog",
  6. [dijit.Dialog],
  7. {
  8. // summary:
  9. // extended version of the dojo Dialog widget with the option to disable
  10. // the close button and supress the escape key.
  11.  
  12. disableCloseButton: true,
  13.  
  14. /* *********************************************************** postCreate */
  15. postCreate: function()
  16. {
  17. this.inherited(arguments);
  18. this._updateCloseButtonState();
  19. },
  20.  
  21. /* *************************************************************** _onKey */
  22. _onKey: function(evt)
  23. {
  24. if(this.disableCloseButton && evt.charOrCode == dojo.keys.ESCAPE) return;
  25. this.inherited(arguments);
  26. },
  27.  
  28. /* ************************************************ setCloseButtonDisabled*/
  29. setCloseButtonDisabled: function(flag)
  30. {
  31. this.disableCloseButton = flag;
  32. this._updateCloseButtonState();
  33. },
  34.  
  35. /* ********************************************** _updateCloseButtonState */
  36. _updateCloseButtonState: function()
  37. {
  38. dojo.style(this.closeButtonNode,
  39. "display",this.disableCloseButton ? "none" : "block");
  40. }
  41. }
  42. );

MySQL, Large Result Sets and OutOfMemory related Headaches

Just in case you run into OutOfMemory Exceptions while requesting a large data chunk from the MySQL: the JDBC driver will load ALL (yes, ALL) rows before passing it to your fancy, agile and low-footprint routine. Tweaking the fetchSize property of a statement won't do any good either... well, not without some voodoo. So, here is how you can get the JDBC driver to get you a nice and tight StreamingResultSet:

  1. Connection connection = dataSource.getConnection();
  2. Statement st = connection.createStatement(java.sql.ResultSet.TYPE_FORWARD_ONLY, java.sql.ResultSet.CONCUR_READ_ONLY);
  3. st.setFetchSize(Integer.MIN_VALUE); // Inter.MIN_VALUE <- and ONLY this value, 1,5 or 100 won't fix your problem.
  4. ResultSet rs = st.executeQuery("select * from someREALLYHugeTable");

Oh, and you clowns out there saying "this is normal, just bump up the memory settings for your JVM" - are you NUTS!? Or do you just like your applications exploding out of nowhere after being in production for some time?

Using .NET reflection to determine if a Class implements a generic interface.

Reflection and me? Big friends. With all the love and hate a good friendship should have. A few days ago it was all about hate again. I had a bunch of service classes, some of them would implement a generic interface... Let's call it IHasAdorable - so a Service-Implementation could look like this:

  1. public class MarketMerchant : IHasAdorable<CheeseBurger>, IProductSeller
  2. {
  3. // defined in IHasAdorable
  4. CheeseBurger BuyAdorable()
  5. {
  6. return new AdorableCheeseburger();
  7. }
  8.  
  9. // defined in IProductSeller
  10. IProduct Buy(String eanCode)
  11. {
  12. if (eanCode.equals("12345"))
  13. return this.bigStackOfSmellyFishburgers.Pop();
  14. else
  15. throw new IAmSoSorryException("We don't have any of these!");
  16. }
  17. }

Now lets assume we want to browse through ALL market merchants and have a look if they have any adorable products. Let's skip the iteration process and pay attention to the probing of all market merchants in order to buy a adorable product from each of them. First attempt might be to use "is":

  1. IProductSeller merchant;
  2. // iteration goes here
  3. if (merchant is IHasAdorable<object>)
  4. {
  5. // this wont work. We are selling something very special, not just a stupid object!
  6. // Casting to IHasAdorable<> won't even compile.
  7. }

This one would not work either, although it makes sense to me:
  1. typeof(IHasAdorable<object>).IsAssignableFrom(merchant)

It seems we need to know what kind of objects a unknown merchant sells before we can "see" if those are also adorable. So here is how I got it working. Let me know if you do know a better solution:
  1. // `1 means there is 1 Generic parameter
  2. Type adorableInterfaceType = merchant.GetType().GetInterface("IHasAdorable`1");
  3.  
  4. if (adorableInterfaceType != null)
  5. {
  6. // yay, the merchant has adorable products, what whould those be!?
  7. Type adorableProductType = adorableInterfaceType.GetGenericArguments()[0];
  8.  
  9. // here is the magic we need to get the correct IHasAdorable Type
  10. // with "filled in" generic type.
  11. Type genericAdorableInterfaceType =
  12. typeof(IHasAdorable<>).MakeGenericType(adorableProductType);
  13. MethodInfo mi = genericAdorableInterfaceType.GetMethod("BuyAdorable");
  14. myBagOfAdorableProducts.Add(mi.invoke(merchant,null));
  15. }

Bon Appétit

Syndicate content