development

warning: Creating default object from empty value in /home/pulsar/webroot/htdocs/modules/taxonomy/taxonomy.pages.inc on line 33.

Tiny RGB Firmware

Work in progress, check back later.

Bind hadoop to a specific network device

quick and dirty

hadoop-env.sh:

  1. #replace eth1:0 with your NIC / alias
  2. bind_ip=$(/sbin/ifconfig eth1:0 | grep 'inet addr:' | cut -d: -f2 | awk '{print $1}')
  3.  
  4. export BIND_OPTS="-Dlocal.bind.address=${bind_ip}"
  5.  
  6. # Command specific options appended to HADOOP_OPTS when specified
  7. export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS $BIND_OPTS"
  8. export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS $BIND_OPTS"
  9. export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS $BIND_OPTS"
  10. export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS $BIND_OPTS"
  11. export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS $BIND_OPTS"
append to hdfs-site.xml:
  1. <property>
  2. <name>dfs.secondary.http.address</name>
  3. <value>${local.bind.address}:50090</value>
  4. <description>
  5. The secondary namenode http server address and port.
  6. If the port is 0 then the server will start on a free port.
  7. </description>
  8. </property>
  9.  
  10. <property>
  11. <name>dfs.datanode.address</name>
  12. <value>${local.bind.address}:50010</value>
  13. </property>
  14.  
  15. <property>
  16. <name>dfs.datanode.http.address</name>
  17. <value>${local.bind.address}:50075</value>
  18. </property>
  19.  
  20. <property>
  21. <name>dfs.datanode.ipc.address</name>
  22. <value>${local.bind.address}:50020</value>
  23. </property>
  24.  
  25. <property>
  26. <name>dfs.http.address</name>
  27. <value>${local.bind.address}:50070</value>
  28. </property>
  29.  
  30. <property>
  31. <name>dfs.datanode.https.address</name>
  32. <value>${local.bind.address}:50475</value>
  33. </property>
  34.  
  35. <property>
  36. <name>dfs.https.address</name>
  37. <value>${local.bind.address}:50470</value>
  38. </property>
you will also need to hardcode the HDFS URL in the core-site.xml by hand. This is not an issue since the same config should be deployed on all nodes in the cluster. Same pattern can be applied to mapred-site.xml - I did not need it.

Load-Sensitive ThreadPool / Queue in Java

Here is a little code-snippet (actually a ready to run jUnit TestCase) which might come handy if you need a fairly open ThreadPool not primarily limited by the number of active threads but rather by a predicted load factor. Latter one might be pretty much everything such as CPU load or a total number of "items" allowed to be processed by the whole ThreadPool at a given time.

If the predicted load is not dynamic enough for you, you might want to add another monitoring thread looking at some indicators (CPU, RAM, I/O) and adjust the LoadTracker's currentLoad value accordingly. Another path would be to skip the monitoring thread and extend the canHandle(load) method of the LoadTracker to respect the current indicator states.

Oh, and please let me know if I am reinventing the wheel, sometimes it is difficult not to.

In retrospect, same pattern could be applied to the Queue beneath the ThreadPool by coupling a LoadTrackableJob with a specific BlockingQueue. I guess you can always make the code / architecture prettier.

  1. public class TestThreadPool extends TestCase
  2. {
  3. private static Log log = LogFactory.getLog(TestThreadPool.class);
  4.  
  5. public void testLoadTracker() throws InterruptedException
  6. {
  7. int maxRunningThreads = 128;
  8. int maxLoad = 500;
  9.  
  10. LoadTracker load = new LoadTracker(maxLoad);
  11. ExecutorService pool = Executors.newFixedThreadPool(maxRunningThreads);
  12.  
  13. for (int i = 0; i < 500; i++)
  14. {
  15. // here you would create your real job and *predict* its impact on the load factor.
  16. // we choose the load to be random.
  17. int predictedJobLoad = (int) Math.round(Math.random() * 10l);
  18. MyJob aJob = new MyJob(load, predictedJobLoad,"job-"+i,this);
  19.  
  20. while (!load.canHandle(predictedJobLoad))
  21. {
  22. log.debug(String.format("WAIT: current load %d and new job is about to be %d", load.get(), predictedJobLoad));
  23. synchronized (this) { this.wait(1000); }
  24. }
  25.  
  26. log.debug(String.format("QUEUE: current load is %d and new job is about to be %d", load.get(), predictedJobLoad));
  27. pool.execute(aJob);
  28. }
  29. pool.shutdown();
  30. pool.awaitTermination(42,TimeUnit.DAYS);
  31. assertEquals(0, load.get());
  32. }
  33.  
  34. private class MyJob implements Runnable
  35. {
  36. private LoadTracker loadTracker;
  37. private int load;
  38. private String jobId;
  39. private Object monitor;
  40.  
  41. public MyJob(LoadTracker loadTracker, int load, String jobId, Object monitor)
  42. {
  43. this.jobId = jobId;
  44. this.loadTracker = loadTracker;
  45. this.load = load;
  46. this.monitor = monitor;
  47. loadTracker.add(load);
  48. }
  49.  
  50. @Override
  51. public void run()
  52. {
  53. log.debug(String.format("RUN: %s with a load of %d", jobId, load));
  54. try
  55. {
  56. Thread.sleep((int) Math.round(Math.random() * 5000l));
  57. }
  58. {
  59. e.printStackTrace();
  60. }
  61. log.debug(String.format("FIN: %s with a load of %d", jobId, load));
  62. loadTracker.remove(load);
  63. if (monitor != null) synchronized (monitor) { monitor.notify(); }
  64. }
  65. }
  66.  
  67. private class LoadTracker
  68. {
  69. private int currentLoad = 0;
  70. private int maxLoad = 0;
  71.  
  72. public LoadTracker(int maxLoad)
  73. {
  74. this.maxLoad = maxLoad;
  75. }
  76.  
  77. private synchronized void add(int load)
  78. {
  79. this.currentLoad += load;
  80. }
  81.  
  82. private synchronized void remove(int load)
  83. {
  84. this.currentLoad -= load;
  85. }
  86.  
  87. public synchronized int get()
  88. {
  89. return currentLoad;
  90. }
  91.  
  92. public synchronized boolean canHandle(int additionalLoad)
  93. {
  94. return ((this.get() + additionalLoad) < maxLoad);
  95. }
  96. }
  97. }

Using a Canvas Element in XUL / Mozilla-Extensions

It's actually pretty simple, but many small things can go wrong. So, here is your small boilerplate code for a XUL overlay which renders a small canvas area in the status bar:

  1. <?xml version="1.0"?>
  2.  
  3. <overlay id="myOverlay"
  4. xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul"
  5. xmlns:html="http://www.w3.org/1999/xhtml">
  6.  
  7. <statusbar id="status-bar">
  8. <statusbarpanel>
  9. <box>
  10. <html:canvas id="myCanvas" width="15" height="15" style="border:1px solid black;"/>
  11. </box>
  12. </statusbarpanel>
  13. </statusbar>
  14. </overlay>

and some simple painting in JS:

  1. var canvas = window.document.getElementById("myCanvas");
  2. var ctx = canvas.getContext("2d");
  3. ctx.fillStyle = "red";
  4. ctx.fillRect(5,5,5,5);

If done right, you will get a result like this

CANVAS in XULCANVAS in XUL

Using .NET reflection to determine if a Class implements a generic interface.

Reflection and me? Big friends. With all the love and hate a good friendship should have. A few days ago it was all about hate again. I had a bunch of service classes, some of them would implement a generic interface... Let's call it IHasAdorable - so a Service-Implementation could look like this:

  1. public class MarketMerchant : IHasAdorable<CheeseBurger>, IProductSeller
  2. {
  3. // defined in IHasAdorable
  4. CheeseBurger BuyAdorable()
  5. {
  6. return new AdorableCheeseburger();
  7. }
  8.  
  9. // defined in IProductSeller
  10. IProduct Buy(String eanCode)
  11. {
  12. if (eanCode.equals("12345"))
  13. return this.bigStackOfSmellyFishburgers.Pop();
  14. else
  15. throw new IAmSoSorryException("We don't have any of these!");
  16. }
  17. }

Now lets assume we want to browse through ALL market merchants and have a look if they have any adorable products. Let's skip the iteration process and pay attention to the probing of all market merchants in order to buy a adorable product from each of them. First attempt might be to use "is":

  1. IProductSeller merchant;
  2. // iteration goes here
  3. if (merchant is IHasAdorable<object>)
  4. {
  5. // this wont work. We are selling something very special, not just a stupid object!
  6. // Casting to IHasAdorable<> won't even compile.
  7. }

This one would not work either, although it makes sense to me:
  1. typeof(IHasAdorable<object>).IsAssignableFrom(merchant)

It seems we need to know what kind of objects a unknown merchant sells before we can "see" if those are also adorable. So here is how I got it working. Let me know if you do know a better solution:
  1. // `1 means there is 1 Generic parameter
  2. Type adorableInterfaceType = merchant.GetType().GetInterface("IHasAdorable`1");
  3.  
  4. if (adorableInterfaceType != null)
  5. {
  6. // yay, the merchant has adorable products, what whould those be!?
  7. Type adorableProductType = adorableInterfaceType.GetGenericArguments()[0];
  8.  
  9. // here is the magic we need to get the correct IHasAdorable Type
  10. // with "filled in" generic type.
  11. Type genericAdorableInterfaceType =
  12. typeof(IHasAdorable<>).MakeGenericType(adorableProductType);
  13. MethodInfo mi = genericAdorableInterfaceType.GetMethod("BuyAdorable");
  14. myBagOfAdorableProducts.Add(mi.invoke(merchant,null));
  15. }

Bon Appétit

"Jar-Hell" - or how do I find a Java-Class in a folder full of Jar-Archives

There you go

  1. #!/bin/sh
  2.  
  3. ########################################################################
  4. ## Scans all jar files within a directory (recursively) for a class
  5. ## name
  6. ## Usage: findClass /tmp/ MyFunnyClass
  7. ########################################################################
  8.  
  9. black='\E[30;47m'
  10. red='\E[31;40m\033[1m'
  11. green='\E[32;47m'
  12. yellow='\E[33;40m'
  13. blue='\E[34;40m\033[1m'
  14. magenta='\E[35;47m'
  15. cyan='\E[36;47m'
  16. white='\E[37;47m'
  17. alias Reset="tput sgr0"
  18.  
  19. cecho ()
  20. {
  21. local default_msg=" "
  22. message=${1:-$default_msg} # Defaults to default message.
  23. color=${2:-$black} # Defaults to black, if not specified.
  24. echo -e -n "$color"
  25. echo -n "$message"
  26. Reset # Reset to normal.
  27. return
  28. }
  29.  
  30. clsln()
  31. {
  32. fillLine " ";
  33. }
  34.  
  35. fillLine()
  36. {
  37. let tw=$(tput cols)-1;
  38. for (( c=0 ; c < $tw; c++))
  39. do
  40. echo -n "$1";
  41. done
  42. echo -e -n '\r'
  43. }
  44.  
  45. echo
  46. echo -n Scanning Folder:
  47. cecho "$1" $yellow
  48. echo -n for Class:
  49. cecho "$2" $yellow
  50. echo
  51.  
  52. fillLine "."
  53. echo
  54. for i in $(find $1 -name '*jar');
  55. do
  56. clsln;
  57. echo -n -e "Scanning :";
  58. cecho $i $blue
  59. echo -n -e '\r';
  60.  
  61. out=$(jar vft $i | egrep $2);
  62.  
  63. if [ "$out" ]
  64. then
  65. clsln;
  66. fillLine "*"
  67. echo
  68. echo -n -e 'Possble hit in file:'
  69. cecho $i $blue
  70. echo
  71. echo "$out" $red
  72. echo
  73. fillLine "*"
  74. echo
  75. echo
  76. fi
  77. done;
  78. clsln;
  79. echo;

(Unix) Timestamp to .NET DateTime ?

Well, this is rather a note to myself:

for Javascript new Date().getTime() value (miliseconds, hence / 1000)

  1. new System.DateTime(1970, 1, 1, 0, 0, 0, 0)
  2. .AddSeconds(ajaxRequest.RequestDate / 1000)

One more reason why I love local initializers

  1. public static IDictionary<RequestTypeEnum, RequestGroupEnum>
  2. RequestTypeToGroup= new Dictionary<RequestTypeEnum, RequestGroupEnum>()
  3. {
  4. {RequestTypeEnum.RegisterAccount, RequestGroupEnum.Account},
  5. {RequestTypeEnum.UnregisterAccount, RequestGroupEnum.Account},
  6. {RequestTypeEnum.RegisterInetAccess, RequestGroupEnum.InetAccess},
  7. {RequestTypeEnum.UnregisterInetAccess, RequestGroupEnum.InetAccess},
  8. {RequestTypeEnum.RegisterMailbox, RequestGroupEnum.Mailbox},
  9. {RequestTypeEnum.UnregisterMailbox, RequestGroupEnum.Mailbox},
  10. {RequestTypeEnum.RegisterRLA, RequestGroupEnum.RLA},
  11. {RequestTypeEnum.UnregisterRLA, RequestGroupEnum.RLA}
  12. };
Syndicate content