java

warning: Creating default object from empty value in /home/pulsar/webroot/htdocs/modules/taxonomy/taxonomy.pages.inc on line 33.

Bind hadoop to a specific network device

quick and dirty

hadoop-env.sh:

  1. #replace eth1:0 with your NIC / alias
  2. bind_ip=$(/sbin/ifconfig eth1:0 | grep 'inet addr:' | cut -d: -f2 | awk '{print $1}')
  3.  
  4. export BIND_OPTS="-Dlocal.bind.address=${bind_ip}"
  5.  
  6. # Command specific options appended to HADOOP_OPTS when specified
  7. export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS $BIND_OPTS"
  8. export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS $BIND_OPTS"
  9. export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS $BIND_OPTS"
  10. export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS $BIND_OPTS"
  11. export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS $BIND_OPTS"
append to hdfs-site.xml:
  1. <property>
  2. <name>dfs.secondary.http.address</name>
  3. <value>${local.bind.address}:50090</value>
  4. <description>
  5. The secondary namenode http server address and port.
  6. If the port is 0 then the server will start on a free port.
  7. </description>
  8. </property>
  9.  
  10. <property>
  11. <name>dfs.datanode.address</name>
  12. <value>${local.bind.address}:50010</value>
  13. </property>
  14.  
  15. <property>
  16. <name>dfs.datanode.http.address</name>
  17. <value>${local.bind.address}:50075</value>
  18. </property>
  19.  
  20. <property>
  21. <name>dfs.datanode.ipc.address</name>
  22. <value>${local.bind.address}:50020</value>
  23. </property>
  24.  
  25. <property>
  26. <name>dfs.http.address</name>
  27. <value>${local.bind.address}:50070</value>
  28. </property>
  29.  
  30. <property>
  31. <name>dfs.datanode.https.address</name>
  32. <value>${local.bind.address}:50475</value>
  33. </property>
  34.  
  35. <property>
  36. <name>dfs.https.address</name>
  37. <value>${local.bind.address}:50470</value>
  38. </property>
you will also need to hardcode the HDFS URL in the core-site.xml by hand. This is not an issue since the same config should be deployed on all nodes in the cluster. Same pattern can be applied to mapred-site.xml - I did not need it.

Load-Sensitive ThreadPool / Queue in Java

Here is a little code-snippet (actually a ready to run jUnit TestCase) which might come handy if you need a fairly open ThreadPool not primarily limited by the number of active threads but rather by a predicted load factor. Latter one might be pretty much everything such as CPU load or a total number of "items" allowed to be processed by the whole ThreadPool at a given time.

If the predicted load is not dynamic enough for you, you might want to add another monitoring thread looking at some indicators (CPU, RAM, I/O) and adjust the LoadTracker's currentLoad value accordingly. Another path would be to skip the monitoring thread and extend the canHandle(load) method of the LoadTracker to respect the current indicator states.

Oh, and please let me know if I am reinventing the wheel, sometimes it is difficult not to.

In retrospect, same pattern could be applied to the Queue beneath the ThreadPool by coupling a LoadTrackableJob with a specific BlockingQueue. I guess you can always make the code / architecture prettier.

  1. public class TestThreadPool extends TestCase
  2. {
  3. private static Log log = LogFactory.getLog(TestThreadPool.class);
  4.  
  5. public void testLoadTracker() throws InterruptedException
  6. {
  7. int maxRunningThreads = 128;
  8. int maxLoad = 500;
  9.  
  10. LoadTracker load = new LoadTracker(maxLoad);
  11. ExecutorService pool = Executors.newFixedThreadPool(maxRunningThreads);
  12.  
  13. for (int i = 0; i < 500; i++)
  14. {
  15. // here you would create your real job and *predict* its impact on the load factor.
  16. // we choose the load to be random.
  17. int predictedJobLoad = (int) Math.round(Math.random() * 10l);
  18. MyJob aJob = new MyJob(load, predictedJobLoad,"job-"+i,this);
  19.  
  20. while (!load.canHandle(predictedJobLoad))
  21. {
  22. log.debug(String.format("WAIT: current load %d and new job is about to be %d", load.get(), predictedJobLoad));
  23. synchronized (this) { this.wait(1000); }
  24. }
  25.  
  26. log.debug(String.format("QUEUE: current load is %d and new job is about to be %d", load.get(), predictedJobLoad));
  27. pool.execute(aJob);
  28. }
  29. pool.shutdown();
  30. pool.awaitTermination(42,TimeUnit.DAYS);
  31. assertEquals(0, load.get());
  32. }
  33.  
  34. private class MyJob implements Runnable
  35. {
  36. private LoadTracker loadTracker;
  37. private int load;
  38. private String jobId;
  39. private Object monitor;
  40.  
  41. public MyJob(LoadTracker loadTracker, int load, String jobId, Object monitor)
  42. {
  43. this.jobId = jobId;
  44. this.loadTracker = loadTracker;
  45. this.load = load;
  46. this.monitor = monitor;
  47. loadTracker.add(load);
  48. }
  49.  
  50. @Override
  51. public void run()
  52. {
  53. log.debug(String.format("RUN: %s with a load of %d", jobId, load));
  54. try
  55. {
  56. Thread.sleep((int) Math.round(Math.random() * 5000l));
  57. }
  58. {
  59. e.printStackTrace();
  60. }
  61. log.debug(String.format("FIN: %s with a load of %d", jobId, load));
  62. loadTracker.remove(load);
  63. if (monitor != null) synchronized (monitor) { monitor.notify(); }
  64. }
  65. }
  66.  
  67. private class LoadTracker
  68. {
  69. private int currentLoad = 0;
  70. private int maxLoad = 0;
  71.  
  72. public LoadTracker(int maxLoad)
  73. {
  74. this.maxLoad = maxLoad;
  75. }
  76.  
  77. private synchronized void add(int load)
  78. {
  79. this.currentLoad += load;
  80. }
  81.  
  82. private synchronized void remove(int load)
  83. {
  84. this.currentLoad -= load;
  85. }
  86.  
  87. public synchronized int get()
  88. {
  89. return currentLoad;
  90. }
  91.  
  92. public synchronized boolean canHandle(int additionalLoad)
  93. {
  94. return ((this.get() + additionalLoad) < maxLoad);
  95. }
  96. }
  97. }

OpenHUG Meeting

_pulsar_: OpenHUG Meeting, wir freuen uns auf zahlreiches Erscheinen! http://linkd.in/drEaBF || http://bit.ly/aERlUU || http://bit.ly/bNGGCt - _pulsar_: OpenHUG Meeting, wir freuen uns auf zahlreiches Erscheinen! http://linkd.in/drEaBF || http://bit.ly/aERlUU || http://bit.ly/bNGGCt [/me Twitters]

Is It Dark Outside

So, here it is. Thanks to http://isitdarkoutside.com you will never have to trouble yourself with the question "is it dark outside?" anymore. This super-advanced Android application will let you focus on more important questions such as "which shoe goes on which foot?" from now on. Thank us later.

Is It Dark Outside QR Code
download APK file here

You are missing some Flash content that should appear here! Perhaps your browser cannot display it, or maybe it did not initialise correctly.

Version 0.1 - 2010-02-07,

  • New icon
  • Public release

Version 0.0.1 - 2010-02-05,

  • better errorhandling (missing location status, no network)
  • Added sunrise and sundset hours to the information dialog

Version 0.0.0 - 2010-02-05
N/A - Dev-Snapshot, link removed

MySQL, Large Result Sets and OutOfMemory related Headaches

Just in case you run into OutOfMemory Exceptions while requesting a large data chunk from the MySQL: the JDBC driver will load ALL (yes, ALL) rows before passing it to your fancy, agile and low-footprint routine. Tweaking the fetchSize property of a statement won't do any good either... well, not without some voodoo. So, here is how you can get the JDBC driver to get you a nice and tight StreamingResultSet:

  1. Connection connection = dataSource.getConnection();
  2. Statement st = connection.createStatement(java.sql.ResultSet.TYPE_FORWARD_ONLY, java.sql.ResultSet.CONCUR_READ_ONLY);
  3. st.setFetchSize(Integer.MIN_VALUE); // Inter.MIN_VALUE <- and ONLY this value, 1,5 or 100 won't fix your problem.
  4. ResultSet rs = st.executeQuery("select * from someREALLYHugeTable");

Oh, and you clowns out there saying "this is normal, just bump up the memory settings for your JVM" - are you NUTS!? Or do you just like your applications exploding out of nowhere after being in production for some time?

"Jar-Hell" - or how do I find a Java-Class in a folder full of Jar-Archives

There you go

  1. #!/bin/sh
  2.  
  3. ########################################################################
  4. ## Scans all jar files within a directory (recursively) for a class
  5. ## name
  6. ## Usage: findClass /tmp/ MyFunnyClass
  7. ########################################################################
  8.  
  9. black='\E[30;47m'
  10. red='\E[31;40m\033[1m'
  11. green='\E[32;47m'
  12. yellow='\E[33;40m'
  13. blue='\E[34;40m\033[1m'
  14. magenta='\E[35;47m'
  15. cyan='\E[36;47m'
  16. white='\E[37;47m'
  17. alias Reset="tput sgr0"
  18.  
  19. cecho ()
  20. {
  21. local default_msg=" "
  22. message=${1:-$default_msg} # Defaults to default message.
  23. color=${2:-$black} # Defaults to black, if not specified.
  24. echo -e -n "$color"
  25. echo -n "$message"
  26. Reset # Reset to normal.
  27. return
  28. }
  29.  
  30. clsln()
  31. {
  32. fillLine " ";
  33. }
  34.  
  35. fillLine()
  36. {
  37. let tw=$(tput cols)-1;
  38. for (( c=0 ; c < $tw; c++))
  39. do
  40. echo -n "$1";
  41. done
  42. echo -e -n '\r'
  43. }
  44.  
  45. echo
  46. echo -n Scanning Folder:
  47. cecho "$1" $yellow
  48. echo -n for Class:
  49. cecho "$2" $yellow
  50. echo
  51.  
  52. fillLine "."
  53. echo
  54. for i in $(find $1 -name '*jar');
  55. do
  56. clsln;
  57. echo -n -e "Scanning :";
  58. cecho $i $blue
  59. echo -n -e '\r';
  60.  
  61. out=$(jar vft $i | egrep $2);
  62.  
  63. if [ "$out" ]
  64. then
  65. clsln;
  66. fillLine "*"
  67. echo
  68. echo -n -e 'Possble hit in file:'
  69. cecho $i $blue
  70. echo
  71. echo "$out" $red
  72. echo
  73. fillLine "*"
  74. echo
  75. echo
  76. fi
  77. done;
  78. clsln;
  79. echo;

Ummm... WHAT!?

Webstart ErrorWebstart ErrorTranslates to "Entryscreen: Error rcv". Yes... I have NO IDEA.

Live Coding Session

I'll be doing some work in java for renoise.com today. If you would like to peek across my shoulder - http://live.yahoo.com/codewut

Combined Access Log Format for a proxed Tomcat

If you need to let your Tomcat write access logs while being proxied by Apache's mod_proxy using the combined format, you will soon notice the lack of the client IP address. Pretty useless if you would like to get some access statistics for that particular instance.

Fortunately the Apache's mod_proxy will add some extra headers with the missing information to each request. Just set up your Log-Configuration in Tomcat as follows:

  1. <Context>
  2. <Valve
  3. className="org.apache.catalina.valves.AccessLogValve"
  4. prefix="access_log"
  5. pattern="%{X-Forwarded-For}i %l %u %t &quot;%r&quot; %s %b &quot;%{Referer}i&quot; &quot;%{User-Agent}i&quot;"
  6. rotatable="false"/>
  7. </Context>

Please note that I'm rotating the Tomcat logs using logrotate. Therefore the "rotatable" attribute is set to "false" above. Just in any case, here is the logrotate config for my setup:

  1. /home/tomcat/logs/*_log
  2. {
  3. weekly
  4. rotate 48
  5. create 640 tomcat www-data
  6. sharedscripts
  7. postrotate /etc/init.d/tomcat restart
  8. endscript
  9. compress
  10. }
Syndicate content