quick and dirty
hadoop-env.sh:
#replace eth1:0 with your NIC / alias bind_ip=$(/sbin/ifconfig eth1:0 | grep 'inet addr:' | cut -d: -f2 | awk '{print $1}') export BIND_OPTS="-Dlocal.bind.address=${bind_ip}" # Command specific options appended to HADOOP_OPTS when specified export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS $BIND_OPTS" export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS $BIND_OPTS" export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS $BIND_OPTS" export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS $BIND_OPTS" export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS $BIND_OPTS"
<property> <name>dfs.secondary.http.address</name> <value>${local.bind.address}:50090</value> <description> The secondary namenode http server address and port. If the port is 0 then the server will start on a free port. </description> </property> <property> <name>dfs.datanode.address</name> <value>${local.bind.address}:50010</value> </property> <property> <name>dfs.datanode.http.address</name> <value>${local.bind.address}:50075</value> </property> <property> <name>dfs.datanode.ipc.address</name> <value>${local.bind.address}:50020</value> </property> <property> <name>dfs.http.address</name> <value>${local.bind.address}:50070</value> </property> <property> <name>dfs.datanode.https.address</name> <value>${local.bind.address}:50475</value> </property> <property> <name>dfs.https.address</name> <value>${local.bind.address}:50470</value> </property>
Here is a little code-snippet (actually a ready to run jUnit TestCase) which might come handy if you need a fairly open ThreadPool not primarily limited by the number of active threads but rather by a predicted load factor. Latter one might be pretty much everything such as CPU load or a total number of "items" allowed to be processed by the whole ThreadPool at a given time.
If the predicted load is not dynamic enough for you, you might want to add another monitoring thread looking at some indicators (CPU, RAM, I/O) and adjust the LoadTracker's currentLoad value accordingly. Another path would be to skip the monitoring thread and extend the canHandle(load) method of the LoadTracker to respect the current indicator states.
Oh, and please let me know if I am reinventing the wheel, sometimes it is difficult not to.
In retrospect, same pattern could be applied to the Queue beneath the ThreadPool by coupling a LoadTrackableJob with a specific BlockingQueue. I guess you can always make the code / architecture prettier.
public class TestThreadPool extends TestCase { private static Log log = LogFactory.getLog(TestThreadPool.class); { int maxRunningThreads = 128; int maxLoad = 500; LoadTracker load = new LoadTracker(maxLoad); ExecutorService pool = Executors.newFixedThreadPool(maxRunningThreads); for (int i = 0; i < 500; i++) { // here you would create your real job and *predict* its impact on the load factor. // we choose the load to be random. MyJob aJob = new MyJob(load, predictedJobLoad,"job-"+i,this); while (!load.canHandle(predictedJobLoad)) { log.debug(String.format("WAIT: current load %d and new job is about to be %d", load.get(), predictedJobLoad)); synchronized (this) { this.wait(1000); } } log.debug(String.format("QUEUE: current load is %d and new job is about to be %d", load.get(), predictedJobLoad)); pool.execute(aJob); } pool.shutdown(); pool.awaitTermination(42,TimeUnit.DAYS); assertEquals(0, load.get()); } { private LoadTracker loadTracker; private int load; private String jobId; private Object monitor; { this.jobId = jobId; this.loadTracker = loadTracker; this.load = load; this.monitor = monitor; loadTracker.add(load); } @Override public void run() { try { } { e.printStackTrace(); } loadTracker.remove(load); if (monitor != null) synchronized (monitor) { monitor.notify(); } } } private class LoadTracker { private int currentLoad = 0; private int maxLoad = 0; public LoadTracker(int maxLoad) { this.maxLoad = maxLoad; } private synchronized void add(int load) { this.currentLoad += load; } private synchronized void remove(int load) { this.currentLoad -= load; } public synchronized int get() { return currentLoad; } public synchronized boolean canHandle(int additionalLoad) { return ((this.get() + additionalLoad) < maxLoad); } } }
_pulsar_: OpenHUG Meeting, wir freuen uns auf zahlreiches Erscheinen! http://linkd.in/drEaBF || http://bit.ly/aERlUU || http://bit.ly/bNGGCt - _pulsar_: OpenHUG Meeting, wir freuen uns auf zahlreiches Erscheinen! http://linkd.in/drEaBF || http://bit.ly/aERlUU || http://bit.ly/bNGGCt [/me Twitters]
So, here it is. Thanks to http://isitdarkoutside.com you will never have to trouble yourself with the question "is it dark outside?" anymore. This super-advanced Android application will let you focus on more important questions such as "which shoe goes on which foot?" from now on. Thank us later.
You are missing some Flash content that should appear here! Perhaps your browser cannot display it, or maybe it did not initialise correctly.
Version 0.1 - 2010-02-07,
Version 0.0.1 - 2010-02-05,
Version 0.0.0 - 2010-02-05
N/A - Dev-Snapshot, link removed
Just in case you run into OutOfMemory Exceptions while requesting a large data chunk from the MySQL: the JDBC driver will load ALL (yes, ALL) rows before passing it to your fancy, agile and low-footprint routine. Tweaking the fetchSize property of a statement won't do any good either... well, not without some voodoo. So, here is how you can get the JDBC driver to get you a nice and tight StreamingResultSet:
st.setFetchSize(Integer.MIN_VALUE); // Inter.MIN_VALUE <- and ONLY this value, 1,5 or 100 won't fix your problem.
There you go
#!/bin/sh ######################################################################## ## Scans all jar files within a directory (recursively) for a class ## name ## Usage: findClass /tmp/ MyFunnyClass ######################################################################## black='\E[30;47m' red='\E[31;40m\033[1m' green='\E[32;47m' yellow='\E[33;40m' blue='\E[34;40m\033[1m' magenta='\E[35;47m' cyan='\E[36;47m' white='\E[37;47m' alias Reset="tput sgr0" cecho () { local default_msg=" " message=${1:-$default_msg} # Defaults to default message. color=${2:-$black} # Defaults to black, if not specified. echo -e -n "$color" echo -n "$message" Reset # Reset to normal. return } clsln() { fillLine " "; } fillLine() { let tw=$(tput cols)-1; for (( c=0 ; c < $tw; c++)) do echo -n "$1"; done echo -e -n '\r' } echo echo -n Scanning Folder: cecho "$1" $yellow echo -n for Class: cecho "$2" $yellow echo fillLine "." echo for i in $(find $1 -name '*jar'); do clsln; echo -n -e "Scanning :"; cecho $i $blue echo -n -e '\r'; out=$(jar vft $i | egrep $2); if [ "$out" ] then clsln; fillLine "*" echo echo -n -e 'Possble hit in file:' cecho $i $blue echo echo "$out" $red echo fillLine "*" echo echo fi done; clsln; echo;
_pulsar_: did I mention me being in love with soapUI - http://www.soapui.org/? #java #dotnet #webservice
Webstart ErrorTranslates to "Entryscreen: Error rcv". Yes... I have NO IDEA.
I'll be doing some work in java for renoise.com today. If you would like to peek across my shoulder - http://live.yahoo.com/codewut
If you need to let your Tomcat write access logs while being proxied by Apache's mod_proxy using the combined format, you will soon notice the lack of the client IP address. Pretty useless if you would like to get some access statistics for that particular instance.
Fortunately the Apache's mod_proxy will add some extra headers with the missing information to each request. Just set up your Log-Configuration in Tomcat as follows:
<Context> <Valve className="org.apache.catalina.valves.AccessLogValve" prefix="access_log" pattern="%{X-Forwarded-For}i %l %u %t "%r" %s %b "%{Referer}i" "%{User-Agent}i"" rotatable="false"/> </Context>
Please note that I'm rotating the Tomcat logs using logrotate. Therefore the "rotatable" attribute is set to "false" above. Just in any case, here is the logrotate config for my setup:
/home/tomcat/logs/*_log { weekly rotate 48 create 640 tomcat www-data sharedscripts postrotate /etc/init.d/tomcat restart endscript compress }