quick and dirty
hadoop-env.sh:
#replace eth1:0 with your NIC / alias bind_ip=$(/sbin/ifconfig eth1:0 | grep 'inet addr:' | cut -d: -f2 | awk '{print $1}') export BIND_OPTS="-Dlocal.bind.address=${bind_ip}" # Command specific options appended to HADOOP_OPTS when specified export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS $BIND_OPTS" export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS $BIND_OPTS" export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS $BIND_OPTS" export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS $BIND_OPTS" export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS $BIND_OPTS"
<property> <name>dfs.secondary.http.address</name> <value>${local.bind.address}:50090</value> <description> The secondary namenode http server address and port. If the port is 0 then the server will start on a free port. </description> </property> <property> <name>dfs.datanode.address</name> <value>${local.bind.address}:50010</value> </property> <property> <name>dfs.datanode.http.address</name> <value>${local.bind.address}:50075</value> </property> <property> <name>dfs.datanode.ipc.address</name> <value>${local.bind.address}:50020</value> </property> <property> <name>dfs.http.address</name> <value>${local.bind.address}:50070</value> </property> <property> <name>dfs.datanode.https.address</name> <value>${local.bind.address}:50475</value> </property> <property> <name>dfs.https.address</name> <value>${local.bind.address}:50470</value> </property>
Here is a little code-snippet (actually a ready to run jUnit TestCase) which might come handy if you need a fairly open ThreadPool not primarily limited by the number of active threads but rather by a predicted load factor. Latter one might be pretty much everything such as CPU load or a total number of "items" allowed to be processed by the whole ThreadPool at a given time.
If the predicted load is not dynamic enough for you, you might want to add another monitoring thread looking at some indicators (CPU, RAM, I/O) and adjust the LoadTracker's currentLoad value accordingly. Another path would be to skip the monitoring thread and extend the canHandle(load) method of the LoadTracker to respect the current indicator states.
Oh, and please let me know if I am reinventing the wheel, sometimes it is difficult not to.
In retrospect, same pattern could be applied to the Queue beneath the ThreadPool by coupling a LoadTrackableJob with a specific BlockingQueue. I guess you can always make the code / architecture prettier.
public class TestThreadPool extends TestCase { private static Log log = LogFactory.getLog(TestThreadPool.class); { int maxRunningThreads = 128; int maxLoad = 500; LoadTracker load = new LoadTracker(maxLoad); ExecutorService pool = Executors.newFixedThreadPool(maxRunningThreads); for (int i = 0; i < 500; i++) { // here you would create your real job and *predict* its impact on the load factor. // we choose the load to be random. MyJob aJob = new MyJob(load, predictedJobLoad,"job-"+i,this); while (!load.canHandle(predictedJobLoad)) { log.debug(String.format("WAIT: current load %d and new job is about to be %d", load.get(), predictedJobLoad)); synchronized (this) { this.wait(1000); } } log.debug(String.format("QUEUE: current load is %d and new job is about to be %d", load.get(), predictedJobLoad)); pool.execute(aJob); } pool.shutdown(); pool.awaitTermination(42,TimeUnit.DAYS); assertEquals(0, load.get()); } { private LoadTracker loadTracker; private int load; private String jobId; private Object monitor; { this.jobId = jobId; this.loadTracker = loadTracker; this.load = load; this.monitor = monitor; loadTracker.add(load); } @Override public void run() { try { } { e.printStackTrace(); } loadTracker.remove(load); if (monitor != null) synchronized (monitor) { monitor.notify(); } } } private class LoadTracker { private int currentLoad = 0; private int maxLoad = 0; public LoadTracker(int maxLoad) { this.maxLoad = maxLoad; } private synchronized void add(int load) { this.currentLoad += load; } private synchronized void remove(int load) { this.currentLoad -= load; } public synchronized int get() { return currentLoad; } public synchronized boolean canHandle(int additionalLoad) { return ((this.get() + additionalLoad) < maxLoad); } } }
It's actually pretty simple, but many small things can go wrong. So, here is your small boilerplate code for a XUL overlay which renders a small canvas area in the status bar:
<?xml version="1.0"?> <overlay id="myOverlay" xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul" xmlns:html="http://www.w3.org/1999/xhtml"> <statusbar id="status-bar"> <statusbarpanel> <box> <html:canvas id="myCanvas" width="15" height="15" style="border:1px solid black;"/> </box> </statusbarpanel> </statusbar> </overlay>
and some simple painting in JS:
var canvas = window.document.getElementById("myCanvas"); var ctx = canvas.getContext("2d"); ctx.fillStyle = "red"; ctx.fillRect(5,5,5,5);
If done right, you will get a result like this
CANVAS in XUL
Reflection and me? Big friends. With all the love and hate a good friendship should have. A few days ago it was all about hate again. I had a bunch of service classes, some of them would implement a generic interface... Let's call it IHasAdorable - so a Service-Implementation could look like this:
public class MarketMerchant : IHasAdorable<CheeseBurger>, IProductSeller { // defined in IHasAdorable CheeseBurger BuyAdorable() { } // defined in IProductSeller IProduct Buy(String eanCode) { if (eanCode.equals("12345")) return this.bigStackOfSmellyFishburgers.Pop(); else } }
Now lets assume we want to browse through ALL market merchants and have a look if they have any adorable products. Let's skip the iteration process and pay attention to the probing of all market merchants in order to buy a adorable product from each of them. First attempt might be to use "is":
IProductSeller merchant; // iteration goes here { // this wont work. We are selling something very special, not just a stupid object! // Casting to IHasAdorable<> won't even compile. }
// `1 means there is 1 Generic parameter Type adorableInterfaceType = merchant.GetType().GetInterface("IHasAdorable`1"); if (adorableInterfaceType != null) { // yay, the merchant has adorable products, what whould those be!? Type adorableProductType = adorableInterfaceType.GetGenericArguments()[0]; // here is the magic we need to get the correct IHasAdorable Type // with "filled in" generic type. Type genericAdorableInterfaceType = MethodInfo mi = genericAdorableInterfaceType.GetMethod("BuyAdorable"); myBagOfAdorableProducts.Add(mi.invoke(merchant,null)); }
There you go
#!/bin/sh ######################################################################## ## Scans all jar files within a directory (recursively) for a class ## name ## Usage: findClass /tmp/ MyFunnyClass ######################################################################## black='\E[30;47m' red='\E[31;40m\033[1m' green='\E[32;47m' yellow='\E[33;40m' blue='\E[34;40m\033[1m' magenta='\E[35;47m' cyan='\E[36;47m' white='\E[37;47m' alias Reset="tput sgr0" cecho () { local default_msg=" " message=${1:-$default_msg} # Defaults to default message. color=${2:-$black} # Defaults to black, if not specified. echo -e -n "$color" echo -n "$message" Reset # Reset to normal. return } clsln() { fillLine " "; } fillLine() { let tw=$(tput cols)-1; for (( c=0 ; c < $tw; c++)) do echo -n "$1"; done echo -e -n '\r' } echo echo -n Scanning Folder: cecho "$1" $yellow echo -n for Class: cecho "$2" $yellow echo fillLine "." echo for i in $(find $1 -name '*jar'); do clsln; echo -n -e "Scanning :"; cecho $i $blue echo -n -e '\r'; out=$(jar vft $i | egrep $2); if [ "$out" ] then clsln; fillLine "*" echo echo -n -e 'Possble hit in file:' cecho $i $blue echo echo "$out" $red echo fillLine "*" echo echo fi done; clsln; echo;
_pulsar_: did I mention me being in love with soapUI - http://www.soapui.org/? #java #dotnet #webservice
Well, this is rather a note to myself:
for Javascript new Date().getTime() value (miliseconds, hence / 1000)
.AddSeconds(ajaxRequest.RequestDate / 1000)
public static IDictionary<RequestTypeEnum, RequestGroupEnum> { {RequestTypeEnum.RegisterAccount, RequestGroupEnum.Account}, {RequestTypeEnum.UnregisterAccount, RequestGroupEnum.Account}, {RequestTypeEnum.RegisterInetAccess, RequestGroupEnum.InetAccess}, {RequestTypeEnum.UnregisterInetAccess, RequestGroupEnum.InetAccess}, {RequestTypeEnum.RegisterMailbox, RequestGroupEnum.Mailbox}, {RequestTypeEnum.UnregisterMailbox, RequestGroupEnum.Mailbox}, {RequestTypeEnum.RegisterRLA, RequestGroupEnum.RLA}, {RequestTypeEnum.UnregisterRLA, RequestGroupEnum.RLA} };
Yesterday I was implementing a generic serializer (for logging purposes). While there is a good number of XML serializers available for .NET - none of them was meant to be used for only the purpose of reading. That means, I would not need deserialization but only some nice human readable output.
Anyway, thats not the point here. What took me about an hour to figure out is how to handle arrays (of natives) via reflection. I kinda feel stupid, but let me show you what i tired:
Object propertyValue = prop.GetValue(o, null); if (propertyValue != null && propertyValue.GetType().IsArray) { //if you need the type, here is how we get it, it's not used //in this example. Type arrayType = propertyValue.GetType().GetElementType(); object[] arr = propertyValue as object[]; for (int i = 0; i < arr.Length; i++) { // serialization logic goes here } } ...
While this might work perfectly for arrays of "real" objects such as string[] - it will fail when your array is composed of native objects such as long[]. The cast to object[] will get you a nice null reference. Yummy.
After that I've been looking for Array.ForEach() which was pretty interesting, but did not really help due to it's generic nature. The solution was much simpler in the end, it just did not cross my mind to cast the propertyValue to Array - sometimes .NET can be pretty mysterious in terms of Autoboxing. Here is how it should be done:
Object propertyValue = prop.GetValue(o, null); if (propertyValue != null && propertyValue.GetType().IsArray) { Array array = propertyValue as Array; foreach (var arrElement in array) { // serialization logic goes here } } ...
Straight forward, simple - makes me feel stupid.