Here is a little code-snippet (actually a ready to run jUnit TestCase) which might come handy if you need a fairly open ThreadPool not primarily limited by the number of active threads but rather by a predicted load factor. Latter one might be pretty much everything such as CPU load or a total number of "items" allowed to be processed by the whole ThreadPool at a given time.
If the predicted load is not dynamic enough for you, you might want to add another monitoring thread looking at some indicators (CPU, RAM, I/O) and adjust the LoadTracker's currentLoad value accordingly. Another path would be to skip the monitoring thread and extend the canHandle(load) method of the LoadTracker to respect the current indicator states.
Oh, and please let me know if I am reinventing the wheel, sometimes it is difficult not to.
In retrospect, same pattern could be applied to the Queue beneath the ThreadPool by coupling a LoadTrackableJob with a specific BlockingQueue. I guess you can always make the code / architecture prettier.
public class TestThreadPool extends TestCase { private static Log log = LogFactory.getLog(TestThreadPool.class); { int maxRunningThreads = 128; int maxLoad = 500; LoadTracker load = new LoadTracker(maxLoad); ExecutorService pool = Executors.newFixedThreadPool(maxRunningThreads); for (int i = 0; i < 500; i++) { // here you would create your real job and *predict* its impact on the load factor. // we choose the load to be random. MyJob aJob = new MyJob(load, predictedJobLoad,"job-"+i,this); while (!load.canHandle(predictedJobLoad)) { log.debug(String.format("WAIT: current load %d and new job is about to be %d", load.get(), predictedJobLoad)); synchronized (this) { this.wait(1000); } } log.debug(String.format("QUEUE: current load is %d and new job is about to be %d", load.get(), predictedJobLoad)); pool.execute(aJob); } pool.shutdown(); pool.awaitTermination(42,TimeUnit.DAYS); assertEquals(0, load.get()); } { private LoadTracker loadTracker; private int load; private String jobId; private Object monitor; { this.jobId = jobId; this.loadTracker = loadTracker; this.load = load; this.monitor = monitor; loadTracker.add(load); } @Override public void run() { try { } { e.printStackTrace(); } loadTracker.remove(load); if (monitor != null) synchronized (monitor) { monitor.notify(); } } } private class LoadTracker { private int currentLoad = 0; private int maxLoad = 0; public LoadTracker(int maxLoad) { this.maxLoad = maxLoad; } private synchronized void add(int load) { this.currentLoad += load; } private synchronized void remove(int load) { this.currentLoad -= load; } public synchronized int get() { return currentLoad; } public synchronized boolean canHandle(int additionalLoad) { return ((this.get() + additionalLoad) < maxLoad); } } }
plenty of videos and slides straight from hadoop world: http://is.gd/h1YeD #base #hadoop #bigdata
I find myself quite often in the need to copy large textfiles over the network. Usually one would go with gzip, either transparently by using the compression switch on scp -C or by archiving a file before pushing it over the wire.
Turns out gzip can compress quite well, but it won't saturate your 100mbit line if you do something like this:
cat bigfile.txt | gzip -c | ssh me@other.side 'cat | gunzip -d > bigfile.txt'
While this has the same effect as scp bigfile.txt me@other.side: it will be helpful to understand the alternatives coming up next.
On a pretty decent machine (Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz) I could get about 60% saturation of the network link. So, what can we do? We could lower the compression level to take load of the CPU and shift it towards the network. We could also use an alternative compression algorithm such as LZO. "lzop" is a free implementation available in most common linux distributions, so this might be the easiest way to go:
cat bigfile.txt | lzop -c | ssh me@other.side 'cat | lzop -cd > bigfile.txt'
My initial tests shown that LZO compression level is about 20% lower than gzip's with default settings. Transfer time was almost cut in half on the other hand. So, how do we get 100% network saturation*AND* great compression? Pigz is a parallel gzip implementation, so instead maxing out only one thread as gzip does, it will use all the available cores and threads your fancy server provides. Downside? No debian stable repository packages available yet. But on the other hand: it does not even require a configure script, how about one header file and 2 "c" files. Most probably your remote connection to the server will take longer to refresh the console output than the compilation process itself.
So, emerge, apt-get install, port install or whatever "pv" and have some fun like this:
me@host:/mnt/data1/import$ cat bigfile.txt | pv | pigz -c | ssh me@otherhost 'cat | unpigz > /mnt/data1/bigfile.txt' 1.83GB 0:00:18 [95.3MB/s] [ <=> ]
Some more numbers: gzip gives me 45 mb/s and lzop 60mb/s
I think "pv" stands for pipe view, it is responsible for the nice stats during the transfer. And yes, this *is* a 100mbit network connection pushing a textfile at ~100mb/s. Nice, isn't it?
You can find pigz over here
While switching my main workstation over to a Mac I ran into an issue with the keyboard shipped with those new iMac. To be honest, I would not even consider calling it a keyboard, must be some torture tool straight from hell. Luckily I have a couple of those ergonomic Microsoft keyboard to my disposal. Now the tricky part: the physical keyboard layout does not correspond with the results on the screen. A problem definitely worth fixing, so here we go: a German custom keyboard Layout. Fixed Keys: "{ } [ ] | ~ @ \"
Install it by dropping the bundle file into "~/Library/Keyboard Layouts/" - you will probably have to create that directory.
A great companion to that layout is the Microsoft driver for that Keyboard which will allow you to remap the ALT, WIN and / APP Key:
While fighting some more problems with Sharepoint 2007 Variations "System" I had to write this comment in my code today:
// note: there is a bug in Sharepoint 2007 when using ", " within a page name - "foo, bar.aspx" for instance. // note: This will render the relationship list entry useless. A healthy object id attribute in a // note: relationship list's entry looks like this: // note: ows_ObjectID='http://mucskysp04:4711/DE/Seiten/Seite mit Sönderzeichen.aspx, /DE/Seiten/Seite mit Sönderzeichen.aspx' // note: once "broken" using the naming-"exploit", the relationship will be represented as: // note: ows_ObjectID='http://mucskysp04:4711/DE/Seiten/variationsseite, mitkommaundleerzeichen.aspx' // note: note the missing relative path. There is no reason to work around this bug in this tool since the original // note: entry in Sharepoint won't be functional anyway (no entries will be shown in the variations dropdown)
So, here it is. Thanks to http://isitdarkoutside.com you will never have to trouble yourself with the question "is it dark outside?" anymore. This super-advanced Android application will let you focus on more important questions such as "which shoe goes on which foot?" from now on. Thank us later.
You are missing some Flash content that should appear here! Perhaps your browser cannot display it, or maybe it did not initialise correctly.
Version 0.1 - 2010-02-07,
Version 0.0.1 - 2010-02-05,
Version 0.0.0 - 2010-02-05
N/A - Dev-Snapshot, link removed
It's actually pretty simple, but many small things can go wrong. So, here is your small boilerplate code for a XUL overlay which renders a small canvas area in the status bar:
<?xml version="1.0"?> <overlay id="myOverlay" xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul" xmlns:html="http://www.w3.org/1999/xhtml"> <statusbar id="status-bar"> <statusbarpanel> <box> <html:canvas id="myCanvas" width="15" height="15" style="border:1px solid black;"/> </box> </statusbarpanel> </statusbar> </overlay>
and some simple painting in JS:
var canvas = window.document.getElementById("myCanvas"); var ctx = canvas.getContext("2d"); ctx.fillStyle = "red"; ctx.fillRect(5,5,5,5);
If done right, you will get a result like this
CANVAS in XUL
dojo.provide("myWidgets.Dialog"); dojo.declare ( "myWidgets.Dialog", [dijit.Dialog], { // summary: // extended version of the dojo Dialog widget with the option to disable // the close button and supress the escape key. disableCloseButton: true, /* *********************************************************** postCreate */ postCreate: function() { this.inherited(arguments); this._updateCloseButtonState(); }, /* *************************************************************** _onKey */ _onKey: function(evt) { if(this.disableCloseButton && evt.charOrCode == dojo.keys.ESCAPE) return; this.inherited(arguments); }, /* ************************************************ setCloseButtonDisabled*/ setCloseButtonDisabled: function(flag) { this.disableCloseButton = flag; this._updateCloseButtonState(); }, /* ********************************************** _updateCloseButtonState */ _updateCloseButtonState: function() { dojo.style(this.closeButtonNode, "display",this.disableCloseButton ? "none" : "block"); } } );
Just in case you run into OutOfMemory Exceptions while requesting a large data chunk from the MySQL: the JDBC driver will load ALL (yes, ALL) rows before passing it to your fancy, agile and low-footprint routine. Tweaking the fetchSize property of a statement won't do any good either... well, not without some voodoo. So, here is how you can get the JDBC driver to get you a nice and tight StreamingResultSet:
st.setFetchSize(Integer.MIN_VALUE); // Inter.MIN_VALUE <- and ONLY this value, 1,5 or 100 won't fix your problem.
Reflection and me? Big friends. With all the love and hate a good friendship should have. A few days ago it was all about hate again. I had a bunch of service classes, some of them would implement a generic interface... Let's call it IHasAdorable - so a Service-Implementation could look like this:
public class MarketMerchant : IHasAdorable<CheeseBurger>, IProductSeller { // defined in IHasAdorable CheeseBurger BuyAdorable() { } // defined in IProductSeller IProduct Buy(String eanCode) { if (eanCode.equals("12345")) return this.bigStackOfSmellyFishburgers.Pop(); else } }
Now lets assume we want to browse through ALL market merchants and have a look if they have any adorable products. Let's skip the iteration process and pay attention to the probing of all market merchants in order to buy a adorable product from each of them. First attempt might be to use "is":
IProductSeller merchant; // iteration goes here { // this wont work. We are selling something very special, not just a stupid object! // Casting to IHasAdorable<> won't even compile. }
// `1 means there is 1 Generic parameter Type adorableInterfaceType = merchant.GetType().GetInterface("IHasAdorable`1"); if (adorableInterfaceType != null) { // yay, the merchant has adorable products, what whould those be!? Type adorableProductType = adorableInterfaceType.GetGenericArguments()[0]; // here is the magic we need to get the correct IHasAdorable Type // with "filled in" generic type. Type genericAdorableInterfaceType = MethodInfo mi = genericAdorableInterfaceType.GetMethod("BuyAdorable"); myBagOfAdorableProducts.Add(mi.invoke(merchant,null)); }