why it is so quiet here? The noise moved over to: http://paul.vc/+
Yay! It's done. Lovely warm distortion and fits great into Eurorack / Doepfer setup. Check sound samples below.
PLS Synth Module Demos by pulsar
Kudos goes to: Ken Stone and Bill and Will over at Dragonflyalley
Just finished mounting the first VCA Module. YAY!
You are missing some Flash content that should appear here! Perhaps your browser cannot display it, or maybe it did not initialise correctly.
and some waveforms. loving the results so far!
You are missing some Flash content that should appear here! Perhaps your browser cannot display it, or maybe it did not initialise correctly.
A little project I am currently working on:
You are missing some Flash content that should appear here! Perhaps your browser cannot display it, or maybe it did not initialise correctly.
if WTF crosses your mind: it's gonna be a dual tube driven voltage controlled amplifier module for my eurorack / doepfer setup. Use google or youtube got get a bit less of that WTF (or more, depends on your tech-affinity). Credits: Ken Stone - http://www.cgs.synth.net/modules/cgs65_vca.html and Bill & Will - http://www.dragonflyalley.com/constructionCGS65tubeVCA.htm
quick and dirty
hadoop-env.sh:
#replace eth1:0 with your NIC / alias bind_ip=$(/sbin/ifconfig eth1:0 | grep 'inet addr:' | cut -d: -f2 | awk '{print $1}') export BIND_OPTS="-Dlocal.bind.address=${bind_ip}" # Command specific options appended to HADOOP_OPTS when specified export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS $BIND_OPTS" export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS $BIND_OPTS" export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS $BIND_OPTS" export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS $BIND_OPTS" export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS $BIND_OPTS"
<property> <name>dfs.secondary.http.address</name> <value>${local.bind.address}:50090</value> <description> The secondary namenode http server address and port. If the port is 0 then the server will start on a free port. </description> </property> <property> <name>dfs.datanode.address</name> <value>${local.bind.address}:50010</value> </property> <property> <name>dfs.datanode.http.address</name> <value>${local.bind.address}:50075</value> </property> <property> <name>dfs.datanode.ipc.address</name> <value>${local.bind.address}:50020</value> </property> <property> <name>dfs.http.address</name> <value>${local.bind.address}:50070</value> </property> <property> <name>dfs.datanode.https.address</name> <value>${local.bind.address}:50475</value> </property> <property> <name>dfs.https.address</name> <value>${local.bind.address}:50470</value> </property>
Here is a little code-snippet (actually a ready to run jUnit TestCase) which might come handy if you need a fairly open ThreadPool not primarily limited by the number of active threads but rather by a predicted load factor. Latter one might be pretty much everything such as CPU load or a total number of "items" allowed to be processed by the whole ThreadPool at a given time.
If the predicted load is not dynamic enough for you, you might want to add another monitoring thread looking at some indicators (CPU, RAM, I/O) and adjust the LoadTracker's currentLoad value accordingly. Another path would be to skip the monitoring thread and extend the canHandle(load) method of the LoadTracker to respect the current indicator states.
Oh, and please let me know if I am reinventing the wheel, sometimes it is difficult not to.
In retrospect, same pattern could be applied to the Queue beneath the ThreadPool by coupling a LoadTrackableJob with a specific BlockingQueue. I guess you can always make the code / architecture prettier.
public class TestThreadPool extends TestCase { private static Log log = LogFactory.getLog(TestThreadPool.class); { int maxRunningThreads = 128; int maxLoad = 500; LoadTracker load = new LoadTracker(maxLoad); ExecutorService pool = Executors.newFixedThreadPool(maxRunningThreads); for (int i = 0; i < 500; i++) { // here you would create your real job and *predict* its impact on the load factor. // we choose the load to be random. MyJob aJob = new MyJob(load, predictedJobLoad,"job-"+i,this); while (!load.canHandle(predictedJobLoad)) { log.debug(String.format("WAIT: current load %d and new job is about to be %d", load.get(), predictedJobLoad)); synchronized (this) { this.wait(1000); } } log.debug(String.format("QUEUE: current load is %d and new job is about to be %d", load.get(), predictedJobLoad)); pool.execute(aJob); } pool.shutdown(); pool.awaitTermination(42,TimeUnit.DAYS); assertEquals(0, load.get()); } { private LoadTracker loadTracker; private int load; private String jobId; private Object monitor; { this.jobId = jobId; this.loadTracker = loadTracker; this.load = load; this.monitor = monitor; loadTracker.add(load); } @Override public void run() { try { } { e.printStackTrace(); } loadTracker.remove(load); if (monitor != null) synchronized (monitor) { monitor.notify(); } } } private class LoadTracker { private int currentLoad = 0; private int maxLoad = 0; public LoadTracker(int maxLoad) { this.maxLoad = maxLoad; } private synchronized void add(int load) { this.currentLoad += load; } private synchronized void remove(int load) { this.currentLoad -= load; } public synchronized int get() { return currentLoad; } public synchronized boolean canHandle(int additionalLoad) { return ((this.get() + additionalLoad) < maxLoad); } } }
_pulsar_: instant <3: http://t.co/1FiXB0q - if you need to plot some data or a function - way to go! - _pulsar_: instant <3: http://t.co/1FiXB0q - if you need to plot some data or a function - way to go! [/me Twitters]
plenty of videos and slides straight from hadoop world: http://is.gd/h1YeD #base #hadoop #bigdata
_pulsar_: OpenHUG Meeting, wir freuen uns auf zahlreiches Erscheinen! http://linkd.in/drEaBF || http://bit.ly/aERlUU || http://bit.ly/bNGGCt - _pulsar_: OpenHUG Meeting, wir freuen uns auf zahlreiches Erscheinen! http://linkd.in/drEaBF || http://bit.ly/aERlUU || http://bit.ly/bNGGCt [/me Twitters]
I find myself quite often in the need to copy large textfiles over the network. Usually one would go with gzip, either transparently by using the compression switch on scp -C or by archiving a file before pushing it over the wire.
Turns out gzip can compress quite well, but it won't saturate your 100mbit line if you do something like this:
cat bigfile.txt | gzip -c | ssh me@other.side 'cat | gunzip -d > bigfile.txt'
While this has the same effect as scp bigfile.txt me@other.side: it will be helpful to understand the alternatives coming up next.
On a pretty decent machine (Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz) I could get about 60% saturation of the network link. So, what can we do? We could lower the compression level to take load of the CPU and shift it towards the network. We could also use an alternative compression algorithm such as LZO. "lzop" is a free implementation available in most common linux distributions, so this might be the easiest way to go:
cat bigfile.txt | lzop -c | ssh me@other.side 'cat | lzop -cd > bigfile.txt'
My initial tests shown that LZO compression level is about 20% lower than gzip's with default settings. Transfer time was almost cut in half on the other hand. So, how do we get 100% network saturation*AND* great compression? Pigz is a parallel gzip implementation, so instead maxing out only one thread as gzip does, it will use all the available cores and threads your fancy server provides. Downside? No debian stable repository packages available yet. But on the other hand: it does not even require a configure script, how about one header file and 2 "c" files. Most probably your remote connection to the server will take longer to refresh the console output than the compilation process itself.
So, emerge, apt-get install, port install or whatever "pv" and have some fun like this:
me@host:/mnt/data1/import$ cat bigfile.txt | pv | pigz -c | ssh me@otherhost 'cat | unpigz > /mnt/data1/bigfile.txt' 1.83GB 0:00:18 [95.3MB/s] [ <=> ]
Some more numbers: gzip gives me 45 mb/s and lzop 60mb/s
I think "pv" stands for pipe view, it is responsible for the nice stats during the transfer. And yes, this *is* a 100mbit network connection pushing a textfile at ~100mb/s. Nice, isn't it?
You can find pigz over here