Archive for the 'Parallell Processing' Category

Going nova: Kx princess denouement 

The  intricacies of this APL story, a story of epic proportions, with richness, contradictions, irony, paradox and most importantly brilliant characters provides everything a storyteller could hope for.  This is the perfect story.  Except…well, a slightly less ephemeral subject would be easier to visualize, but never mind.

The dashing Pierre Kovalev and Oleg Finkelshteyn (left) take after their mentor, Arthur Whitney and are men of few words, big brains and deep hearts.  They are working on performance optimization, a field where those trained in Array Programming Language thinking have an advantage, a mental edge.  Seriously, Professor, take note.

The up and coming Peter Bell, Carlos Butler and Tisean Jeffers (right) from Loughborough University are all bright stars ready for action with poise and affability which indicates there are great things in their future reminiscent of APL’s Brat Pack. Peter, by the way, gets extra points for recognizing me from YouTube, which means, folks, he is out there exploring and paying attention.  Go easy and think about what this says about him, rather than me.

Fintan Quill (left) radiates a calm competence with just a hint of characteristically Irish charm. He’s a bit like Simon, although Simon isn’t Irish.  Fintan has a long history with Array Programming and recently joined the Kx team in New York. Lucky Kx!

And if I really WERE a Princess, and to be candid, I’m certainly not. And in actuality, I get along better with Queens than Princesses, but if I were a Princess, I would give David Katz (below) my scarf, any day.

He has the ultimate, you just never know with APL, story. Having written his Master’s thesis on APL a relatively long time ago, and then finding not much APL action in Michigan back in the day, he went on to more mainstream  Software Engineering.  And then, several years later his resume was plucked out of the pile because of that APL thesis.  AND that’s why I had the pleasure of meeting him in Ireland.  Incidentally, he half volunteered and was half drafted to help me with the next phase of this project, which is getting it financed.

Where are all the women?  Well, it turns out that Janet, Victoria, Alla and Holly are all running around faster than the speed of light keeping everyone in business.   So far, I haven’t caught them sitting still long enough to grab a picture.  But they’re there.  You can bet on it.  And guess what I’m learning?  If they weren’t, these guys wouldn’t be here either.

In  conclusion, the support, encouragement and congeniality from all the people I met in Ireland, pictured and not pictured, have left me burning a little brighter. The realization of a documentary about APL is far from a sure thing, but these folks just nudged it one step further along.

Thank you.

(PS I’m still showing everyone who will tolerate listening to me, J on my iPhone.  I mean everyone. ) 


Kx Princess Report #2

Where were we? Oh, yes, my commitment to Simon to find the K-Club in Straffan just outside of Dublin. And to arrive with something to show.

I flew in a day early to relax and see the city. This was enough time to learn that the Irish can give uncomfortably vague directions that miraculously work. For example, “It’s across the road,” did in fact turn out to be true.

The event was a Kx International Users conference. The tricky business about these conferences is that they are TOP SECRET. In fact, one of my new found friends revealed to me all kinds of interesting details about his personal life and when I asked if they were a secret, because… er… I am making a documentary… He said, no. BUT! What he had just explained to me about his work, now THAT, was absolutely secret.

Don’t get too excited. The only people who would understand any possible trade secret I may have learned, are the ultra-geek of geeks. I don’t expect to be kidnapped and held for ransom anytime soon. So, no kiss and tell. That’s how it goes. I can say that the spirit of innovation which began in the 1950’s with Dr Iverson at Harvard carries on. And simple is still the more difficult and best route. Even after all these years.

It was a real treat to finally meet Arthur Whitney. I expect that if I say too much right now, he’ll never talk to me again, which would be tragic. BUT! It was great. And Janet Lustgarten is the sharpest executive I’ve ever met. She’s an awesome role model for anyone, and is especially inspiring for women in tech. It was a pleasure to spend time with these folks.

Next time I’ll tell you about Carlos, Oleg, Peter, Pierre and Tisean. They share first prize for inspiring super ultra-core fans. What I mean by that, is when I ask myself, Why the heck am I doing this anyway? I just need to see their smiling faces and my batteries re-charge.


Jaxon and me on Falkoff’s one liner

The following is adapted from email conversations with Greg Jaxon, a Compiler Engineer from Illinois, USA who studied at Syracuse University.  He is an active contributor to the APL LinkedIn online forum and it turns out he met my dad at Minowbrook in 1980. I needed a little help to conclude my, “Where were you…” miniseries, and Greg graciously stepped up to the plate.

My dad, incidentally, sends his regards from Manitoulin Island.  Though he still controls his farm house with his iPhone, he doesn’t miss the Internet connection. 

To give a little bit of context, I was born in 1965 to very young and idealistic parents who believed that the 60’s really were going to change things.  In 1966, IBM whisked my family off to NY, USA from Edmonton, Alberta, Canada.  We subsequently moved to the Philadelphia, PA, USA area and ended up living in a small college town called Swarthmore.

Greg Jaxon writes:

One non-programming thing that has always intrigued me about the “APL community” and which has been formative for me politically and personally is our early and frequent use of consensus decision-making.  Perhaps your Dad could start that thread of the story, since (as I understand it) the group at Philadelphia took on this Quaker practice to form the exact definitions in the first APL implementation.

On Day 1 of the X3J10 APL standards effort the topic of voting came up right away. As that work progressed we used a few unorthodox voting schemes to tease out where consensus could be found – a lot of preference ranking and approval threshold measurement. It was clear that the intellectual descendants of the first 6 had the same passion for getting the hive mind to function optimally – to not marginalize the difficult corner opinions, not to cave in to majority rule. I’m convinced this is why APL is so very good – it hasn’t compromised on anything important – instead it found and fixed all the problems until no more could be found.  It’s not just good enough to get by…

The Minnowbrook conferences also echo this emphasis on cooperative agreement. Trade Secrets come out of their closets there – mostly I think out of the sheer joy of meeting other live humans who understand the topics (these are the uber-geeks of an already too geeky computing subculture).

This got my attention.   Swarthmore is in the heartland of Quaker territory.  I was educated by Quakers.  And Greg must have read Adin Falkoff’s, The Design of APL.

I belong to the generation uncomfortably sandwiched between the boomers and their children.  My attitude is formed more from the dress in black, hard core music generation, than the Flower Child generation but I still have strong ties to the Quakers and have remained connected to them up here in Canada.   To my good fortune, I started programming APL as a teen and unlike many of my peers, I’ve had a career from the get-go.  But still, the irreverence of my generation stuck.  In other words, I’m a little cynical.

The first time I read Adin Falkoff’s, The Design of APL, the line about Quaker Consensus jumped right out of the text.  (like: WHAT?  Where the hell does that come from? Consensus? At IBM?) And as I move through this project, I am learning a lot more about business, I have been chipping away at 50+ years of Computer History, and naturally, my gaze falls upon the history of IBM.  Which is also American corporate history.  And patent history.  An intellectual property law history.  I’m still pondering… What on earth is a reference to Quaker process doing in an IBM publication?

Greg responds:

My history lesson on this: Penn was a Flower Child of a famous military officer; he joined the Quakers who were emphatically not the Church of England, nor easily governed by any hierarchical law. Through consensus they sought God’s natural Laws for their community. Penn acquired his North American woods to settle the King’s debt to his late father. But by the time he got with the English aristocracy programme, his Woods were full of Quaker hippies.

For many years he sent governors and magistrates and others to try to collect rents or taxes, and the resident Friends politely declined to impose these on themselves. So your Quakers were the original American libertarians struggling to understand God’s intention for human Law.

To find Harvard mathematicians (arguably in search of much the same kind of revelation) adopt this practice, is interesting.  To see it grow into APL, itself a quaint minority language with an uncannily natural place near the heart of Computer Science’s new fascination with parallel execution models, cooperating independent processes, and clean data abstraction,  … is perhaps a recurrent story in the history of ideas. Your Dad’s “shared variables” ideas combine “message passing” with “shared memory” approaches to parallelism, a synthesis sorely missing in modern parallel languages.

There… my contribution to a historical explanation, I can cite “Conceived in Liberty” by historian Murray Rothbard for this summary of the Quaker colonies.

Wow.  Now THAT gives me a lot to think about.  On this crazy filmmaking journey, I’m paying careful attention to the stories we tell ourselves, about ourselves, our culture and “progress”.  And by we, I also mean people, not just us.

And, sadly, this is the one year anniversary of Adin Falkoff’s death, the man who wrote those words about Quaker Consensus at IBM in 1973.



It took forever to get through immigration and I’m having a really bad hair day (sorry Monica) – BUT AMERICA HERE I AM.

You can follow this little story on twitter #arrayStories


Predictions from the past

In 1998, Dennis E. Shasha and Cathy A. Lazere wrote a book called  Out of Their Minds: The Lives and Discoveries of 15 Great Computer Scientists. This book has become my most useful guide and reference.  In the postscript, Shasha and Lazere make some predictions about the upcoming 25 years.  As we are almost half way to the 25 year mark right about now, I’ve been giving their predictions some thought.  In particular, I’m contemplating three points with respect to the Array Processing Language family, namely:

Specialized languages that can meld components written in different languages will become popular (page 252).

Software design that will make parallel processors behave like a single very fast and very reliable computer presents… [a great challenge] (page 250).

… [New] programming languages will always catch people’s attention.  But like beautiful images, the innovative and influential ones will remain rare (page 251).

I’m also getting ready to speak at Ryerson University on Thursday at the Undergraduate Computer Science Program awards ceremony!  All the while, praying to the technology gods to please help me be entertaining, smart and a worthy array language ambassador to the next generation of Computer Scientists.


Individualism, parallel processing and Condor

Thank goodness it’s someone else’s job to get deep down into technical nitty-gritty and  make things go.  It’s my job to DREAM and make huge semantic leaps.  And that’s how I assert my own individuality on the universe, make my markOr dent, in the words of Steve Jobs re-purposed by Hugh McLeod.

Recently, I was talking to someone interesting and important in the APL community about what it takes to be an APL programmer these days. He said, It’s more than just being proficient in Array programming.  You need to be able to listen and understand what the customer wants, and do THAT.

And this reminded me of another conversation between two former APLers who are both very successful in business and technology now outside of the Array language community:  One says to the other, It took me a long time to realize it’s important to do what my boss wants me to do… Yeah! says the other, Me too!

Oh! Individualism! The Array programmers’ tragic flaw.  Where other languages require a whole army to assemble around a problem, for us it’s just one. We don’t need to assemble.  APL makes us independent.

But what if we want to tackle really big problems.  And I mean huge gigantic impossible to imagine they-are-so-big problems? I mean problems like parallel processing problems. 

What the heck is a parallel processing problem anyway?

I asked Peter Keller if he would explain to me a parallel processing problem, because I read a lot of discourse on the subject, but only see very small hints about why I should care.  Not enough, really, for me to sink my teeth into.

As it turns out, his answer was really cool.  He’s working on the Condor project, which is not an Array language project, but it could be.   Here’s an excerpt of what he wrote to me:

In the use cases that I know of, and for which my small contributions are most likely to be used, it would be data processing for high energy physics.

Basically, modern particle accelerators (like the large hadron collider) produce interesting particle event data in the gigabits per second range for ten or more years straight. This data gets stored and routed to entire countries or political organizations to be processed on the vast physics grids to look for statistical correlations in the data or to see if it matches predicted behavior.

The mathematical models are very complex, sometimes being in pipelines of dozens to hundreds of programs and have to be run upon billions (possibly trillions?) of events whose subsequent data-in varying sizes of kilobytes to terabytes, needs to be moved to the right place, etc, etc, etc. Each scientific research group has their own set of mathematical models, each with their own data pipelines, and there are many groups. The processing of a single event may take 5 minutes or much longer like an hour depending upon the type of analysis being performed on it.

These pipelines already exist… and each one can take 6+ months in real time to run on hundreds to thousands of computers (all going up and down, networks failing, disks filling up, etc, etc, etc). Condor and other batch schedulers are the means by which these disparate workflows are executed (often at the same time in a pool of machines). We provide a robust layer to get the work completed in the face of all kinds of failures. We try very hard to make it that one or two people can manage hundreds of parallel workflows on thousands of machines with little to no human intervention.

High! Energy! Physics! I want to understand high energy physics!

Peter also suggested that I go ask the big physics project entities why they are not using APL or any of the APL Array Languages descendants. I expect that asking is the next best thing to being there.  So, I will.

Now I shall keep a close eye on my stats to see if  I’ve caught your attention.

Oh! I’m still getting hits on my naked Austrian post, by the way.


Analytics Plugin created by Jake Ruston's Wordpress Plugins - Powered by Laptop Cases and r4.