It is hard to think of a problem more ubiquitous than the diagonalization of a matrix.I will discuss today a statistical approximation method for finding the eigenvalues of a symmetric matrix.

Matrix diagonalization is fundamental to quantum mechanics as well as to most other branch of physics and mathematics. As result considerable effort has been expended on obtaining efficient numerical approximations to the eigenvaues of a matrix. In general, for a matrix of size this takes operations, although in principle should suffice.If the matrix is sparse-i.e., has very few non-zero elements-much more efficient methods are available.

I have not seen todays method anywhere before, but it was suggested by a paper on the Principal Component Analysis of large rectangular matrices:

Dimitris Achlioptas and Frank McSherry.Fast Computation of Low Rank Matrix Approximations. STOC 2001, pages 611-618.

If anyone knows of this method being used before, please leave a comment or let me know by email: I am new to this subject. I only talk of the basic idea here. If the method works in practice numerically I will post here those results as well. My undergraduate student Jason Robin is checking it out on some examples. Also, I have decided not to spend time giving rigorous proofs until it is clear that the method (i) is new and (ii) works in practice.

Sum Over Paths

Let be an symmetric matrix; the spectrum of
is (unordered, allowing for multiplicities) the multi-set of its eigenvalues ;i.e.,roots of the characteristic polynomial

The same information arises as the poles of the function (the Stieljes transform of )

Another convenient way of packaging this information is as the spectral density

If the eigenvalues remain bounded as $n$ becomes large, this can approach a continuous probability density on the real line. Clearly

Conversely, the spectral density is determined by the poles ( or more generally the discontinuity across the branch cut) of .

Yet another way to package the information about the eigenvalues of a matrix are the moments of the spectral density:

This has a graphical interpretation. Think of a complete unoriented graph with vertices labelled by ; i.e., every unordered pair is an edge. Then the last expression is the sum over all closed paths of length , the contribution of each edge being a factor equal to the matrix element corresponding to it. Or, we can take a `grand canonical’ version: sum over all paths with a weight for each edge. This gives the Stieljes transform of the spectral density which is also the generating function of the moments:

Sampling the Matrix

Thus finding the moments of a matrix is the same as averaging over all closed paths in the graph of a matrix. We can now think of estimating this average by taking a sample of paths, a common strategy in averaging over large sets. We must sample paths at random such that the probability of picking a path depends only on its length. This is the discrete version of a well known idea in quantum mechanics, the Feynman path integral; also related is the Wiener integral over Brownian paths. The matrices are replaced there by the kernel of the Schr\”odinger or heat equation respectively. In addition to its conceptual elegance, this point of view had led to practically useful approximation methods. The Monte-Carlo simulation methods of lattice gauge theory are sampling methods applied to path integrals.

Even for finite matrices, we might get useful approximation methods.Suppose we take the original matrix and replace each of its entries independently by a random variable which is zero with probability and equal to with probability . Thus . More generally

if each edge is distinct from the others. If remains fixed as , most paths will be non-intersecting in this sense. If we average over many such randomizations, the answer for the moments should therefore be unchanged.

The number of nonzero entries in any sample will be about . If is held fixed as this is a sparse matrix that can be diagonalized in about steps. This can be repeated a large number times and the average spectral density determined. This should be faster,  than the  direct diagonalization of the matrix.

Actually often it is the largest (or smallest) eigenvalue that is of interest. In this case we can get a sample of values for these variables: finding the largest eigenvalue of a sparse matrix is quite fast by the  Lanczos method.  Tracy-Widom theory gives us the a priori probability distribution of these random variables and we can use a statistical inference based on it to get a good estimate of the largest eigenvalue of the original matrix.

In fact we could go further. We choose an edge at random and replace the entry there with such that the average is ; this can reduce further the memory required to store .But this saving is a constant factor independent of the size of the matrix, so it may not be worth
it.

The Jaataka tales are a collection of parables about the 500 lives of the Buddha until he achieved Nirvana, salvation. After that there are no more re-incarnations. The stories proceed from simple morality tales in which the Bodhisatva ( the soul of the Budha) was alive in the body of a lower life-form: a rabbit, an elephant and so on. Until he attains human form and the stories get more sophisticated. Still retaining the essential simplicity of parables. Various versions of these stories have been told and retold over many generations all over the Eastern World.

One of these stories is a stripped down version of the Ramaayana, one of the two great Indian epics. Unlike in the other Jaataka tales, the Bodhisatva does not appear as a character. In his place is Rama, but no claim is made that Rama was an earlier form of the Buddha.

Rama and Seetha

A long time ago there ruled in Varanasi a benign and just ruler named Dasaradha.

“ Hey, wait a minute, Dasaradha was King of Ayodhya. Not Varanasi. Duh.”

“Yes, that is in our Ramayana. This is the Buddhist version. It is like people abroad think everyone in America lives in Manhattan.”

“Whatever.”

He had 16,000 wives. The oldest of them he made the queen.

“ Now we know which of the two Dasaradhas was busier. Which one was happier you think?” A glare from she who shall not be named. Need to get back to the story.

She had two sons named Rama and Lakshmana. And a daugher named Seetha.

“ I wasn’t going to say anything. But Seetha was Rama’s wife. Not his sister!”

“ Yes, but again, that is in the Valmeeki Ramayana. The Buddhists were the first to have priests who didn’t marry. May be they wanted a hero like themselves.”

The King was devastated when the Queen died unexpectedly. After some months of grieving Dasaratha returned to state matters. Eventually he installed one of his other wives as the Queen. In time he came to adore this young and pretty new queen. They had a son named Bharatha.

He was so pleased with his wife that he told her she could ask for any wish and he would grant it. She recieved his offer humbly, but said she would rather ask some day when she had need of something. She was quite happy with things as they are now.

When Bharatha was seven she went to the King and reminded him of his promise. She wanted him to name her son as the crown prince and successor in place of Rama. The King couldn’t believe his ears. His whole body shook “How dare you ask me that! What do you want me to do to my two older sons? Kill them?” The Queen ran away in tears into her room.

But she persisted and schemed to get her way. Dasaratha got very worried.

“Yeah we know all this.” “See, it is not a completely different story.”

Dasaratha called in the palace astrologer and asked him how long he had left to live. “ Twelve years.” Then he told his older sons ‘Rama, Lakshmana, things are getting really bad for you guys here. I even fear for your life. You should go way for twelve years and return to claim your country after my time. That is how long I have to left to live.” Rama and Lakshmana protested. But eventually they came to see the wisdom of their father’s words. Seetha insisted on leaving with them in their exile. They established an Ashram in the forest, where they lived on fruits and berries and things that grow under the ground.

“ You mean like potatoes.” “Yeah like that, except that potatoes themselves were not introduced to India until much later.” “ Did they make French Fries?” “ Not exactly. Probably more like a barbecue.”

Dasaratha could not contain his grief. He lived a broken man and died after only nine years. Bharatha’s mother immediately wanted him crowned King. But Bharatha remembered his brother Rama fondly. He refused to become King. Instead he set off with the Royal Seal and the four divisions of his army to the forest. Meeting with Rama, he told him of Dasaratha’s death.

Rama was worried how it would affect his brother and sister, who had just returned from gathering berries. He asked them to immerse themselves in a pond of fresh water. When they came up after the dip, he told them the sad news. Both of them fainted. After reviving them they were told again. They fainted again. Only the fourth time they were able to take it. Even then they cried for days and days after that.

Bharatha asked Rama “ How come you are so unmoved by our father’s death? Why aren’t you mourning like them?” Rama said,

“ Life is finite. No point in grieving for something you always knew you would lose. The strong and the weak, rich and poor, scholar and ignoramus, all die. The fruit that is ripe will fall from the tree. The one you see in the morning may not be around in the evening. Just as you pour water to control fire, the wise learn to control their emotions. ”

He then asked Bharatha to go back and rule Varanasi for three years. After the twelve years that his father asked him to be away was up, he would return. And rule the Kingdom for many, many happy years.

“Hey where is the war?What happened to Ravana? Why didn’t he steal Seetha?” “Hello.. Hanuman??”

“I guess the Budhists edited all that out. They don’t want a war hero. By the way, our Rama cried.”

Our new address is
http://www.sgrajeev.com

Please visit us there. There are several  more posts there already.

Please update your links.

Most of what we know of the Spartans is from their arch-rivals, the Athenians. So we have to be a little skeptical of what we hear. Still, we know that they were a city state that was dominated by a small tribe of warriors. They had an underclass of agricultural workers,the helots. These were descendants of the messenians whom they had subjugated in earlier wars. The young men and women of Sparta were separated early on. The men received military training. Women received education as well, unusual in ancient societies. Spartan way of life was austere, based on a system of honor that emphasized valor above all else. They are most famous for the battle of Thermopylae, in which a small band of Spartan braves fought off the invading Persian army.

The Spartans had a contempt for open discourse, the endless speculation on every aspect of life and the universe that was so much a part of Athenian culture. It looked like a waste of time to them; you could say they were “goal oriented”. For a long time the Spartans led a self-contained life, living off the wealth produced by their serfs, largely leaving their neighbors alone. Young Spartan men proved their valor by assaulting and killing in the dark of night any helot who showed any sign of creativity or organizational ability. A terrible situation for the helots but, as it turns out, also for their lords.

For, the lords lived in fear of their serfs who outnumbered them ten to one. Their isolation and natural suspicion of outsiders-the other Greek city states- only compounded this feeling of insecurity. Spartans started to turn their military might on their neighbors. They conquered nearby states, who became their allies in wars with the farther states. Eventually they defeated the Athenians as well, to form a large federation. It looked like brawn had defeated brains.

Then the allies of the Spartans started to notice that they were being taken advantage of. In battle, the spartans would always take the flank that faced the weak divisions of the opponent; the allies always faced the tough veterans. The only thing the Spartans inspired was fear. Their inability to articulate a vision, or their lack of vision, meant that all they could do was replicate their tribal practices in the new lands they had conquered. Whether or not the people in the new lands wanted this. Spartans had only one response to every protest- total warfare.

They did not learn from their mistakes: every battle was fought in the same formation, so that the next time the enemy knew exactly what to expect. Soon their allies turned against them. They figured out that the oppressed helots were the weak link in Spartan security. Once the helots got inspired by the ideals of Athenian freedom and democracy, and learned to organize, they overthrew the Spartans with the help of their former allies.

The Athenians were no match against the Spartans in military might. But their imagination and and eloquence eventually won over the whole Greek civilization. They continued to inspire intellectuals for centuries to come in Europe and the world over. Athenian literature, their philosophy, their democratic values form the foundation of western civilization. Spartans are still used as a model in the militaries. The basic training that all recruits go through is a modernized version of the Spartans training regime for their young men. But other than that, they are an obscure footnote to history.

Indeed, Sparta seems much smaller today than it actually was; and Athens much larger.

Original: April 2005

Tags: history, athens, sparta

Living in Rochester, one hears a lot about how science is done at Xerox and Kodak. Xerox was a little better at it, but their experience was still rather painful. It looks like the company has recovered from some of its stumbles and has posted a reasonable record in recent years.

The standard model of industrial research is that knowledge gained by basic research has to pass through a long chain to be turned into profit. First, it has to be converted to some useful devices; then to a marketable product; then it has to be manufactured, advertised and sold. A single error in any of these steps is fatal, no matter how good the original research. Linear pathways cause blockages, as anyone stuck in a traffic jam knows.

Well-meaning managers have developed sophisticated mathematical models using the Black-Scholes theory of options to justify the value of basic research to their superiors. It is a hard sell though, because each later step in the chain turn out to be about ten times more valuable than the previous one. So the models end up showing instead the relative insignificance the original research contribution .

What is wrong with this analysis? The mere use of mathematics cannot turn a bad model into a good one.

Knowledge is like a gas. Once created, it will expand in all dimensions. Not just the one you want. It will find its way to those who seek it. A savvy competitor can simply take it, especially if you are slow to realize its full potential. Conversely, you need to be ready to jump on someone else’s bright idea. Any resources you spend on hoarding and containing your knowledge are likely to be wasted. Small nimble groups have an advantage in this non-linear process.

The most famous example in modern times is another branch of Xerox (Paolo Alto, not Rochester). Much of the technology that sparked the computer revolution was created by the brilliant scientists there: the personal workstation, the ethernet, even the mouse. But it was Apple computer that first brought the ideas to market. Part of the legend of Silicon Valley is that Steve Jobs was given a tour of Xerox Paolo Alto and jumped at the opportunity. He was twentyfour years old at that time.

So is the lesson that Xerox should have hermetically sealed its research facilities and not give tours to any long-haired young guys? That would not have worked, because somebody else would have got to it, somehow. In hindsight, it is clear that they should have hired that enthusiastic young man. Or at least invested in his company. As Hewlett or Packard would have done.

No system that is based on containing and monopolizing knowledge can succeed in the long term. Especially not in our age of instant and widespread communications. The only way to master knowledge is to be smart, to never stop learning, and have a system nimble enough to adapt as changes happen. A linear product chain cannot do that.

Or does this mean that one shouldn’t do basic research? No, it just means that the people managing the research have to be as quick on their feet as the people creating it. There should be room for changes of direction, not just small course corrections. The knowledge created by basic research must have multiple outlets. It should not be hoarded, it should be widely disseminated. Its real use may be very different from the original intent. So researchers must be willing to follow their ideas to unfamiliar fields. Managers must be tolerant of such excursions by their researchers.

Although universities may be better at creating knowledge than corporations, the recent success of Hotmail, Google, PayPal, EBay, NetFlix, YouTube etc. and the continued success of Hewlett-Packard, Boeing,Intel, AMD and GE shows that it is still possible in industry. It just can’t be done linearly.

Mira Nair’s movie `Namesake’ is about a man with an odd name
(Gogol) for an Indian. I have my own situation to deal with.
My name is usually written as Sarada G. Rajeev.

To most Americans it is just another foreign-sounding name; the first name vaguely sounding like a brand of tea. (I have been asked if my family owns that tea company: Indian,tea natural association.)

The real trouble starts when people of Indian origin hear the
name. They know that Sarada is a woman’s name. The more
well-informed will also know that it is in fact one of the
names of Saraswati, the Goddess of wisdom. So what am
I doing with it as my first name? Also, why do I insist on
being called Rajeev? Contrary to modern American custom
where even presidents want to be known by the informal
first name.

The truth is, Rajeev is my given name. I would say Christian
name if I were Christian. But because of an odd custom in
South India-prevalent around the time of my birth- it appears
at the end of my list of names, so it has become my `last name’.
You see, most Indians use their caste or clan name as their last
name: why there are millions of Agarwals, Singhs and Patels.

My life in America would have been simpler if my
parents followed that tradition and named me Rajeev Nair or
Rajeev Pillai. For, Nair is my caste and Pillai the usual title
used by men of my caste. Nairs slightly higher in the totem pole
of castes can use the title Menon. If you can claim descent from
a teacher of the martial arts (Kalari Payattu), you can call
yourself a Kurup. But I would have been plain old Rajeev Pillai
or Rajeev Nair. My father was Gangadharan Pillai, his brother is
Chellan Nair. We use Pillai and Nair interchangeably, but Nair
has acquired a certain cachet these days. In spite of its association with a hair removal product.

It was also the tradition to attach your mother’s and your
father’s name in front of your own, so I would more likely
have been S. G. Rajeev Pillai. The S. standing for Sarada Amma,
my mother and the G. for Gangadharan Pillai my father.

But in the late fifties and early sixties, it became politically
incorrect for people in Kerala to use caste names to identify
themselves. The left wing movement was at its zenith, my father
an ardent believer in eradicating the caste system. So like many
others, my caste name was chopped off and I became S. G. Rajeev.
Once that name goes on the Secondary School Certificate (High
school diploma) it might as well be tatooed on your forehead.
It continues on the Indian passport and from there to the US
Resident Alien card (green card) and later the US passport.

Thus Rajeev became my `last’ name. Now I have to explain to
every new friend why I should be called Rajeev and not `Sarada’.

And, to continue the confusion my children have Rajeev as their
last names! Some day they will have to explain why their last name sounds like the first name of a former Indian prime minister.

At least Gogol was a great writer. My most famous namesake is an
Indian politician with a questionable legacy.

I can’t even say `My father was a fan’ of that Rajiv.

Vedaanta is the end of all knowledge. End as in goal, or as in the ultimate kind of knowledge. It is a theory of what knowledge itself is. What practical use could it be? Volumes have been written on how to translate the abstract concepts of Vedaanta to every day life. The ultimate authority in `modern times’ (only about a few hundred years ago) is Sankara Achaarya. His Vivekachoodaamani and Bhajagovindam are attempts
to explain this most abstruse of all branches of classical Indian philosophy to the masses; or at least to laymen.

So I sat my ten year old son down one Sunday morning. Started him on the Vivekachoodaamani the modestly titled “Crown Jewel of all Wisdom”.
After the initial prayer to Govinda, which went off well enough, we got into the basic question of ;
`Who am I’?
`Daddy’ came the reply.
`But who is Daddy?’
`You of course.’
`Yeah but who am I really?
`I said, you are my daddy.’
`Oh, then who are you?’
Starting to lose patience: `What do you mean?’
Things are not going too well.

I decided on a different tack. `Your soul is the universe. Ayam aatma brahma’.
`Ok.’ Eyes glazing over.’
`Are you your body?’
`Whatever.’
`Sit up straight!’
`Uh?’
`I said sit up straight and pay attention.’
`Okay…’
`What is the difference between the space inside a pot and the space outside the pot?’
`Which pot do you mean?’

‘ It doesn’t matter which pot; just any pot.’

` How am I supposed to answer if you don’t tell me which pot?’
Eyes tearing up.

Oh well, time to stop for the day.

The next Sunday we start on the Bhaja Govindam.
Perhaps a song would be easier to understand.
bhaja govindam bhaja govindam
govindam bhaja moodhamathe

`Pray to Govind, Pray to Govind
Pray to Govind, you idiot!’
Giggle. `You said idiot.’
`Yes, he means you are an idiot until you start to pray.’

Sampraapte sannihite kaale
nahi nahi rakhshati dukr.njkarane

`When you are on your death bed
your knowledge
of Sanskrit grammar wont save you’
`Oh Ok.I wasn’t going to learn it anyway.’
`He means no knowledge of any kind except devotion to God will help you then.’
`Great!So why are we doing this every Sunday?’
`Don’t be a smartass!”
`Ok.’

evam maamsavadaati vikaaram
manassi vichinthaya vaaram vaaram

`When you get lust for women in your mind,
remind yourself over and over that the body is mere flesh.’
`Did he ever have a girlfriend?’
`No he was celibate all his life.’
`What is celibate?’
`It means he never had a girlfriend.’
`Ok’
`He is saying not to get attached to all that.’
`How would he know?’
`May be he had experiences in his previous life?’
`Or may be he didn’t. That is why he is putting it down.’

Better change the subject.
`What are the six kinds of temptation?
We talked about this three weeks ago.’
`I forget. I do remember the names of Vishnu you asked me to memorize. Viswam vishnu vashadkaaro..’ ;
`Yeah, yeah but that is not what I am asking now.’
`Eyes tearing up..I tried so hard to learn them..you know dad this stuff is real hard for me’

Time to stop for the day. Clearly I am not getting through.

The next Sunday I begin: `let us learn the malayaalam alphabet.
`Ok.’ Unexpected enthusiasm.
`Get a pencil.’
`Sure.’ Runs for a pencil
Even remembers to get paper, no second trip needed.
`Let us start, Dad.’
ka….kaa..ki..kii….
Go through all sixteen vowel forms of ka.

Next week we go on to the next consonant kha.
Tries to read the title page of vivekachuudaaamani ..guesses correctly that the first symbol is `vi’.

Do I have a natural linguist here? Another Norbert Wiener who will learn ten languages by the age of seventeen?
Wow.

His mom smiles.
`He is just so glad you aren’t talking Vedaanta.Just so relieved that all he has to learn is a strange alphabet.’

Ha! I knew it! Vivekaananda was right! There is nothing more useful than Vedaanta.

April 2005