You don’t want to use it too often.
For a friend who spoke another language before English, I will sometimes email or text in their first language. It’s so good as a surprise.
No having to think English into Croatian.
Deep learning is both a new term and an idea that has been around more than seven decades old.
It all began with neural nets. These two terms are used interchangeably even though they do not mean the same thing. There is a difference.
First, let’s remember that this story begins with a friendship between two men that would lead to great science.
They would go on to make history. And in the end, they would taste tragedy.
But here, as in their lives, the ideas come first.
How does deep learning differ from neural nets?
Deep learning is at the bleeding edge of artificial intelligence. Machines in deep learning teach themselves to absorb data and learn from it. They do not need a third party to “train” them.
Neural nets, or artificial neural networks, recreate the brain’s neuronal networks to copy how humans learn. ANN’s are the answer to, “What happens if you could model the mind via the logical calculus?”
They do need a third party to help train them.
More than anything, it’s sweet to see what radical transformation can come from such humble beginnings.
It all starts with a little boy locked overnight inside a library.
The boy who read Bertrand Russell
The story of neural nets starts on an afternoon in 1935. A 12-year old bookworm flees bullies by ducking into the local branch of the Detroit public library. This was a go-to move for him.
He escaped into the stacks from peers who wanted to beat him up. While hiding, he taught himself Greek, Latin, logic, and mathematics. Those skills were of little value in his world.
Motor City during Prohibition had no use for a 12-year old autodidact. Neither did his father, who pushed Walter Pitts to quit school.
Pitts started at page one and did not stop until he had read all three volumes, cover to cover. It came to more than two thousand pages. He did not stop there.
Seeing that he believed he had found several mistakes, he felt it was his duty to let Russell know about them.
Russell replies with Pitts's invitation to come to Cambridge as a graduate student to study with him. Of course, he could not do any such thing. He was 12-years old.
He does stay in correspondence with Russell.
When Pitts hears that Russell is to lecture at the University of Chicago, he runs away from home and heads to Chicago, never to return home again.
To the manor born
Philosophy and psychology at Yale University followed. He came from a family of lawyers, doctors, and theologians. He was from the other side of the tracks from Pitt. Yet they would spend their working lives with each other.
And they would be happier for it. The two would leave their mark on thinking machines for generations to come, as well as on each other for the rest of their lives.
Before all of that can happen, the two thinkers must meet.
Two Titans Meet
When they first meet, McCulloch is 42, Pitts is all of 18, and he has no place to live. He was said to thrive on ice cream, whiskey, and late nights — rarely sleeping before 4 am.
He worked low-wage jobs, loitered around the University of Chicago, and snuck into Russell’s lectures.
He met Jerome Litvin in one of those lectures. Litvin would introduce him to McCulloch.
They found they shared a mutual love for the work of Gottfried Leibniz, the 17th-century philosopher.
Leibniz developed what he hoped would become an alphabet of human thought. Each symbol would represent a concept, and with them, he wanted to synthesize all possible human thought.
Russell and Whitehead inspired McCulloch with their mission to model mathematics via logic. He tried to use Leibniz’s logical calculus to model the brain in a similar epic swing for the philosophy fences.
The attempt began with the most basic proposition: it is either true or false.
Then came and — the conjunction. After that was or — the disjunction. Next was not — the negation. And so they worked up the ladder of logic, transforming logic gates into a symbolic language.
First comes the logic, then comes the neurons from which the logic will spark.
Turing tips them to the possibility
British mathematician robust thinker Alan Turing wrote a paper theorizing a Turing Machine that could compute any function as long as the function could be solved in a finite number of steps.
McCulloch worked with the idea that the brain itself was a kind of Turing Machine. He believed it used logic encoded in neurons to compute. Pitts was all in for the idea at once, and thus, so was McCulloch.
McCulloch had the concept. Pitts had the mathematics needed. Modulo arithmetic, the modality Pitts chose, studies how addition, subtraction, multiplication, division, and roots work when applied to integers.
What exactly is modulo arithmetic?
Modulo arithmetic deals with the zero, the counting or natural numbers, and the additive inverse or negative numbers. When they reach a certain value, they “wrap around” or circle back.
Friedrich Gauss derived the modern approach to these numbers in his Disquisitiones Arithmeticae, published in 1801. An easy example of a modulo is a 12-hour clock. They partition a day into two 12-hour periods. If it is 5 PM now, in nine hours, it will be 2 am.
Though five plus nine is 14, clocks “wrap-around” or reset every 12 hours. Since the hour numbers restart after 12: o’clock, that’s arithmetic modulo 12. In other words, according to Gauss, 14 is congruent to 2 modulo 12. That means 14:00 on a 24-clock is equal to 2:00 on a 12-hour clock.
Pitts understood that time t+1 circling back to before time = t, or when 2:00 went back before 12:00, it was not a paradox. Time was not part of the equation. If you saw a cocker spaniel in their model, the signal runs from the eyes to the brain.
You could start at a neuron and retrace the signal. Say the chain was a loop instead. It would then spin out of time, on and on. In other words, chains that spin as loops correspond to memory.
They did good work
The two men developed the first mechanistic theory of mind, adopted the first computational approach to neuroscience, and made a robust argument for the brain as a data processor. They wrote their findings in a legendary paper called, “A Logical Calculus of Ideas Immanent in Nervous Activity.”
Published in the Bulletin of Mathematical Biophysics, they proposed that it was possible “to know how we know,” outside the shadows cast by Sigmund Freud. From this start, Pitts would help create the conditions that led to cybernetics and AI.
Pitts and McCulloch would go to the Massachusetts Institute of Technology to help find the first program to study mind and intelligence, with a twist.
This program would include the perspective of and concepts from anthropology, linguistics, philosophy, artificial intelligence, neuroscience, and psychology as part of the blend — what we now call cognitive science.
The frog’s eyes broke Pitts at the end
However, the natural world, both animals and humans, would drive him to despair. He wanted to see if the eyes of a frog were as he theorized. That is if they had one to one mapping of the image as seen.
The found was that there was contrast, mapping, but that nature took shortcuts that were not the stuff of pure logic.
Rather, it smacked of analog conveniences. The frogs would be a tragedy, oddly, of his intellectual journey. Destitution, isolation, alcohol abuse; had also taken a toll on his mind too. This was more than the eyes of a frog.
Pitts would leave this mortal coil four months apart. One was broken and with a cirrhotic liver. The other? He was broken-hearted at the loss of his friend. Their final act may have been less glorious than those with preceded it. Still, so much of what we now take for granted happens only by dint of their good labor.