James Gurney on Vangobot

James Gurney (author and illustrator of Color and Light and Dinotopia) regularly explores the science behind various aspects of painting at his blog. Here he looks at a system that uses image parsing to choose from among various painting techniques in different parts of the same image.

visual sugar

This painting by Leonid Afremov has, a couple of times, appeared on the front page of the website Reddit. Reddit has a system for voting up submissions, so something that a lot of people like will rise to the top. It is relatively rare for any painting to be upvoted so highly, especially more than once.
From an artistic standpoint, the painting has little to offer. Everything that is good about this painting is good on the surface. It uses a variety of simple tricks to attract the viewer. It is the visual equivalent of McDonalds' food-- it tastes good because it has abundant fat and sugar, which have a direct connection to the pleasure centers in the brain. Here are a few of the tricks:
The colors are all saturated.
There is a rainbow progression from yellow at the center of the light, through orange, red, and purple to blue in the sky. This is a comforting pattern.
The lines of perspective and brighter center draw the eye to the couple, who are expressing love and comfort. (The artist doesn't really have a great feel for perspective, though. What is going on with that bench in the foreground?)
There is a cute dog.
The reflections in the puddles and canal make a kind of symmetry.
The palette-knife strokes allow pleasing contrasts between adjacent colors, and add a kind of visual texture to the image.
These are predictable responses of people to images. Good artists make use of the same techniques. Van Gogh's Starry Night, for example, uses some of the same techniques of contrast, saturation, and lines guiding the eye. I think that anyone, human or otherwise, learning to be an artist has to pass through the stage where they are learning these tricks. Eventually, they become tools that can be deployed as desired with an understanding of the effect they will have on the viewer, in order to get across a more subtle emotional message.

Machinamenta in Old English

Reading a little about the etymology in Lord of the Rings, I came across the word that they used in Old English to translate the Latin word "machinamenta." It was Orþanc, and that p/b shaped letter (called thorn) is pronounced 'th.' This meaning of ingenious devices and siege engines was the reason Tolkein picked the name Orthanc for the tower of Saruman (whose name similarly means 'cunning mind.')

Gödel and Leibniz

Leibniz gets a whole chapter in Machinamenta; I love the story of how he imagined the potential of computers so early. I had thought the rediscovery of Leibniz's digital aspirations (Gregory Chaitin, for example, speaks highly of him in this context) was a recent phenomenon, though-- Charles Babbage was certainly inspired by Leibniz's writings, but as far as I knew his influence on the first electronic computers was pretty far removed. However, I now think that there may have been some influence by way of Kurt Gödel.

Gödel was fascinated by Leibniz's ideas, to the point that others felt he was obsessed: he checked out every book on Leibniz from his university library. He believed (correctly, I would say) that Leibniz's most important ideas (the characteristica universalis) had been nearly forgotten by society; but he also believed that this was due to a shadowy conspiracy meant to prevent the intellectual advancement of mankind. While one could make up a marvelous conspiracy theory about this, involving Newton, the Illuminati, the Invisible College, and so forth, it was more likely due to the fact that many of Leibniz's writings have never been published, and that Leibniz himself never completed the project.

At any rate, Gödel wanted to achieve Leibniz's dream of an exact, computational philosophy, able to come to provable conclusions. Gödel wrote, "There are systematic methods for the solution of all problems (also art, etc.)" Leibniz believed that the natural world arose out of a network of binary relations.  This idea of a mathematical world underlying the world we see, a kind of Platonic realism, was appealing to Gödel as well, and he saw his work as pointing in that direction. Gödel  like Leibniz, believed that the study of mathematics could tell us ultimate truths about the nature of reality. Since, as he proved, it is impossible to prove certain true facts about the mathematical universe, those truths must exist, he thought, somewhere outside of proof.

Gödel's more mathematical ideas were very important to people like Alan Turing, Stanislaw Ulam, and John Von Neumann. Gödel's famous proof of the incompleteness theorem needs to be able to make statements about mathematics using mathematics itself, and this required the invention of something very much like a programming language. Turing's key paper "On Computable Numbers, with an Application to the Entscheidungsproblem" uses the incompleteness theorem, proved five years before, to prove that it is impossible to decide algorithmically whether a given Turing machine will ever halt.

There are two main ideas I tried to get across in Machinamenta, that I laid out in the introduction. One is the idea of the kaleidoscope pattern, which I'm not going to go into here. The other is that the history of computers is not just the history of the development of mechanical math machines. There has also been, for a long, long time, a desire to make machines that can take ideas, and combine them with other ideas, to come up with new ideas. You see this in divination machines, which inspired Ramon Llull. Lllull's own devices inspired Leibniz to develop a much more ambitious and realistic plan. This in turn was taken up by Babbage and Gödel, who were direct influences on the people who built the first electronic computers. The dream of machine intelligence was already fully present through this chain of influence at the birth of the computer.

AI and children's drawings

I've been reading about attempts to write software that approximates the steps that children go through when they create drawings. This gets at some of the fundamental differences between how machine currently generate images in "artistic" styles (using brushstroke filters in Photoshop, for example) and  how an artist paints. By trying to shortcut past the early representational stages, we have failed to capture some of the important things about what it means for a person to create a painting.
One of the best papers I've found is called Thoughtful Drawings: A Computational Model of the Cognitive Nature of Children’s Drawing. It describes a piece of software called Rose (Representation Of Spatial Experience). The author states, "Rose is not intended to be a model of any part of a child’s mind. Rose is a representation of personal ideas about just a few vital elements of the human experience of drawing."
Rose takes as input a 3D form, composed of triangular surfaces.

From this form, it recognizes certain parts that are joined together-- the legs, the neck, and the tail all joined to the body, and the head joined to the neck. Each of these body parts is fit with a cylinder.

Then the program attempts to draw the shape. It attempts to create closed curves (by moving a "pen" around with an imperfect control algorithm) whose length and width are determined by the proportions of the cylindrical body parts, and are connected in the same way.

This projection doesn't take into account perspective, viewpoint or occlusion: it simply copies the connectivity of the graph of how the body parts join together. This seems to me to be how children begin to draw. We could test this by giving children an unfamiliar shape and seeing if the drawings tend to follow this rule. I would also like to know when a child will use a closed curve and when they will simply use a line to represent a part.
I would like to do something similar to this, but add in an extra step. I would like to give the program a computer vision capability that allows it to look at its own drawings and see how much they resemble realistic line drawings of the same subjects (or, more practically, semantic contour detection on photos of the same subjects). Depending on how good the resemblance is, it could choose what lines to keep and what lines to erase and try again. This extra judgement step is an important aspect of how people create art.

lichen and wildfires

lichen

wildfires in Russia, seen from space at night

the similar shapes betray a similar growth pattern

Pareidoloop

This is a program that satisfies, at a very simple level, the creation/evaluation loop that I felt would be necessary for any machine to make something we would be able to find creative. It randomly places triangles  until it finds a face using face detection software. A few results are pictured above. I see a lot of character in these faces that I think wouldn't have shown up in a more direct synthesis from a face model.
The author is Phil McCarthy.
The face detection method is the really rich part of this program that does most of the heavy lifting. It is based on this paper: High-Performance Rotation Invariant Multiview Face Detection by Chang Huang, Haizhou Ai, Yuan Li, and Shihong Lao, from Tsinghua University. The paper advances both machine learning techniques and optimiztion methods to make it fast. The training set is Labeled Faces in the Wild. The faces in the training set form the algorithm's idea of what a face should be.
The program was inspired by Greg Borenstein running a face tracker on the flickr pool Hello Little Fella!, and by Evolution of the Mona Lisa by Roger Alsing.

The name of the program, Pareidoloop, comes from the word pareidolia used to describe our tendency to see faces in the clouds, or in the rocks.  I put this poster by Bev Doolittle on my bedroom wall as a kid, that plays with this idea:



(The poster came from Ranger Rick magazine, which I had a subscription to for years.  I learned a ton of science from that magazine; it built up a lot of the structural framework in my memory that later ideas I was taught about ecology, botany, and so forth attached to.)

Procedural content

Video games sometimes make use of generative art techniques to create more natural feeling worlds at a lower cost in the time of the designers. This is referred to as procedural generation of content.


procedural planets:

procedural cities:

procedural trees:

procedural deserts:

The artists who make and use this software consider them as simply more powerful tools.  

the greatest and most consistent magicians

"It may surprise you that I, the Hard-SF Guy, believe there’s magic. But let’s define it as the use of incantations to create vivid subjective realities in other people’s heads. That’s what most magic has always been. The shaman might not really be able to make it rain. But if his shtick was good, he would get fed! By that light, we authors, especially in science fiction, are the greatest and most consistent magicians. We concoct long incantations — chains of spaces and black squiggles (a million of them in Existence) — and skilled recipients of the spell (well-educated readers) proceed to scan those squiggles with their eyes, decrypting them swiftly into clever dialogue, deep emotions and insight, or star-spanning explosions. This partnership of spell-weaver and incantation-user is stunning, and remains far more effective for the full-rich texture of invented worlds than any competing medium."
--David Brin interview with Frank Catalano

Babbage's Siege Engine

The title of this blog, Machinamenta, means basically the same thing as another Latin word, machina. They both mean "machine," but there are a few kinds of machines that machina is used for most often. One of those is siege engines, the huge contraptions wheeled up to a castle gate or wall to try to breach it.
In 1823, Charles Babbage was trying to get the government to fund development of his computer. This was a very unusual idea at the time, and the politicians were uncomfortable with it, wondering what they would be expected to fund next.
The prime minister, Robert Peel, wrote in a letter "'Aut haec in nostros fabricata est machina muros.  Aut aliquis latet error.' I should like a little consideration before I move in a thin house of country gentlemen, a large vote for the creation of a wooden man to calculate tables from the formula x^2+x+41. I fancy Lethbridge's face on being asked to contribute."
The Latin quote is from the Aeneid, where Laocoon is urging the Trojans not to accept the wooden horse that the Greeks have built into into the walls of Troy: 'It is an engine set against our walls... or some other trickery lies hidden within.' Peel is saying that the difference engine is a siege engine, bringing change that is far too strange for the 'country gentlemen' in Parliament to be willing to support. 
(The formula he mentions will be a familiar one to anyone who has ever played with finding patterns in prime numbers. The difference engine was able to calculate tables of results for any second degree polynomial using the method of differences, a technique that only requires successive additions.) 

The illusion of meaning

http://www.freakonomics.com/2012/06/12/paying-for-transparently-useless-advice/

A good proof is like a story. Both are like reality run backwards: disparate things flow to an ordered final state.

Take-the-Best

This is continuing the ideas in a previous post, Franklin's Perceptron. I am trying to find out whether any ancient methods of artificial intelligence actually worked. Any ancient methods of AI must have been very simple, but a surprising finding in machine learning has been that simple models can often perform better than more complex models.
One example of this is the rule called take-the-best. Here's how it works:
Suppose you want to predict whether one team or another will win a soccer game. (Maybe you're going to place a bet.) You have various pieces of information about the teams: the ages of the players, the number of wins each team has had this season, and so forth.  You also have this same kind of data from the whole league from a few seasons ago, along with which teams won and which teams lost. We call this the training data.
The take-the-best rule goes like this:
See how well each piece of information does at predicting wins on the training data. For example, the team with the most past victories won 75% of the time.  The team with the oldest players won 64% of the time (say this is a youth soccer league) and so forth.
Now, to make your prediction, just choose the piece of information that does the best job at predicting wins, and go with that.  Ignore all the rest of the data. (If that piece of information says they are both as likely to win, you can use the next best predictor as a tiebreaker.)
Now it's easy to see that this is a good quick-and-dirty method that should work pretty well.  The surprising thing is that in tests with real data, this method usually does better than statistical regression, artificial neural networks, and all kinds of other complicated techniques people have come up with to solve this kind of problem.  It's as if the food they served on the dollar menu at McDonald's was not just the cheapest food, it was also the tastiest and the most healthy!
 The reason it works so well seems to be that it avoids overfitting.  It's easy for more complex methods to find patterns where there are none, and be misled by noise.  Here's a paper that explains more.
What I'm looking for now is people who advocated this method, the longer ago the better. Even better would be if they built some kind of device to do this for them. And if not, it's at least would make a good basis for a science fiction story set in an alternate past.

The First Search Engine

(Excerpt from Machinamenta)

The idea of punched cards for storing data was taken up by a Russian inventor, Semyon Korsakov, in 1832. Korsakov worked in the police ministry and kept extensive records on the populace. He devised a way of recording data by means of holes punched in cards. If two cards shared many of the same holes, it meant that the data they contained was similar. This fact could be used for creating a kind of search engine, where a plate with pins would rapidly scan across hundreds of records, only falling in and stopping when the pins could fall through all of the holes, indicating an exact match. Korsakov was excited about his ideas, and thought that they could be used to enhance human intelligence in the same way that the microscope and telescope had been used to enhance human sight. He wrote,


"machines intellectuelles would limitlessly strengthen the power of our thought, as soon as distinguished scientists apply their knowledge to studying the principles of this process and compose the tables necessary for its application in various fields of human knowledge." 

His designs for the ideoscope and homeoscope were released to the world (open source fashion) rather than patenting them to encourage their widespread use and further development. Unfortunately the Russian Academy of Science didn’t see the potential and little resulted from the inventions. He is today better remembered by the homeopathic medicine community for his remedies, than by the information science community for his ingenious method of searching through the database of those remedies.

For more about him, check out the Wikipedia article (and also the references at the end of the article):
http://en.wikipedia.org/wiki/Semen_Korsakov

I like this because it is basically the original version of Google. We don't think of search engines as AI, usually, but they really are automating, in a spectacular way, something that was only possible before through human intellectual effort. Like topics in philosophy, once something is solved we no longer call it artificial intelligence.

Film and Photography as Art

I was reading an article by Berys Gaut on Cinema as Art.What brought me to this book is the possibility of a naturally occuring camera.  The author imagines a natural camera obscura (a hole into a cave) and some naturally occuring photographic salts on the floor of the cave. If a photograph is created of a passing elephant, we would call that a real photograph. But if paint were to fall onto the floor because it was blown by the wind, we wouldn't call that a real painting. That's because a painting, Gaut supposes, necessarily involves intentionality on the part of the artist, while the photograph does not.
Could one create artificial intentionality in a robot that puts paint to canvas?
He points out that in a sense, any film is a documentary, in the sense that it records what the actors were doing at certain moments in time. Except special effects blur this line, as do animations, I would think.
Anyway, there may be a much older connection between artistic representation and the camera.  Watch this video about stone age cameras!

How can we compose better music?

I just read this excellent article about the causes of musical frission, what makes a particular moment in a piece of music strike you as just right. It would be nice to see these ideas built into an automatic composition program like Melody Generator. In the book I mentioned how a sunset can induce strong feelings through an unguided process.  I don't see any reason we couldn't make a system that generates beautiful music like the atmosphere generates beautiful sunsets.

In fact, it might be cool just to make an artificial sunset generator using a fluid simulation and the physics of light passing through clouds.  This is an image I made years ago using POV-ray and Perlin noise functions:

 I think it gets the cloud shapes about right, though the lighting needs a lot of work.

Physiological and Musico-Acoustic Correlates of the Chill Response

Martin Guhn, Alfons Hamm and Marcel Zentner

Music Perception: An Interdisciplinary Journal
Vol. 24, No. 5 (June 2007), pp. 473-484 

Anima in Anime

Anima in Anime: the nature of the soul in Japanese animation.

I just thought that would be a fun essay to read. It would discuss Ghost in the Shell, The Spirits Within, Paprika, maybe... what else do you think would be good?
I read a book about Ki (the Japanese symbol for breath and spirit) when I was trying to learn Kanji while living in Nagoya. Since my English reading material at the time was so limited, I have an almost supernatural memory for anything I read at the time.
Edit: I've just finished watching the anime series Mushishi. In these stories, the spirits are more like insects or fungus. They are a natural phenomenon that sometimes affect people when their lives intersect. I highly recommend it. 

19

This looks like an interesting resource on 19th century views of the body as machine.
http://www.19.bbk.ac.uk/index.php/19/issue/view/70

Llull, Bruno, Borges, and the fourth dimension

Charles Hinton worked on visualizing the fourth dimension.  There are some interesting connections between his techniques and and other visualization methods like the Ars Memoria. (Ramon Llull, you'll recall, invented a way of recombining words to make new ideas called the Ars Combinitorica, which influenced Leibniz's grand AI project. Jorge Borges, the scholar and author, commented on Llull and on ontologies.)  Check out the links below.

http://www.waggish.org/2010/charles-hinton/

http://ideafoundlings.blogspot.com/2009/10/maelstroms-from-renaissance.html

http://higherspace.wordpress.com/2010/03/09/borges-on-hinton/

Also, Boole's daughter (I discuss both her parents in the book) was adept at visualizing the fourth dimension.
http://www.agnesscott.edu/lriddle/women/stott.htm

Aeolian harps

This is an Aeolian harp, driven by the wind.  I talk about them a bit in the book, but you really need to listen to one to understand the description.
Steve Cameron is an artist who built one as designed by Athanasius Kircher. Here is a link to his videos.

Here is another one to listen to. Does it remind anyone else of the Myst games?

Natural magic

Ken Perlin (my advisor when I was at NYU) gave a talk recently in Hong Kong. Notice how the tech demo resembles a magic show.  The connection between stage magic and science/technology that was pervasive at the time of Athanasius Kircher and has never really gone away.

Cabinet magazine published an interesting interview on this topic with Simon During.  During contends that magic (in the sense of stage magic) has had a huge influence on the film industry, advertising, and art. All of these take advantage of illusions.

AARON

AARON is Harold Cohen's artistic program.  It was available as a screensaver from Kurzweil for a while. I mentioned in the book that Cohen didn't consider his program creative, but he is interested in figuring out what he would need to do to make it creative.  In an article in Wired magazine, he says:

"All the time in the back of mind is the question of machine autonomy. The problem I would face if I ever figured out how to do it, in giving Aaron as it were its own head, is I might hate what it does...If the program did a drawing in August that it couldn't have done when I stopped programming it in January, then I'll consider it creative."

It wouldn't be hard to achieve what he is asking for in a trivial way-- the program could take in new imagery from the web to learn new leaf shapes, for example.  But I suspect more than this would be needed for Cohen to feel satisfied. When he says, "I might hate what it does" that might be okay if other people liked it. The question is if we can set a program free and yet give it enough of a human sense of aesthetics that any people will like what it produces.

Another good article is from a course page at MIT about genetics and culture.  This describes a little of how the code works, and the design process behind the program. 

Cohen's own writings on the subject

Wikipedia article