Poetry generation with rhyme, meter, and phrase constraints

In mid-February 2020, I was stuck in a small, crowded airport for several hours on the way home from a meeting in upstate New York. (This was the last international meeting I would attend before Covid-19 shut everything down.) I had been working with GPT-2 to create text adventures in a fundamentally different way than AI Dungeon 2. So while I was in the airport and on the plane ride home, I wrote a program that generated rhyming poetry using GPT-2. I had already written another program to create rhyming word pairs using word2vec, so I knew that rhymes could be checked using the CMU pronunciation dictionary.
This first program worked on a token-by-token basis. Only whole words which were also tokens could be generated. The last token-word in the second line had to rhyme with the last token-word in the first line. The selection was made more efficient by dividing the list of words to be generated into sets of rhyming words that all rhymed with each other, a trick I had used for the word-pairs program earlier.
While the results were interesting, I was unsatisfied because it lacked meter constraints.So in May I again used the CMU pronunciation dictionary to generate all possible continuations  of a prompt over a certain probability using a depth-first search method. It followed the most probable continuations, and when it wasn't able to satisfy a meter, rhyme, or probability constraint, it backed up and tried again.
This worked pretty well, but it was slow. So again I created something like the rhyme sets, but this time for meter. Any word in the pronunciation dictionary that could be consistent with the next three stresses was allowed in the set associated with those stresses. Since line ends also needed to be accounted for, I also included one- and two-stress sets.
The biggest remaining problem was that sentences often ended in the middle of lines, and the poem had no notion of ending at the end of the poem-- it was as if it were clipped arbitrarily from a longer poem. So I added in what I call phrase constraints. This says that a line must end in some punctuation (? ,! , . , , , ; , or :) and that punctuation must be sufficiently probable. If not, it must back off and try again. This does an adequate job of making the meaningful phrases line up with the lines of the poem. (It prevents some perfectly acceptable poems from being generated, though.)
If you want to try it out, here is the github repository.

GPT-3 and creativity

Here's a sample generated by GPT-3:

Sample #99

If your personal philosophy is a little bit Taoist and a little bit Marxist, there's a new place in LA to scratch that particular philosophical itch. Tao Mao is the only East-meets-West neo-communist tiki bar in LA, a three-story Chinatown bordello of poi dogs and palm trees. To get there, you enter from a rear alley and walk past the washboard abs of dancers in the Bumpin' Uglies Go-Go Bar to the karaoke bar called Red 7, and from there climb a red-lit staircase to the restaurant.

Tao Mao, "the great origin," is the Chinese name for the Milky Way galaxy, as seen from the Earth. But it's also the name of a star in that galaxy. And in Mandarin, Tao Mao is pronounced "Dow Mao," which sounds like "D.M.A.," or "doctor of philosophy." And that is exactly what Tao Mao is—a tiki bar where all the drinks have the names of philosophers.

You may be wondering: Is Tao Mao a tiki bar for philosophers? Or a bar for philosophers that serves tiki drinks? Or is it both? Does it matter?

Actually, no, it doesn't. Tao Mao is a philosophical tiki bar, that's all you need to know. The more you think about it, the less sense it makes.

Before I try to explain Tao Mao, I should mention that the bartender told me the bar's name is pronounced "Dow Mao." And he is a white dude. And yet... all the cocktail names are in Mandarin. And he only speaks English. So what the fuck do I know? Maybe I'm the crazy one.

I can tell you that the people who seem most at home at Tao Mao are little clusters of friends—mainly white dudes who look like they'd be hanging out at some over-designed tiki bar that was actually inspired by Tao Mao. For example, I met two TV writers who were there for the first time. The owner was there with his mom. The first time I visited, the woman in the next booth to me was there with her sister, and

The name "Tao Mao" is not the name of a real bar. So the wordplay is its own invention. One of my favorite things about GPT-3 is that when it fails to continue a list like you want, a common failure mode is to start explaining what all the things in the list have in common, or otherwise analyse the list (because that's a common thing people do when they write a list on the web). 

New music generation model from OpenAI includes singing

https://openai.com/blog/jukebox/

Randomization as an advantage in divination

Sometimes being random can be an advantage. When you are playing a game against an opponent who is trying to guess your strategy, having the ability to use randomization to be unguessable is critical. In hunting, the caribou are trying to guess the strategy of the hunters to elude them. From The Secret of Our Success by anthropologist Joseph Heinreich:

"Traditionally, Naskapi hunters decided where to go to hunt using divination and believed that the shoulder bones of caribou could point the way to success. To start the ritual, the shoulder blade was heated over hot coals in a way that caused patterns of cracks and burnt spots to form. This patterning was then read as a kind of map, which was held in a pre-specified orientation. The cracking patterns were (probably) essentially random from the point of view of hunting locations, since the outcomes depended on myriad details about the bone, fire, ambient temperature, and heating process. Thus, these divination rituals may have provided a crude randomizing device that helped hunters avoid their own decision-making biases."

(Thanks for the book review, Scott Alexander)

OpenAI's Musenet

Musenet uses the same 'transformer' architecture as GPT-2-- it is a general method for continuing any sequence. You can even feed it an image as a scan-line and it will finish up the image in a fairly realistic way. You can find some nice fugues here:
https://twitter.com/rewonfc

GPT-2

GPT-2, a text generator by OpenAI, has been making waves the last few weeks. The Turing Test is basically a kind of constrained text generation, so this area is particularly important in artificially creative machines that are meant to demonstrate intelligence. It seems to have motivated OpenAI to change their business model completely.

The post where they show off the kind of results they are getting with the model trained on a large dataset

You can try it yourself here, trained on a smaller dataset
If you haven't used a colaboratory before, it takes a little poking around to get how it works. What you need to do is run the first cell, get everything initialized, run the second cell, and then enter the text you want it to riff off at the prompt.
Some of the things I tried entering at the prompt, and getting it to generate:
computer code (it closes parentheses)
SCP artifacts (one of my favorites)
Shakespeare
Lovecraft
Bible (it creates sequential verses)
recipes
Star wars fanfiction
love letters

Here is material from a non-existent role playing game manual (unedited, except for line breaks):

Amulet of the Planeswalker 
Wondrous item, uncommon (requires attunement)
 You learn the art of andether and control the ever losing hearts of planeswalkers. In this celestial dance, you perceive your planes at will through the concepts of psionics. 

Amulet of the Heart 
Wondrous item, rare (requires attunement) 
This amulet increases mobility and allies' movement speeds. You gain an extra Strength score, as early as 1 level. 

Amulet of the Rathbone 
Wondrous item, rare (requires attunement) 
This amulet increases movement speeds and increases attack bonus against flying creatures by 10 percent. Known by creatures that receive different skills from you or others fanged, you are hit by a flurry of powerful and piercing attacks while studying this amulet. 

Amulet of the Spellcasters 
Wondrous item, uncommon (requires attunement) 
The amulet of death abides by a vague spell that negates the effects of spells. Spells you cast may damage your mortal senses by a plus-d3 to any target of the spell, even if the surrounding allies perceive you as their own. On an attack roll of +1, you may stun the target for 30 seconds. On all scores, the target may now remain stunned or staggered. Further, when you cast a spell other than Seismic Prayer, you may cause the target's save DC and the caster's Cha modifier to equal your spell level before you take the spell. Use of the Ward cost 1d8 damage and Anticipation cost 4d8 damage. 

Amulet of Shadows 
Wondrous item, rare (requires attunement) 
Hand-to-Hand Combat changes the appearance of your vision to resemble the appearance of any other rogue. 
Style: While wielding a staff of silver, you sense the motions of the luminous, maddening movement of the 23-foot-radius (88 cubits) green dragon and can pass through on every turn. While the wearer is dead, enemy spells or creatures that are within 30 feet gain the illusion of you. 10 If 1 want the same effect, roll a d6 to determine the desired effect. 

Amulet in the Mystic Alchemical 
You can re-energize this amulet with the follow enchantment. It will easily hold 300 magicka in it's holder. Fiendish Stone-Sands Replaceting this item causes the Inquisitor to hydrate the ground around Daisuki in his decaying room. It cures 2000 ft of fur poisoning. 

If you want to see various options for the next word, and how probable it thinks they are, you can try it at the AllenAI institute site. Try always clicking the top word, versus always clicking the second word or the tenth word to get a taste of how you could introduce variety into the generated sentences.

You can get better results by fine-tuning the model on a dataset that contains only the kind of thing you are trying to generate. Gwern has been playing with poetry generation. Scott Alexander had some commentary on this.
Mario Klingemann has been generating wise quotes.

Style Transfer for Jewelry Design

I've been playing a lot with style transfer this year, and one of my favorite results has been from copying jewelry styles. I've done a little cleanup on these-- I only generate one half and mirror it, make sure it has a clean background, and I use the "healing brush" tool here and there. Here are a few examples (click to see larger):

People are always interested to see the style images I used, so here are a few: