Artist & Critic

DCGANCharRNNGPT-2YOLOLatent spaceStyleGAN

Blueprint for the self-critiquing AI artist — Generated landscape art by DCGAN, critic by CharRNN and GPT-2 (2019)

A painter and a critic in one system: it generates a landscape painting, reads the emotion in it, and writes criticism from that reading. The manifesto covers why.

Iteration 1

The painter is a DCGAN, modified from the TensorFlow example to output 128×128 colour and trained on about 14,000 WikiArt landscape paintings over two weeks on Colab. For the critic, I took the emotion tags used in museum collection metadata and mapped them onto a valence–arousal model, then grouped them into five representative states — sublime, joyous, serene, monotonous, sombre.

Emotion words plotted on valence and arousal axes, with five groups circled — Emotion tags placed on the valence–arousal metric and grouped into five states, with the sublime added for how often it appears in historical landscape painting

Output from the trained DCGAN was then sorted into those five classes, and a CNN captioner learned to label a new painting with one of them. A CharRNN trained on landscape criticism wrote the sentence.

Generated paintings sorted into five affective classes with example images for each — Generated paintings sorted into the five classes as training data for the classifier

Iteration 2

I replaced the captioner with YOLO, annotating the visual features that make up each affective state — theatrical lighting, vast space, tiny human figures — so detection worked on parts of a painting rather than the whole. It reads real paintings more reliably than generated ones, where 128 pixels leave little for the detector to find.

YOLO bounding boxes over real landscape paintings and DCGAN-generated images — YOLO detections on real paintings, left, and on DCGAN output, right

The critic became a fine-tuned GPT-2, trained on Ruskin, Beatty, Howes and other writing on landscape and psychology, prompted with the detected affect and left to complete the sentence. I also tried StyleGAN transfer learning for the painter, though the DCGAN produced the images I kept.

End-to-end run: generated painting, detected affect vector, preset prompt, completed critique — One pass through the system: a painting is generated, its affect detected, and a preset prompt completed into a critique

Walking the latent space

Each painting the DCGAN can produce corresponds to a 100-dimensional vector. Adding a scaled random vector to it returns variations of the same image, and the size of the scalar controls how far they drift. Interpolating between two vectors gives a continuous transition from one painting to another, passing through every intermediate state.

Latent space arithmetic: small and large variations from one vector, and an interpolation between two — Scaled noise added to a single vector gives variations of one painting; interpolating between two vectors transitions from one to another

Some of those paths run along the affective categories, so a painting can be moved gradually from joyous through serene to sublime while its composition holds. This is the part that makes the rest of the system possible: if criticism is going to change the next painting, there has to be some way to move through the space deliberately rather than sample from it at random.

A walk through the latent space, at the generator's native 128×128

The StyleGAN transfer-learning run has a latent space of its own. I kept the DCGAN for its more abstract aesthetics, which ended up being more artistic. With the limited time I trained it, StyleGAN was still more realistic, and would have required more fine-tuning.

Two walks through the StyleGAN latent space

Future steps

A visual interface would let someone generate a painting, see its detected emotion, and shuffle prompts into a critique. Formal analysis would sit alongside affect detection, so the reading covers composition as well as mood. Translating the critique back into visual features closes the loop: an aesthetic goal is set by a person — a painting with vivid colours and a telling contrast, conveying hope — each result is scored to guide the critic, and that advice becomes arithmetic on the latent vectors of earlier iterations, so every painting is a revision of the last.

The full loop: aesthetic goal, painter, critic, and latent vector arithmetic feeding the next iteration — The complete design, with each critique becoming arithmetic on the previous iteration's vectors

The latent space is the part I carried into the vocabulary of forms — a continuous design space you can search and steer, where work comes from navigating possibilities rather than issuing instructions.

References

Texts and data

Beatty, J. W. (1922). The relation of art to nature. Project Gutenberg. https://www.gutenberg.org/ebooks/53477

Howes, E. P. (1905). The psychology of beauty. Project Gutenberg. https://www.gutenberg.org/ebooks/3751

Ruskin, J. (1871). Lectures on landscape: Delivered at Oxford in Lent Term, 1871. Project Gutenberg. https://www.gutenberg.org/ebooks/20019

WikiArt. (n.d.). Landscape paintings [Data set]. https://www.wikiart.org/

Technical

Karpathy, A. (2015, May 21). The unreasonable effectiveness of recurrent neural networks. https://karpathy.github.io/2015/05/21/rnn-effectiveness/

Karras, T., Laine, S., & Aila, T. (2019). StyleGAN [Computer software]. NVIDIA. https://github.com/NVlabs/stylegan

Mu, A. (n.d.). TrainYourOwnYOLO [Computer software]. https://github.com/AntonMu/TrainYourOwnYOLO

OpenAI. (2019, February 14). Better language models and their implications. https://openai.com/blog/better-language-models/

TensorFlow. (n.d.). Deep convolutional generative adversarial network [Tutorial]. https://www.tensorflow.org/tutorials/generative/dcgan

TensorFlow. (n.d.). Image classification [Tutorial]. https://www.tensorflow.org/tutorials/images/classification