The Vocabulary of Forms

VGG16LDAt-SNELatent spaceGestalt

The full pipeline from generating compositions through annotation, topic modelling, and prediction — The pipeline end to end: compositions generated, annotated, grouped into topics, then predicted from image features

Designers describe compositions in words — balanced, crowded, restless — without much account of what in the arrangement produces them. This project learns that mapping: given a composition it has not seen, predict the language a person would use for it. Research assistant work led by Andrew Witt at the Harvard Graduate School of Design, funded by the Harvard Data Science Initiative, 2019–2020.

Building the dataset

No corpus pairing controlled compositions with free-form description existed, so the first step was constructing one. Object silhouettes and simple icons were reduced to 32×32 pixelated figure-ground images and scaled back to 256×256, which strips colour, texture and semantic detail and leaves only the spatial relationships. Grasshopper scripts then generated square compositions parametrically, varying the classical Gestalt grouping attributes — proximity, similarity, continuity, common fate — along with the later additions of common region and uniform connectedness. A DCGAN trained on those compositions supplied more, and interpolating between latent vectors filled in the intermediate forms between the ones specified by hand.

An icon of palms and a hill reduced to a blocky figure-ground composition — Reduction to 32×32 leaves the spatial relationships and discards everything nameable in the image

Annotating

Two hundred compositions went to Mechanical Turk, each annotated by ten workers giving three words apiece — freely chosen rather than selected from a fixed list, so the vocabulary came from the annotators rather than from the study. That yields thirty terms per composition. Tags were left unnormalised, with no stemming or synonym merging, so the variation between annotators stayed in the data rather than being collapsed by hand.

Grouping the words

The thirty-term documents were vectorised over a 150-word vocabulary and modelled with latent Dirichlet allocation into seven topics. Each composition is then represented not by a single label but by a seven-dimensional topic vector — the proportion of each latent theme it carries — which keeps the ambiguity of description rather than forcing one term to win. Projecting those vectors with t-SNE gives a two-dimensional map in which compositions cluster by how they are described.

t-SNE projection of the seven LDA topics, each labelled with its top three words — The seven topics projected with t-SNE, each labelled by the words that weight it most heavily

Predicting from the image

Visual features come from VGG16 pretrained on ImageNet, taking the 4096-dimensional activations of the second fully connected layer for each 224×224 image. A linear support vector regressor wrapped in a multi-output regressor maps those 4096 features onto the seven-dimensional topic vector, evaluated with repeated k-fold cross-validation reporting mean absolute error. For an unseen composition the model returns a position in topic space, and inverse-transforming that position recovers its top-weighted terms.

The same vectors load into the TensorBoard embedding projector, where the space can be queried directly — selecting a composition and retrieving its nearest neighbours in topic space rather than in pixel or feature space.

Gestalt composition–word pairs: each point is a composition positioned by its LDA topic vector and labelled with its dominant terms

Future steps

The corpus is deliberately narrow: abstract figure-ground compositions, a 150-word vocabulary, a single untrained annotator pool. Widening any of them changes what the mapping can describe — real design artefacts rather than synthetic compositions, a larger vocabulary, or annotators with design training whose terms would carry more discrimination. The method transfers to any visual domain where people already have language for what they see and no account of where it comes from.

References

Perceptual organization

Palmer, S. E. (1992). Common region: A new principle of perceptual grouping. Cognitive Psychology, 24(3), 436–447.

Palmer, S. E., & Rock, I. (1994). Rethinking perceptual organization: The role of uniform connectedness. Psychonomic Bulletin & Review, 1(1), 29–55.

Wertheimer, M. (1923). Untersuchungen zur Lehre von der Gestalt II. Psychologische Forschung, 4, 301–350.

Methods

Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022.

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, É. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825–2830.

Radford, A., Metz, L., & Chintala, S. (2016). Unsupervised representation learning with deep convolutional generative adversarial networks. International Conference on Learning Representations. https://arxiv.org/abs/1511.06434

Simonyan, K., & Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations. https://arxiv.org/abs/1409.1556

Smilkov, D., Thorat, N., Nicholson, C., Reif, E., Viégas, F. B., & Wattenberg, M. (2016). Embedding projector: Interactive visualization and interpretation of embeddings. https://arxiv.org/abs/1611.05469

van der Maaten, L., & Hinton, G. (2008). Visualizing data using t-SNE. Journal of Machine Learning Research, 9, 2579–2605.