Next, we explore how GPT-4 can generate and identify objects in different modalities, such as vector
graphics, 3D scenes, and music. We show that GPT-4 can understand and manipulate multi-modal informa-
tion despite being trained solely on text.
2.2 Vision
When prompting the model to generate images of objects such as a cat, a truck or a letter in the alphabet
using Scalable Vector Graphics (SVG), the model produces code which usually compiles to rather detailed
and identifiable images (Figure 2.4). See Appendix B.2 for the rerun of various examples by ChatGPT.
Figure 2.4: SVGs generated by GPT-4 for the classes automobile, truck, cat, dog.
2.2.1 Image generation beyond memorization
One may hypothesize, however, that the model simply copied the code from training data, where similar
images appear. Given that the model was trained solely on textual content, one may further argue that there
is no reason to expect that it would understand visual concepts, let alone that it would be able to create,
parse and manipulate images. Yet, the model appears to have a genuine ability for visual tasks, rather than
just copying code from similar examples in the training data. The evidence below strongly supports this
claim, and demonstrates that the model can handle visual concepts, despite its text-only training.
In the first example we prompted the model to draw a person by combining the shapes of the letters Y,
O and H (see Figure 2.5 for the exact prompt and the result).
GPT-4
Produce TikZ code that draws a person composed from letters in the
alphabet. The arms and torso can be the letter Y, the face can be the
letter O (add some facial features) and the legs can be the legs of the
letter H. Feel free to add other features.
The torso is a bit too long, the arms are too short and it looks like the
right arm is carrying the face instead of the face being right above the
torso. Could you correct this please?
Please add a shirt and pants.
Figure 2.5: A stick figure composed of alphabet symbols
16
剩余153页未读，继续阅读
