"通用人工智能的火花：GPT-4早期实验批注"

需积分: 5 126 浏览量更新于2024-03-24 收藏 6.97MB PDF 举报

The early experiments with GPT-4, an advanced large language model, have sparked new possibilities in the field of artificial general intelligence (AGI). Developed by a team of researchers from Microsoft Research, including S´ebastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang, GPT-4 has demonstrated remarkable capabilities across various domains and tasks. These experiments have challenged our understanding of learning and cognition, pushing the boundaries of AI research. The team's work on GPT-4 has shown significant progress in developing a truly universal artificial intelligence that can perform a wide range of tasks with high accuracy and efficiency. The abstract of the research highlights the potential of large language models in advancing AI research and their impact on shaping the future of AGI. The team's efforts in refining GPT-4 have paved the way for new possibilities in harnessing the power of artificial intelligence to revolutionize various industries and improve human life. Overall, the early experiments with GPT-4 mark a significant milestone in the journey towards achieving artificial general intelligence and have ignited a spark of innovation in the field of AI research. The collaborative efforts of the researchers at Microsoft Research demonstrate the potential of large language models in pushing the boundaries of artificial intelligence and unlocking new opportunities for the future.

Next, we explore how GPT-4 can generate and identify objects in diﬀerent modalities, such as vector

graphics, 3D scenes, and music. We show that GPT-4 can understand and manipulate multi-modal informa-

tion despite being trained solely on text.

2.2 Vision

When prompting the model to generate images of objects such as a cat, a truck or a letter in the alphabet

using Scalable Vector Graphics (SVG), the model produces code which usually compiles to rather detailed

and identiﬁable images (Figure 2.4). See Appendix B.2 for the rerun of various examples by ChatGPT.

Figure 2.4: SVGs generated by GPT-4 for the classes automobile, truck, cat, dog.

2.2.1 Image generation beyond memorization

One may hypothesize, however, that the model simply copied the code from training data, where similar

images appear. Given that the model was trained solely on textual content, one may further argue that there

is no reason to expect that it would understand visual concepts, let alone that it would be able to create,

parse and manipulate images. Yet, the model appears to have a genuine ability for visual tasks, rather than

just copying code from similar examples in the training data. The evidence below strongly supports this

claim, and demonstrates that the model can handle visual concepts, despite its text-only training.

In the ﬁrst example we prompted the model to draw a person by combining the shapes of the letters Y,

O and H (see Figure 2.5 for the exact prompt and the result).

GPT-4

Produce TikZ code that draws a person composed from letters in the

alphabet. The arms and torso can be the letter Y, the face can be the

letter O (add some facial features) and the legs can be the legs of the

letter H. Feel free to add other features.

The torso is a bit too long, the arms are too short and it looks like the

right arm is carrying the face instead of the face being right above the

torso. Could you correct this please?

Please add a shirt and pants.

Figure 2.5: A stick ﬁgure composed of alphabet symbols

GPT-4

You will be given a name of an object (such as Car, Chair, Elephant) and a letter in the alphabet. Your

goal is to first produce a 1-line description of how that object can be combined with the letter in an

image (for example, for an elephant and the letter J, the trunk of the elephant can have a J shape, and

for the letter A and a house, the house can have an A shape with the upper triangle of the A being the

roof). Following the short description, please create SVG code to produce this (in the SVG use shapes

like ellipses, triangles etc and polygons but try to defer from using quadratic curves).

Round 1: A pig and the letter H.

Description: The pig can be lying on its back with its four legs forming the two

vertical bars of the H, and its curly tail forming the horizontal bar.

Round 2: A house and the letter U.

Description: A house with a U shape, where the two vertical lines of the U are the

walls and the horizontal line is the floor. The roof is a triangle above the U.

Round 3: A chair and the letter K.

Description: The back of the chair can have a K shape, with two vertical lines and

a diagonal line connecting them. The seat of the chair can be a rectangle.

Round 4: A dog and the letter Q.

Description: The dog is sitting with its tail curled around its body, forming a Q

shape. The Q is filled with orange and has a black outline.

Round 5: A car and the letter E.

Description: The car has an E shape on its front bumper, with the horizontal lines

of the E being lights and the vertical line being the license plate.

Figure 2.6: Images combining alphabet letters and objects.

The letters of O, H and Y are created using draw-line and draw-circle commands and the model manages

to position them in a way that results in a reasonably-looking stick ﬁgure. It is plausible that the training

data contains information about the geometric shapes of diﬀerent letters, and perhaps the fact that the letter

Y could look like a torso with arms pointing upwards can also be inferred from the training data. Arguably, it

is much less obvious that the model would be able to infer from the training data what is a reasonable way to

position those letters in order to draw a reasonably-looking stick ﬁgure. In a second iteration, we prompted

the model to correct the proportions of the torso and arms, and place the head in the center. Finally, we

asked the model to add a shirt and pants (see Figure 2.5 for the exact prompt and the result). To further

probe the model’s grasp of geometric concepts, we also asked it to create images that blend objects with

letters of the alphabet. The model had to ﬁrst invent a reasonable way of merging the object and the letter,

and then produce the image. The results, shown in Figure 2.6, demonstrate that GPT-4 can usually preserve

the identity of both the object and the letter and combine them in a creative way.

2.2.2 Image generation following detailed instructions (`a la Dall-E)

To further test GPT-4’s ability to generate and manipulate images, we tested the extent to which it can follow

detailed instructions on creating and editing ﬁgures. This task requires not only generative skills, but also

interpretive, compositional, and spatial skills.

The ﬁrst example instructs the model to generate a 2D image with the description “A frog hops into a

bank and asks the teller, ‘Do you have any free lily pads?’ The teller responds, ‘No, but we do oﬀer low

interest loans for pond upgrades.’ ”. We made several attempts to generate the image, each time, the gen-

eration matches the description with the key objects frog, teller, bank, and the two texts. We picked the

most visually appealing version. Inspired by the standard image generation workﬂow, we then ask GPT-4 to

upscale the ﬁgure by adding more details. GPT-4 adds a bank sign, some windows, a car, a traﬃc light, a

few clouds, and makes the frog hold a ﬂower. Finally, we ask GPT-4 to perform various tasks, such as adding

a few objects relative to the existing objects, recoloring some objects and changing the z-order of some ob-

jects. GPT-4 does all tasks correctly. The ﬁnal result is shown in Figure 2.7 (a) and the prompt in Figure B.4.

Our second example is an attempt to generate a 3D model using Javascript. We instruct GPT-4 with the

prompt “A fantasy landscape of ﬂoating islands, waterfalls, and bridges, with a dragon ﬂying in the sky and a

castle on the largest island.” Similar to the 2D experiment, we ask GPT-4 to modify the 3D model in various

ways, such as adding, relocating, recoloring objects and changing the trajectory of the dragon. Again, GPT-4

does many of the tasks correctly. The ﬁnal result is shown in Figure 2.7 (b) and the prompt in Figure B.5.

It is a 3D animation with multiple dragons is circling above the islands.

Figure 2.7: Examples of 2D, 3D images generated according to instructions.

2.2.3 Possible application in sketch generation

Text-to-image synthesis models have been widely explored in recent years, but they often suﬀer from a lack of

spatial understanding capabilities and the inability to follow complex instructions [GPN

22]. For example,

given a prompt such as “draw a blue circle on the left and a red triangle on the right”, these models may

produce images that are visually appealing but do not match the desired layout or colors. On the other hand,

GPT-4 can generate code from a prompt, which can be rendered as an image, in a way that is true to the

instructions to a higher degree of accuracy. However, the quality of the rendered image is usually very low.

Here, we explore the possibility of combining GPT-4 and existing image synthesis models by using the GPT-4

output as the sketch. As shown in Figure 2.8, this approach can produce images that have better quality and

follow the instructions more closely than either model alone. We believe that this is a promising direction for

leveraging the strengths of both GPT-4 and existing image synthesis models. It can also be viewed as a ﬁrst

example of giving GPT-4 access to tools, a topic we explore in much more depth in Section 5.1.

GPT-4 and Stable Diﬀusion

Prompt: A screenshot of a city-building game in 3D. The screenshot is showing a terrain

where there is a river from left to right, there is a desert with a pyramid below the river, and

a city with many highrises above the river. The bottom of the screen has 4 buttons with the

color green, blue, brown, and red respectively.

GPT-4:

Stable Diﬀusion v2.1: Stable Diﬀusion v2.1

with GPT-4’s sketch:

Figure 2.8: Combining GPT-4 and stable diﬀusion

2.3 Music

The data on which the model was trained also contains musical information encoded as ABC notation.

This is a system that uses letters, numbers and symbols to represent musical pitches, durations, chords and

other elements in a compact and readable way. We are interested in exploring how well the model has

acquired musical skills from this exposure, such as composing new melodies, transforming existing ones, and

understanding musical patterns and structures.

When instructed to generate a short tune (Figure 2.9), and the model was able to produce valid ABC

notation. The tune had a clear structure, the time signature was consistent between bars and the notes

followed increasing and decreasing patterns. The tune also used a consistent set of notes within the melody,

and the rhythm had a repetitive pattern. However, the model did not seem to obtain the skill of understanding

harmony. In fact, consecutive notes in the generated tunes are almost always adjacent to each other (namely,

the note following C will almost typically be either B or D), and testing on 10 generated tunes, we were not

able to extract any clear chords or arpeggios.

We then asked the model to describe the tune in musical terms. It was able to successfully give a technical

description of the structure in terms of repetitions, the rising or descending parts of the melody and to some

extent the rhythm. However, it seems that the descriptions of the harmony and chords are not consistent

with the notes (in fact, it refers to sequences of adjacent notes, which do not form valid chords, as arpeggios).

We then asked the model to manipulate the melody in two ways. First, we instructed to change a certain

rising sequence to a descending one, which it did successfully. Then we asked the model to convert the tune

to a duet adding a bass voice. The model successfully extends the ABC notation with a second staﬀ which

has compatible rhythm and is played on a lower octave, however there is a lack of harmony between the two

voices.

In summary, the model was able to produce valid tunes in ABC notation and, to some extent, explain and

manipulate their structure. However, we could not get the model to produce any nontrivial form of harmony.

It should be noted that ABC notation is not a very widely used format, and in fact the model was not able

to produce even the most well-known tunes in ABC notation (such as Ode to Joy, F¨ur Elise or Greensleeves,

all of which are abundant online in that format), nor was it able to recognize these tunes.

GPT-4

Can you compose a short tune (say four to eight bars) using ABC notation?

There are many possible ways to compose a short tune in abc notation, but here is one example:

X:1

T:Example Tune

M:4/4

L:1/8

K:C

This tune has two four-bar phrases, each repeated once, forming an A part and a B part. The key is C

major.

Can you describe the tune in musical terms?

Here is one possible description:

The tune starts with a rising arpeggio of the tonic chord (C major), followed by a descending scale that

returns to the tonic. This pattern is repeated, but the second time the scale continues to the higher

tonic note. The first phrase ends with a long tonic note.

The second phrase begins with a similar pattern, but using the subdominant chord (F major) instead of

the tonic. The second time, the scale continues to the higher subdominant note. The phrase ends with a

stepwise descent to the tonic, followed by a long tonic note.

The A part and the B part have a similar structure, but use different chords and ranges. The tune has a

simple and catchy melody, with a clear sense of tonality and rhythm.

I like part A, but maybe part B is just a bit too similar to part A, making the song repetitive. How

about beginning part B with a descending arpeggio instead of a rising one?

Sure, that would create some contrast and variation. Here is one possible way to do that:

X:1

T:Example Tune

M:4/4

L:1/8

K:C

That’s much better. Now how about making this into a duet, can you add a second staff for this with a

bass accompanying the melody?

One possible way to write a second staff for this tune is to use root notes and fifths of the chords, as

well as some passing notes, to create a simple accompaniment. Here is an example in abc notation, with

the second hand bass on the lower staff:

X:1

T:Example Tune

M:4/4

L:1/8

K:C

V:1

V:2 clef=bass

The sheet music of the final tune:

Figure 2.9: Generating and modifying a tune in ABC notation.

剩余153页未读，继续阅读

RunForSon

粉丝: 2
资源: 11

"通用人工智能的火花：GPT-4早期实验批注"

"探索人工通用智能：GPT-4早期实验报告

"GPT-4早期实验：通用人工智能的火花

微软GPT-4早期实验：探索人工通用智能的火花

微软 GPT-4 报告 人工通用智能的星星之火GPT-4的早期实验.pdf

人工通用智能GPT-4的早期实验

人工通用智能的星星之火-GPT-4的早期实验

通用人工智能的火花：GPT-4早期实验

人工通用智能的火花 GPT-4的早期实验 （中文）.pdf

通用人工智能的火花：GPT-4早期实验.zip

微软：人工通用智能的火花 GPT-4的早期实验

最新资源

微软 GPT-4 报告人工通用智能的星星之火GPT-4的早期实验.pdf

人工通用智能的火花 GPT-4的早期实验（中文）.pdf