1
00:00:00,900 --> 00:00:04,500
ChatGPT seemed to come out of nowhere in November 2022.

2
00:00:04,960 --> 00:00:08,560
But pull on any piece of it, and you find something built for another reason.

3
00:00:11,099 --> 00:00:18,320
Take the artificial neuron. In 1943, the psychiatrist Warren McCulloch wanted to know how brain tissue could do logic.

4
00:00:18,660 --> 00:00:22,899
His partner, Walter Pitts, was a teenage runaway and self-taught logician.

5
00:00:23,260 --> 00:00:30,460
Their neuron fires or doesn't, and networks of them can compute logic. A theory of the mind, not a plan for a machine.

6
00:00:30,760 --> 00:00:36,500
That logic came from George Boole, who in 1854 tried to write down the laws of thought.

7
00:00:36,880 --> 00:00:45,700
In 1986, cognitive scientists modeling how people learn popularized a way to train those networks. One was Geoffrey Hinton, Boole's great-great-grandson.

8
00:00:47,159 --> 00:00:50,780
Then it stalled. Training takes absurd amounts of arithmetic.

9
00:00:51,759 --> 00:00:53,780
The arithmetic came from video games.

10
00:00:53,780 --> 00:01:01,560
Nvidia was founded in 1993 to make graphics chips for gamers, chips that do simple math on thousands of pixels at once.

11
00:01:01,920 --> 00:01:06,920
The same math neural networks need. Researchers started disguising their equations as graphics.

12
00:01:07,420 --> 00:01:13,219
In 2012, Hinton's student Alex Krizhevsky trained AlexNet on two gaming cards in his bedroom.

13
00:01:14,040 --> 00:01:18,599
AlexNet learned from ImageNet, a million labeled photos gathered by Fei-Fei Li.

14
00:01:18,959 --> 00:01:28,640
By her math, labeling ImageNet by hand would take one person nineteen years. So she used Mechanical Turk, which Amazon built to weed out duplicate product pages.

15
00:01:29,060 --> 00:01:33,019
It's named after an 18th-century chess machine with a human hidden inside.

16
00:01:33,579 --> 00:01:38,280
ImageNet had 49,000 people inside, in 167 countries.

17
00:01:38,939 --> 00:01:43,299
AlexNet won, and Google bought Hinton's three-person company for 44 million dollars.

18
00:01:44,540 --> 00:01:46,459
Google could pay that because of ads.

19
00:01:46,959 --> 00:01:54,780
Noam Shazeer joined Google in 2000 and helped build a system that learned which words go together. Google used it to match ads to web pages.

20
00:01:55,280 --> 00:01:59,260
In 2017, Shazeer co-wrote the paper that introduced the transformer.

21
00:01:59,620 --> 00:02:08,539
His team wanted better translation. Older models read one word at a time. The transformer reads them all at once, exactly what graphics chips do best.

22
00:02:08,879 --> 00:02:10,400
It's the T in GPT.

23
00:02:11,460 --> 00:02:16,960
Google published it. OpenAI, co-founded by Hinton's student Ilya Sutskever, used it for GPT-2.

24
00:02:18,199 --> 00:02:24,300
That took good text, and sorting the web by hand would cost a fortune. So OpenAI borrowed Reddit.

25
00:02:24,740 --> 00:02:32,360
Reddit exists because Y Combinator rejected its founders' first idea, ordering food by phone, and told them to build a link-sharing site.

26
00:02:32,680 --> 00:02:40,379
Upvotes were built to rank those links. OpenAI kept every page linked from Reddit with at least three karma. Basically, three upvotes.

27
00:02:41,560 --> 00:02:47,259
Forty-five million links, chosen by strangers clicking a little arrow, never knowing an AI would read them.

28
00:02:47,860 --> 00:02:53,919
Take away any one of these, and I suspect we'd get a very different AI, or a much later one.

29
00:02:54,439 --> 00:02:56,560
Every piece was built for something else.

30
00:02:57,099 --> 00:03:04,699
And the internet of human writing it learned from, made by people talking to each other, not to machines, is probably something we can't make twice.

