This decade of data is what drove the university’s new experiment in artificial intelligence.
Dr. Finn and her team built a neural network, a mathematical system that can learn skills from vast amounts of data. By pinpointing patterns in thousands of cat photos, a neural network can learn to identify a cat. By analyzing hundreds of old phone calls, it can learn to recognize spoken words. Or, by examining the way teaching assistants evaluate coding tests, it can learn to evaluate these tests on its own.
The Stanford system spent hours analyzing examples from old midterms, learning from a decade of possibilities. Then it was ready to learn more. When given just a handful of extra examples from the new exam offered this spring, it could quickly grasp the task at hand.
“It sees many kinds of problems,” said Mike Wu, another researcher who worked on the project. “Then it can adapt to problems it has never seen before.”
This spring, the system provided 16,000 pieces of feedback, and students agreed with the feedback 97.9 percent of the time, according to a study by the Stanford researchers. By comparison, students agreed with the feedback from human instructors 96.7 percent of the time.
Mr. Pham, an engineering student at Lund University in Sweden, was surprised the technology worked so well. Although the automated tool was unable to evaluate one of his programs (presumably because he had written a snippet of code unlike anything the A.I. had ever seen), it both identified specific bugs in his code, including what is known in computer programming and mathematics as a fence post error, and suggested ways of fixing them. “It is seldom you receive such well thought out feedback,” Mr. Pham said.
The technology was effective because its role was so sharply defined. In taking the test, Mr. Pham wrote code with very specific aims, and there were only so many ways that he and other students could go wrong.
But given the right data, neural networks can learn a range of tasks. This is the same fundamental technology that identifies faces in the photos you post to Facebook, recognizes the commands you bark into your iPhone and translates from one language to another on services like Skype and Google Translate. For the Stanford team and other researchers, the hope is that these techniques can automate education in many other ways.
