Artificial Intelligence models that generate entirely new content are creating a world of opportunities for entrepreneurs. And engineers are learning to do more with less.
Those were some takeaways from a panel discussion at the Intelligent Applications Summit hosted by Madrona Venture Group in Seattle this week.
“Big data is not a priority anymore, in my opinion,” said Stanford computer science professor Carlos Guestrin. “You can solve complex problems with little data.”
Engineers are more focused on fine tuning off-the-shelf models, said Guestrin, co-founder of Seattle machine learning startup Turi, which was acquired by Apple in 2016. Off-the-shelf models like DALL-E and GPT-3 can hallucinate images or text from initial prompts.
Such new “foundation” AI models were built off of massive datasets that can now be adapted. They are the basis for emerging startups that generate written content, interpret conversations, or assess visual data. They will enable a host of use cases, said Oren Etzioni, technical director of the Allen Institute for Artificial Intelligence (AI2). But they also need to be tamed so that they are less biased and more reliable.
“A huge challenge of these models is that they hallucinate. They lie, they generate — they invent things,” said Etzioni, also a venture partner at Madrona.
Guestrin and Etzioni spoke at a fireside chat moderated by UW computer science professor Luis Ceze, who is also a Madrona venture partner and CEO of Seattle AI startup OctoML.
OctoML was chosen for a new top 40 list of intelligent application startups assembled by Madrona in collaboration with other firms. Startups on the list have raised more than $16 billion since their inception, including $5 billion since the start of this year.
Read on for more highlights from the discussion.
New AI models are changing how engineers work
Engineers are used to building distinct AI models with unique tech stacks for individual tasks, such as a predicting airfares or medical outcomes — and they are accustomed to front-filling the models with massive training datasets. But now, using less data as input, engineers are elaborating on foundation models to build specific tools, said Guestrin.
“We are totally changing, with large language models and foundation models, how we think about developing applications, going beyond this idea of big data,” said Guestrin. He added that engineers are using “task-specific, habituated small datasets for fine-tuning prompting that leads to a vertical solution that you really care about.”
Added Etzioni: “Now, with foundation models, I build a single model, and then I may fine tune it. But a lot of the work is done ahead of time and done once.”
AI has become “democratized“
AI tools are becoming more accessible to engineers with less specialized skill sets and the cost of building new tools is beginning to come down. The general public also has more access through tools like DALL-E, said Guestrin.
“I’m in awe of how large language models, foundation models, have enabled others beyond developers to do amazing things with AI,” said Guestrin. “Large language models give us the opportunity to create new experiences for programming, for bringing AI applications to a wide range of people who never thought they could program an AI.”
Bias is still an issue
Bias has always dogged AI models. And it remains an issue in newer generative AI models.
As an example, Guestrin pointed to a story-making tool that created a different fairy tale outcome depending on the race of the prince. If the tool was asked to create a fairy tale about a white prince, it described him as handsome and the princess fell in love with him. If it was asked to create a story with a Black prince, the princess was shocked.
“I worry about this a lot,” said Guestrin about bias in AI models and their ability to in turn affect societal biases.
Etzioni said newer technology under development will be better at stripping out bias.
Guestrin said engineers need to consider the problem at all steps of development. Engineers’ most important focus should be how they evaluate their models and curate their datasets, he said.
“Thinking that addressing the gap between our AI and our values is just some salt we can sprinkle on top at the end, like some post-processing, is a bit of a limited perspective,” added Guestrin.
Human input will be central to improving models
Etzioni made an analogy to internet search engines, which in their early days often required users to search in different ways to get the answer they wanted. Google excelled at honing output after learning what people clicked on from billions of queries.
“As people query these engines and re-query them and produce things, the engines are going to get better at doing what we want,” said Etzioni. “My belief is very much that we are going to have humans in the loop. But this is not an obstacle to the technology.”
AI also can’t predict its own best use-cases. “If you ask GPT-3, ‘what’s the best use for building new startups,’ you’re going to get garbage,” said Etzioni.
Improving reliability is a focus
“These models, despite being amazing, are brittle. They can fail in catastrophic ways,” said Ceze.
Researchers should learn how to better define their goals and ask how to test and evaluate systems systematically to make them more fail-safe, said Guestrin. He added that researchers should be “bringing more of that software engineering mindset.”
Learning how to make AI models more reliable is a major focus of research at Guestrin’s group at Stanford and at the AI2.
“It’s going to be an extremely long time before you have a GPT-3-based app running a nuclear power plant. It’s just not that kind of technology,” said Etzioni. “That’s why I think that the analogy to web search engines is so profound. If we have human-in-the-loop and if we have rapid iteration, we can use highly unreliable technology in a very empowering way.”