ChatGPT gets “eyes and ears” with plugins that can interface AI with the world

On Thursday, OpenAI announced a plugin system for its ChatGPT AI assistant. The plugins give ChatGPT the ability to interact with the wider world through the Internet, including booking flights, ordering groceries, browsing the web, and more. Plugins are bits of code that tell ChatGPT how to use an external resource on the Internet.

Basically, if a developer wants to give ChatGPT the ability to access any network service (for example: “looking up current stock prices”) or perform any task controlled by a network service (for example: “ordering pizza through the Internet”), it is now possible, provided it doesn’t go against OpenAI’s rules.

Conventionally, most large language models (LLM) like ChatGPT have been constrained in a bubble, so to speak, only able to interact with the world through text conversations with a user. As OpenAI writes in its introductory blog post on ChatGPT plugins, “The only thing language models can do out-of-the-box is emit text.”

Bing Chat has taken this paradigm further by allowing it to search the web for more recent information, but so far ChatGPT has still been isolated from the wider world. While closed off in this way, ChatGPT can only draw on data from its training set (limited to 2021 and earlier) and any information provided by a user during the conversation. Also, ChatGPT can be prone to making factual errors and mistakes (what AI researchers call “hallucinations”).

To get around these limitations, OpenAI has popped the bubble and created a ChatGPT plugin interface (what OpenAI calls ChatGPT’s “eyes and ears”) that allows developers to create new components that “plug in” to ChatGPT and allow the AI model to interact with other services on the Internet. These services can perform calculations and reference factual information to reduce hallucinations, and they can also potentially interact with any other software service on the Internet—if developers create a plugin for that task.

What kind of plugins are we talking about?

In the case of ChatGPT, OpenAI will allow users to select from a list of plugins before starting a ChatGPT session. They present themselves almost like apps in an app store, each plugin having its own icon and description.

OpenAI says that a first round of plugins have been created by the following companies:

• Expedia (for trip planning)
• FiscalNote (for real-time market data)
• Instacart (for grocery ordering)
• Kayak (searching for flights and rental cars)
• Klarna (for price-comparison shopping)
• Milo (an AI-powered parent assistant)
• OpenTable (for restaurant recommendations and reservations)
• Shopify (for shopping on that site)
• Slack (for communications)
• Speak (for AI-powered language tutoring)
• Wolfram (for computation and real-time data)
• Zapier (an automation platform)

In particular, the Zapier plugin seems especially powerful since it grants ChatGPT access to an existing software automation system, or as Zapier puts it: “You can ask ChatGPT to execute any of Zapier’s 50,000 actions (including search, update, and write) with Zapier’s 5,000+ supported apps, turning chat into action. It can write an email, then send it for you. Or find contacts in a CRM, then update them directly. Or add rows to a spreadsheet, then send them as a Slack message. The possibilities are endless.”

OpenAI is also hosting three plugins itself, a web browser (that can grab info from the web in a manner similar to Bing Chat), a code interpreter for executing Python programs (in a sandbox), and a retrieval tool that allows access to “personal or organizational” information sources hosted elsewhere (basically, fetching information from documents).