Microsoft’s Copilot can now see what you see in real time – and speak to you about it

Microsoft Copliot has proven a worthy competitor to ChatGPT. As ChatGPT continues to evolve, so has Copilot, but recently, it has been trailing behind. However, this latest update might be enough to return its competitive edge. 

On Thursday, Microsoft made Copilot Vision, an experience in which Copilot can view and understand the context of what you’re doing online to provide verbal real-time assistance, available in preview for Pro subscribers through Copilot Labs.  

Also: Google Photos unveils its ‘Spotify Wrapped’ style end-of-year recap. How to access it

When the user enables Copilot Vision, it can read along with you, discuss issues you are having while browsing, analyze your site, and provide insights based on what it sees. It’s essentially an assistant for all your browsing needs — on-call whenever you need it. 

I had the opportunity to demo the feature at a NYC Microsoft Copilot and Windows Event in October where I witnessed real-world applications that showcased some of Copilot Visions’ assistance value. 

Also: OpenAI’s 12 days of product drops and demos advances – what’s new on day 2

In my demo, the user asked Copilot Vision for assistance with picking outfit inspiration from Pinterest. Copilot Vision suggested an outfit from all the options on the page and offered encouragement when the user said he didn’t think he had what it took to pull the outfit off. This interaction took place verbally, making it a natural experience. 

Other real-life use cases Microsoft provided for Copilot Vision included helping users with holiday shopping by pointing out products on the page that match what they are looking for or helping them plan a day at the museum by highlighting important information to know before leaving for the trip. 

The new experience lives on Microsoft Edge, tucked at the bottom of the browser. Copilot Vision is rolling out first to a limited number of Copilot Pro subscribers in the United States through Copilot Labs and will initially only work on a select number of websites. As it collects feedback, Microsoft says it will expand access.   

Also: The best AI chatbots of 2024: ChatGPT, Copilot, and worthy alternatives

For users concerned about privacy, Microsoft says it will be entirely opt-in, giving the user control over whether they want Copilot to see their browsing activity. Even for those who do choose to opt-in, all the data shared with Copilot during that session, including what you say and the context you share, is deleted, according to the blog post. 

The Copilot Pro subscription is $20 per month. It includes other perks, such as priority access to the latest models, more access to Copilot Voice, and Copilot in select Microsoft 365 apps, including Word, Excel, PowerPoint, OneNote, and Outlook.