Microsoft Copilot can now 'see' your screen—Why it's a game-changer
What's the story
Microsoft is testing a new feature for its Copilot app on Windows, Copilot Vision.
The upgrade enables users to share their screens or apps with the AI assistant.
Initially limited to Microsoft's Edge browser, the capability is now being expanded to any app on a user's PC.
The feature was demonstrated at Microsoft's 50th anniversary celebration last week, where it helped navigate Minecraft and Clipchamp video editor settings.
Functionality
Copilot Vision can help you use complex apps
With Copilot Vision, users can receive step-by-step assistance within complex software such as Adobe Photoshop.
The AI can highlight specific features on the screen, offering real-time guidance to streamline workflows.
Copilot Vision also extends its functionality to analyzing images and web pages. This enables users to gain insights or perform actions based on the visual content displayed.
Distinction
Copilot Vision: More than just a snapshot tool
While Copilot Vision may look similar to Microsoft's Recall feature, which automatically takes screenshots with user permission, it is fundamentally different.
The new feature works more like sharing an app or entire desktop during a Microsoft Teams call.
This difference highlights the unique capabilities of Copilot Vision and its potential to revolutionize the way you interact with your PCs.
Additional feature
Microsoft's Copilot now supports file search
Along with Copilot Vision, Microsoft is also testing a file search feature in Copilot on Windows. This lets users query the AI assistant about the contents of files on their PC.
The file search function supports a range of formats including .docx, .xlsx, .pptx, .txt, .pdf and .json files. It lets users easily find documents they have recently worked on.
Accessibility
Copilot features available on iOS and Android
Both Copilot Vision and file search features are available via the Copilot app on Windows, without the need for a full Copilot Plus PC. The new capabilities are also available on iOS and Android.
Microsoft has started testing the features with Windows Insiders, ahead of a broader rollout to all Windows 11 users in the near future.