Windows 11 is getting its third feature update, and this new version isn't like any other since it introduces a slew of new AI innovations we have never seen before.
As part of version 24H2 (2024 Update), the operating system introduces the new Windows Copilot Runtime, which makes it possible to run many of the AI features, such as Windows Recall, Live Captions, Auto Super Resolution, Voice Clarity, Studio Effects, and other AI features for some of the inbox apps.
In this guide, I will dive into the AI features expected to become available as part of the Windows 11 2024 Update. (At the time of this writing, it's still unclear when these features will be available, but the company says that some of them are expected to roll out sometime in November 2024.)
What AI changes are coming to Windows 11 2024 Update
In this feature update, Microsoft is adding nearly a dozen features powered by AI, most of which work locally without an internet connection.
Windows Copilot Runtime
Let's start with the Windows Copilot Runtime, which is basically the heart of AI on Windows 11. It's a "layer" in the operating system that integrates more than 40 different AI models capable of running locally on the device.
Some of these AI models include Phi Silica, a Small Language Model (SLM) from Microsoft that offers many of the benefits found in large language models (LLMs). In addition, you will find models like Screen Region Detector, Optical Character Recognizer, Natural Language Parser, and Image Encoder.
As part of this integration, the company is also making available the Windows Copilot Library, which is a collection of APIs (Application Programming Interfaces) that allows developers to integrate AI capabilities into their applications easily. This library is always ready, passively running on the computer, to enable the new AI features.
Although there are some AI features, such as Copilot, that are cloud-dependent, the Windows Copilot Runtime enables many of the features to run locally without sending any data to the cloud.
Windows Recall
Recall is perhaps the most anticipated and controversial feature coming to Windows 11 (version 24H2). The feature constantly takes screenshots of anything you're doing and uses multiple AI models running on-device with the help of an NPU (Neural Processing Unit) to record (literally). The feature then analyzes the content and context and stores everything on the device inside the new Windows Semantic Index.
Whenever you need to find something, you can launch the "Recall" app to search with natural language or use the timeline slider to scroll back in time. As you select the snapshot, the screenray will make it easier to interact with the content and copy text or images.
You can also use the available controls to open the application or website, share, delete, or open the snapshot with the Snipping Tool.
The feature requires storage to store the snapshots and the index database. The default storage allocation is 25GB for a computer with a 256GB hard drive, and the allocation will increase depending on the size of the storage drive.
In the Settings app, the system adds a new "Recall & snapshots" page to control the feature state, storage, and filters to exclude websites and apps for participating in the Recall experience.
It's important to note that Windows Recall won't track websites when using InPrivate or Incognito on Chromium-based browsers, such as Microsoft Edge and Google Chrome, and you can use the filters settings to exclude certain websites you don't want the feature to track.
Although the feature can be useful in many cases, it's still controversial because it fails to gain users' trust. There are many questionable security and privacy concerns, some of which Microsoft has acted to address.
When the company first unveiled the feature, the Recall data was stored on-device unencrypted for anyone to grab, but new changes to the feature will encrypt this data and require authentication to use the app to search any using Windows Recall.
This feature will be available exclusively for Copilot+ PCs later this year.
Click to Do
On Windows 11, "Click to Do" is a new AI feature that can understand the contents on the screen and offer relevant action at any time, similar to "Circle to Search" on Android devices.
To activate "Click To Do," hold the "Windows key" and click anywhere on the screen to bring up the overlay that provides contextual suggestions based on what it sees. For example, if you've selected some text, Click To Do might offer to summarize or rewrite it. If you've chosen an image, it could suggest blurring the background or erasing an object.
Some key features include contextual actions based on what's on the screen. Search for anything on the screen with a dedicated search bar, and it uses the Windows Recall integration to make sense of the content on the screen.
This feature will be available exclusively for Copilot+ PCs later this year.
Windows Search with AI
Microsoft also plans to update the Windows Search feature, allowing you to use natural language to search on the Windows Search experience from the Taskbar, File Explorer, and the Settings app.
In other words, you will no longer have to rely on keywords from the file name, metadata, or content. You will be able to describe the file, photo, or settings, and the feature should be able to find it.
This feature will be available exclusively for Copilot+ PCs later this year.
Windows Studio Effects
Windows Studio Effects is a collection of features that uses the NPU to apply special effects on the webcam or microphone. This feature isn't entirely new, but now, starting with version 24H2, Microsoft will start making it broadly available for compatible devices.
As part of the effects, depending on the support available on the device, you can access Background Blur, Eye Contact, Auto Framing, Portrait light, Creative filters, and Voice Focus.
You can access the features from the Quick Settings flyout using the "Studio effects" page and turning on or off the feature you want to use.
You can also control these settings from Settings > Bluetooth & devices > Camera. Select the camera from the list, and the available features will appear in the "Camera Effects" settings.
Background effects include standard and portrait blur. Eye Contact connects your eyes to make it seem you're looking at the camera, and you can choose the standard or teleprompter mode.
Automatic framing zooms and crops the field of view to keep the person centered on the screen. Portrait light improves the lighting condition to improve viewability in low-light environments. Creative filters add fun animations and flares, and Voice Focus filters background noise during video calls.
The effects available on your device will depend on the peripherals and capability of the NPU.
Live Captions
Live Captions is another feature that has been around for some time on Windows 11. With the release of the 2024 Update, the feature is now powered by AI and includes live translations.
Live translation is an accessibility feature that aims to help anyone understand audio through automatic transcription and translation from any audio source on Windows 11. For example, live translation will work to translate a YouTube video or voice call.
This feature will be available exclusively for Copilot+ PCs.
Voice Clarity
Microsoft is also adding Voice Clarity, a new feature that uses AI to analyze and remove real-time background noise during conversations or recordings.
The feature works automatically, so there is no setting to manage it. Originally, this feature was reserved for Surface devices, but it'll be available as part of the 2024 Update without an NPU.
Since this feature will be available on devices without an NPU, it indicates that not every AI feature will require hardware for existing computers.
Auto Super Resolution
This is another highly anticipated feature. Auto Super Resolution (Auto SR) is a feature that will use on-device AI to upscale games to improve frame rates and image quality.
The feature works similarly to NVIDIA DLSS Super Resolution, AMD FidelityFX Super Resolution, and Intel XeSS, but Auto SR upscale games automatically without the need for developers to update their games.
When the feature is turned on, Auto SR will process the upscale process to the NPU (Neural Processing Unit) rather than the GPU (Graphic Processing Unit). As a result, you will not only experience a better gaming experience but also energy efficiency.
The setting is available through the "Graphics" page, and you can turn it on globally on the computer or per application.
If you launch a game, and it's compatible with Auto SR, the system will show a toast notification letting you know that the support is available, with an option to turn it on from the Settings app.
Microsoft has also created a website with Qualcomm to list the games compatible with the ARM version of Windows 11. The page also shows whether the game is also compatible with Auto Super Resolution. However, at launch, less than a dozen games will be supported, including Borderlands 3, Control (DX11), Dark Souls III, God of War, Kingdom Come: Deliverance, Resident Evil 2, Resident Evil 3, Sekiro Shadows Die Twice, Sniper Ghost Warrior Contracts 2, and The Witcher 3.
It's important to note that Auto Super Resolution (Auto SR) is not the same as Super Resolution (SR), which usually refers to the upscaling technologies from Intel, AMD, and Nvidia.
In relation to the Auto SR feature, Microsoft, in collaboration with Intel, AMD, and Nvidia, also created the new DirectSR, a new technology that allows game developers to integrate existing upscaling AI technologies into a single set of APIs instead of manually integrating each technology.
This feature will be available exclusively for Copilot+ PCs.
Restyle Image for Photos
The Photos app now includes a new "Restyle Image" feature that uses on-device AI to help you change the style of any photo using different photographic techniques, similar to filters you are used to using on phones.
You can even use the text prompt to describe and instruct Photos on how you want to change the background and other parts of the image.
In addition, Microsoft is also bringing the "Image Creator" from the Paint app to Photos to convert a text prompt into an image with AI. This particular feature uses the Copilot AI technology to use the cloud to create images using the OpenAI DALL-E language model.
Photos with Super Resolution
As part of the AI features coming to the Photos app, the software giant announced that you will soon be able to upscale images using AI on your computer.
According to the company, the feature will use the NPU on your Copilot+ PC to upscale and sharpen images up to 8x the original size. In addition, with Super Resolution, you will be able to restore low-quality pictures.
Microsoft explains that the feature is capable of modifying the image with new details without ruining the contents of the image.
These features will be available exclusively for Copilot+ PCs later this year.
Cocreator for Paint
In addition to the "Image Creator" feature, the Paint app is adding "Cocreator," a new feature that scans your drawing and helps you create your artwork.
You can use this feature with a text prompt and use the slider option to adjust the creativity intensity.
This particular feature uses a diffusion-based algorithm to draw a high-quality image with minimum effort on your part.
This feature will be available exclusively for Copilot+ PCs.
Paint with Generative Fill and Erase
If you use the Paint app, you will be getting two new features, including "Fill" and "Erase," that leverage the power of AI to make editing and manipulating images easier than ever before.
The "Fill" feature allows you to select a part of your image and enter a text prompt, and the AI will generate new content to fill the designated area.
Initially, this feature will be exclusive to Copilot+ PCs with Qualcomm processors later this year.
The "Erase" feature allows you to select and erase an area of the image, similar to Magic Eraser for Android.
This feature is expected to be available to everyone, regardless of whether your computer is a regular or Copilot+ PC.
Copilot app
In the Windows 11 2024 Update (as well as on version 23H2), the software giant is changing the Copilot AI integration with a new app instead of the side panel that slides in from the right side.
The app is actually the Copilot.Microsoft.com version of the chatbot AI wrapped around an app frame. The experience introduces a new design similar to the ChatGPT web interface with a left-side panel for your prompt history and other options, such as starting a new conversation, plugins, and notebook.
You can still change the conversation style with the option at the top of the page.
As a result of these changes, you can no longer use Copilot to change settings, the "Copilot" button is no longer part of the System Tray, and Microsoft is also removing the "Copilot" setting from the Settings app.
The company also plans to update the Copilot intelligence with OpenAI GPT-4o, which can process text, images, audio, and video as input and output.
Since Copilot relies on the cloud to analyze and process responses, it's a feature available to everyone, and it's not exclusive to Copilot+ PC.
Copilot suggestions
Although the Copilot app can no longer be used to change system settings, Microsoft is also introducing "Copilot suggestions."
Copilot suggestions will appear on applications when the chatbot AI knows it can be helpful. For example, in the Settings app, you may find suggestions at the top of the page to fix problems, enable features, and more.
On File Explorer, hovering over an image can bring up a Copilot prompt interface. You can use this prompt to ask questions about the image and take action, such as removing the background, creating a different image, or resizing it.
If you receive an email, you may see a toast notification with a suggestion to summarize the document attachment and more.
Important: As a result of Microsoft gradually rolling out new features and improvements for Windows 11 versions 23H2 and 24H2, not all the changes will be available on day one with the release of the 2024 Update. Also, some improvements meant for version 24H2 will eventually be ported to version 23H2. The company is even working on changes that will first be available on version 23H2, and then they will be ported to version 24H2. Also, it is worth noting that the feature update became partially available on June 18, 2024, for Copilot+ PCs, but with only some of the intended features, excluding Windows Recall, as it has been postponed until further notice. In October, the feature update will become more broadly available for existing devices. Finally, AI features may require new hardware (such as NPU) to be available.
More resources
For more helpful articles, coverage, and answers to common questions about Windows 10 and Windows 11, visit the following resources:
