Windows 11 is getting its third feature update, and this new version isn't like any other, since it introduces a slew of new AI innovations we have never seen before.
As part of version 24H2 (2024 Update), the operating system introduces the new Windows Copilot Runtime, which makes it possible to run many of the AI features, such as Windows Recall, Live Captions, Auto Super Resolution, Voice Clarity, Studio Effects, and other AI features for some of the inbox apps.
In this guide, I will dive into the AI features expected to become available as part of the Windows 11 2024 Update.
What AI changes are coming to Windows 11 2024 Update
In this feature update, Microsoft is adding nearly a dozen features powered by AI, most of which work locally without an internet connection.
Windows Copilot Runtime
Let's start with the Windows Copilot Runtime, which is basically the heart of AI on Windows 11. It's a "layer" in the operating system that integrates more than 40 different AI models capable of running locally on the device.
Some of these AI models include Phi Silica, a Small Language Model (SLM) from Microsoft that offers many of the benefits found in large language models (LLMs). In addition, you will find models like Screen Region Detector, Optical Character Recognizer, Natural Language Parser, and Image Encoder.
As part of this integration, the company is also making available the Windows Copilot Library, which is a collection of APIs (Application Programming Interfaces) that allows developers to integrate AI capabilities into their applications easily. This library is always ready, passively running on the computer to enable the new AI features.
Although there are some AI features, such as Copilot, that are cloud-dependent, the Windows Copilot Runtime enables many of the features to run locally without sending any data to the cloud.
Windows Recall
Recall is perhaps the most anticipated and controversial feature coming to Windows 11 (version 24H2). The feature constantly takes screenshots of anything you're doing and uses multiple AI models running on-device with the help of an NPU (Neural Processing Unit) to record (literally). The feature then analyzes the content and context and stores everything on the device inside the new Windows Semantic Index.
Whenever you need to find something, you can launch the "Recall" app to search with natural language or use the timeline slider to scroll back in time. As you select the snapshot, the screenray will make it easier to interact with the content and copy text or images.
You can also use the available controls to open the application or website, share, delete, or open the snapshot with the Snipping Tool.
The feature requires storage to store the snapshots and the index database. The default storage allocation is 25GB for a computer with a 256GB hard drive, and the allocation will increase depending on the size of the storage drive.
In the Settings app, the system adds a new "Recall & snapshots" page to control the feature state, storage, and filters to exclude websites and apps for participating in the Recall experience.
It's important to note that Windows Recall won't track websites when using InPrivate or Incognito on Chromium-based browsers, such as Microsoft Edge and Google Chrome, and you can use the filters settings to exclude certain websites you don't want the feature to track.
Although the feature can be useful in many cases, it's still controversial because it fails to gain users' trust. There are many questionable security and privacy concerns, some of which Microsoft has acted quickly to address.
Sure, according to the company, its approach to Recall follows the most rigorous security guidelines, such as using the Microsoft Pluton security processor, Windows Hello authentication, and device encryption using BitLocker, but there are a lot of things that don't feel right.
For example, Windows Recall doesn't moderate the information it finds during the snapshot process, meaning that any sensitive information will become instantly searchable.
When the company first unveiled the feature, the Recall data was stored on-device unencrypted for anyone to grab, but new changes to the feature will encrypt this data and require authentication to use the app to search any using Windows Recall.
You can learn more about Windows Recall in this article that I wrote detailing how the feature works.
Windows Studio Effects
Windows Studio Effects is a collection of features that uses the NPU to apply special effects on the webcam or microphone. This feature isn't entirely new, but now, starting with version 24H2, Microsoft will start making it broadly available for compatible devices.
As part of the effects, depending on the support available on the device, you can access Background Blur, Eye Contact, Auto Framing, Voice Focus, Portrait light, Creative filters, and Voice Focus.
You can access the features from the Quick Settings flyout using the "Studio effects" page and turning on or off the feature you want to use.
You can also control these settings from Settings > Bluetooth & devices > Camera. Select the camera from the list, and the available features will appear in the "Camera Effects" settings.
Background effects include standard and portrait blur. Eye Contact connects your eyes to make it seem you're looking at the camera, and you can choose the standard or teleprompter mode.
Automatic framing zooms and crops the field of view to keep the person centered on the screen. Portrait light improves the lighting condition to improve viewability in low-light environments. Creative filters add fun animations and flares, and Voice Focus filters background noise during video calls.
The effects available on your device will depend on the peripherals and capability of the NPU.
Live Captions
Live Captions is another feature that has been around for some time on Windows 11. With the release of the 2024 Update, the feature is now powered by AI and includes live translations.
Live translation is an accessibility feature that aims to help anyone understand audio through automatic transcription and translation from any audio source on Windows 11. For example, live translation will work to translate a YouTube video or voice call.
Voice Clarity
Microsoft is also adding Voice Clarity, a new feature that uses AI to analyze and remove real-time background noise during conversations or recordings.
The feature works automatically, so there is no setting to manage it. Originally, this feature was reserved for Surface devices, but it'll be available as part of the 2024 Update without an NPU.
Since this feature will be available on devices without an NPU, it indicates that not every AI feature will require hardware for existing computers.
Auto Super Resolution
This is another highly anticipated feature. Auto Super Resolution (Auto SR) is a feature that will use on-device AI to upscale games to improve frame rates and image quality.
The feature works similarly to NVIDIA DLSS Super Resolution, AMD FidelityFX Super Resolution, and Intel XeSS, but Auto SR upscale games automatically without the need for developers to update their games.
When the feature is turned on, Auto SR will process the upscale process to the NPU (Neural Processing Unit) rather than the GPU (Graphic Processing Unit). As a result, you will not only experience a better gaming experience but also energy efficiency.
The setting is available through the "Graphics" page, and you can turn it on globally on the computer or per application.
If you launch a game, and it's compatible with Auto SR, the system will show a toast notification letting you know that the support is available, with an option to turn it on from the Settings app.
Microsoft has also created a website with Qualcomm to list the games compatible with the ARM version of Windows 11. The page also shows whether the game is also compatible with Auto Super Resolution. However, at launch, less than a dozen games will be supported, including Borderlands 3, Control (DX11), Dark Souls III, God of War, Kingdom Come: Deliverance, Resident Evil 2, Resident Evil 3, Sekiro Shadows Die Twice, Sniper Ghost Warrior Contracts 2, and The Witcher 3.
It's important to note that Auto Super Resolution (Auto SR) is not the same as Super Resolution (SR), which usually refers to the upscaling technologies from Intel, AMD, and Nvidia.
In relation to the Auto SR feature, Microsoft, in collaboration with Intel, AMD, and Nvidia, also created the new DirectSR, a new technology that allows game developers to integrate existing upscaling AI technologies into a single set of APIs instead of manually integrating each technology.
Restyle Image for Photos
The Photos app now includes a new "Restyle Image" feature that uses on-device AI to help you change the style of any photo using different photographic techniques, similar to filters you are used to using on phones.
You can even use the text prompt to describe and instruct Photos on how you want to change the background and other parts of the image.
In addition, Microsoft is also bringing the "Image Creator" from the Paint app to Photos to convert a text prompt into an image with AI. This particular feature uses the Copilot AI technology to use the cloud to create images using the OpenAI DALL-E language model.
Cocreator for Paint
In addition to the "Image Creator" feature, the Paint app is adding "Cocreator," a new feature that scans your drawing and helps you create your artwork.
You can use this feature with a text prompt and use the slider option to adjust the creativity intensity.
This particular feature uses a diffusion-based algorithm to draw a high-quality image with minimum effort on your part.
At least initially, Cocreator will be a feature for Copilot+ PCs.
Copilot app
In the Windows 11 2024 Update, the software giant is changing the Copilot AI integration with a new app instead of the side panel that slides in from the right side.
The app is actually the Copilot.Microsoft.com version of the chatbot AI wrapped around an app frame. The experience introduces a new design similar to the ChatGPT web interface with a left-side panel for your prompt history and other options, such as starting a new conversation, plugins, and notebook.
You can still change the conversation style with the option at the top of the page.
As a result of these changes, you can no longer use Copilot to change settings, the "Copilot" button is no longer part of the System Tray, and Microsoft is also removing the "Copilot" setting from the Settings app.
The company also plans to update the Copilot intelligence with OpenAI GPT-4o, which can process text, images, audio, and video as input and output.
Since Copilot relies on the cloud to analyze and process responses, it's a feature available to everyone, and it's not exclusive to Copilot+ PC.
Copilot suggestions
Although the Copilot app can no longer be used to change system settings, Microsoft is also introducing "Copilot suggestions."
Copilot suggestions will appear on applications when the chatbot AI knows it can be helpful. For example, in the Settings app, you may find suggestions at the top of the page to fix problems, enable features, and more.
On File Explorer, hovering over an image can bring up a Copilot prompt interface. You can use this prompt to ask questions about the image and take action, such as removing the background, creating a different image, or resizing it.
If you receive an email, you may see a toast notification with a suggestion to summarize the document attachment and more.
Important: The Windows 11 2024 Update has become partially available on June 18, 2024, for Copilot+ PCs, but it won't include all the intended features, including Windows Recall, as it has been postponed until further notice. In September or October, the feature update will become more broadly available for existing devices. As a result, some of the new features and changes may not be available to everyone on day one. Also, some features (especially AI features) may require new hardware to be available.
More resources
For more helpful articles, coverage, and answers to common questions about Windows 10 and Windows 11, visit the following resources:
