AI Assistants in the Future: Security Concerns and Risk Management

To better understand how the digital assistant matrix works, we will show how it maps three different digital assistants categories: one from the previous generation of hardware assistants (Amazon Alexa, Apple Siri), one from the present offering (Google Astra, ChatGPT 4o) and even one hypothetical future offering straight from a cinematic universe (Stark Industries’ E.D.I.T.H.) that, while fictional, presents an interesting portrait of future developments and shows the flexibility of our approach. The section “Digital Assistant Capabilities Scoring Table” shown on the sidebar and in the appendix serves as a reference for the values depicted in the tables.

First-generation digital assistants: Amazon Alexa, Siri, Cortana, Google Assistant

First released in 2014 alongside the first Amazon Echo device, Alexa represents the first generation of DAs to enter the consumer market, with capabilities that are very much comparable to its competitors such as Google Assistant, Microsoft’s Cortana (which is currently being phased out), and Apple’s Siri. However, while these assistants paved the way for the technology to gain wider user acceptance, they show the shortcomings of their older age.

Figure 3. Mapping the capabilities of the first-generation digital assistants

In terms of capabilities, these DAs represent the baseline of our matrix: for example, we can see that Alexa, while able to provide a conversational interface, has a very basic level of agency, being limited to running some home automation upon user request. The same can be said for Cortana or Siri, both of which are able to perform basic interactions with a local computer or mobile phone.

They also possess basically zero reasoning ability, acting like a flat conversational agent for the user. This requires a very limited psychological acceptance from the user, who often can barely withstand listening to what the assistant says. The notable capability that these first-generation DAs have that chart higher than the baseline level is ubiquity, since Siri or Google Assistant can run on a phone and be with the user the whole time.

Current-generation digital assistants: ChatGPT, Google Astra

In recent years, DAs have undergone evolutions that brought us a new generation of assistants that are more capable from a conversational standpoint.

Examples include ChatGPT, with its latest conversation engine GPT-4o, as well as more integrated systems like Microsoft Copilot and the upcoming Google Astra.

While not yet fully integrated into the Google ecosystem at the time of writing, we can take Google Astra’s release video at face value to examine its capabilities as a next-generation DA.

Figure 4. Mapping the capabilities of the current-generation digital assistants

Compared to the first generation of DAs, we have an increase in conversational capabilities thanks to the use of LLMs behind the scenes, with the assistant now being able to deal with multimodal input (i.e., “explain what I am seeing right now”) and output (i.e., “draw me a picture of something”). The assistant is also able to retain a short memory of the ongoing conversation, allowing the user to have a more complex interaction.

We see, however, that we are still lagging in terms of reasoning and agency: while this new generation of assistants can, for example, fetch information from multiple sources, it cannot yet interact with different providers to create and carry out a complex plan (such as a full travel itinerary), but we believe we are not far from that reality.

Hypothetical future DA: E.D.I.T.H.

While we wait for the industry to catch up on some capabilities, we tried to apply our matrix to what Hollywood thinks could be a possible future for DAs.

In the Marvel Cinematic Universe, Tony Stark’s creation, E.D.I.T.H (Even Dead I’m The Hero), is an assistant developed by the genius billionaire to assist his protégé, Peter Parker, who interacts with it via a pair of augmented reality (AR) sunglasses.

Basing our consideration to what we can infer from the movies, we can assign a fairly high score to the E.D.I.T.H. ecosystem integration, since the assistant is not only able to interact with multiple services at once, but even hack into those who are not available. Furthermore, the assistant sports a particularly high level of agency, being able to coordinate and control a whole fleet of armed drones, which comes from a staggering level of psychological acceptance, where our protagonist is willing to delegate to the assistant even actions with potential criminal consequences (however, we won’t spoil any more than this).

The figure below compares all the first and current generation digital DAs with what would be the hypothetical capabilities offered by a fictional assistant such as E.D.I.T.H.

Figure 5. Comparing the capabilities of current DAs to a future hypothetical E.D.I.T.H.

Digital assistants will become an integral part of our daily lives. They will offer convenience and efficiency in managing tasks and accessing information. However, to function effectively, these assistants often collect, process, and store a wide range of personally identifiable information (PII).

To understand the privacy implications and enhance data protection measures, it is crucial to recognize the various types of PII typically collected by digital assistants. These categories could include:

1. Names: DAs require the names of users to personalize interactions and provide tailored experiences.
2. Dates of birth/birthplace: This information is used for identity verification, customization, and security purposes.
3. Current and previous addresses: Address data is used to offer location-based services such as weather updates, navigation, and local recommendations.
4. Email addresses:Email addresses are essential for account creation, communication, and authentication purposes.
5. Phone numbers: Phone numbers serve as unique identifiers and are used for communication, verification, and recovery processes.
6. Account credentials: DAs may store credentials for emails, social media accounts, messaging apps, and banking services to facilitate seamless integration and enhanced functionality.
7. Communication/Chat history: This data is used to improve interaction quality, learn user preferences, and provide continuity in conversations.
8. Social maps: A social map outlines a user's relationships and interactions, providing insights into social connections and communication patterns.
9. Social Security Numbers/Tax IDs: Some services may request this information for identification and verification purposes, e.g. an accounting DA.
10. Name and address of doctors: Health-related services may require this information to manage appointments, prescriptions, and health records.
11. Health information/Health records: DAs handle health information to provide reminders, track fitness goals, and offer medical advice.
12. Insurance information: Insurance details are stored to facilitate claims processing and policy management.
13. Travel-related information and accounts: This data is used to streamline booking processes and enhance travel experiences.

The collection and management of these diverse types of PII by DAs raise significant privacy concerns. To address these concerns, it is essential to ensure robust data protection measures, transparency in data handling practices, and compliance with relevant privacy regulations.

As DAs become more sophisticated and more commonplace, they will increasingly be targeted by attackers exploiting API abuse vulnerabilities. This threat involves using a DA as an intermediary to access APIs and process sensitive data. To mitigate this risk, DAs must implement strong authentication and authorization mechanisms, such as OAuth, to manage user consent and data sharing permissions. Users must explicitly authorize which service providers their DA can interact with, and what information can be shared, mirroring current API security challenges like domain takeovers and unauthorized access.

DAs will interact with other digital services, potentially introducing security risks. A compromised DA can request sensitive information under false pretenses, such as credit card numbers or social security numbers. This highlights the need for robust security protocols to verify the authenticity of DAs that are accessing personal data. Only trusted and vetted DAs should be granted access to sensitive information, minimizing the risk of malicious activity within the DA network.

As DAs become more integrated into our daily lives, they face growing security threats that take advantage of their unique capabilities and trusted position within user interactions. The potential threats include the creation of malicious “custom skills” (i.e., any additional integration that can be added to a DA, such as home automation controls and external services integration), manipulating DA interactions to deliver misleading information, and exploiting APIs to alter the data that DAs rely on. Furthermore, the rise of immersive environments like the Metaverse introduces new vulnerabilities in which avatars can mimic trusted personas to deceive users.

To address these threats, we came up with threat categories, which are summarized in a non-exhaustive list shown below. Each threat is assigned a number within a range of 0 – 6, depending on the levels that we previously covered. This range shows where the threats exist in each category, and in a few cases, you will see that they only exist in one level, not a range. Threats that have an upper range of 1 to 3 are denoted in orange, while those that have an upper range of 4 to 6 are denoted in red.

Skill squatting

Figure 6. The capabilities of DA-based threats that use skill squatting

Skill squatting takes advantage of the phonetic similarities between skill names or how users might mispronounce or slightly alter a skill’s name. This attack involves creating malicious skills with names that are very similar to legitimate skills. When a user attempts to invoke a legitimate skill but mispronounces its name, the digital assistant might inadvertently invoke the malicious skill instead. As an example, a skill named Howtel instead of Hotel would demonstrate a skill squatting threat.

Trojan skill

Figure 7. The capabilities of DA-based threats that use trojan skills

Trojan skills are a type of malicious software that targets voice-activated DAs. These apps appear legitimate and useful at first glance, often mimicking popular functions or offering helpful features. However, once installed, they secretly perform harmful actions that put the user’s privacy and security at risk. For example, they might steal sensitive information such as passwords or personal data, record conversations, or execute unauthorized commands without the user's knowledge or consent. Trojan skills take advantage of the user’s trust in DA platforms and exploits vulnerabilities for installation. This type of attack emphasizes the importance of thorough review processes, ongoing monitoring, and user education to protect against hidden threats from malicious apps in increasingly connected digital environments.

Malicious Metaverse avatar DA phishing

Figure 8. The capabilities of DA-based threats that use metaverse avatar DA phishing

Malicious metaverse avatars appear as legitimate DAs. These fake DAs mimic the appearance, behavior, and speech patterns of authentic assistants using advanced AI techniques. They are designed to deceive users into trusting them, which can lead to sensitive information, such as passwords, personal data, or financial details, being shared.

Additionally, these malicious avatars might manipulate users into performing actions that compromise their security or privacy. The immersive nature of the Metaverse makes it challenging for users to recognize malicious intent. Therefore, robust authentication mechanisms, constant monitoring, and user awareness are essential in preventing these sophisticated avatar phishing attacks and ensuring the integrity and security of interactions within the Metaverse (as we discussed in a previous publication).

Digital assistant social engineering (DASE)

Figure 9. The capabilities of DA-based threats that use DASE

Social engineering can be performed by proxy through DAs. Attackers can manipulate the output generated by DAs to indirectly deceive users into taking harmful actions. For instance, a malicious actor might instruct the DA to relay a message like, “Amazon has requested your permission for this,” knowing that users are likely to trust this communication due to their confidence in both their DA and the brand.

Although this type of attack can be executed with any form of DA, the likelihood of the threat increases as users develop greater trust in these systems, leading them to accept information and requests without skepticism. As DAs take on more sophisticated tasks, such as managing user finances, the severity of the threat rises, given that the potential consequences of a successful attack grow significantly.

To mitigate these risks, it is essential to enhance DA security, implement rigorous verification processes for communications, educate users about the dangers of uncritical trust in AI-driven interactions, and regularly update security protocols to address emerging vulnerabilities.

The Future of Malvertising

Figure 10. The capabilities of DA-based threats that use malvertising

The future of malvertising will experience an evolution, particularly within the framework of distributed advertising. Unlike traditional malvertising, which exploits vulnerabilities in ad networks to serve malicious content, DA-based malvertising takes advantage of the personalized nature of modern recommendation engines. By subtly corrupting the knowledge base of a user’s social network, attackers can manipulate the DA’s recommendation algorithms to steer users towards harmful content.

This tactic is particularly insidious because it capitalizes on the trust users place in their social and community-driven recommendations, which typically see less scrutiny from content protection systems focused primarily on individual user behavior. The success of DA-based malvertising hinges on a deep understanding of community-level user interactions, making it a sophisticated and challenging threat to counter.

The future of spear phishing

Figure 11. The capabilities of DA-based threats that use spear phishing

This threat involves compromised DAs tricking users into providing information, such as financial data, by using PII it already has access to. For example, the DA might use information on previous purchases or places visited by the individual to create a plausible scenario (e.g., “There is a problem with a previous purchase, please provide financial information to fix it”). This approach can work when the DA has some level of agency, but doesn't have the authority to proceed with financial operations on its own.

Semantic Search Engine Optimization (SEO) abuse

Figure 12. The capabilities of DA-based threats that abuse SEO

Semantic SEO abuse represents a growing threat in manipulating DAs, as the world moves towards real-time search and retrieval-augmented generation (RAG) models. Unlike traditional keyword-based indexing, semantic indexing focuses on understanding the meaning and context of information. Malicious actors can exploit this by injecting content that appears semantically relevant to what a DA is searching for but is actually misleading the user. This type of abuse aims to manipulate the DA' s results to present content that seems like the correct answer, thereby enticing users to visit a URL that may be harmful or deceptive.

As digital assistants become more reliant on semantic understanding to deliver information, attackers can craft pages that superficially align with user queries, but instead redirect users to undesirable outcomes. This manipulation not only undermines user trust in DAs but also poses significant security risks.

To mitigate these threats, it is essential to develop advanced filtering and verification mechanisms that can detect and prevent semantic manipulation, ensuring that DAs deliver accurate and reliable information. As semantic indexing becomes more prevalent, the challenge will be to maintain the integrity of search results in the face of increasingly sophisticated attempts to game the system.

Agents Abuse

Figure 13. The capabilities of DA-based threats that abuse agents

Agent abuse in DAs occurs when attackers exploit DA interaction APIs rather than directly manipulating the DA itself. This type of abuse is similar to targeted advertisements on social network platforms like Facebook, where malicious actors promote counterfeit products or scam services.

By manipulating the APIs that DAs rely on for information retrieval and decision-making, attackers can feed deceptive data into the system. If a DA recommends a product or service based on manipulated API responses, users are more likely to trust and act on this recommendation, believing their DA to be a reliable intermediary. This exploitation takes advantage of the inherent trust users place in their DAs, making them susceptible to scams and low-quality products.

As DAs evolve to become more integrated into daily decision-making, it becomes even more important to ensure the integrity of the APIs they interact with. Safeguards such as robust validation mechanisms, enhanced transparency of data sources, and regular audits can help mitigate the risk of agent abuse, preserving the trust relationship between users and their DAs.

Malicious External DAs

Figure 14. The capabilities of DA-based threats that use malicious external DAs

DAs often serve as intermediaries between users and APIs, possessing delegated authority to call and process data. This setup introduces significant security risks, as the DA, rather than the user, determines which APIs to interact with and how to manage the information being exchanged. OAuth or similar authorization mechanisms in a DA context would require a clear delineation of which providers the DA can communicate with, and the specific data authorized for sharing, such as personal information or health data.

This opens the door to a class of attacks where malicious external DAs can abuse vulnerabilities like those seen in today's API security breaches, like domain takeovers. These attacks can be particularly dangerous since users typically focus on the provider rather than the underlying API interactions. The potential for DAs to accept natural language queries further complicates the risk landscape, as it creates new avenues for API misuse and manipulation.

As a result, a comprehensive understanding and adaptation of existing API security frameworks to the DA environment is crucial for mitigating these emerging threats. As a note, task complexity in the chart for this section uses a 1-6 ranking, since we consider 1 to be disinformation, and people will take action from 2 through 6. On the Ecosystem Integration, we figured that 3 indicates that users installed the DA themselves, and anything above that would involve the DA installing another DA on its own.

To address digital assistant-based threats, a holistic approach that incorporates multiple layers of defense is essential. For example, implementing robust authentication mechanisms to ensure user identity verification safeguards users against attacks by employing multiple layers of security, making unauthorized access to DAs significantly more challenging. The combination of strong passwords and continuous monitoring limits an attacker’s ability to exploit vulnerabilities in the DA or manipulate it for malicious purposes. These measures ultimately help protect user data and privacy.

In addition, encrypting sensitive data serves as a fundamental security measure for protecting user information that is accessed or processed by DAs. Encryption would prevent unauthorized individuals from accessing or understanding the contents found within a DA, even if they manage to gain access to the systems. This protection can be applied at various levels: encrypting data at rest (what is stored on the system or devices), in transit (during communications between DAs and servers, or even DA to DA communication), and within the DAs internal memory. Utilizing strong encryption standards ensures that the data remains secure even if it is intercepted by malicious actors. By prioritizing data encryption, DAs can significantly reduce the risk of sensitive information being compromised, in turn safeguarding privacy and maintaining user trust in the system.

Furthermore, it is important to consider how the communication paradigm will shift in an environment predominantly tailored around DAs. We are looking at a multi-agentic architecture leveraging natural language for a large number of requests and responses, where one piece of polluted information might impact the whole planning chain. It becomes important therefore to proof the API ecosystem from attacks like prompt injections and information poisoning, something that, if not mitigated, could only be detected far off the processing chain.

Finally, employing advanced threat detection and mitigation techniques is crucial for proactively defending DAs against evolving cyber threats. These techniques go beyond traditional signature-based detection by utilizing machine learning algorithms and behavioral analysis to identify malicious activities in real-time.

Systems can be trained to recognize patterns indicative of known attack vectors — including the threats that are laid out within this entry, further evolving with the landscape of threats that do not yet have a defined signature. By continuously monitoring system logs that keep user privacy in mind, these systems can detect anomalies and suspicious activities, triggering alerts and responses to mitigate the threat.

Our analysis of the digital assistant ecosystem has revealed both the complexities and opportunities presented by this emerging technology.

We have identified several significant categories of security threats to DAs, including various attack vectors that can be exploited by malicious actors. These risks emphasize the need for prioritizing security considerations in the design and implementation of DAs.

One particular aspect that deserves an in-depth look is how the internet middle layer will have to evolve to support the agentic architecture that DAs will implement in order to bridge the gap between services and users, while also providing the latter with a smooth and intuitive interaction.

Collaboration between developers and users is crucial in promoting a culture of security awareness, responsible behavior, and best practices when using DAs. It is essential for users and developers to be able to recognize the types of personally identifiable information collected by DAs. Understanding these categories can help DA developers create more effective data protection measures, ensuring user-sensitive information remains secure.

The DA ecosystem has the potential to revolutionize human-technology interactions. However, addressing technical, social, and security challenges is important for unlocking its full potential. By prioritizing user needs, fostering innovation, and addressing security concerns, we can create a brighter future for all.

This appendix serves as a hands-on resource, allowing readers to experiment with the tool introduced in this study. It provides a summary of the capabilities we identified for digital assistants (DAs), along with their corresponding levels, presented in a table format. This layout makes it easier for the reader to select the appropriate level of capabilities when mapping DAs or assessing potential threats.

The list of DAs and threats that are included in this study is not meant to be exhaustive. Instead, it serves as a demonstration of how our tool can assist in analyzing various DAs and understanding their potential vulnerabilities.

Therefore, this appendix can also be seen as a starting point for further expanding the research we have undertaken with this study.

A1. Agency

One of the characteristic traits of DAs should be the ability to carry out actions on behalf of the user, and this skill tries to capture exactly that. We have identified five skill levels, namely:

• A1.0 – Informational only. The DA is information-only (i.e., only answers questions).
• A1.1 – Single commands. Every action possible by the DA has, in fact, a 1:1 mapping with specific commands that can be issued (“set an alarm for 8:30AM”).
• A1.2 – Action chaining. The DA is able to chain together multiple actions from one specific command, e.g., a command like “good night” triggers a whole routine of multiple actions (lowers the shades, turns off the lights, etc.).
• A1.3 – Goal proactive. The DA can suggest a goal to the user and a set of actions to make it happen, however, actions are still subject to user confirmation.
• A1.4 – Full unsupervised. The DA can take actions in the interest of the user, fully unsupervised.

A2. Task complexity

Task complexity classifies how much a DA will be able, in terms of usability and agency, to fully replace a phone or a PC in daily usage. Take as an example task, booking a hotel:

• A2.1 – Scripted answers The DA is a simple question answering service that provides simple scripted answers. For example, a DA being tasked to book a hotel would simply return a list of URLs for booking services (the scope is very limited possibilities of action on a fixed domain).
• A2.2 – Instructional. The DA has knowledge of certain domains and can generate a list of instructions on what do to in a specific domain, but cannot act upon them. For example, for the previous tasks the DA would return a list of instructions for the booking, but would not actually book the hotel itself.
• A2.3 – Autonomous actions. The DA can carry out certain actions based on user input (e.g., query availabilities for hotels).
• A2.4 – Single app replacement. The DA has the capability to be on par with an equivalent phone app. For example, the DA would be able to assist the user and carry on all the actions required to complete a full search and hotel booking, with all the associated support (reminders, check-in, rebooking).
• A2.5 – Multi app replacement. The DA would be able to implement and integrate the capabilities of several equivalent mobile apps. For example, acting as a full travel planner, searching for hotels in specific areas close to the proposed sightseeing spots that ranked best on sites like Tripadvisor, booking accommodation and restaurants, and calculating travel costs, among others.
• A2.6 – OS replacement. The DA would have enough capabilities to basically replace whole mobile platforms altogether (i.e., the DA becomes the new fully-supported platform for users).

A3. Required mental shift

This capability focuses on the users, in particular, the level of psychological engagement required to promote the adoption of a certain DA:

• A3.1 – Trust of DA output. A willingness to listen to what the DA says.
• A3.2 – Trust of DA instructions. A willingness to trust the DA instruction without validation.
• A3.3 – Basic actions delegation. A willingness to delegate basic actions to the DA on the user’s behalf.
• A3.4 – Financial actions delegation. A willingness to delegate financial actions to the DA.
• A3.5 – Liable actions delegation. A willingness to trust the DA to perform actions in which the user can be held legally liable for.

B1. Conversational complexity

In this section, we categorize how complex a conversation the DA can sustain, organized into the following levels:

• B1.0. The DA cannot engage in conversation.
• B1.1. The DA can engage in a simple Q&A conversation.
• B1.2. The DA can engage in a Q&A conversation that involves memory and context.
• B1.3. The DA can engage in a natural conversation — including emotional content — with the ability to perform small talk and display wit.
• B1.4. The DA can engage in a spontaneous purposeful conversation; it might initiate a conversation on its own initiative, with a specific goal in mind.
• B1.5. The DA can engage in spontaneous social conversation, initiating a conversation for purely social purposes.

B2/B3. Input/Output multimodality

With the evolution of AI models, we are seeing DAs supporting an increasing number of media both as an input and an output.

For input multimodality, we have the following levels:

• B2.1 – Text-only. The DA interacts via text-only interface.
• B2.2 – Speech to Text. Users can speak to the DA, but the voice is turned into text before being processed, losing all the meta-language information about voice, intonation, etc.
• B2.3 – Multimedia. The DA integrates images, video, and audio for relatively basic tasks such as reading the text on a picture, commenting on a song, reading handwriting, and identifying the actions in a video.
• B2.4 – Speech. The DA possesses native audio processing, i.e. using the user speech including information on intonation, inflection, etc.
• B2.5A – Facial input. The DA understands facial expressions and micro-expression, moving towards real emotional computing.
• B2.5B – Biometric signals. The DA integrates biometric information such as heart rate, and blood pressure.
• B2.6 – Real-time embodiment. The DA possesses the ability to calculate and understand position, depth perception, context, and audiovisual awareness.
• B.7 – BMI. The DA acts as an actual brain machine interface

For output multimodality, the levels are:

• B3.0 – No output. The DA provides no output.
• B3.1A – Text. The DA generates text.
• B3.1B – Text to speech. The DA possesses text to speech capabilities, i.e., it still generates text, but it is read out (although missing all voice information).
• B3.2 – Multimedia. The DA can generate picture, audio, and video output.
• B3.3 – Real speech. The DA generates actual spoken audio, while considering characteristics such as emotional information and vocal intonation.
• B3.4 – Realistic avatar. The DA can appear in the form of a human-looking virtual avatar.
• B3.5 – Enhanced mixed reality. The DA can superimpose information on top of an actual real time point-of-view video.

B4. Ubiquity

The ubiquity capability identifies how much and how well the DA integrates and follows users in their daily activities:

• B4.1 – Fixed. The DA exists as a desktop application or a hardware assistant, confined to a fixed place which requires the user to specifically reach out and interact with.
• B4.2 – Mobile. The DA can be accessed by users via a device that they constantly keep close to them, but also require them to reach out for (such as a smartphone).
• B4.3 – Wearable. The DA always accessible and is always with the user without the need to reach for it (such as a smart watch or smart glasses).
• B4.4 – BMI. The DA is literally augmenting the user’s flow of thoughts as a brain-machine interface (BMI).

C1. Planning and logical reasoning

Despite a seemingly straightforward definition, this capability is actually one of the most complex since it attempts classify the depth of a DA’s logical reasoning. We rank the levels as follows:

• C1.0 – No planning. The DA possesses no logical reasoning whatsoever.
• C1.1 – Basic query. The DA possesses logical reasoning that is somewhat comparable in complexity to a SQL query.
• C1.2 – 1-2 steps inference. The DA is able to reliably infer a chain of thought for a few steps.
• C1.3 – Single domain planning. The DA is capable of logically planning in a specified domain alone. This is similar to the capability of models like AlphaGo.
• C1.4 – Multi domain planning. The DA is capable of logically planning across multiple domains simultaneously.
• C1.5 – New domain learning. The DA is capable of autonomously learning new domains and can reason within them.
• C1.6 - New domain discovery. The DA is capable of autonomously discovering new domains to be learned to carry out a task or solve a problem, learn the new domains, and use the learned information for planning.

C2. User Knowledge

The user knowledge skill represents the level of knowledge a DA can acquire and retain about its user. We have identifed five skill levels for this:

• C2.0 – No prior knowledge. The DA works as a stateless answering box, retaining no information about the user.
• C2.1 – Static knowledge. The DA has some basic knowledge about the user, generally in the form of personal settings (language, location, etc.).
• C2.2 – Dynamic knowledge. The DA has knowledge about users acquired over time via interaction with them, learning their habits and adapting over time.
• C2.3 – Community. The DA integrates knowledge about the user's social circle, place in social groups, and details of associates, among others. This is information that it can learn through interactions, either with users themselves, or with other DAs.
• C2.4 – Blind spot. While levels 2 and 3 cover information about the user that is known to the user itself, this level involves a form of information that the DA can derive and learn that is not known, even to the users themselves. It approaches a level very close to serendipity — as a reference, think about a fitness band that can diagnose heart conditions previously unknown to the user.

C3. Ecosystem integration

Ecosystem integration defines how well the DA is able to access external information and integrate it into a wider ecosystem. It can be considered complementary to Task Complexity, but focused only on the ability to integrate with a larger ecosystem.

We define the levels as following:

• C3.0 – None. The DA has no access to external information.
• C3.1 – Skill-based. The DA has some access to external information and services, limited to a specific skill or fixed domain.
• C3.2 – Natively chaining multiple providers or provide API. The DA can link together sequential data derived from multiple sources, without the help of external support frameworks, such as LangChain or IFTTT. Alternatively, it can expose its own API. For example, the DA is able to retrieve a list of hotels for a certain time, locate them, then based on their position, query nearby restaurants, all natively.
• C3.3 – Talk to Other Assistant. In a scenario where DA might become specialized tools for specific services, a personal DA should be able to directly interface with other DAs and compile queries in natural language to retrieve the specified information. Alternatively, it should be able to interact with a human agent (i.e., customer support or reservation staff) to obtain a service natively.
• C4.4 – Integration discovery. Similar to the planning and reasoning level (to which this skill is complementary), at this level, the DA should be able to autonomously consult an information directory and find new services and mechanisms to access. It uses those services to augment its own capabilities. For example, it can provide answers to queries such as “Book a hotel, I need a taxi to get there from the airport, is there a taxi service in the area? How do I contact them and perform a transaction?”

The following table outlines the progressive capabilities of DAs across three key areas: User-perceived capabilities, interaction capabilities, and intrinsic capabilities.

Each capability represents a specific stage in a DA's evolution, progressing from basic functionalities to advanced, autonomous behaviors.

As these capabilities develop, DAs become better at performing tasks, interacting with users, and integrating into daily life. This evolution also demands varying levels of trust and psychological engagement from users. The table provides a structured view of this progression, detailing how each capability evolves over time — from rudimentary to highly sophisticated forms — ultimately shaping the overall user experience and interaction with the DA.

Digital Assistant Capabilities Scoring Table