AI Integration in the Metaverse with XR Creator Studio

The AI Integration Microservice provides a centralized solution for connecting AI agents in the metaverse to providers such as OpenAI and Azure AI.

To build an AI-powered agent in XR Creator Studio, you’ll need to connect a supported AI model (from OpenAI or Azure) by configuring the API key, assistant ID, or endpoint.

What is an AI Agent?

An AI Agent is a configurable digital entity powered by a language model (like OpenAI or Azure AI) that can interact with users in immersive environments such as the metaverse. Its core components include:

• Model: The underlying AI engine (e.g., GPT-4, GPT-3.5) that drives its intelligence.
• Memory: Determines whether the agent retains conversation history temporarily (volatile) or across sessions (persistable).
• Interaction Instructions: Custom system prompts that define the agent’s personality, role, and behavioral constraints, like “You are a helpful assistant specializing in company products.”

AI Agent Examples

• ChatGPT (OpenAI): A versatile conversational agent used for writing, coding, research, and general assistance.

• Google Bard (Gemini): A chatbot focused on creativity, productivity, and real-time information retrieval.

• Meta AI (Facebook): Integrated into Messenger and Instagram, offering smart replies, search, and recommendations.

• Microsoft Copilot: Agents Embedded in Microsoft 365 apps like Teams, Outlook, and Word. These agents assist with tasks like summarizing emails, drafting documents, generating reports, and automating workflows2.

Using Completion vs Assistant

The choice between them depends on your project’s needs—whether you want fast, lightweight interactions or a more robust, context-aware assistant.

• Completion-Based Agents
  These rely on prompt-response interactions. You provide a system prompt, and the model generates replies. Ideal for lightweight, reactive agents with optional memory.

• Assistant-Based Agents
  Built using OpenAI’s Assistant API, these agents support advanced features like file search, persistent memory, and tool integration. Best suited for complex tasks requiring deeper context and continuity.

Supported Agent types

XR Creator Studio supports three distinct types of AI Agents, each integrated through a different method and offering unique capabilities:

• Azure AI Completion Agent
  Connects via an endpoint and API key. Offers voice options, proximity detection, and event triggers.

• OpenAI Completion Agent
  Uses OpenAI’s API and model name. Supports volatile or persistable memory, voice synthesis, and scripting events.

• OpenAI Assistant Agent
  Built with OpenAI’s Assistant API. Allows file uploads and tool usage, enabling deeper contextual understanding and long-term memory.

AI Agent Event System

When Emit Events is enabled, the AI Agent can trigger various interactions:

Event	Description
user-enter	Triggered when a user user enters the proximity region of the agent. This event is dispatched only if ‘Emit events’ is enabled and ‘Enable proximity’ is activated in the AI Agent component.
user-leave	Triggered when a user leaves the AI Agent’s proximity region. It is dispatched if 'Emit events' is enabled and ‘Enable proximity’ is being used.
agent-talk-start	Emitted when the agent begins speaking. When the audio is played with the play button, it is also emitted whenever the voice synthesis sequence starts.
agent-talk-talking	Dispatched during the agent's speech in each frame, sending the modulated amplitude of the sound wave. When subscribed to this event, the sound wave arrives as a float value as a parameter of the event handlers, amp is similar to the value used in MUDz for the scale animation of avatars when they speak. Here, we would take it to send the audio value of the agent speaking.
agent-talk-end	Triggered when the agent finishes speaking.
agent-thinking	Dispatched when the agent is thinking, i.e., when the dots are shown in the chat.

These are especially useful when working with our Scripting API.

warning

Make sure the Observable option is enabled so these corresponding in‑world scripting events can execute properly.