Building AI Agents in the MUD XR Platform
Step into the frontier of immersive intelligence. In this hands-on workshop, participants will learn how to build intelligent, interactive AI agents inside the MUD XR Platform using XR Creator Studio.
Building AI Agents in the MUD XR Platform
Step into the frontier of immersive intelligence. In this hands-on workshop, participants will learn how to build intelligent, interactive AI agents inside the MUD XR Platform using XR Creator Studio. Leveraging the power of CoPilot’s API, this session will guide you through creating fully contextual, visually embodied agents capable of dynamic conversation and world interaction.
By the end of the session, you'll have the skills to build your own AI-powered agents from scratch with persistent memory, custom instructions, and animated 3D presence.
instructor
Transforming innovator with a proven track record in streamlining solutions for complex engineering challenges. As Principal Software Developer at Microsoft’s Mixed Reality team, I build best-in-class data observability solutions spanning hardware to user experience. Previous roles include leading telemetry and business intelligence systems at Magic Leap, developing Spatial Computing solutions in education, and pioneering mobile technology test systems at Motorola. My career also includes hands-on work at NASA’s Kennedy Space Center on the International Space Station and the Mission to Mars prototype.
A passionate community leader, I serve as a board director for the MUD Foundation, driving education initiatives to enhance diverse student outcomes in STEAM. Skilled in software, hardware, and cloud integrations, I bring expertise in Unity, C#, Python, mobile devices, and global supply chain engineering. Fluent in cross-cultural collaboration, I leverage my curiosity and technical insight to design, advocate for, and implement impactful innovations.
What You'll Learn
-
Set up an Azure Developer account.
-
Generate and securely manage your API Key.
-
Understand basic prompt engineering and usage limits.
-
Add and configure the AI Agent Logic component in your XR scene.
-
Explore agent triggers, event handling, and communication models.
-
Define system-level prompts to shape your agent’s personality, tone, knowledge scope, and behavioral rules.
-
Use XR Creator Studio's text interface to dynamically edit and test system prompts in real-time.
-
Understand the importance of context retention for multi-turn conversations.
-
Implement Persistables in XR Creator Studio to store and recall information during interactions.
-
Configure memory scope: local, session-based, or persistent across scenes.
-
Attach 3D character models to your AI agents.
-
Animate the agents with gestures, facial expressions, and idle behaviors to enhance immersion.
-
Use interaction triggers to coordinate animation states with conversation events.
-
Duration: 1 hour
-
Skill Level: Beginner to Intermediate
-
Tools Used: XR Creator Studio, OpenAI Dashboard, MUD XR Platform and 3D Asset Library (models + animations provided)
By the end of the workshop, you will be able to:
-
Set up an AI agent in the MUD XR Platform using XR Creator Studio.
-
Write custom instructions that define how your agent thinks and responds.
-
Implement persistent memory to make your agents smarter and more consistent.
-
Give your agent a body with a 3D model and lifelike animations.
-
Deploy and test your AI agent in a live XR environment.
-
XR Developers & Designers.
-
AI Enthusiasts and Prompt Engineers.
-
Game Developers.
-
Educators & Interactive Storytellers.
-
Metaverse Builders.
-
Laptop with XR Creator Studio installed (installation instructions will be provided ahead of time).
-
OpenAI account (we’ll help you set one up if needed).
-
Curiosity and creativity!
XR Controls Overview
Getting started? Select your device to see the basic controls.
Use arrows or W, A, S, D
Use W, A, S, D + Hold Shift
Use Q and E or left mouse button
Right mouse button
Left mouse button
Use the joystick
Touch + Drag
Touch
Left controller thumbstick
B or Y buttons to move faster
Right controller thumbstick
Right controller trigger
Left or Right controller trigger
SUPPORT MUD FOUNDATION EVENTS AND PROGRAMS
Tech-Centered Art, Art Creation, and XR Art Access: Together, let's build the future of art on digital media. By supporting our art + tech programs, our vibrant art-tech community, and the XR tech infrastructure of our non-profit community partners, we're ensuring that our programs remain accessible to everyone, becoming a collaborative ecosystem for all!
Your contributions go to the MUD Foundation INC, a 501(c)(3) organization based in Miami, Florida, and are used at our discretion for charitable purposes. They are tax-deductible in the U.S. to the fullest extent permitted by law.
Sponsors
