Meta/Oculus Avatars Networked with Normcore

I wanted to share my setup for using meta avatars with normcore. I put this setup together with help of the official meta docs as well as a variety of articles and posts. Reference this code at your own risk :slight_smile: but I hope this will help some folks :heart:

This assumes that you have the meta avatar SDK installed as well as a basic avatar setup with normcore. Also make sure to follow the meta docs on how to get oculus platform permissions for your app.

First, initialize the Oculus platform and fetch the user’s avatar ID for later use (SessionManager.metaUserID is where I store it and it will be used to load the local user’s avatar and allow remote users to get the same asset as well). I do this as my app starts up.

      public class MetaAvaratInit : MonoBehaviour {
        private void Awake() {
            StartCoroutine(SetupOvrPlatform());
        }

        private IEnumerator SetupOvrPlatform() {
            // Ensure OvrPlatform is Initialized
            if(OvrPlatformInit.status == OvrPlatformInitStatus.NotStarted) {
                OvrPlatformInit.InitializeOvrPlatform();
            }
            while(OvrPlatformInit.status != OvrPlatformInitStatus.Succeeded) {
                if(OvrPlatformInit.status == OvrPlatformInitStatus.Failed) {
                    Debug.LogError("Error initializing OvrPlatform");
                    yield break;
                }
                yield return null;
            }
            Users.GetLoggedInUser().OnComplete(message => {
                if(!message.IsError) {
                    SessionManager.metaUserID = message.Data.ID.ToString();
                } else {
                    var e = message.GetError();
                }
            });
        }
    }

The bulk of my custom code is in the MetaAvatarControlle script. It is a realtime component that synces the avatar asset Id as well as avatar animation. It also handles instantiation of the meta avatar for local and remote clients. I’m using addressables to load my avatar entity prefab and I use a prefab for remote as well as local avatars. Probably cleaner ways of doing this, but I had issues with timings of avatar entity creation and avatar loading and found this to be the most robust solution. Both avatar prefabs use my custom AvatarEntity class (RevelMetaAvatarEntity, see below) and are configured according to the meta docs for local and remote avatar entities. I’m using Normcore’s avatar setup and I’m calling the MetaAvatarController.InitLocal() function at the end of CreateAvatarIfNeeded function of the avatar manager. The remote avatar gets instantiated when the metaID comes through over the network OnUserMetaIdDidChange.

public class MetaAvatarController : RealtimeComponent<MetaAvatarModel> {

        public OvrAvatarLipSyncBehavior lipSyncBehavior;

        [SerializeField] private AssetReference metaAvatarAsset;
        private AsyncOperationHandle<GameObject> metaAvatarHandle;
        private GameObject metaAvatarObject;

        [SerializeField] private AssetReference metaAvatarAssetRemote;
        private AsyncOperationHandle<GameObject> metaAvatarHandleRemote;
        private GameObject metaAvatarObjectRemote;

        private byte[] byteArray = new byte[0];
        private RevelMetaAvatarEntity avatarEntity;
        private bool remoteLoaded = false;

        public bool isLocal = false;

        private void OnDestroy() {
            if(metaAvatarHandle.IsValid()) Addressables.Release(metaAvatarHandle);
            if(metaAvatarHandleRemote.IsValid()) Addressables.Release(metaAvatarHandleRemote);
        }

        protected override void OnRealtimeModelReplaced(MetaAvatarModel previousModel, MetaAvatarModel currentModel) {
            if(previousModel != null) {
                previousModel.userMetaIdDidChange -= OnUserMetaIdDidChange;
                previousModel.avatarAnimationDataDidChange -= OnAvatarAnimationDataChanged;
            }
            if(currentModel != null) {             
                currentModel.userMetaIdDidChange += OnUserMetaIdDidChange;
                currentModel.avatarAnimationDataDidChange += OnAvatarAnimationDataChanged;
                if(currentModel.userMetaId != "notSet") {
                    InitRemote(currentModel.userMetaId);
                }
            }
        }

        private void OnUserMetaIdDidChange(MetaAvatarModel model, string metaId) {        
            if(!isLocal) {
                InitRemote(metaId);
            }
        }

        private void OnAvatarAnimationDataChanged(MetaAvatarModel model, byte[] data) {        
            if(!remoteLoaded) return;
            if(isLocal) return; //only apply data for remote     
            avatarEntity.SetPlaybackTimeDelay(0.1f);
            avatarEntity.ApplyStreamData(data);
        }

        public void InitLocal() {
            isLocal = true;
            PlayerController.playerController.lipSyncBehavior = lipSyncBehavior;
            StartCoroutine(WaitForUserId());
        }

        private IEnumerator WaitForUserId() {
            while(string.IsNullOrEmpty(SessionManager.metaUserID)) yield return null;
            model.userMetaId = SessionManager.metaUserID;
            StartCoroutine(CreateAvatarLocal());
        }

        private IEnumerator CreateAvatarLocal() {
            metaAvatarHandle = Addressables.LoadAssetAsync<GameObject>(metaAvatarAsset.RuntimeKey);
            yield return metaAvatarHandle;
            metaAvatarObject = metaAvatarHandle.Result;
            avatarEntity = Instantiate(metaAvatarObject, transform, false).GetComponent<RevelMetaAvatarEntity>();
            avatarEntity._userId = Convert.ToUInt64(SessionManager.metaUserID);
            avatarEntity.OnUserAvatarLoadedEvent.AddListener(OnUserAvatarLoaded);
            avatarEntity.Init();
        }

        private void InitRemote(string metaId) {
            StartCoroutine(CreateAvatarRemote(metaId));
        }

        private IEnumerator CreateAvatarRemote(string metaId) {
            metaAvatarHandleRemote = Addressables.LoadAssetAsync<GameObject>(metaAvatarAssetRemote.RuntimeKey);
            yield return metaAvatarHandleRemote;
            metaAvatarObjectRemote = metaAvatarHandleRemote.Result;
            avatarEntity = Instantiate(metaAvatarObjectRemote, transform, false).GetComponent<RevelMetaAvatarEntity>();
            avatarEntity._userId = Convert.ToUInt64(metaId); //set remote users avatarID
            avatarEntity.OnUserAvatarLoadedEvent.AddListener(OnUserAvatarLoadedRemote);
            avatarEntity.Init();
        }

        private void OnUserAvatarLoadedRemote(OvrAvatarEntity avatarEntity) {
            remoteLoaded = true;
        }

        private void OnUserAvatarLoaded(OvrAvatarEntity avatarEntity) {
            //avatar done
            if(isLocal) {
                StartCoroutine(StreamAvatarData(avatarEntity));
            }
        }

        private IEnumerator StreamAvatarData(OvrAvatarEntity avatarEntity) {
            while(true) {
                byteArray = avatarEntity.RecordStreamData(avatarEntity.activeStreamLod);
                model.avatarAnimationData = byteArray;
                yield return new WaitForSecondsRealtime(0.05f);
            }
        }
    }

Avatar Model for avatar sync:

public partial class MetaAvatarModel {
    [RealtimeProperty(1, true, true)] private string _userMetaId = "notSet";
    [RealtimeProperty(2, false, true)] private byte[] _avatarAnimationData = new byte[0];
}

My custom AvatarEntity class - I tried to keep it minimal, you can look at the sampleavatarentity class that comes with the avatar SDK for another (more convoluted) example. At first I set the references to the body tracking as well as the lip sync context if the avatar entity is local. This will allow for hand/head tracking and lipsync. I’m using the sample input manager that is part of the AvatarSDKManager prefab (see meta docs). More on the avatar lip sync below. The Init() function gets called from the MetaAvatarController with different timings based on whether it is a local or remote avatar.

    public class RevelMetaAvatarEntity : OvrAvatarEntity {

        protected override void Awake() {
            if(IsLocal) {
                _bodyTracking = PlayerController.playerController.sampleInputManager;
                SetLipSync(PlayerController.playerController.lipSyncBehavior);
            }
            base.Awake();
        }

        public void Init() {
            StartCoroutine(LoadAvatar());
        }

        private IEnumerator LoadAvatar() {
            // Ensure OvrPlatform is Initialized and we have a token
            if(OvrPlatformInit.status == OvrPlatformInitStatus.NotStarted) {
                OvrPlatformInit.InitializeOvrPlatform();
            }
            while(OvrPlatformInit.status != OvrPlatformInitStatus.Succeeded) {
                if(OvrPlatformInit.status == OvrPlatformInitStatus.Failed) {
                    Debug.LogError("Error initializing OvrPlatform");
                    yield break;
                }
                yield return null;
            }

            var hasAvatarRequest = OvrAvatarManager.Instance.UserHasAvatarAsync(_userId);
            while(!hasAvatarRequest.IsCompleted) { yield return null; }
            LoadUser();
        }
    }

Lasty, the avatar lipsync component can be added to your avatar with the following setup:

You then need to reference this component in your RealtimeAvatarVoice class, or wherever you process your avatar voice in order to avoid conflicts between mic permisisons. This is what I added to the SendMicrophoneData() function:

 //pass on audio data to lip sync processor
            if(didGetAudioData) {
                metaLipSyncContext.ProcessAudioSamples(_microphoneFrameData, 1);          
            }

That’s it, good luck.

1 Like