PS5 crash after connecting to room

Bug Reports
1

PS5 build crashes when connecting to room

Development Environment:
Windows
Unity Version - 2021.3.23f1
Normcore Version - 2.4.2

Deployment/Build platform:
PS5 / PSVR2

Issue

I’m getting a consistent crash when connecting to a room on PS5. I’m afraid I don’t have a repro case at the moment but wondering if the stack trace might help to diagnose it?

Crashdump:

Location: libkernel.sprx!0x000000083227CCCC (Module: 0x000000083227C000 + 3276 bytes)
coreview (prospero)> process analysis
process analysis
Crash Info

Exception Info
  Thread : 101208
  Address : 0x000000083227CCCC
  Category : 0x00000000
  Code : 0x0000000D
  Description : SIGPIPE: Write on a pipe with no reader.

Location: libkernel.sprx!0x000000083227CCCC (Module: 0x000000083227C000 + 3276 bytes)

Stack
    0 0x000000083227CCCC libkernel.sprx!0x000000083227CCCC
    1                    [Frames below may be incorrect and/or missing, no symbols loaded for libkernel.sprx]
    2 0x0000000094AE1763 RealtimeClient.prx!rtc::AsyncTCPSocketBase::FlushOutBuffer()
    3 0x0000000094AE1B24 RealtimeClient.prx!rtc::AsyncTCPSocket::Send(void const*,unsigned long,rtc::PacketOptions const&)
    4 0x0000000094D2D1B8 RealtimeClient.prx!cricket::TCPPort::SendTo(void const*,unsigned long,rtc::SocketAddress const&,rtc::PacketOptions const&,bool)
    5 0x0000000094B69DA2 RealtimeClient.prx!cricket::Connection::OnSendStunPacket(void const*,unsigned long,cricket::StunRequest*)
    6 0x0000000094B77363 RealtimeClient.prx!cricket::StunRequest::OnMessage(rtc::Message*)
    7 0x0000000094A3BBA0 RealtimeClient.prx!rtc::Thread::Send(rtc::Location const&,rtc::MessageHandler*,unsigned int,rtc::MessageData*)
    8 0x0000000094B75FE2 RealtimeClient.prx!cricket::StunRequestManager::SendDelayed(cricket::StunRequest*,int)
    9 0x0000000094B6B201 RealtimeClient.prx!cricket::Connection::Ping(long)
   10 0x0000000094B57C2E RealtimeClient.prx!cricket::P2PTransportChannel::CheckAndPing()
   11 0x0000000094B5B2DF RealtimeClient.prx!webrtc::webrtc_new_closure_impl::SafetyClosureTask<cricket::P2PTransportChannel::CheckAndPing()::$_13>::Run()
   12 0x0000000094A3BF46 RealtimeClient.prx!rtc::Thread::QueuedTaskHandler::OnMessage(rtc::Message*)
   13 0x0000000094A3AD5F RealtimeClient.prx!rtc::Thread::Dispatch(rtc::Message*)
   14 0x0000000094A3BA2F RealtimeClient.prx!rtc::Thread::Run()
   15 0x0000000094A3B546 RealtimeClient.prx!rtc::Thread::PreRun(void*)
   16 0x00000008322834E0 libkernel.sprx!0x00000008322834E0

Source Info
  Not available

Disassembly
  rtc::AsyncTCPSocketBase::FlushOutBuffer()
      0000000094AE1750 mov        rdi,qword ptr [r14+1C8h]
      0000000094AE1757 mov        rsi,rbx
      0000000094AE175A mov        rdx,r12
      0000000094AE175D mov        rax,qword ptr [rdi]
      0000000094AE1760 call       qword ptr [rax+30h]
  >   0000000094AE1763 mov        r13d,eax
      0000000094AE1766 test       eax,eax
      0000000094AE1768 jle        0000000094AE17A2h
      0000000094AE176A mov        eax,r13d
      0000000094AE176D cmp        r12,rax


Module Info
  Name : libkernel.sprx
  Path : /irwnwip2H3/common/lib/libkernel.sprx
  Version : 1.1 (SDK 6.50.00.10-00.00.00.0.1)

Unity log:

51:19:04	Log: BidirectionalClient: Connected!
51:19:04	Log: Sending message: {"messageType":"Request","requestData":{"appKey":"db612a3e-26ff-4adc-ae51-ff925145da9b","clientOffer":"dj0wDQpvPS0gNTEyMzEyNzQ0NjkyOTMyNjk3NCAyIElOIElQNCAxMjcuMC4wLjENCnM9LQ0KdD0wIDANCmE9Z3JvdXA6QlVORExFIDANCmE9ZXh0bWFwLWFsbG93LW1peGVkDQphPW1zaWQtc2VtY
51:19:04	W50aWM6IFdNUw0KbT1hcHBsaWNhdGlvbiA5IFVEUC9EVExTL1NDVFAgd2VicnRjLWRhdGFjaGFubmVsDQpjPUlOIElQNCAwLjAuMC4wDQphPWljZS11ZnJhZzp0K1o5DQphPWljZS1wd2Q6S1cwLzI2Ujd5QUNaaEcxd1lKVXI4UmNEDQphPWljZS1vcHRpb25zOnRyaWNrbGUNCmE9ZmluZ2VycHJpbnQ6c2hhLTI1NiA0NTpGNjo3RDoxMjo3MTo1QzpGODpBOTo0RDp
51:19:04	BNjowODoyRTo3NDo0RjpCMTo3MTozRTpFNTpCNDoyNzpFRDo3MDpDMzpDQTpBNDo0Qzo1Njo3QTo3MTo5ODo2QTo5MQ0KYT1zZXR1cDphY3RwYXNzDQphPW1pZDowDQphPXNjdHAtcG9ydDo1MDAwDQphPW1heC1tZXNzYWdlLXNpemU6MjYyMTQ0DQo=","normcoreVersion":"2.8.0","roomName":"0d8fda75-3935-fb72-9482-746757a60000"},"reque
51:19:04	stGUID":"c48691d2-2d34-f1ce-b750-68306d962789","requestType":"ConnectToRoom"}
51:19:05	Log: Received message: {"messageType":"RequestResponse","requestGUID":"c48691d2-2d34-f1ce-b750-68306d962789","status":"success","responseData":{"response":"PingClusters","clusters":[{"name":"do-nyc1-us-nyc1","address":"167.99.234.155"},{"name":"do-sfo3-us-sfo3","address":"1
51:19:05	64.92.111.146"},{"name":"do-ams3-eu-ams3","address":"161.35.157.240"}]}}
51:19:05	Realtime: Pinging clusters to determine server with lowest latency for this client.
51:19:05	UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
51:19:05	UnityEngine.Logger:Log(LogType, Object)
51:19:05	UnityEngine.Debug:Log(Object)
51:19:05	Normal.Realtime.Room:SetConnectionState(ConnectionState)
51:19:05	Normal.Realtime.<ConnectToServer_Coroutine>d__88:MoveNext()
51:19:05	UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr) (at C:\build\output\unity\unity\Runtime\Export\Scripting\Coroutines.cs:17)
51:19:05	
51:19:05	Log: Sending message: {"messageType":"Request","requestData":{"appKey":"db612a3e-26ff-4adc-ae51-ff925145da9b","clientOffer":"dj0wDQpvPS0gMTg0MDA4NDU4MDcwNTA0MDU1NCAyIElOIElQNCAxMjcuMC4wLjENCnM9LQ0KdD0wIDANCmE9Z3JvdXA6QlVORExFIDANCmE9ZXh0bWFwLWFsbG93LW1peGVkDQphPW1zaWQtc2VtY
51:19:05	W50aWM6IFdNUw0KbT1hcHBsaWNhdGlvbiA5IFVEUC9EVExTL1NDVFAgd2VicnRjLWRhdGFjaGFubmVsDQpjPUlOIElQNCAwLjAuMC4wDQphPWljZS11ZnJhZzo5ejRXDQphPWljZS1wd2Q6c2RQejdxTmk4YkNEc1FEVVFJWVcyU0ZnDQphPWljZS1vcHRpb25zOnRyaWNrbGUNCmE9ZmluZ2VycHJpbnQ6c2hhLTI1NiA5NzpCNTo3NDozNDpFOTo4MTpBMTpDOTpDRDo
51:19:05	zODo5ODpCMzpFMDpCMDo1NToxQTo4RTo2MzoxODozNTo5QjpBNjo5NDpDNzpFQTpDQzozNTpDRDoxNzoyOTpFRTo1Rg0KYT1zZXR1cDphY3RwYXNzDQphPW1pZDowDQphPXNjdHAtcG9ydDo1MDAwDQphPW1heC1tZXNzYWdlLXNpemU6MjYyMTQ0DQo=","clusterPingResults":[{"address":"167.99.234.155","name":"do-nyc1-us-nyc1","ping":1
51:19:05	61},{"address":"164.92.111.146","name":"do-sfo3-us-sfo3","ping":161},{"address":"161.35.157.240","name":"do-ams3-eu-ams3","ping":161}],"normcoreVersion":"2.8.0","roomName":"0d8fda75-3935-fb72-9482-746757a60000"},"requestGUID":"b9f54d69-798d-f5ce-9d82-d5d605484054","requestT
51:19:05	ype":"ConnectToRoom"}
51:19:06	Log: Received message: {"messageType":"RequestResponse","requestGUID":"b9f54d69-798d-f5ce-9d82-d5d605484054","status":"success","responseData":{"response":"FoundRoom","serverAnswer":"dj0wDQpvPS0gNzk5MTMyNzY2NTE3OTMxODU3MCAyIElOIElQNCAxMjcuMC4wLjENCnM9LQ0KdD0wIDANCmE9Z3JvdXA
51:19:06	6QlVORExFIDANCmE9ZXh0bWFwLWFsbG93LW1peGVkDQphPW1zaWQtc2VtYW50aWM6IFdNUw0KbT1hcHBsaWNhdGlvbiA0Nzc5OCBVRFAvRFRMUy9TQ1RQIHdlYnJ0Yy1kYXRhY2hhbm5lbA0KYz1JTiBJUDQgMTYxLjM1LjEwMi4yOQ0KYT1jYW5kaWRhdGU6MTYxMDI2MjExMSAxIHVkcCAyMTIyMjYwMjIzIDE2MS4zNS4xMDIuMjkgNDc3OTggdHlwIGhvc3QgZ2VuZ
51:19:06	XJhdGlvbiAwIG5ldHdvcmstaWQgMSBuZXR3b3JrLWNvc3QgNTANCmE9Y2FuZGlkYXRlOjQwMDkzMjY4MiAxIHVkcCAyMTIyMTk0Njg3IDE2MS4zNS4xMDIuMjkgNDA1NzAgdHlwIGhvc3QgZ2VuZXJhdGlvbiAwIG5ldHdvcmstaWQgMiBuZXR3b3JrLWNvc3QgNTANCmE9Y2FuZGlkYXRlOjI1NTg0MTk2NjEgMSB1ZHAgMjEyMjEyOTE1MSAxNjEuMzUuMTAyLjI5IDQ
51:19:06	wOTQ1IHR5cCBob3N0IGdlbmVyYXRpb24gMCBuZXR3b3JrLWlkIDMNCmE9Y2FuZGlkYXRlOjI5MDA1NjY2MTQgMSB1ZHAgMjEyMjA2MzYxNSAxNjEuMzUuMTAyLjI5IDQ3MTcyIHR5cCBob3N0IGdlbmVyYXRpb24gMCBuZXR3b3JrLWlkIDQNCmE9Y2FuZGlkYXRlOjE2NjIyMTU0IDEgdWRwIDIxMjE5OTgwNzkgMTYxLjM1LjEwMi4yOSA1Mjc3NiB0eXAgaG9zdCBnZ
51:19:06	W5lcmF0aW9uIDAgbmV0d29yay1pZCA1DQphPWNhbmRpZGF0ZTo1NTcxMzkxNDMgMSB0Y3AgMTUxODI4MDQ0NyAxNjEuMzUuMTAyLjI5IDQ2MDM3IHR5cCBob3N0IHRjcHR5cGUgcGFzc2l2ZSBnZW5lcmF0aW9uIDAgbmV0d29yay1pZCAxIG5ldHdvcmstY29zdCA1MA0KYT1jYW5kaWRhdGU6MTc2NDM4NDIxMCAxIHRjcCAxNTE4MjE0OTExIDE2MS4zNS4xMDIuMjk
51:19:06	gNDMwNDEgdHlwIGhvc3QgdGNwdHlwZSBwYXNzaXZlIGdlbmVyYXRpb24gMCBuZXR3b3JrLWlkIDIgbmV0d29yay1jb3N0IDUwDQphPWNhbmRpZGF0ZTozODcwMzkyNDA1IDEgdGNwIDE1MTgxNDkzNzUgMTYxLjM1LjEwMi4yOSA1OTM2OSB0eXAgaG9zdCB0Y3B0eXBlIHBhc3NpdmUgZ2VuZXJhdGlvbiAwIG5ldHdvcmstaWQgMw0KYT1jYW5kaWRhdGU6MzUyNjE0O
51:19:06	DMwMiAxIHRjcCAxNTE4MDgzODM5IDE2MS4zNS4xMDIuMjkgNTQ0NjEgdHlwIGhvc3QgdGNwdHlwZSBwYXNzaXZlIGdlbmVyYXRpb24gMCBuZXR3b3JrLWlkIDQNCmE9Y2FuZGlkYXRlOjIxMTcyMjI2MTAgMSB0Y3AgMTUxODAxODMwMyAxNjEuMzUuMTAyLjI5IDQwNjY1IHR5cCBob3N0IHRjcHR5cGUgcGFzc2l2ZSBnZW5lcmF0aW9uIDAgbmV0d29yay1pZCA1DQp
51:19:06	hPWljZS11ZnJhZzpFRjJvDQphPWljZS1wd2Q6QW0vdnJwRnFyTVFDVmd4aFN2Yk5Ea2gzDQphPWljZS1vcHRpb25zOnRyaWNrbGUNCmE9ZmluZ2VycHJpbnQ6c2hhLTI1NiAyM
51:19:05*	#
51:19:05*	# A user thread receives a fatal signal
51:19:05*	#
51:19:05*	# signal: 13 (SIGPIPE)
51:19:05*	# thread ID: 101208
51:19:05*	# thread name:
51:19:05*	# proc ID: 169
51:19:05*	# proc name: eboot.bin
2

I haven’t seen this one before. Nice catch! We don’t use TCP within webrtc, so I imagine this is a bug in how webrtc is configured on PlayStation given it’s trying to send a TCP packet. Happy to look into it for you. Do you have any tips on how I can recreate this issue?

Max

3

I’ll see if I can get a small test case together. Strangely we’ve not had this problem until just recently, and haven’t been changing any of the Normcore connection code either, so it could be hard to get a repro case for.

4

This does look like a race condition of sorts, which I’m sure will be hard to repro. Keep me posted! I can dig into this too and see if I can find out what code paths lead to this stack trace.

Max

5

After some more testing one thing I have found is that the PS5 seems only crash when it’s the first player to attempt to connect to the room. If another player on PC has already connected to the room there wasn’t a problem. I wondered if the Normcore servers might be rejecting the PS5 as the first player for some reason - the PS docs say the SIGPIPE termination occurs when the destination disconnects the connection.

Regarding TCP, the Normcore docs do say that TCP is used as a fallback. I couldn’t find any reason why it would be unable to use UDP though - I tried disabling all firewalls etc.

I’ll try next to creat a small repro case I suppose, hopefully that will help figure it out.

6

I’ve emailed a small repro project to you. I did notice that removing my didConnectToRoom event listener made the crash stop happening. So I might be able to work around it for now if that fix works in the main project.

7

I got it! Thanks for putting that together.

I’ve just pushed Normcore v2.8.0 that should work on PlayStation. Since a lot has changed since Normcore v2.4.2 and v2.8.0 with respect to libwebrtc, will you give it another shot and let me know if you’re still seeing this bug?

Max