Players Experiencing Crashes & Freezes In 5.12

Issue Description:

Each OS exposes some functionality to games like ours, which we use to draw things to the screen: this is called the graphics API. Everything seen in dragons (including 3D and 2D content) is rendered via this graphics API. Since the game was first created, we’ve been using OpenGLES as the graphics API for Dragons (we started out with OpenGLES2, and started to use OpenGLES3 with the release of Atlas). 

As a result of Apple’s deprecation of OpenGL, the engine team has been converting over to the Metal graphics API. These changes were added to an iOS-only binary to be delivered in early January 2020 in 5.12.

After the release of 5.12, users began failing to launch the game. Non-bundled assets on distribution builds were being downloaded to the right place in the client, but the client did not look in the right place for them; this resulted in iOS users who had installed the new binary to crash.

Timeline of Events

• 1/7/20

• 12pm PT: Release DRIs (Directly Responsible Individuals) begins working through checklist items to set the new client and DCON live

• 4:54pm PT - Release ticket is sent to RE (Release Engineer) to begin iOS staged rollout to 1%

• 6:24pm PT - App is live in the app store

• 6:25pm PT- RE checks store and notices 5.12 is available and began requesting for others to check their device to confirm suspicion that more than 1% of users may be able to obtain the new client

• 6:30pm PT - After updating and playing through 5.12, crashing behavior is encountered. Engineer DRI is alerted and begins looking at crash reports

• 6:35pm PT - DRIs begin investigating new crash groups and notify engineers of the issue to make sure they were aware

• 6:38pm PT - QA begins to attempt reproduction on debug builds

• 6:42pm PT - RE finds out and confirms that users can still update to the version manually if they chose to find the new version in the app store after it has propagated

• 6:50pm PT - Engineer DRI is unable to find any meaningful information from crash logs or new crash groups and continues investigation

• 6:55pm PT - Senior Engineer requests for reproduction on debug build but QA is unsuccessful at recreating the issue on all 5.12 builds available from buildsv2 page

• 6:59pm PT - RE pauses staged rollout of 5.12

• 7:01pm PT - Dev Manager (DM) informs PX that some 5.12 users have been crashing and that investigation is underway

• 7:02pm PT - DM begins sifting through forums for more data points and hypothesizes that the problem is impacting existing users and not new users which was supported by QA’s fresh installs of the store build and crashing when upgrading elder gamestates to the store build

• 7:03pm PT - Engineer continues to look at crash reports of top crash groups while Senior Engineer and Engineer DRI attempt to build 5.12 distribution builds locally for debugging

• 7:41pm PT - RE works with Senior Engineer with getting set up to build distribution

• 8pm PT - QA begins testing different client, DCON, and A/B Test configurations and continue to crash with distribution 5.12 builds only on existing gamestates

• 8:15pm PT -Engineer DRI gets up to speed, trying to build distribution builds

• 9pm PT - QA and engineers start to build distribution builds for deeper investigation

• 9:30pm PT - While engineers investigate distribution build, RE begins creating 5.13 build without metal for QA to test the viability of upgrading a metal distro client to a non-metal distro client to ensure another submission would fix the crashing

• 10pm PT - Senior Engineer and Engineer DRI inform the team that users on distribution build are not able to load assets for char_dragon.lvl 

• 10:30pm PT - DM begins creating DCON 739 to regenerate the live DCON to see if assets were just not uploaded correctly but QA continues to crash on distro build with the DCON changes

• 10:53pm PT - DM tries removing the references to the lvl file but QA continues to crash on distro build with the DCON changes (slightly different crash than original - not char_dragon, but some other asset)

• 11pm PT - QA confirms that not only elder users receive the crash, new users that installed 5.12 who eventually began calling for non-bundled assets also crash

• 11:14pm PT - Engineers confirm that resolving the issue would require a binary change and start working on the fix. Non-bundled assets were downloaded but not being loaded by the 5.12 client. 

• 11:16pm PT - DM informs RE that a submission will be necessary tonight but will explore temporary fix options with Engineers to see if we could avoid using an expedite submit

• 11:25pm PT - DM makes the call to not add the fix to 5.13 but to instead submit 5.11 (the last known stable build) as 5.13 to rollback metal changes and give QA more time to test metal before a re-release

• 11:28pm PT - 5.13 Submission build is created

• 11:41pm PT - DM and QA prepare for submission and send submission ticket to RE

• 1/8/20

• 12:00am PT - QA passes submission items to RE

• 12:21am PT - Team was unable to come up with any acceptable temporary fix so DM writes expedite reasoning and messaging to RE

• 12:31am PT - DM informs PX that War Declarations will be blocked as no immediate solution is available 

• 12:36am PT - DM and PX sync on messaging to update players in the forums

• 1am - Engineers and DM discuss the possibility of temporarily replacing the reference to char_dragon.lvl file with a bundled level file

• 1:20am PT - DM launches a bandaid fix to new 5.12 installs so that they could enter the game and begin playing through the early game

• 1:34am PT - DM sets up an in-game pop-up that is displayed before the crash to 5.12 users that informs players that we are undergoing maintenance and that 5.13 will be available to download soon

• 3:15am PT - War Dragons 5.13 begins waiting for Apple review

• 7:34am PT- 5.13 Apple approval received, DM begins working through the release checklist. Eventually QA and Engineer DRI are contacted to complete the remaining QA and engineer release items

• 8:59am PT - Android 5.13 submission build is created

• 9:21am PT- RE is notified to begin the release process and Android RE is alerted that an Android 5.13 submission is being prepared so that both platforms would be in versioning parity to support cross-platform playing 

• 12:05pm PT - 5.13 is live in iOS app store.

• 12:07pm PT -Engineers are requested to cancel started wars and refund trophies

• 12:08pm PT - Android RE begins releasing 5.13 to Android users

• 1:37pm PT - Existing wars are cancelled by Engineers

• 3:30pm PT -Engineers, QA and DM meet to discuss strategy to test distribution build before re-releasing metal