Great news, Soldiers! (Or at least it sounds great?). Expression Games announced via another Server and Stability Statement on their Discord and Steam page, that they have a major break through on the issues that are plaguing Hell Let Loose concerning both client and server stability. You can take a peek at their official message on their website, here.

It sounds as though they can replicate the issue, they are working hard to resolve it, and hopeful they may have an update before the weekend rolls around!

Stay safe out there, soldiers! -Hobbit

October 23rd, Server and Stability Statement

Thu 23 Oct Update

Thank you to everyone who has continued to support us and share crash reports over the past few days. Your feedback and detailed reports have played a huge role in helping us make real progress.

Today, we’re pleased to share that we’ve taken a major step forward in our investigation. The team has been able to consistently reproduce the crash, which brings us much closer to identifying the exact cause. We know how important stability is to your experience, and we remain fully committed to resolving this issue as quickly as possible.

TLDR Summary

• The team has made a major breakthrough in the investigation.
• We have now found a consistent way to reproduce the crash, allowing us to focus on a much narrower area of the game’s systems.
• This issue has been extremely difficult to track down, as reproducing it in a QA or development environment was nearly impossible until now.
• With this progress, the team is closing in on the root cause and continuing to work around the clock toward a fix.
• We will share more information before the weekend so stay tuned.

Status of the Investigation

Latest Update

Today marks a significant step forward in resolving the current crashing issues.

As of this afternoon, the team has discovered a consistent way to trigger a crash. This breakthrough allows us to focus on a much smaller area of the game’s systems to pinpoint the exact cause.

This particular issue has been a needle in a haystack to uncover. The conditions required to reproduce it are so specific that encountering it within a QA or development environment is almost impossible. Finding a reliable way to make it happen marks a major turning point in our investigation.

The team continues to work around the clock to determine the root issue, and have been focused on getting this resolved as soon as possible.

We will be sharing another update tomorrow and hope to have more information available before the weekend.

From the Dev Team: Understanding the Issue

As part of our update yesterday, we shared more behind-the-scenes details about the process of tracking down the cause of the recent crashes. The response from the community has been incredibly supportive, and it’s been great to see players taking such an interest in how we’re approaching our investigation. With that in mind, we wanted to expand a little further on how we got here and what we’ve been doing to resolve the issue.

Here is an explanation from Kieran, Studio Technical Director at Expression Games:

“The release build for Update 18 was starting to be finalised from 10 September, over a month ago. This means the underlying issue responsible for the current crashes has likely existed since that point, and possibly even earlier within our development environment.

Prior to yesterday afternoon (22 October), we had only observed a handful of unrelated crashes internally, and none provided indicators of the issue that has since appeared in the live environment.

The challenge we face is that this specific crash occurs when something has already gone wrong before the actual crash point. As a result, the crash dumps we collect, both from our internal testing and from the community, show where the crash happens within PhysX but provide little insight into how the physics engine reached that unstable state.

Our next step is to fully understand what conditions lead to the crash so we can reliably reproduce it. Once we can reproduce it consistently, we can verify a fix and roll it out confidently to everyone.

The good news is that on Wednesday 22 October, we successfully triggered the crash on a live server outside of the development environment for the first time. This discovery allowed us to focus our search on what causes the crash. By attaching development tools to clients playing on live servers, we were then able to collect detailed debugging information whenever a crash occurs, helping us better understand its cause.

Using this approach, as of this afternoon we have now been able to reproduce the crash within the development environment. This marks a major step forward, and we are closing in on the root cause.

We would like to thank everyone who has been submitting crash logs and reports during this process. Your contributions have been an invaluable resource in our investigation. We’re confident that once we’ve confirmed the exact issue, we’ll be able to prepare and deliver a fix as soon as possible. Your support means a great deal to me and to the wider development team.”

END OF WED 22 OCT UPDATE