Star Citizen – Bad Server Code
Welcome to some more Star Citizen, looking at whats to be done about Star Citizen’s Bad Servers, Server Meshing and a Dev Response with a spectrum thread How can server meshing etc be built upon server code this bad? Started by Dreamy:
“Hello,
I think we all know pretty well how bad the server code currently is. Only two keywords are necessary to summarize the situation: desync and 30k.
Server meshing and item cache (required for full persistence) were announced by CR during his 2018 CititzenCon keynote as the key milestones on the “road to release”.
But how can you technically implement those technologies on top of such bad server code?
Is that the reason why we’ve been getting no news on the progress of those milestone technologies?
Or will server meshing actually improve the situation by decreasing the load on the individual overloaded servers?
What can be done and what is currently being done or planned to improve the server code issues?
Thank you.”
CI’s Lead Network Programmer Clive Johnson responded:
Building Server Meshing on top of unstable servers or where players experience desync is not ideal but is the only realistic way to do it.
I have previously posted about what causes server crashes (most often the cause of the infamous 30000 disconnection error) and what we are doing to address them.
Server instability could be even more detrimental for Server Meshing than it currently is for separate instances simply because a server crash could affect more clients than are connected to a single server. We’re planning to address this in a couple of ways. First, Server Meshing will try to minimise the connections between servers, breaking up the universe into isolated islands of simulation, so that when a server in one “island” crashes it cannot affect those servers and clients in other islands. Second, we’ll be adding server crash recovery so when a server does crash a new server will spin up and restore the state of the crashed server from persistence (clients of the crashed server will probably still be 30k’d back to the main menu but then given an option to join the new server). Both of the above are big reasons why full persistence and S-OCS are required before Server Meshing can work. Of course, we’ll also continue fixing server crashes as quickly as we can once they have been discovered.
Desync is often used to describe a wide range of issues including stuttering, teleporting, rubber banding, high latency, as well as true desync, where objects are in completely different states on different clients and won’t recover. Different symptoms have their own causes and without the correct information it is often difficult to reproduce these bugs internally. When reporting desync issues to the Issue Council please be as specific as possible as to what it is you have actually seen and provide videos where you can (preferably with the r_displaySessionInfo = 1 QR code enabled). What we do know, however, are the major causes for these different types of desync. “True desyncs” are usually caused by a logic error in some piece of code and the only way to fix them is to reproduce the bug and track down where the logic error is occurring. Stuttering, teleporting and rubber banding, are most often caused by low server frame rate or by sudden frame spikes (a frame that takes a lot longer to process than the frames immediately before or after it) on a client or the server. The best way to fix these is optimising the code to improve average frame rate or eliminate the spikes. But, as part of our work on Server Meshing, we’ve been working on ways to make the network code more resilient to low frame rates and frame spikes. My team (network) also has tasks to fix the spikes occurring in our code (but there are also spikes in other systems that their teams will need to address). High latency can be caused by low server frame rates and high bandwidth between clients and server. Additionally some frame spikes can cause high bandwidth (and therefore latency) or can be caused by them. We do have tasks to optimise bandwidth and look at other techniques for reducing or hiding latency. Some are on other teams and will be rolled out gradually according to their schedules. The ones on my team are currently scheduled for after we’ve got Server Meshing working but, as with everything in game development, plans may change and we’ll continue to monitor the situation.
He went on to explain & summarize:
Server Meshing itself should of course help with desync issues caused by server performance since each server will be focused on only a relatively small part of the overall simulation, spreading the load out over multiple servers. It’s not a silver bullet though and we’ll still need to optimise the code.
Beyond what Clive Said we do know that CI are working on a multitude of Server Tech and More robust code for the game… with plans to both increase player caps in servers, have better more stable servers and have a more enjoyable less janky pvp / multi-crew experience. A lot of work has gone in to getting some of that done by the end of they year OR at least that’s the target… tho we are not going to see Server Meshing until next year according to CI.