 Back To News
UC 2 Server instability solved 26 December 2005
UC Game server 2
we have been encountering some server instability recently on uc game server 2. although players can login to play on other servers hwoever it is still not an excuse for us, therefore i believe i owe everyone an appology for this as well as to explain what actually happened.
we implemented a new backup system to allow cross server backups for more redundacy. there is a daily backup that copies the entire game over (a couple of gig) from our main gc server to the uc2 server. there isnt any notice-able lag on our main gc servers (cause its much later) compared to uc2 server therefore uc2 suffered slight lag.
this triggered our special anti-lag system to refresh the server, unfortunately due to a mistake on our side the anti-lag system configuration refreshed iis and not the game itself therefore if any lag occured on uc2 the game could never be refreshed.
server refresh & anti-lag system in detail (you can skip this)
how refresh occurs is, the game will be unloaded from the server memory (ram) to free up more ram. then the game will be reloaded into th server memory. this is because of how windows fuction (sad to say) that it does not automatically free up server memory automatically once it is not being in use. therefore after sometime the server memory used will grow and eventually the game will slow down. by righ this should occur within a 48 hours period that is why we have a auto-server refresh once every 24 hours. sometimes due to certain reasons if the server should lag the `anti-lag` system auto kicks in and refreshes the game server to speed things up (to avoid lag). now the problem is, when the server unloads the memory and dont loads it back up, the game can not start, causing a `hung` situation as the game is waiting for its memory to be loaded.
new backup system explained in detail (you can skip this as well)
we are in the process of implementing a new backup system. how our old backup system work is, all the most important data is backup once a day. non-important ones is backup once a month (bad!!). to improve things, backup will be done on an incremental system. this means data that changes very often (ex: planet systems, ministers, resources, etc) will be backup 2x a day, and non essential ones (ex: cheat tracking, balancing stats) will be backup once a week. this new backup system also allows us to recreate the entire game within a couple of minutes if should the main servers encounter any physical problems (ex: exploded, melted or something). i am a pretty paranoid person so its good to plain for the unforseeable and also to have an insurance `in case` right ? (this is all the stuff i`m doing in the back that no one sees.. heh)
Server upgrades
to accomodate our backup system, redudancy and to cater for gc 1.07 update, we will be getting 2 additional physical servers. after we have 100% implemented all of this in place, we would have a far greater redudancy system to ensure that at least 2-3 game servers is running at any particular time for both gc & uc. our current system (uc redundacy) only allows either uc1 & uc3 or uc2 (2 redundancy compared to 4 which we are planning). to cut a long story short, this will give us 100% uptime on our games unless if the datacenter is facing technical issues.
other reasons:
maintenance
this also makes it easier for us to do maintenance. its pretty hard for us to do server maintenace when we only have 2 redundancy, so if we switch off 1, this is bad cause the other doesnt have any backup if it should have any problems.
age
as of jan 2006, our main server would be 2.5 years old. there is no pcs (here) of that age that is still alive, all of it has died due to old age. somehow electronics (pc) does not last very long especially the el-cheapo ones we got. in the event they do pass away due to old age we want to be certain that we have at least something there.
redundancy
as explained earlier, we need more redundancy to ensure 100% uptime, so whenever you want to login to play you can be sure that you can login. our current redundancy: uc1 & uc3 ^^ uc2, we are planning for uc3 ^^ uc2 ^^ uc1 ^^ uc0. gc will join uc to support multiple server after the 1.07 update.
multiple-ver
in the event that any of our update will cause server instability or bugs, more servers means we can do multiple ver. we can update say uc4 first, if there is no problems after 1 week it goes to uc3 and so on. this will ensure if you have problems on a server that has the latest update you can switch to a server that has an older update. our current multi-ver are: uc2 -> uc1 & uc3. we are planning on uc3 -> uc2 -> uc1 & uc0
changes
o http://uc.gamestotal.com will be renamed uc game server `ver 0` or `stable version` from uc game server 3 (to avoid confusion)
o new uc game server 3 will be added. it will replace uc2 as the one that will contain the latest version/updates.
o gc will have the same configuration, gc0 gc1 gc2 gc4 after 1.07 update.
|
|
