|
|
Post by DaHart85 on May 10, 2016 15:43:27 GMT -5
Good Afternoon All,
There was a catastrophic hardware failure on the devices that held the game up. In talking among the admins, we have some ideas to try and prevent/minimize this in the future, but to get us back up now will be mostly on AP. With our setup being limited on backup capabilities, restoration has become more of a rebuild this time and AP is working on that even though it's mid work-week, so it should be back online within the next couple days.
There may be an issue with the rent files as he works the rebuild, so when/if you run into any issues after the game comes back online, your points of contact will be Sand/Despair/Ego for getting things fixed. We'll make a point to be even more available than we normally are so we can get you fixed up and running again.
We appreciate your patience during this and the game will be back online as soon as it reasonably can be. There likely will be something for mortals in the way of a quest or something fun once it is back online. Stay tuned, will try and keep this string updated as we delve into it further, no post basically means nothing new to add since the last post. Have good evening. :-)
-Sand, DEMI
|
|
|
|
Post by DaHart85 on May 11, 2016 19:24:49 GMT -5
The game is expected to be up this evening. It looks like there will be a loss of the rent file since about January, please talk to the aforementioned DEMI+ so that we can get you straightened out. Your honesty and cooperation will be quite helpful as we work on getting everyone operational. Talk to us about remorts/gear loads/etc. in that timeframe.
-Sand
|
|
|
|
Post by syllix on May 11, 2016 21:21:06 GMT -5
Wow. That is going to be an insane amount of memory. Wish i kept better logs right now.
|
|
|
|
Post by DaHart85 on May 11, 2016 21:59:39 GMT -5
We'll get it sorted. Got some things we're going to work on once it's back online.
Quick update: It should be up tomorrow.
|
|
otep
New Member
Posts: 6
|
Post by otep on May 12, 2016 12:34:18 GMT -5
GOD DAMN... Well thanks of the upgrades... But At this point so many players rely on the "upgraded" rentfiles these days and dont plan for "catastophic failures" got a feeling were all going to lose a lot of poop because we didnt keep better lists/files as players. That is on us, of course. As a player though im sick to death of losing gear for ANY reason esp ones outside of my control. Is there any way we can get something going so the players can keep better track of their resources if the mud goes down and starts eating stuff? I have a feeling im going to lose a LOT of stuff with little in the way of knowing whats gone, and thats going to make me not want to play once i start realizing just what is missing 2 months down the road.
At this point i would advise anyone seriously playing the game to make in depth lists via google sheets to record ALL chars with stats remorts and gear/all excess gear... Seems like thats all we can really do to insure our eq in the event of "catastrophic failures."
|
|
otep
New Member
Posts: 6
|
Post by otep on May 12, 2016 12:34:51 GMT -5
just becoming too difficult to keep track of gear we have vs what we remember having in the past.
|
|
otep
New Member
Posts: 6
|
Post by otep on May 12, 2016 12:36:56 GMT -5
oh and can you uncensor the proboards since were all adults here? because the work poop i find more offensive than sh*t. time to come to grips with life and admit we mudders are old no need to protect the younger crowd. No more over caffeneited college kids here...
|
|
|
|
Post by syllix on May 12, 2016 13:01:44 GMT -5
Otep -
think this will reset you to before your DT?
|
|
|
|
Post by DaHart85 on May 12, 2016 15:08:25 GMT -5
Haha, I can't say I've run into the language filter issue. At the moment, let's cross our fingers. AP is trying to recover the rent data if we can but typically this type of data recovery costs ~$1500 an hour for a reason. So we may get lucky yet, cross your fingers, rub the rabbit's foot, etc. We've come up with a few ideas of how to try and prevent this from occurring again and we're looking into that. With a limited budget, time, and staff, having a perfect system is not possible so we consistently have to do a "best" approach, but we think we can still be setup better than we were and lean into another strategy from this. We're going to try another method to help with recovery and prevent the loss of data from occurring. Not what happened here, but an example of what I mean. I can have things spread across 6 servers in 3 different racks on 3 different power supplies in 3 different parts of the building with a generator backup. If the building burns to the ground or someone cuts the fiber line outside, none of that matters.  Can't promise something won't happen again, obviously, but think we have a few ideas that could severely aid in improving the recovery process and make it a bit more seamless and easier on both players and admins. |
|
|
|
Post by sunman on May 12, 2016 17:41:02 GMT -5
KISS strategy? Weekly zipped encrypted rent files upload to google drive?
Or since you are already collecting our email accounts and selling them to ashley madison, why not an option to email us our latest rent file?
Just tossing free ideas to account for on site catastrophic failures.
Sonny
|
|
|
|
Post by Austinpowers on May 12, 2016 18:27:49 GMT -5
Hi all...
Despair and Sand have been doing a great job keeping everyone abreast.
Long story short. We were having a lot of issues with memory allocation and I saw some raid controller issues. I tried my damnest to head it off, but when I moved the backup server we ran out of space. At that point the backup server crashed, than the raid controller than the game server. Luckily I had the dev server on another controller. I could recover back to Jan. 28th.
I have lost a lot of sleep, and money to get this fixed. I have put another solution into play. I put my Intel box up and added 16G more memory to 64G. Received my 3 new raid controllers, one was not compatible with VMware.
The backup server is now on another box, and its own controller. I have put the jedi server and Dev server on the INTEL server on separate controllers.
I have set up the backups to be direct connect. We will now do weekly Git backups off site. I am working to set that up with SilverKing. I have learned from this, and with the money I have invested in this It hurts when this happens.
I about cried when I saw it all happen. I thought today i could pull the miracle and pull the data of the backup server, but alas it was just another slap in the face. I could see the files in lost+found but since its a backup server, they are symbolically linked and i just could not follow it through the 500,000+ directories i saw.
All I can say is I am so sorry this happened. I will have us back up tomorrow, before I work. It is hard to put a 70+ hour work week, followed by another 70+ and have your Girlfriend going in for major surgery in early June.
Again I am very sorry.
AP
|
|
Sati
New Member
Posts: 11
|
Post by Sati on May 12, 2016 20:32:44 GMT -5
Dear Austinpowers,
I hope your gf's surgery goes well. And honestly, we as players (myself especially) should do a better job of keeping up with our own crap. Luckily(?), I haven't been around much the past 4 or 5 months because my kids are all insane; I can't imagine where they get it from. Anyway, we all know that you are doing about a million times more for the good of this game than ANYONE ever has, and believe me, we are all very appreciative. Thank you!
Dear Sunman,
Stop blaming Jedi for your Ashley Madison indiscretions.
See you guys soon!
-Amanda =)
|
|
lechem
New Member
I am the poster child for "No texting while walking."
Posts: 17 
|
Post by lechem on May 14, 2016 8:31:50 GMT -5
|
|
|
|
Post by Austinpowers on May 14, 2016 12:20:28 GMT -5
KISS strategy? Weekly zipped encrypted rent files upload to google drive? Or since you are already collecting our email accounts and selling them to ashley madison, why not an option to email us our latest rent file? Just tossing free ideas to account for on site catastrophic failures. Sonny Sonny I now have a backup strategy with GitHUB that is private. We have automated this so that if a change to a file in the jedimud directory changes it is automatically updated to the repository. |
|
|
|
Post by Austinpowers on May 14, 2016 12:22:15 GMT -5
Dear Austinpowers, I hope your gf's surgery goes well. And honestly, we as players (myself especially) should do a better job of keeping up with our own crap. Luckily(?), I haven't been around much the past 4 or 5 months because my kids are all insane; I can't imagine where they get it from. Anyway, we all know that you are doing about a million times more for the good of this game than ANYONE ever has, and believe me, we are all very appreciative. Thank you! Dear Sunman, Stop blaming Jedi for your Ashley Madison indiscretions. See you guys soon! -Amanda =) As crazy as I may sound and look, this is not a half bad idea. I am working on a weekly sql pull that will send you: The characters specific stats you own, and your rent file. I just do not know if I can do it all as one file or do i have to do multiple emails. I will work on this. |
|
|
|
Post by Austinpowers on May 14, 2016 19:01:05 GMT -5
we are back up. Please please work with Sand/Despair/Ego for character or rent assistance. I am wipped out physically and mentally from this week.
|
|
