SQL Server Trouble

News updates, status reports, and version information from the FC Team

SQL Server Trouble

Postby weasel on Mon Jan 05, 2009 2:03 am

Hey everyone,

Our last server upgrade went well and our web server is now top-notch without any problems. We can easily survive a sudden spike of traffic without the game slowing down!

But that's just the front end server - the one that feeds you the website and the game and the forums. This server asks our database server for what information to display... And the database server has been hiding under the sheets crying for the last few weeks. It simply can't handle the stress.

There are likely two problems at fault here:

- The code was originally written and optimized for a low-traffic game, and does not scale well to larger scale levels of traffic like we have now. SQL Optimization is something that there are specialists for, and if we end up needing to rewrite all of our queries and reconfigure the server, we will make sure we do whatever it takes to get this working right.
- The box the SQL queries were running against was misreporting it's activity levels. We would see figures that looked like it was not loaded too heavily but in reality it was bursting at the seams. We now know that with the current code and the current levels of traffic, we need a beefier SQL server as well (though this can be mitigated by making the above changes).

Whatever the specific problem ends up being, we are working hard to find a solution to get the game up and running at full speed again.

As an aside, the day the game was out (Friday/Saturday) was because of a power issue at the hosting facility. They were rewiring the main power lines that entered the building and I believe there was a problem with their backup generators. Many sites went down along with our own.

On top of all this, it's "Christmas Vacation" time for myself, Pacifist, our team at inXile, and at our hosting facility. Co-ordinating speedy repairs has been a bit of a nightmare and communication isn't at it's best. It looks like everyone is going to be back in the office - together! - around January 5th (today), so we'll probably see some definitive action in this coming week.

Thanks for being so patient with this, our first very successful indie game. :)
User avatar
weasel
 
Posts: 2469
Joined: Sat Jul 05, 2008 4:03 pm

Re: SQL Server Trouble

Postby CannedFruit on Mon Jan 05, 2009 2:39 am

Well I can say at my end FC either works very well or not at all, which is a change from when it used to run consistently but badly!

Right now It's working speedily but then I'm on GMT time so it's about 2am In America, which might explain the the speed!

Anyway, that's my say (and I'm not complaining or anything, the game is good enough by far to make up for this in my eyes!)
CannedFruit
 
Posts: 195
Joined: Wed Sep 10, 2008 9:52 am
Location: London, England

Re: SQL Server Trouble

Postby dazzellerr on Mon Jan 05, 2009 11:24 am

thanks for the heads up Weasel. i think i speak for everyone when i say thank you and the team for what they are doing. damn game is so addicting, i find myself not having time to work on my racecar. ;(
dazzellerr
 
Posts: 134
Joined: Sat Nov 22, 2008 4:38 pm

Re: SQL Server Trouble

Postby drnull on Mon Jan 05, 2009 2:06 pm

Is there any way the applet could be coded to NOT lock up when the save fails? Just a simple connection timeout or something? Then pop up a message saying "save failed, please try again later". I can't even begin to tell how many :science: designs I have lost thanks to that. ;)
drnull
 
Posts: 159
Joined: Fri Oct 03, 2008 1:05 pm

Re: SQL Server Trouble

Postby a.r.l on Mon Jan 05, 2009 2:23 pm

sometime me when i want go see my latest contraption, the my "design" bubble just dont load
It do the same thing to a other computer

do something
User avatar
a.r.l
 
Posts: 16
Joined: Fri Sep 19, 2008 3:36 pm

Re: SQL Server Trouble

Postby flipmstr2 on Mon Jan 05, 2009 2:26 pm

I hate to complain, but I have been doing my part by not adding to the traffic for the past few months. Since the FCResource went away, and the server crawling to all but a complete halt I have given up on this for a while. I like a good challenge and this game is great for that, however, I don't enjoy the challenge of hoping the game will run when I log on. I came back hoping the new year would see things moving faster, unfortunately things still haven't seemed to have improved. Sorry for venting, I guess I just miss the game.
flipmstr2
 
Posts: 1
Joined: Thu Aug 21, 2008 10:39 pm

Re: SQL Server Trouble

Postby weasel on Mon Jan 05, 2009 5:02 pm

Mike (our server guy at inXile) has made a bunch of changes to the SQL configuration and the server appears to be running much better now. We are hoping this lasts but will be monitoring it over the next few days to see what happens.

I know the interface is kind of... indie, but the save screen isn't "locked up"... You can always hit the "save" button again if it is taking more than 30 seconds or so. I've had a contraption half built on my laptop not save, but I left the window open and put my laptop into sleep mode.. walked around town for a day.. popped into a cafe, got online, then hit "save" again and it worked.
User avatar
weasel
 
Posts: 2469
Joined: Sat Jul 05, 2008 4:03 pm

Re: SQL Server Trouble

Postby agenttrotter on Mon Jan 05, 2009 5:03 pm

I, like many people here, get FURIOUS when my favorite game, Fantastic Contraption, won't load. But it seems weasel and the gang are doing everything they can- after all, when the servers mess up- they're loosing money. Our frustration over loosing designs and gameplay is NOTHING compared to the frustration of loosing money- through lost ad revenue and a lack of subscribers.

I'm kind of confused, though- as to how they still have problems. I mean- there are SO MANY people on here- each one represents :10bux: . Only rarely do you see regulars like minisimp, phil, mr contraption, klli, etc. Most people seem like random noobs- and there is a endless supply of them. I don't know how much servers cost nowadays, but damn- they must be made of gold. Not to mention all those pop-up ads and banners...

Does the FC staff have an expensive drug habit we don't know about? :)

But all joking aside...

...with this, our first very successful indie game.


Why does it have to be "indie"? Have you ever considered selling the whole thing to a company with better servers? Are there companies that buy games like this? What about Sierra. They made "the incredible machine"...
User avatar
agenttrotter
 
Posts: 158
Joined: Fri Oct 17, 2008 6:43 pm
Location: kevinquinnyo

Re: SQL Server Trouble

Postby Afronanny on Mon Jan 05, 2009 5:53 pm

I was wonder what was up the other day when FC wouldn't even show the forums ("can't find the server") for a whole day. I was entering withdrawal! But all joking aside...lol
User avatar
Afronanny
OUTRAGED!!!1
 
Posts: 1625
Joined: Thu Sep 18, 2008 5:32 pm
Location: Eating your pie. Don't bother to stop me, I'm finished now.

Re: SQL Server Trouble

Postby drnull on Mon Jan 05, 2009 7:31 pm

weasel wrote:I know the interface is kind of... indie, but the save screen isn't "locked up"... You can always hit the "save" button again if it is taking more than 30 seconds or so. I've had a contraption half built on my laptop not save, but I left the window open and put my laptop into sleep mode.. walked around town for a day.. popped into a cafe, got online, then hit "save" again and it worked.

Wow, that's great to know, thanks! I'll keep that in mind next time it looks "locked up". I have tried doing this in the past, though, and never seen it work, but I wasn't very persistent, thinking that it was locked up.
drnull
 
Posts: 159
Joined: Fri Oct 03, 2008 1:05 pm

Re: SQL Server Trouble

Postby weasel on Mon Jan 05, 2009 7:50 pm

agenttrotter wrote:Why does it have to be "indie"? Have you ever considered selling the whole thing to a company with better servers? Are there companies that buy games like this? What about Sierra. They made "the incredible machine"...


good observation, and you are pretty close to the truth!

The game would be dead now if we didn't get a boatload of help from inXile. They now own the rights to the game and moved it to their superfast servers. The problem isn't in the server though, so much as it's in our code - the database queries in the code probably have some flaws in them that naturally use up much more server time than they should, and that's what we're looking into to solve problems in the future.
User avatar
weasel
 
Posts: 2469
Joined: Sat Jul 05, 2008 4:03 pm

Re: SQL Server Trouble

Postby dazzellerr on Mon Jan 05, 2009 9:34 pm

get rid of the clouds in the background of the game perhaps? a little less "load" on the game itself trying to run that particular code. i imagine that is why there is a limit of pieces you can use, so someone does not use 1 billion pieces and crash the entire intreweb lol. seems that the simpler the graphics, the faster the program will run. ?-?
dazzellerr
 
Posts: 134
Joined: Sat Nov 22, 2008 4:38 pm

Re: SQL Server Trouble

Postby weasel on Tue Jan 06, 2009 1:10 pm

There is some truth to that too - if the graphics are simplified, it will run faster. Even on my speedy machine, a large contraption will bog down if I don't click the "simple graphics" button.

However!

There are around 3 different performance factors:

1. Local game speed. This game speed is purely determined by the speed of your computer (particularly your processor), how many active objects are on the screen (and if they are animated or not - the simple graphics button fixes this), and how fast Flash can draw things to the screen. Not much we can do about the last one. ANY flash application with hundreds of objects, even if perfectly coded, will end up going a bit slower on even modern machines.

2. Web Server Speed. This speed is how long it takes the HTML page to appear on your screen when loading the game or the FCResource. If you hit F5, the speed of this server determines how long until your page refreshes. This was never particularly a problem for us.

3. Database Speed. The web server requests data from the database - when loading or saving levels, showing you a list of levels, or when using the FCResource. This is where our problem lay. If the database server is going very slow, it takes a long time for the Web Server (#2) to get back to you, and in turn makes your flash application (#1) appear to be going very slow.

As you can see it's a bit of an intricate network we have going, but the cornerstone to it all is the Database Server. Without the data coming out of it at a timely speed, everything else is caught tapping it's fingers and waiting for things to load.

We've made a database configuration change yesterday and things seem to be a thousand times better than last week. We are monitoring the performance and we have noted a few saving timeouts still; we have a few other SQL configuration changes planned which will hopefully smooth out all these other problems.
User avatar
weasel
 
Posts: 2469
Joined: Sat Jul 05, 2008 4:03 pm

Re: SQL Server Trouble

Postby ~Jon~ on Tue Jan 06, 2009 6:45 pm

Off topic: Haha Agenttrotter I have The incredible machine made by Sierra, i noticed the similarity right when i first saw FC ^_^

On topic: Well I think that FC is in good hands and should be fixed soon. Hopefully in the next year ill have FC on my Nintendo Wii!
Last edited by ~Jon~ on Wed Jan 14, 2009 7:30 pm, edited 1 time in total.
User avatar
~Jon~
 
Posts: 1151
Joined: Mon Oct 27, 2008 4:41 pm
Location: Gtalk me @ bluej75@gmail.com

Re: SQL Server Trouble

Postby VERYDUMBNAME on Wed Jan 07, 2009 11:56 am

agenttrotter wrote:I'm kind of confused, though- as to how they still have problems. I mean- there are SO MANY people on here- each one represents :10bux: .

incorrect less than 1% pay
User avatar
VERYDUMBNAME
 
Posts: 1503
Joined: Fri Oct 24, 2008 12:52 pm
Location: uʍop ǝpısdn sı uǝǝɹɔs ɹnoʎ sıɥʇ pɐǝɹ uɐɔ noʎ ɟı

Re: SQL Server Trouble

Postby radar12564 on Fri Jan 09, 2009 3:47 pm

My computer is running slow! :( I can't find any problems!!! :canada:
User avatar
radar12564
 
Posts: 55
Joined: Mon Dec 22, 2008 2:19 pm
Location: penny lane next to lucy

Re: SQL Server Trouble

Postby jofjltn4 on Sun Jan 11, 2009 3:48 pm

Down again? Stuck on saving screen.

(Tweaked out water-only champ solve...I'm sure I can just throw that together again.)

:angry:

J
jofjltn4
 
Posts: 951
Joined: Wed Sep 17, 2008 10:51 pm
Location: I'm too stupid to figure any of this out...

Re: SQL Server Trouble

Postby thebrain* on Mon Jan 12, 2009 1:26 pm

FC always run smooth on the normal levels but on player levels and when saving its pathetic! The game is wonderful but the slowness is mean to me :sigh:
User avatar
thebrain*
 
Posts: 161
Joined: Fri Jan 09, 2009 7:23 pm
Location: God bless America!

Re: SQL Server Trouble

Postby weasel on Mon Jan 12, 2009 3:02 pm

Slowness should be gone, we are still tweaking the server though so there is the occasional minute or two outage while we restart things. Shouldn't be more than once a day though.
User avatar
weasel
 
Posts: 2469
Joined: Sat Jul 05, 2008 4:03 pm

Re: SQL Server Trouble

Postby squeegee on Wed Jan 14, 2009 10:33 pm

Slowness should be gone, we are still tweaking the server though so there is the occasional minute or two outage while we restart things. Shouldn't be more than once a day though.
Is that the only explanation we will see for FC being down from 12/11 through 12/13? Really? You can't do better?

Can you at the very least acknowledge that the site was down for that period and, you know, be sorry and all?
squeegee
 
Posts: 6
Joined: Mon Jul 21, 2008 4:42 pm

Next

Return to The Dev Dish



Who is online

Users browsing this forum: No registered users