SQL Server Trouble

[weasel]

Hey everyone,

Our last server upgrade went well and our web server is now top-notch without any problems. We can easily survive a sudden spike of traffic without the game slowing down!

But that's just the front end server - the one that feeds you the website and the game and the forums. This server asks our database server for what information to display... And the database server has been hiding under the sheets crying for the last few weeks. It simply can't handle the stress.

There are likely two problems at fault here:

- The code was originally written and optimized for a low-traffic game, and does not scale well to larger scale levels of traffic like we have now. SQL Optimization is something that there are specialists for, and if we end up needing to rewrite all of our queries and reconfigure the server, we will make sure we do whatever it takes to get this working right.
- The box the SQL queries were running against was misreporting it's activity levels. We would see figures that looked like it was not loaded too heavily but in reality it was bursting at the seams. We now know that with the current code and the current levels of traffic, we need a beefier SQL server as well (though this can be mitigated by making the above changes).

Whatever the specific problem ends up being, we are working hard to find a solution to get the game up and running at full speed again.

As an aside, the day the game was out (Friday/Saturday) was because of a power issue at the hosting facility. They were rewiring the main power lines that entered the building and I believe there was a problem with their backup generators. Many sites went down along with our own.

On top of all this, it's "Christmas Vacation" time for myself, Pacifist, our team at inXile, and at our hosting facility. Co-ordinating speedy repairs has been a bit of a nightmare and communication isn't at it's best. It looks like everyone is going to be back in the office - together! - around January 5th (today), so we'll probably see some definitive action in this coming week.

Thanks for being so patient with this, our first very successful indie game.

[CannedFruit]

Well I can say at my end FC either works very well or not at all, which is a change from when it used to run consistently but badly!

Right now It's working speedily but then I'm on GMT time so it's about 2am In America, which might explain the the speed!

Anyway, that's my say (and I'm not complaining or anything, the game is good enough by far to make up for this in my eyes!)

[dazzellerr]

thanks for the heads up Weasel. i think i speak for everyone when i say thank you and the team for what they are doing. damn game is so addicting, i find myself not having time to work on my racecar. ;(

[drnull]

Is there any way the applet could be coded to NOT lock up when the save fails? Just a simple connection timeout or something? Then pop up a message saying "save failed, please try again later". I can't even begin to tell how many designs I have lost thanks to that.

[a.r.l]

sometime me when i want go see my latest contraption, the my "design" bubble just dont load
It do the same thing to a other computer

do something

[flipmstr2]

I hate to complain, but I have been doing my part by not adding to the traffic for the past few months. Since the FCResource went away, and the server crawling to all but a complete halt I have given up on this for a while. I like a good challenge and this game is great for that, however, I don't enjoy the challenge of hoping the game will run when I log on. I came back hoping the new year would see things moving faster, unfortunately things still haven't seemed to have improved. Sorry for venting, I guess I just miss the game.

[weasel]

Mike (our server guy at inXile) has made a bunch of changes to the SQL configuration and the server appears to be running much better now. We are hoping this lasts but will be monitoring it over the next few days to see what happens.

I know the interface is kind of... indie, but the save screen isn't "locked up"... You can always hit the "save" button again if it is taking more than 30 seconds or so. I've had a contraption half built on my laptop not save, but I left the window open and put my laptop into sleep mode.. walked around town for a day.. popped into a cafe, got online, then hit "save" again and it worked.

[agenttrotter]

I, like many people here, get FURIOUS when my favorite game, Fantastic Contraption, won't load. But it seems weasel and the gang are doing everything they can- after all, when the servers mess up- they're loosing money. Our frustration over loosing designs and gameplay is NOTHING compared to the frustration of loosing money- through lost ad revenue and a lack of subscribers.

I'm kind of confused, though- as to how they still have problems. I mean- there are SO MANY people on here- each one represents . Only rarely do you see regulars like minisimp, phil, mr contraption, klli, etc. Most people seem like random noobs- and there is a endless supply of them. I don't know how much servers cost nowadays, but damn- they must be made of gold. Not to mention all those pop-up ads and banners...

Does the FC staff have an expensive drug habit we don't know about?

But all joking aside...

...with this, our first very successful indie game.

Why does it have to be "indie"? Have you ever considered selling the whole thing to a company with better servers? Are there companies that buy games like this? What about Sierra. They made "the incredible machine"...

[Afronanny]

I was wonder what was up the other day when FC wouldn't even show the forums ("can't find the server") for a whole day. I was entering withdrawal! But all joking aside...lol

[drnull]

I know the interface is kind of... indie, but the save screen isn't "locked up"... You can always hit the "save" button again if it is taking more than 30 seconds or so. I've had a contraption half built on my laptop not save, but I left the window open and put my laptop into sleep mode.. walked around town for a day.. popped into a cafe, got online, then hit "save" again and it worked.

Wow, that's great to know, thanks! I'll keep that in mind next time it looks "locked up". I have tried doing this in the past, though, and never seen it work, but I wasn't very persistent, thinking that it was locked up.

[weasel]

Why does it have to be "indie"? Have you ever considered selling the whole thing to a company with better servers? Are there companies that buy games like this? What about Sierra. They made "the incredible machine"...

good observation, and you are pretty close to the truth!

The game would be dead now if we didn't get a boatload of help from inXile. They now own the rights to the game and moved it to their superfast servers. The problem isn't in the server though, so much as it's in our code - the database queries in the code probably have some flaws in them that naturally use up much more server time than they should, and that's what we're looking into to solve problems in the future.

[dazzellerr]

get rid of the clouds in the background of the game perhaps? a little less "load" on the game itself trying to run that particular code. i imagine that is why there is a limit of pieces you can use, so someone does not use 1 billion pieces and crash the entire intreweb lol. seems that the simpler the graphics, the faster the program will run. ?-?

[weasel]

There is some truth to that too - if the graphics are simplified, it will run faster. Even on my speedy machine, a large contraption will bog down if I don't click the "simple graphics" button.

However!

There are around 3 different performance factors:

1. Local game speed. This game speed is purely determined by the speed of your computer (particularly your processor), how many active objects are on the screen (and if they are animated or not - the simple graphics button fixes this), and how fast Flash can draw things to the screen. Not much we can do about the last one. ANY flash application with hundreds of objects, even if perfectly coded, will end up going a bit slower on even modern machines.

2. Web Server Speed. This speed is how long it takes the HTML page to appear on your screen when loading the game or the FCResource. If you hit F5, the speed of this server determines how long until your page refreshes. This was never particularly a problem for us.

3. Database Speed. The web server requests data from the database - when loading or saving levels, showing you a list of levels, or when using the FCResource. This is where our problem lay. If the database server is going very slow, it takes a long time for the Web Server (#2) to get back to you, and in turn makes your flash application (#1) appear to be going very slow.

As you can see it's a bit of an intricate network we have going, but the cornerstone to it all is the Database Server. Without the data coming out of it at a timely speed, everything else is caught tapping it's fingers and waiting for things to load.

We've made a database configuration change yesterday and things seem to be a thousand times better than last week. We are monitoring the performance and we have noted a few saving timeouts still; we have a few other SQL configuration changes planned which will hopefully smooth out all these other problems.

[~Jon~]

Off topic: Haha Agenttrotter I have The incredible machine made by Sierra, i noticed the similarity right when i first saw FC ^_^

On topic: Well I think that FC is in good hands and should be fixed soon. Hopefully in the next year ill have FC on my Nintendo Wii!

[VERYDUMBNAME]

I'm kind of confused, though- as to how they still have problems. I mean- there are SO MANY people on here- each one represents .

incorrect less than 1% pay

[radar12564]

My computer is running slow! I can't find any problems!!!

[jofjltn4]

Down again? Stuck on saving screen.

(Tweaked out water-only champ solve...I'm sure I can just throw that together again.)

:angry:

J

[thebrain*]

FC always run smooth on the normal levels but on player levels and when saving its pathetic! The game is wonderful but the slowness is mean to me

[weasel]

Slowness should be gone, we are still tweaking the server though so there is the occasional minute or two outage while we restart things. Shouldn't be more than once a day though.

[squeegee]

Slowness should be gone, we are still tweaking the server though so there is the occasional minute or two outage while we restart things. Shouldn't be more than once a day though.

Is that the only explanation we will see for FC being down from 12/11 through 12/13? Really? You can't do better?

Can you at the very least acknowledge that the site was down for that period and, you know, be sorry and all?