| World of KJ https://www.worldofkj.com/forum/ |
|
| DB Issue https://www.worldofkj.com/forum/viewtopic.php?f=3&t=53268 |
Page 1 of 1 |
| Author: | Eagle [ Fri Aug 28, 2009 6:24 am ] |
| Post subject: | DB Issue |
What a mess! Should be fixed now ... basically the posts table and it's 1.5 million records crashed, had an index error and wasn't recovering properly. Spent all yesterday working on it and a process I ran overnight seems to have fixed it, thank goodness. Sorry for the downtime. |
|
| Author: | Argos [ Fri Aug 28, 2009 6:50 am ] |
| Post subject: | Re: DB Issue |
Did torrino bring you coffee? |
|
| Author: | Bradley Witherberry [ Fri Aug 28, 2009 8:11 am ] |
| Post subject: | Re: DB Issue |
Sounds like sabotage. I wonder who has sufficient admin privileges to pull such a stunt... |
|
| Author: | Eagle [ Fri Aug 28, 2009 8:17 am ] |
| Post subject: | Re: DB Issue |
Nah, nothing like that. Looks like the DB crashed partly through an insert, and because the record was partly inserted, indexes were out of wack and everything went bonkers. |
|
| Author: | Jedi Master Carr [ Fri Aug 28, 2009 9:37 am ] |
| Post subject: | Re: DB Issue |
That really sucks, I really appreciate your work. I am sorry you had to spend so much time fixing it though. |
|
| Author: | Jiffy [ Fri Aug 28, 2009 10:22 am ] |
| Post subject: | Re: DB Issue |
Thanks for all your work to fix it! |
|
| Author: | dolcevita [ Fri Aug 28, 2009 10:53 am ] |
| Post subject: | Re: DB Issue |
Eagle wrote: Nah, nothing like that. Looks like the DB crashed partly through an insert, and because the record was partly inserted, indexes were out of wack and everything went bonkers. What the hell does that even mean?  Thanks for all the (unexpected) hours of working through partially inserted indexes and whatnot. |
|
| Author: | Eagle [ Fri Aug 28, 2009 11:12 am ] |
| Post subject: | Re: DB Issue |
Heh, I'll try and explain better for anyone interested ... basically ... each post has a unique number: Post 1, Post 2, Post 3, etc. When someone makes a post, the database will automatically give that post the next unique number. Unfortunatly, something happened in the middle of an insert, meaning a record was entered into the database as Post 4, but the database got amnesia and doesn't remember inserting it. Well, the next time someone tried to post, the database again tried to insert the new post as Post 4 and was like "Hey, who the hell are you! There's not supposed to be a Post 4 here!" That's the basic issue, and fixing that issue isn't as simple as you'd expect. Each time a post is made, the posts table is LOCKED temporarily, and UNLOCKED when the insert completes. Well, since it never finished inserting correctly, the table remained LOCKED, which causes all kinds of headaches. All attempts to unlock the table resulted in the fail whale. The only way to fix these two problems is a series of repairs on the database, but because the table is so huge (~1.5 million records), attempts to repair it were causing the HTTP service to crash. To get around that, I had to run the repair via command line, but part of the repair process is to build a temporary index file which was too big for the directory it builds in. Meaning it ran out of space and couldn't fully rebuild the database. To give the specific partition more space, I'd have to reformat the entire server, and I didn't want to do that. So I then decided to rebuild the index in memory, which was very SLOW, but seemed to be working. Unfortunatly, when that process finished, it threw an error saying the database was still marked as crashed, and rebuilding the index wasn't sufficient to repair it. Yeesh!!! So I had to rebuild not only the table indexes, but also the file holding all the data of the database. I kicked that process off before bed, it finished early this morning. Ta Da! Bet you wish you hadn't asked, heh. |
|
| Author: | torrino [ Fri Aug 28, 2009 11:13 am ] |
| Post subject: | Re: DB Issue |
Thanks Karl-izzle! |
|
| Author: | Riggs [ Fri Aug 28, 2009 11:14 am ] |
| Post subject: | Re: DB Issue |
Thanks, Eagle 
|
|
| Author: | Rev [ Fri Aug 28, 2009 11:40 am ] |
| Post subject: | Re: DB Issue |
Thx Karl  I luv this place n don't ever want it to die. |
|
| Author: | Libs [ Fri Aug 28, 2009 11:48 am ] |
| Post subject: | Re: DB Issue |
Yay it's back |
|
| Author: | Box [ Fri Aug 28, 2009 12:29 pm ] |
| Post subject: | Re: DB Issue |
Thanks for your hard work fixing this Eagle! |
|
| Author: | Excel [ Fri Aug 28, 2009 12:32 pm ] |
| Post subject: | Re: DB Issue |
Thanks brosef |
|
| Author: | Mr. Reynolds [ Fri Aug 28, 2009 12:48 pm ] |
| Post subject: | Re: DB Issue |
Muchas gracias! I missed my KJ lurking yesterday and with all the threads about the site disappearing I was a bit worried. great that it's back up. |
|
| Author: | Riggs [ Fri Aug 28, 2009 12:55 pm ] |
| Post subject: | Re: DB Issue |
Yeah I thought it's over now, I'm really glad it's not. |
|
| Author: | Passionate Thug [ Fri Aug 28, 2009 1:35 pm ] |
| Post subject: | Re: DB Issue |
Thanks Eagle! No matter how much guff you get sometimes from certain posters, I've always admired your hard work. |
|
| Author: | Riggs [ Fri Aug 28, 2009 2:13 pm ] |
| Post subject: | Re: DB Issue |
Passionate Thug wrote: Thanks Eagle! No matter how much guff you get sometimes from certain posters, I've always admired your hard work. Seconded. |
|
| Author: | Korrgan [ Fri Aug 28, 2009 3:29 pm ] |
| Post subject: | Re: DB Issue |
All lies. The truth of August 26th has been documented and made public knowledge. viewtopic.php?f=10&t=53274 Never forget. |
|
| Author: | Dr. Lecter [ Fri Aug 28, 2009 9:45 pm ] |
| Post subject: | Re: DB Issue |
Yeah, Karl, thanks a lot! |
|
| Author: | Argos [ Sat Aug 29, 2009 2:37 am ] |
| Post subject: | Re: DB Issue |
Thank you very much, Eagle. I will dedicate my first daughter to you. |
|
| Author: | STEVE ROGERS [ Sat Aug 29, 2009 9:33 pm ] |
| Post subject: | Re: DB Issue |
Eagle wrote: What a mess! Should be fixed now ... basically the posts table and it's 1.5 million records crashed, had an index error and wasn't recovering properly. Spent all yesterday working on it and a process I ran overnight seems to have fixed it, thank goodness. Sorry for the downtime. Hmmm.. Must've crashed from the AVATAR trailer Premiere and it's overwhelming popularity.. 
|
|
| Page 1 of 1 | All times are UTC - 5 hours [ DST ] |
| Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group http://www.phpbb.com/ |
|