When Mythtv Servers Go Belly Up
Oh, the irony. A little over a week ago I found myself thinking that I hadn’t written a blog post in quite a while and I realized that it was because my setup was running so smoothly. There was just nothing to write about because I hadn’t needed to do anything with it for quite a while. Little did I know, that was about to change.
It all started off like any other night. Things were running smoothly and it was time to sit down to a bit of TV watching. After thinking to myself how well things were going, I thought that I would log into my Mythtv server and see how long things had been running between reboots. This is when things got a little funky. First, I tried VNC to get a remote desktop: Connection Refused! This happens every now and then, no big deal. I figured that I’d just log in over SSH and relaunch the X-server. Unfortunately, that didn’t work either. Uh oh. That’s not normal. At this point, I was starting to feel a bit concerned. As casually as I could, I headed off to the server to see what was up. Normally, I would just hit the reset button and connect to it remotely again but this time I decided to pull out a keyboard and mouse for some direct access. Anyone want to guess what I saw? That’s right, garbage: Major screen corruption. OK, whatever, I could just reboot it and everything would be fine. That’s what I thought at first. After hitting the reboot everything appeared to be running fine. The system booted right up without a sweat. But, as soon as it got to the login screen, it shut off. What the…?! So, I hit the power button again and this time it lasted for about two seconds before shutting off. I hit it again and it lasted about one second. No good! Something was obviously wrong here.
So now it’s time to put on my computer debugging hat. What does it mean if something works for a bit and then later stops, and each time the duration of workingness (yes, I will assume that’s a word) gets shorter and shorter? I figured it was a thermal issue and let the system cool for a while. After about half an hour, I tried again. I hit the power button and the system ran for about ten seconds. I hit it again and it was back to one. Hmm… I felt I was on to something so I let it sit overnight. At this point I returned upstairs and was forced to utter the words that all of us dread: “Honey, the backend is down. And it might not be back up for a bit”. As you can imagine this did not go over well and I was inspired to get things working again as quickly as possible. I need not worry about sleeping…
I tried turning things on again the next morning after letting it sit all night thinking about what a bad computer it was being, but it did not help. It still would only power up for a few seconds and then shut down again. So, now it was time to start removing hardware to see if I could find the problem. First thing to go were the hard drives because they were easy to disconnect. That didn’t help. Next went the tuners: two PVR150s and a PVR350. No difference there either. Next, it was time to start pulling memory. That did not have any affect either. At this point, all I had left was the mother board, processor, power supply and a couple of fans. I took a power supply from another system and plugged it in to the same result: power for a second and then death. So, I knew it was either the motherboard or processor. Since I suspected a thermal issue from the start, I removed the CPU heatsink and reapplied the thermal grease. This may have helped cool things a bit because the grease that was there had gotten a bit dried out but it had no affect on the overall problem that I was seeing.
At this point, I was stumped so I decided to try another tactic. I pulled out my development system, plugged in all of the tuners, hard drives and the power supply from my Mythtv server and powered it up. Needless to say, everything started right up. Luckily, I was able to get this far just in time to start recording shows for the evening. It was a hack but it was running. And this is where things are sitting now. I’m thinking about buying some new components to replace what died on me so if you have suggestions, leave a comment below.
This happened to me twice.
This happened to me twice. both times it was “fixed” by changing the power supply. ultimately the mobo just died on me, went to sleep seemingly healthy and never awoke.
Agree with Joe – Had this
Agree with Joe – Had this happen with my HTPC server few months back. If I was lucky, it would login to Windows but reboot after few minutes. More often than not, it would keep rebooting without even getting to the login screen. Tried all the steps you have already listed and then was able to get few more months out of it by swapping the power supply. In the end however, it just stopped working (no POST, nothing). Swapped the motherboard (different brand at that too) and was back in business.
One way to get an idea if motherboard might be the issue is to look for bulging or blown capacitors. I got lot of good information/help from the folks/forums at badcaps.net – was able to actually fix an LCD monitor that had blown power supply capacitors with $5 new capacitors from digikey and help from folks at badcaps.net – but that’s a different story 🙂 Although I ordered replacement capacitors for the motherboard, havent had the time to replace those yet (lot more capacitors to replace than on the LCD monitor)
Good luck – it certainly is no fun being in this kind of situation!
-Topper
Blown caps are an extremely
Blown caps are an extremely common cause for LCD panels going bad. I have repaired at least half a dozen panels with blown caps on the power supply board. It is not surprising given the amount of heat generated by the panel and hanging around inside the panel.
It is possible that this could be a cause for motherboard instability though if it is old enough where the caps are starting to fail, it’s probably time to build a new system anyway.
‘”Honey, the backend is
‘”Honey, the backend is down. And it might not be back up for a bit”. As you can imagine this did not go over well and I was inspired to get things working again as quickly as possible. I need not worry about sleeping…’
Hahaha! Heard that one before. Best wishes getting it up and running again.
Just for my information, how long was the system operational before this happened?
I started the whole HTPC
I started the whole HTPC scene in 2004. I think I jumped into Mythtv in early 2005 and built this dedicated server in 2005 or 2006. I think I had this running stably for 5+ years.
Oh the pain! I had a similar
Oh the pain! I had a similar situation, but it was my power supply. I pulled everything, hoping and praying it wasn’t the motherboard – luckily I could borrow a PS from another machine. Under warranty still, too, and just before vacation, so very little apparent downtime.
Really makes you wonder – “Is now a good time for a stand-alone solid-state machine (Apple TV, Roku, etc..) that has LAN access to a shared folder or NAS?”
That’s the reason I picked up an Apple TV 2.0, jailbroke it, and put on XMBC. It’s VERY close to being usable, especially since I don’t watch any live TV and just use TED for everything. It just isn’t reliable enough yet.. maybe 3.0, but that’ll probably be un-jailbreakable, unfortunately…
Unfortunately for me it was
Unfortunately for me it was not a power supply issue. I have a number of them floating around and that was the first thing that I tried. I have not looked for blown capacitors yet although that is definitely something to look into. I have no problem soldering on new components (I’ve got nothing to lose at this point, right?). I’ve had problems in the past where a system goes to sleep and doesn’t wake up again. I had to RMA one motherboard twice for that problem (and actually that’s the same board that is currently running my server…)
I’ve always been of the
I’ve always been of the mind-set that 5 years is the maximum usable life for PC type computers of any sort. I set this milestone based on what I consider the maximum life of a good quality HDD. By that time Moore’s Law will have readily taken care of the rest of the hardware as well.
From the sound of things your server has served (no pun intended) you well. Maybe it’s time to set aside some coin to build a new and improved box. You know as well as I do that most of today’s lowest cost components are for the most part a significant notch above the performance level of five year old components. A home server doesn’t need cutting-edge components after all so a replacement should be reasonably affordable.
It’s not my place to suggest to where you open your wallet but a guide on building a Linux home server would be a nice addition to MR (hint, hint, nudge, nudge).
Joking aside I hope that all of your data is fully and readily recoverable and good luck with whatever direction you choose to go regarding this matter.
I am looking at getting some
I am looking at getting some new components to rebuild my server with but it is taking a bit of time to decide what I want and put together the funds for it. My current setup has served me well.
I would like to say that the hard drive that I’m using for my OS is a 8GB IDE drive from about 10 years ago. It was old when I built this server. Fortunately/unfortunately, it has been working very reliably for me. The only problem with it is that it’s loud. Because of that I’m actually working on migrating it to a slightly less old (5 years) 120GB SATA drive. It’s much quieter but I ran into an issue getting the boot sector copied correctly so it’ll take some manual intervention and perhaps a hammer to get it working properly.
I’m also happy to say that all of my data is safe. Nothing was lost in the crash in that respect. My raid array was a software array so no dependence on the hardware in the system, and my TV drives were just individual drives. Nothing crashed, nothing was lost. Big thumbs up there.
“Because of that I’m actually
“Because of that I’m actually working on migrating it to a slightly less old (5 years) 120GB SATA drive.”
That’s a bold move IMO. Storage is so cheap (relative to applicable financial demographics of course) that my suggestion would be to simply buy a new $30 HDD to act as the OS drive. Again I’m not your budget director but on a personal level I’ve had enough HDD issues develop in the past that I cannot help myself but warn against using old HDDs for any purpose that requires risk mitigation of some sort (OK so serving solely as an OS drive isn’t necessarily going to tube your data HDD…but I digress)…
Either way your data storage disks are still in tact and your media is safe…that’s the bottom line.
I’m happy to say that the
I’m happy to say that the migration went well in the end. There were a few gotchas that I ran into but in the end I have two working copies of my MythTV server OS. And I have to say that the “new” drive is MUCH quieter. I now have to stop and listen to make sure if the server is on. Before I could just casually walk past the room… I’ll probably do another blog post or short article detailing the process I went through.