This post follows up Fireproofing Pin*bot and Other Early System 11 Games, which was posted a few weeks back. In covering this fix, I will revisit to note that that Pinsider Inkochnito has now introduced a “Bridge Board” which he designed to solve the same problem that is normally corrected by installing two fuses in line with the +18 and +25 volt power supply regulated by two bridge rectifiers on System 3 thru 11 Williams games (and some Data East games too).
Inkochnito, who may be better known for making available faithfully reproduced apron cards for our classic games, has spearheaded the production of this board which can replace parts that already exist in the backbox. To use my Pin*Bot as an example, I would need to remove the fuse blocks I installed (obviously), the two mounted bridges and the large capacitor that sits near the bottom of the box, all indicated in the first picture of the gallery below. These parts are all “on-board” the new unit. Screw-down connectors are used to affix the wires (which I’m not all that fond of) so no soldering skills are required for installation and the board has two LED lights to alert you when exit power is present for the two line voltages. The bridges are mounted to the back of Inkochnito’s board so that they retain the same cooling properties from the metal backbox plate as the original bridges. Mounting the board is achieved using the original bridge mounting holes. Installation couldn’t be simpler. Installation notes can be found here (pdf).
What to remove
Installed, via Pinsider “Cheddar”
The “Add Two Fuses” fix has been the standard for years, and I don’t see much reason for my Pin*Bot to have a whole new board installed in it, even though the price is affordable at $49.00USD+$7.00USD economy shipping within the US. If I notice lights dimming (sign of a bad cap) or if my installed fuses blow (sign of a bad BR) perhaps then I would consider this board as a suitable replacement. Even at an affordable price, the game could be fixed with suitable replacement parts for much less, but as I understand, some of the Data East caps are pricy and hard to find, so in that case, it may be cheaper to use the board. Folks doing a complete overhaul or restoration of games from this period may want to jump straight to this solution, as it looks much cleaner and may save you a few steps down the road anyhow. Restoration of this era of game is normally done for the love of the game rather than profit (after dealing with a blown out playfield, new plastics, cabinet repaint, board fixes, etc. there really isn’t all that much meat left on the bone). What’s another $49.00 when you are doing it for the love of pinball in the first place?
The Inkochnito Bridge Board is available in the US from North American distributor Big Daddy Enterprises, and directly from its creator in Europe by emailing inkochnito (at) kpnplanet.nl.
I spoke of the annoying reset issue in a post earlier in the day, and promised a follow up interview with Rob Kahr, whose WPC MPU daughterboard designed to stifle resets has caused quite a stir in the community and has brought many troublesome WPC machines back online. When I first saw the daughterboard, I’ll admit, I dismissed it immediately, taking the stance that it was one step above a hack solution that just ignored a larger problem in the machine. After doing more research, and corresponding with Mr. Kahr, I was wrong to be dismissive. The daughterboard has the potential to be a powerful device to own (even more so if Mr. Kahr implements some possible enhancements, outlined in the interview). Perhaps one doesn’t belong in every WPC machine just yet, but it would be handy tool to use for troubleshooting reset issues or a temporary fix on league night when your machine craps out. I’d like to thank Mr. Kahr for being gracious enough to provide answers to my questions with absolutely no notice at all. That’s the Credit Dot way. Future interviewees be warned. Mr. Kahr’s daughterboards are available through his website, and can frequently be found on Pinside with user ID “rkahr”.
Credit Dot: I briefly discussed, in layman’s terms, HOW resets occur in the prior post. Maybe you could briefly give some information as to what your product does to solve or ease the reset issue?
Rob Kahr: Glad you covered “how” because there seem to be a million ways resets can occur and my daughterboard does not address all of them. For example, I had one guy [private message] me explaining how his transformer was out of spec – he had it re-wound and his resets were cured! Fantastic creativity. My board won’t fix grounding issues. It won’t magically deliver power if either the 18-volt or the 12-volt digital supplies are shot. It doesn’t fix battery damage on an MPU. It does offer relief to a very common situation where the game will play sometimes but reset when load is heavy or supply voltage drops. Note I dubbed the board the WPC MPU Power Fix Daughterboard–not the WPC Reset Fix Daughterboard.
My daughterboard specifically addresses power supply-caused resets and is designed based on my analysis of the linear 5 Volt supply on the Power Driver Board (PDB). The load on the power supply spans multiple boards and multiple functions on those boards. I measured the 5Volt load of the MPU board, DMD, and the sound board empirically via current measurements to determine individual board power use. I tabulated over 90 PDB connection points to the linear 5 Volt supply. I don’t have a Fliptronics board so I reviewed the schematic to determine it likely draws a small load compared to the other boards. Additionally the linear 5 Volt supply can be passed to the backbox and playfield via PDB jumpers J117 to J119. There is A LOT of demand on the linear 5 Volt power supply.
Anyone who has read about or worked to solve reset problems recognizes the many sources for weakness in the supply itself – what I call the foundation of the 5 Volt supply. Best maintenance practices are well documented in this realm and I’m not looking to discredit any of those maintenance strategies. Indeed, one needs a functioning linear 5 Volt supply in your WPC pinball whether or not my daughterboard is installed. Taking a system view, however, there are a myriad of permutations that weaken the overall ability to drive 5 Volts from the supply. Worse, issues that may not alone frequently drop voltage levels to trigger resets are additive so a combination of problems within the foundation and/or the load may contribute to the reset. And nothing is static in real-world operations. Power draws change through the cycle of a game; environmentals such as temperature fluctuate.
All of this means there is going to be some variability in the 5 Volt supply and at some point those variations will trigger resets via the watchdog circuit on the MPU board. [The watchdog chip] serves a valid purpose and should not be disabled. Its board (the MPU board), however, should be offered the cleanest power signal possible and in designing my daughterboard I set out to do exactly that.
The daughterboard literally removes the MPU from the PDB linear 5 Volt supply. All of the functions on the MPU are instead driven by a new, switching regulator supply included on the daughterboard. The watchdog circuit, as a 5 Volt function on the MPU continues to serve it’s role of protecting the ASIC, the processor, and the PROM chips by monitoring the new switching supply. Because (1) the new supply develops a very reliable, clean, temperature tolerant voltage and (2) the MPU load when isolated is more static, the watchdog is no longer driven to reset for off-MPU board fluctuations. Further, the reduced loading on the linear 5 Volt supply allows for more tolerance to imperfections in the foundations of the supply. Again, the linear supply needs to be maintained but it now has more capacity or “give”. No longer does the pinball community have to worry about every hundredth of a volt when conducting WPC repairs.
CD: What do you say to critics who dub your product as a “Band-Aid” solution? That it ignores a much larger problem by temporarily patching it up and ignoring it?
RK: I get that logic, but I really think I attacked this problem from a more holistic viewpoint. This little daughterboard fundamentally changes the calculus that is power distribution within WPC era games. Why do NASCAR drivers change tires mid-race? If they would just slow down they could get a lot more miles out of those tires! Obviously they are pushing the limits of the tires on every turn to eek out a faster time. The WPC 5-volt rail is similar in that it runs close enough to design thresholds that when it is not pristine problems surface. By reducing the 5-volt load, my daughterboard backs the design away from that bleeding edge and that is better for the MPU and everything that remains on the PDB-derived 5-volt supply.
In any case, in between soldering and shipping daughterboards, I have been tinkering with a couple of ideas to better address this concern. I have developed two prototype circuits for detecting the pinball machine’s reset condition directly on the daughterboard. The idea would be run your game with this enhanced daughterboard – if your PDB 5-volt rail is stable, the daughterboard burns a green LED. If, at any point the daughterboard detects what would have driven the MPU to reset, the daughterboard turns off green and burns a red LED. I latch the indication, so once the machine has triggered the daughterboard to turn red, the only path to get back to green is to turn the game off and reboot. All this detecting and latching happens without interrupting gameplay so you get the benefits of the daughterboard without losing the “early warning” indicator. If all goes well with prototype testing I could start shipping such a device this summer.
So, what would I say to critics? Thank you for challenging me to be better.
CD: When using your daughterboard, will other components of the power supply come under stress, which could lead to failure over time?
RK: Electrically, no. The 12-volt digital is so lightly loaded in the WPC design that you could easily run two MPUs simultaneously via two daughterboards without issue.
Thermally, no. The 7812 will run a little bit warmer because a larger load is being driven. I’ve run some tests that show the 7812 with the original WPC-design heat sink easily stay within the device’s spec while driving the daughterboard and MPU.
Mechanically, maybe. The daughterboard attaches where there was previously just a header connector. Either is a lever that could damage the header pins connection if fussed with over the long term; the daughterboard extends that lever by about an inch so you could more easily disturb the MPU power pins. Most backboxes remain closed so there really isn’t a lot of opportunity to apply force to this lever.
CD: What is your technical background?
RK: I am a degreed electrical engineer (Bachelors from Penn State; Masters from Virginia Tech). As a pastime I have a long history of reviving electronics. I got my first fixer-upper pinball machine in 2008.
CD: How did you devise this plan of attack on resets?
RK: I’ve been fighting resets in my Party Zone (which I revived from completely dead) for about 2 years. I did get the 5V to be quite stable through re-pinning, crimping and new caps, but would get occasional very brief collapses in the voltage down below 3V and immediately back to 5V. These were about 10 [milliseconds] in duration so you couldn’t see them on a meter but I could see them clearly on my O-scope (which, I also revived from the dead a few years ago!). Anyway, I found that I could eliminate this collapse by unplugging the GI power from the PDB -or- unplugging power to the sound board. Either would eliminate the collapse, so I reasoned that neither caused it; rather the load of both caused it. I replaced the LM323K on the PDB and even did the 11 ohm jump-up to mitigate it but neither solved the resetting, so I lived with no GI on my PZ… at some point I’d rather just play the game! But, being twice degreed in electrical engineering and an avid tinkerer, I never wholly got over my failure.
Now, one of my goals in working machines is to leave no exhaust–meaning whatever I do should be able to be undone in the future. For example, when I did the resistor trick/hack, rather than cut any traces, I isolated the heat sink from the PDB using vinyl nuts/bolts and I soldered the 11 ohm resistor to the ground test point… both easy to undo. No way was I going down the watchdog bypass or computer power supply routes.
Thus my requirements–to get beyond the PZ reset problem without damaging the machine for the next guy. It hit me shortly after leaving the York show (where I almost sold the PZ) that I need to take a complete systems view of the problem. I’ve studied the WPC schematics many times. I measured the power consumption of the boards downstream from the PDB on both 12V and 5V. And I thought hard about all the best practices. And of course, the regular flow of “reset problem” threads on RGP routinely reminded me the problem is rather widespread. I concluded the 5V rail at the SYSTEM level just doesn’t have enough tolerance built into the design. The connectors, the caps, the rectifier, the watchdog, the heat-spewing regulator, the overall loading – the whole system is just fragile. How could I add more tolerance?
Looking at the schematics, the 12V line is much more lightly used and I started thinking of ways to shift the extensive 5V load to the 12V rail. Eventually it hit me that a plug-compatible (and thus completely un-doable) solution that generated a unique 5V supply to drive only the MPU could be very effective. So, in January I built my first prototype. It used an LM7805 to generate the new 5V supply from the 12 V line. This prototype worked but it required a significant heat sink as linear power supplies simply burn off extra voltage to drive their output – the step-down from 12V to 5V is quite large. Not pretty, but I did learn the 12 volt line had enough power to drive the 5V for the MPU and generate quite a bit of burn off heat.
I considered making the daughterboard so that it plugged into the output side of the PDB instead of the input to the MPU but that moved a lot more load to the 12 volt line – I wanted to achieve better balance. In my measurements the MPU pulls about 2 watt (about 0.4 amp at 5 v) – that’s a nice chunk but not overwhelming.
I did some research on voltage regulators and dug deep into Mouser and Jameco’s extensive library of data sheets and came up with a couple of switched charge pumps that are drop-in replacements for 7805’s. These puppies are pretty cool – because they don’t burn off extra power they can deliver a lot of current without needing a heat sink. The one I ended up choosing (the ezSBC) isn’t available at either supply house but it can deliver 1 amp at 70 deg Celsius without heatsinking. And, I’m only loading it well below capacity. 1/4 watt at 12 volts is about 0.17 amp – that’s the current impact to the 12V rail and I can measure that with my meter so I’m confident in the calculation. Prototype 2 is what is in my youtube video [below, ed] – it is a home-etched board with male and female connectors and it works swimmingly.
By now I knew I was on to something pretty good. It doesn’t eliminate the need for sound maintenance practices but it does (1) reduce the load on the PDB 5V rail, (2) reduce sensitivity in the 5V system, (3) stop in-game resets on systems that are “almost there”. This can greatly reduce the temptation to hack boards which is great for the future of pinball and I can offer them at a fraction the cost of replacement boards… or a fraction of the value of the time spent chasing gremlins… I saw a niche.
CD: You also offer a 12 volt regulator to replace the 7812 on the power driver board. Can you describe, again in layman’s terms, the need for the 12 volt regulator and how it works in relation to the 5 volt reset issue?
RK: The ezSBC switching regulator is a direct, pin-compatible replacement for the 7812 on a Power Driver Board. Do look at the pictures on ezSBC’s web site so it don’t get installed backwards. The 7812 (and like it the 7805 for 5 volt loads) is a linear regulator. In the pinball machine it takes the 18 volt supply as an input and provides a very stable reference voltage at 12 volts for loads up to 1 amp. It accomplishes this by “burning off” the difference between the input and output voltages as heat. The load is important because a 7812 with very little load (and thus very little current draw) will not produce a lot of heat. As the load grows however, the amount of heat expelled increases because more current flows through the regulator. Essentially the input and output are wired together through a semiconductor (silicon) and are always connected. The ezSBC switching regulator operates differently in that it rapidly connects and disconnects the input and output to transfer power to the load. Because the device is either “on” (i.e., conducting) or “off” (not conducting), it does not burn power like a linear regulator and therefore does not need a heat sink. Think of a fire in a fireplace. You burn logs and the room warms up. But heat also goes up the chimney instead of into the house. That’s analogous to a linear regulator. Now think of ventless gas log fireplace. No longer is heat going up the chimney. That’s analogous (sort of) to the switching regulator. It would be more analogous if the gas log set had a chimney that opened and closed 300,000 times per second… but it’s the best analogy I’ve dreamt up so far. All that said, switching out the 7812 for a switching regulator is just a “nice to have”. I was kinda expecting some nay-sayers when releasing the daughterboard (I’m used to RGP moreso than Pinside) and I wanted to have a good answer as a mitigation for the increased load on the 12 volt line. So, in some ways it is more of a “political afterthought” rather than an imperative. Or, think of it like driving a Prius instead of Camry. Both get you from A to B, but the Prius does so with less gasoline. Your pinball will play the same whether or not you swap out the 7812; will you feel better knowing it burned a little less power when the DMD flashes “Game Over”?
CD: Who do you see as the main buyers of your daughterboard? Casual players? Tinkerers?
RK: Based on feedback from purchases the spectrum spans from folks who have never opened their backbox to professionals repair shops who maintain machines for others. Some are sitting in tool boxes waiting for a need; others are installed in machines and forgotten about. One was returned because the individual still had resets – he used my board to rule out a 5-volt rail problem, fixed a diode problem elsewhere and then sent the daughterboard back. Go figure.
I personally think it is more interesting to ask where along the well-documented path of best practices for addressing WPC resets should different experience levels reach to my daughterboard. Said another way, if I were the author of Pinwiki, where would I insert “install a daughterboard”? For some it would be very near the end (if at all). For others it should be inserted before a soldering iron is plugged in. And for yet another subset it should be shortly after the “open the backbox” step. I am confident that for everybody it should be ahead of the resistor hack – especially if traces are being cut to separate the regulator from ground. Otherwise it’s debatable – I don’t know where exactly I would insert it into Pinwiki.
CD: Can you give me an idea of how many boards have been sold thus far?
RK: 324 daughterboards have sold as of 24 April; 42 of those 324 were sold with the switching regulator add-on. Aside from North America I have sent boards to Europe, South America, and Australia. There was a smidgen of interest from an individual in Africa but to my knowledge none have hit that continent yet (nor to Asia). I have just under 200 ready for the 2014 Allentown show but that number could drop if there is a run on sales prior to the show. I haven’t really thought too much about post-show, so if I sell out at the show I may be out of stock for a week.
CD: Are you a regular attendee at the Allentown show?
RK: If you watch TNT Amusements videos from last year’s show, he does a tour where he goes around complimenting machine after machine after machine. Then he gets to my machine – an Elektra playfield I had installed in a Mata Hari cabinet and he just kind of pauses. He doesn’t say anything mean but he can’t come up with anything good to say either. It is flipping hilarious! Anyway, I completely assembled that machine from spare parts that came from other acquisitions. It’s the only machine I’ve ever sold (and I might have done so just to prove I’m not a hoarder).
CD: How do you handle production? Do you buy the individual components and solder all the boards yourself?
RK: I build and test all the daughterboards myself. Pre-build, I test each regulator with a small load in a jig I created that mimics the 12 volt PDB supply. Post build, I test each assembled daughterboard in another jig with a heavy load. A sampling also end up in my Party Zone for a one-game test prior to being released.
The soldering itself isn’t very exciting to write about – the fact that I have components on both sides of the board would make it more difficult to automate the assembling. At the rate I’ve been selling it doesn’t make sense to automate anyway and although repetitive I still enjoy making them. Not sure I would feel the same about soldering if it wasn’t my idea I was building.
The upstream supply chain is a little interesting – you need parts on hand but you don’t want a ton of cash tied up in inventory. I order the components from online supply houses (mostly Mouser, but they sometimes run out of stock which sends me scrambling), I get my shipping supplies from papermart.com, the boards themselves come from a experssPCB.com (high quality, but high priced). And the regulators I use are from ezSBC.com – a great small business in California. I frequently have partially assembled boards waiting for something that is “in the mail”.
Packaging and shipping is less fun… bubble wrap, paper, and tape – oh my. I really want to find a better way to ship overseas (eBay global shipping is a total ripoff from the customer viewpoint) but I haven’t come up with a solution yet. Anyway, why is the Engineering team worried about shipping? Oh yea, all of this is just me.
CD: All good pinball interviews ask this, so I’d be remiss if I didn’t–what do you have in your current collection? Obviously the WPC test patient Party Zone, what else?
RK: My wife bought me my first machine for me in 2008 as a 39th birthday present–she wanted to hold out for 40 but she knew I was trolling Craigslist pretty heavily for a machine. She got me a Cue Ball Wizard–best present I’ve ever gotten. About a month later I got a Mata Hari out of an estate sale for $60. No back glass and completely dead. Looks and plays great now – I put a CPR playfield in it last fall. I got a Bride of Pinbot from Ebay for a song, from a guy located in Atlanta, and met up with him at my first Allentown show back when it was the Pinball Wizards Convention–2009 I think? I have a Firepower I got in September 2013 that I had to completely rebuild the cabinet and the electronics. Did a Max2K clear coating on the playfield and it is really nice now too. And I have a Xenon that I might bring to Allentown to see if I can sell… I just don’t like it that much. I also got a skee ball machine about 2 years ago. Most guests walk right past the pinballs and go directly to the skee ball.
CD: This daughterboard is turning out to be quite the project! Sounds you are actually getting people back to what matters most–playing pinball! Thanks for making time for me on no notice.
RK: These won’t make me rich but the pride I have in creating something that others actually want is a tremendous treasure for me. I am grateful to Pinside itself for connecting me to my first customers and to the wonderful users of Pinside who so warmly welcomed my daughterboard. Thank you!
If you own a WPC game, its almost a given that you have experienced a reset problem at one time or another. My most recent experience with this annoyance was with my White Water. I had “5X the Fun” running and had just started multiball. Big points were coming my way. Sure enough, with coils firing and flippers flipping, the machine’s power supply collapsed under its own weight, and I experienced the dreaded reset. I hung my head and softly cursed, knowing I had a long journey of troubleshooting ahead of me.
To dumb it down a bit, a reset occurs when the +5 volt power supply, which runs the “brains” of the machine, diverts from an acceptable range of voltage. This normally happens when the flippers are fired, stressing the +5 volt line and causing a dip in voltage. To protect the brains of the pinball machine, a watchdog chip was inserted in line with the +5 volts. When this chip recognizes a flux in power that doesn’t agree with the +5 volts range it should normally be getting, the game automatically restarts, thus easing the load. The flipper stress on the +5 volts is caused by a weak link upstream in the chain…and its a very long chain with a lot of elements that could be out of order. In the end, the watchdog chip congratulates itself on a job well done…however, it turns into a vicious cycle of resets and much weeping and swearing will commence on the part of you the owner.
With my reset issue, I luckily had Clay’s WPC repair guide and PinWiki as references, not to mention helpful colleagues over on Pinball Revolution. I guess the first advisable thing to do is not panic. Do not immediately assume the worst. For those who like to jump to conclusions, there is a stigma that these resets are caused by components located at BR2/C5. To quote PinWiki on their introduction to fixing reset problems:
“A long time ago……in a pinball galaxy far, far away…a kindly fellow was playing a nice game of pinball, hitting all the shots, and earning multiball after multiball. Just as he was about to beat the game’s high score to date, the WPC game MPU reset. “Not-A-Finga” he yelled in some ancient, dead, language. A visitor, from the neighboring planetary system noted that, “I once fixed that by replacing BR2 and C5”. And lo…the mantra was born. Fortunately, through advances in both our knowledge of these game systems, and the application of clearer thinking, we’ve come to realize that leaping to the “solution” of replacing components before doing real testing is not advisable. Most pinball owners do not possess the skill, experience, and tools that would allow them to work on circuit boards without damaging an expensive, and sometimes irreplaceable board.”
After troubleshooting with PinWiki (highly recommended) and my multimeter, I was able to determine that it was a simple connector issue between the power driver board and the CPU. A relief indeed! Reseating the connector fixed the problem temporarily, however the pins and connectors were swapped out to fix the solution long-term. If I had taken the board out and blindly replaced the BR2/C5 as the mantra above states, the issue probably would have been “fixed”, not because I swapped out the above components but because the connectors would have been reseated in the board removal process. The resets would have resurfaced and I’d be right back at square one.
Without a doubt, resets are an absolute pain in the ass. With so many novices joining the hobby, myself included to a certain extent, everyone is looking for a one-step, easy solution that requires less work than removing boards and intricate soldering. No easy solution does exist, resets occur for thousands of different reasons. However, a piece of technology is now on the market that appears to ease the burden on that overtaxed +5 volt power supply and has become the reset elixir for many hobbyists. This product is called the WPC MPU Power Fix Daughterboard and has sprung from the mind of Rob Kahr. Its introduction has not come without criticism. I had the opprotunity to speak with Mr. Kahr about his new product, the “power” it harnesses, and what it can do for the community through an email interview and can be found in a post appearing later this evening.