I spoke of the annoying reset issue in a post earlier in the day, and promised a follow up interview with Rob Kahr, whose WPC MPU daughterboard designed to stifle resets has caused quite a stir in the community and has brought many troublesome WPC machines back online. When I first saw the daughterboard, I’ll admit, I dismissed it immediately, taking the stance that it was one step above a hack solution that just ignored a larger problem in the machine. After doing more research, and corresponding with Mr. Kahr, I was wrong to be dismissive. The daughterboard has the potential to be a powerful device to own (even more so if Mr. Kahr implements some possible enhancements, outlined in the interview). Perhaps one doesn’t belong in every WPC machine just yet, but it would be handy tool to use for troubleshooting reset issues or a temporary fix on league night when your machine craps out. I’d like to thank Mr. Kahr for being gracious enough to provide answers to my questions with absolutely no notice at all. That’s the Credit Dot way. Future interviewees be warned. Mr. Kahr’s daughterboards are available through his website, and can frequently be found on Pinside with user ID “rkahr”.

———————

Credit Dot: I briefly discussed, in layman’s terms, HOW resets occur in the prior post. Maybe you could briefly give some information as to what your product does to solve or ease the reset issue?

Rob Kahr: Glad you covered “how” because there seem to be a million ways resets can occur and my daughterboard does not address all of them. For example, I had one guy [private message] me explaining how his transformer was out of spec – he had it re-wound and his resets were cured! Fantastic creativity. My board won’t fix grounding issues. It won’t magically deliver power if either the 18-volt or the 12-volt digital supplies are shot. It doesn’t fix battery damage on an MPU. It does offer relief to a very common situation where the game will play sometimes but reset when load is heavy or supply voltage drops. Note I dubbed the board the WPC MPU Power Fix Daughterboard–not the WPC Reset Fix Daughterboard.

My daughterboard specifically addresses power supply-caused resets and is designed based on my analysis of the linear 5 Volt supply on the Power Driver Board (PDB). The load on the power supply spans multiple boards and multiple functions on those boards. I measured the 5Volt load of the MPU board, DMD, and the sound board empirically via current measurements to determine individual board power use. I tabulated over 90 PDB connection points to the linear 5 Volt supply. I don’t have a Fliptronics board so I reviewed the schematic to determine it likely draws a small load compared to the other boards. Additionally the linear 5 Volt supply can be passed to the backbox and playfield via PDB jumpers J117 to J119. There is A LOT of demand on the linear 5 Volt power supply.

Anyone who has read about or worked to solve reset problems recognizes the many sources for weakness in the supply itself – what I call the foundation of the 5 Volt supply. Best maintenance practices are well documented in this realm and I’m not looking to discredit any of those maintenance strategies. Indeed, one needs a functioning linear 5 Volt supply in your WPC pinball whether or not my daughterboard is installed. Taking a system view, however, there are a myriad of permutations that weaken the overall ability to drive 5 Volts from the supply. Worse, issues that may not alone frequently drop voltage levels to trigger resets are additive so a combination of problems within the foundation and/or the load may contribute to the reset. And nothing is static in real-world operations. Power draws change through the cycle of a game; environmentals such as temperature fluctuate.

All of this means there is going to be some variability in the 5 Volt supply and at some point those variations will trigger resets via the watchdog circuit on the MPU board. [The watchdog chip] serves a valid purpose and should not be disabled. Its board (the MPU board), however, should be offered the cleanest power signal possible and in designing my daughterboard I set out to do exactly that.

The daughterboard literally removes the MPU from the PDB linear 5 Volt supply. All of the functions on the MPU are instead driven by a new, switching regulator supply included on the daughterboard. The watchdog circuit, as a 5 Volt function on the MPU continues to serve it’s role of protecting the ASIC, the processor, and the PROM chips by monitoring the new switching supply. Because (1) the new supply develops a very reliable, clean, temperature tolerant voltage and (2) the MPU load when isolated is more static, the watchdog is no longer driven to reset for off-MPU board fluctuations. Further, the reduced loading on the linear 5 Volt supply allows for more tolerance to imperfections in the foundations of the supply. Again, the linear supply needs to be maintained but it now has more capacity or “give”. No longer does the pinball community have to worry about every hundredth of a volt when conducting WPC repairs.

CD: What do you say to critics who dub your product as a “Band-Aid” solution? That it ignores a much larger problem by temporarily patching it up and ignoring it?

RK: I get that logic, but I really think I attacked this problem from a more holistic viewpoint. This little daughterboard fundamentally changes the calculus that is power distribution within WPC era games. Why do NASCAR drivers change tires mid-race? If they would just slow down they could get a lot more miles out of those tires! Obviously they are pushing the limits of the tires on every turn to eek out a faster time. The WPC 5-volt rail is similar in that it runs close enough to design thresholds that when it is not pristine problems surface. By reducing the 5-volt load, my daughterboard backs the design away from that bleeding edge and that is better for the MPU and everything that remains on the PDB-derived 5-volt supply.

In any case, in between soldering and shipping daughterboards, I have been tinkering with a couple of ideas to better address this concern. I have developed two prototype circuits for detecting the pinball machine’s reset condition directly on the daughterboard. The idea would be run your game with this enhanced daughterboard – if your PDB 5-volt rail is stable, the daughterboard burns a green LED. If, at any point the daughterboard detects what would have driven the MPU to reset, the daughterboard turns off green and burns a red LED. I latch the indication, so once the machine has triggered the daughterboard to turn red, the only path to get back to green is to turn the game off and reboot. All this detecting and latching happens without interrupting gameplay so you get the benefits of the daughterboard without losing the “early warning” indicator. If all goes well with prototype testing I could start shipping such a device this summer.

So, what would I say to critics? Thank you for challenging me to be better.

CD: When using your daughterboard, will other components of the power supply come under stress, which could lead to failure over time?

RK: Electrically, no. The 12-volt digital is so lightly loaded in the WPC design that you could easily run two MPUs simultaneously via two daughterboards without issue.

Thermally, no. The 7812 will run a little bit warmer because a larger load is being driven. I’ve run some tests that show the 7812 with the original WPC-design heat sink easily stay within the device’s spec while driving the daughterboard and MPU.

Mechanically, maybe. The daughterboard attaches where there was previously just a header connector. Either is a lever that could damage the header pins connection if fussed with over the long term; the daughterboard extends that lever by about an inch so you could more easily disturb the MPU power pins. Most backboxes remain closed so there really isn’t a lot of opportunity to apply force to this lever.

CD: What is your technical background?

RK: I am a degreed electrical engineer (Bachelors from Penn State; Masters from Virginia Tech). As a pastime I have a long history of reviving electronics. I got my first fixer-upper pinball machine in 2008.

CD: How did you devise this plan of attack on resets?

RK: I’ve been fighting resets in my Party Zone (which I revived from completely dead) for about 2 years. I did get the 5V to be quite stable through re-pinning, crimping and new caps, but would get occasional very brief collapses in the voltage down below 3V and immediately back to 5V. These were about 10 [milliseconds] in duration so you couldn’t see them on a meter but I could see them clearly on my O-scope (which, I also revived from the dead a few years ago!). Anyway, I found that I could eliminate this collapse by unplugging the GI power from the PDB -or- unplugging power to the sound board. Either would eliminate the collapse, so I reasoned that neither caused it; rather the load of both caused it. I replaced the LM323K on the PDB and even did the 11 ohm jump-up to mitigate it but neither solved the resetting, so I lived with no GI on my PZ… at some point I’d rather just play the game! But, being twice degreed in electrical engineering and an avid tinkerer, I never wholly got over my failure.
Now, one of my goals in working machines is to leave no exhaust–meaning whatever I do should be able to be undone in the future. For example, when I did the resistor trick/hack, rather than cut any traces, I isolated the heat sink from the PDB using vinyl nuts/bolts and I soldered the 11 ohm resistor to the ground test point… both easy to undo. No way was I going down the watchdog bypass or computer power supply routes.

Thus my requirements–to get beyond the PZ reset problem without damaging the machine for the next guy. It hit me shortly after leaving the York show (where I almost sold the PZ) that I need to take a complete systems view of the problem. I’ve studied the WPC schematics many times. I measured the power consumption of the boards downstream from the PDB on both 12V and 5V. And I thought hard about all the best practices. And of course, the regular flow of “reset problem” threads on RGP routinely reminded me the problem is rather widespread. I concluded the 5V rail at the SYSTEM level just doesn’t have enough tolerance built into the design. The connectors, the caps, the rectifier, the watchdog, the heat-spewing regulator, the overall loading – the whole system is just fragile. How could I add more tolerance?

Looking at the schematics, the 12V line is much more lightly used and I started thinking of ways to shift the extensive 5V load to the 12V rail. Eventually it hit me that a plug-compatible (and thus completely un-doable) solution that generated a unique 5V supply to drive only the MPU could be very effective. So, in January I built my first prototype. It used an LM7805 to generate the new 5V supply from the 12 V line. This prototype worked but it required a significant heat sink as linear power supplies simply burn off extra voltage to drive their output – the step-down from 12V to 5V is quite large. Not pretty, but I did learn the 12 volt line had enough power to drive the 5V for the MPU and generate quite a bit of burn off heat.

I considered making the daughterboard so that it plugged into the output side of the PDB instead of the input to the MPU but that moved a lot more load to the 12 volt line – I wanted to achieve better balance. In my measurements the MPU pulls about 2 watt (about 0.4 amp at 5 v) – that’s a nice chunk but not overwhelming.

I did some research on voltage regulators and dug deep into Mouser and Jameco’s extensive library of data sheets and came up with a couple of switched charge pumps that are drop-in replacements for 7805’s. These puppies are pretty cool – because they don’t burn off extra power they can deliver a lot of current without needing a heat sink. The one I ended up choosing (the ezSBC) isn’t available at either supply house but it can deliver 1 amp at 70 deg Celsius without heatsinking. And, I’m only loading it well below capacity. 1/4 watt at 12 volts is about 0.17 amp – that’s the current impact to the 12V rail and I can measure that with my meter so I’m confident in the calculation. Prototype 2 is what is in my youtube video [below, ed] – it is a home-etched board with male and female connectors and it works swimmingly.

By now I knew I was on to something pretty good. It doesn’t eliminate the need for sound maintenance practices but it does (1) reduce the load on the PDB 5V rail, (2) reduce sensitivity in the 5V system, (3) stop in-game resets on systems that are “almost there”. This can greatly reduce the temptation to hack boards which is great for the future of pinball and I can offer them at a fraction the cost of replacement boards… or a fraction of the value of the time spent chasing gremlins… I saw a niche.

CD: You also offer a 12 volt regulator to replace the 7812 on the power driver board. Can you describe, again in layman’s terms, the need for the 12 volt regulator and how it works in relation to the 5 volt reset issue?

RK: The ezSBC switching regulator is a direct, pin-compatible replacement for the 7812 on a Power Driver Board. Do look at the pictures on ezSBC’s web site so it don’t get installed backwards. The 7812 (and like it the 7805 for 5 volt loads) is a linear regulator. In the pinball machine it takes the 18 volt supply as an input and provides a very stable reference voltage at 12 volts for loads up to 1 amp. It accomplishes this by “burning off” the difference between the input and output voltages as heat. The load is important because a 7812 with very little load (and thus very little current draw) will not produce a lot of heat. As the load grows however, the amount of heat expelled increases because more current flows through the regulator. Essentially the input and output are wired together through a semiconductor (silicon) and are always connected. The ezSBC switching regulator operates differently in that it rapidly connects and disconnects the input and output to transfer power to the load. Because the device is either “on” (i.e., conducting) or “off” (not conducting), it does not burn power like a linear regulator and therefore does not need a heat sink. Think of a fire in a fireplace. You burn logs and the room warms up. But heat also goes up the chimney instead of into the house. That’s analogous to a linear regulator. Now think of ventless gas log fireplace. No longer is heat going up the chimney. That’s analogous (sort of) to the switching regulator. It would be more analogous if the gas log set had a chimney that opened and closed 300,000 times per second… but it’s the best analogy I’ve dreamt up so far. All that said, switching out the 7812 for a switching regulator is just a “nice to have”. I was kinda expecting some nay-sayers when releasing the daughterboard (I’m used to RGP moreso than Pinside) and I wanted to have a good answer as a mitigation for the increased load on the 12 volt line. So, in some ways it is more of a “political afterthought” rather than an imperative. Or, think of it like driving a Prius instead of Camry. Both get you from A to B, but the Prius does so with less gasoline. Your pinball will play the same whether or not you swap out the 7812; will you feel better knowing it burned a little less power when the DMD flashes “Game Over”?

CD: Who do you see as the main buyers of your daughterboard? Casual players? Tinkerers?

RK: Based on feedback from purchases the spectrum spans from folks who have never opened their backbox to professionals repair shops who maintain machines for others. Some are sitting in tool boxes waiting for a need; others are installed in machines and forgotten about. One was returned because the individual still had resets – he used my board to rule out a 5-volt rail problem, fixed a diode problem elsewhere and then sent the daughterboard back. Go figure.

I personally think it is more interesting to ask where along the well-documented path of best practices for addressing WPC resets should different experience levels reach to my daughterboard. Said another way, if I were the author of Pinwiki, where would I insert “install a daughterboard”? For some it would be very near the end (if at all). For others it should be inserted before a soldering iron is plugged in. And for yet another subset it should be shortly after the “open the backbox” step. I am confident that for everybody it should be ahead of the resistor hack – especially if traces are being cut to separate the regulator from ground. Otherwise it’s debatable – I don’t know where exactly I would insert it into Pinwiki.

CD: Can you give me an idea of how many boards have been sold thus far?

RK: 324 daughterboards have sold as of 24 April; 42 of those 324 were sold with the switching regulator add-on. Aside from North America I have sent boards to Europe, South America, and Australia. There was a smidgen of interest from an individual in Africa but to my knowledge none have hit that continent yet (nor to Asia). I have just under 200 ready for the 2014 Allentown show but that number could drop if there is a run on sales prior to the show. I haven’t really thought too much about post-show, so if I sell out at the show I may be out of stock for a week.

CD: Are you a regular attendee at the Allentown show?

RK: If you watch TNT Amusements videos from last year’s show, he does a tour where he goes around complimenting machine after machine after machine. Then he gets to my machine – an Elektra playfield I had installed in a Mata Hari cabinet and he just kind of pauses. He doesn’t say anything mean but he can’t come up with anything good to say either. It is flipping hilarious! Anyway, I completely assembled that machine from spare parts that came from other acquisitions. It’s the only machine I’ve ever sold (and I might have done so just to prove I’m not a hoarder).

CD: How do you handle production? Do you buy the individual components and solder all the boards yourself?

RK: I build and test all the daughterboards myself. Pre-build, I test each regulator with a small load in a jig I created that mimics the 12 volt PDB supply. Post build, I test each assembled daughterboard in another jig with a heavy load. A sampling also end up in my Party Zone for a one-game test prior to being released.

The soldering itself isn’t very exciting to write about – the fact that I have components on both sides of the board would make it more difficult to automate the assembling. At the rate I’ve been selling it doesn’t make sense to automate anyway and although repetitive I still enjoy making them. Not sure I would feel the same about soldering if it wasn’t my idea I was building.

The upstream supply chain is a little interesting – you need parts on hand but you don’t want a ton of cash tied up in inventory. I order the components from online supply houses (mostly Mouser, but they sometimes run out of stock which sends me scrambling), I get my shipping supplies from papermart.com, the boards themselves come from a experssPCB.com (high quality, but high priced). And the regulators I use are from ezSBC.com – a great small business in California. I frequently have partially assembled boards waiting for something that is “in the mail”.

Packaging and shipping is less fun… bubble wrap, paper, and tape – oh my. I really want to find a better way to ship overseas (eBay global shipping is a total ripoff from the customer viewpoint) but I haven’t come up with a solution yet. Anyway, why is the Engineering team worried about shipping? Oh yea, all of this is just me.

CD: All good pinball interviews ask this, so I’d be remiss if I didn’t–what do you have in your current collection? Obviously the WPC test patient Party Zone, what else?

RK: My wife bought me my first machine for me in 2008 as a 39th birthday present–she wanted to hold out for 40 but she knew I was trolling Craigslist pretty heavily for a machine. She got me a Cue Ball Wizard–best present I’ve ever gotten. About a month later I got a Mata Hari out of an estate sale for $60. No back glass and completely dead. Looks and plays great now – I put a CPR playfield in it last fall. I got a Bride of Pinbot from Ebay for a song, from a guy located in Atlanta, and met up with him at my first Allentown show back when it was the Pinball Wizards Convention–2009 I think? I have a Firepower I got in September 2013 that I had to completely rebuild the cabinet and the electronics. Did a Max2K clear coating on the playfield and it is really nice now too. And I have a Xenon that I might bring to Allentown to see if I can sell… I just don’t like it that much. I also got a skee ball machine about 2 years ago. Most guests walk right past the pinballs and go directly to the skee ball.

CD: This daughterboard is turning out to be quite the project! Sounds you are actually getting people back to what matters most–playing pinball! Thanks for making time for me on no notice.

RK: These won’t make me rich but the pride I have in creating something that others actually want is a tremendous treasure for me. I am grateful to Pinside itself for connecting me to my first customers and to the wonderful users of Pinside who so warmly welcomed my daughterboard. Thank you!

Further Reading:
KAHR.US – WPC MPU Power Fix
KAHR.US – Detailed Technical Discussion
Pinside – Announcing new product to help resolve resets in WPC era games