Adventures in Equipment Repair

[Galahad]

If you've been following the fuse board thread, you'll know that the Holy Grail Labs main manufacturing robot (which we named George) broke at the beginning of the month. I've finally got time to fix it so I figured I'd document it for anyone interested - hopefully doing that here is alright. There are a few general EE forums (and at least one specific to circuit board assembly) but I don't know anyone there so it's not as fun.

George is an APS Gold-Place / Novastar L60 from 2002, the control PC runs windows 98. It's designed for lower volume manufacturing and prototyping, which suits our use case very well. I may upgrade the hardware to a modern motion controller at some point but what it has works (except for being broken) so that's a longer term project.



We got George at the beginning of April this year, and the only big issue so far has been a loose camera lens. I didn't realize that was the cause of all kinds of calibration drift until it fell off during an assembly run and I haven't had issues after putting it back on.

The way this works is basically like a 3d printer except with a vacuum nozzle instead of a hot glue gun - the gantry head (with the googly eyes) goes over to the side, picks a part off a reel with the vacuum nozzle, centers it with little fingers, moves over and places it on the circuit board (held by the two horizontal bars in the center of the table). Modern pick and place machines use a computer vision system instead of mechanical centering but the computing power wasn't there in 2002.

Here's a close up of the vacuum nozzle holder in the lowered position with the fingers closed (power is off):



Here's a side-on shot of the centering fingers - they don't touch at the same z-height (I'd estimate about 15-25 thou different) which I will be correcting along with the electronics fixes:



George was (thankfully) optioned with glass slide position DROs for the x and y axes but most motions are actually done dead-reckoning - probably to save cycle time since checking the DROs can be slow. This is reasonably accurate because the axis control servos all have relative position encoders, but the position can drift if you miss encoder ticks. The camera on the gantry head exists only to make the operator's life much easier, the control software doesn't actually do any image processing in this model (the later LS60s had automatic fiducial recognition for board position and angle adjustment but that's done by hand on George).

Power and data has to get to the gantry head from the control board, but the range of motion is quite large (on the order of 3'x4' in this model). The way this is solved on George (and most 3d printers and other pick and place machines) is running all the wires through cable chains, which keeps the wires from getting eaten by any of the mechanical bits. The problem is that moving the gantry back and forth constantly - which you do using the machine - can work harden the copper and cause intermittent or total failure of some wires. The guy we bought George from warned me about this and said "make sure you get really flexy wires so it lasts longer when you replace anything".

George had been acting a little bit like he was missing encoder ticks occasionally - not enough to be concerning but enough I'd noticed. I agreed to do a manufacturing run of 600 small PCBs (in panels of 50) for a friend of mine, which was a significantly wider range of motion in both the x and y axes than I typically use doing fuse boards. At least one of the wires completely broke during the motion test, since the z-axis servo couldn't lift the vacuum nozzle and sat there vibrating (after launching the nozzle across the table) when we started the actual placement. I was glad I had the warning from the previous guy, I would have been very worried if I didn't have any idea what had happened.

Here's a top-down photo of the gantry head with the cover removed:



The translucent tube going to the bottom is the vacuum line, and the blue ribbon cable is the original wiring to the gantry head. Almost everything connects through terminal blocks at the bottom of the circuit board, which is a good thing for both servicing and strain relief. The x-axis DRO wiring does not go through the circuit board (maybe the camera as well), I am not entirely sure why but possibly because it's factory optional. The unpopulated terminal blocks in the bottom right are (most likely) for an optional solder paste / glue dispenser, but it might be for the optional laser centering thingy that could be used instead of the fingers. I'm not totally sure because I don't have a wiring diagram or manual for the L60 - only the LS60 - and finding information on an uncommon piece of equipment from two decades ago can be difficult. I sent the manufacturer an email, we'll see if I can get a copy of the correct manuals.

One important thing to notice in the above photo is the additional 3 gray cables going into the cable chain that are spliced into some things. If you looked at that and thought to yourself "self, those look like ethernet cables" you would be correct! You know what consumer ethernet cables are really bad at? Constant flexing back and forth! I can't fault the previous owner too much, he needed the fastest repair in his full time production setting, not necessarily the correct repair. Additionally, the factory ribbon cable isn't a good solution and the wire routing is horrible.

As I was taking photos and unplugging things to remove the circuit board at least one more wire broke, so if I didn't have to do these repairs before I definitely need to now. It's a bit hard to see but the broken end of a black wire is circled in this pic:



The gantry has a second, smaller circuit board to drive the x-axis motor (which is itself stationary):



I don't know what completely smooth-brained engineer decided to place a vertical connector exactly underneath the axis limit sensor when you've got all that slack, but now I need to either remove the limit sensor (bad) or remove the circuit board entirely to undo that plug. It's not a commonly accessed plug, but come on.

Current goals for this round of repairs:

1. Redesign both gantry circuit boards to:
   • improve plug position layout
   • use multiple connectors instead of a ribbon cable to improve servicing
   • change the terminal blocks to a better connector (if there's a meaningful improvement to be had)
2. use wires that are appropriately "flexy" so I don't need to worry about this for a while
3. redo gantry wiring layout inside of George so it isn't completely unserviceable garbage
4. fix the centering finger alignment
5. Possible scope creep:
   • redo y-axis wiring (not on gantry) so it doesn't have wire nuts connecting the servo to, you guessed it, yet another hacked up ethernet cable. Everywhere I look on George there's garbage-tier ethernet cables with the ends cut off and it bothers me.
   • identify the camera board and investigate upgrades - it's currently usable but I wouldn't describe it as good

I generally take a "do it right, do it once" approach with this sort of project, so we will see where this goes. Hopefully George isn't sick for too long.

[Shawn D.]

This is pretty neat! Sleuthing out all the issues can be pretty frustrating.

I suppose one could still use less-flexy wires if you can loop them in a manner so there's less bending strain. What characteristics do "flexy" wires have that's better (i.e. metallurgy, strain relief, geometry)?

How do you program the sequences? Something like CNC g-code?

[Panici]

Thanks for sharing!

Having to redo improper repairs is frustrating, but as you said sometimes the "get it running now, fix it properly later (re: never)" is true for production.

[gadget73]

"Nothing more permanent than a temporary fix" is the phrase that comes to mind.


The flexy wires I know of are usually high strand count. Ethernet cable is solid, really sucks for lots of bending. In real thin stuff that doesn't need to carry any sort of current, you may even end up with Litz wire, which is basically copper foil strands around thread which is all braided together. Headphone wire was sometimes this kind of construction. Litz is absolutely miserable stuff to work with though.

[stuartinmn]

Thanks for sharing. Depending on Windows 98 must be a little scary.

The flexy wires I know of are usually high strand count.

Early in my career I designed test equipment that had a lot of repetitive motion and like you said we used wire with a high strand count; the individual strands were very fine, not much bigger than a human hair. The insulation was also different (if I remember correctly it was Teflon instead of PVC). The stuff was like wet noodles when compared to regular machine tool wiring.

[Ju@n]

That looks like a home built 3d printer or cnc, but on steroids!
Thanks for sharing, it's really interesting

[Galahad]

This is pretty neat! Sleuthing out all the issues can be pretty frustrating.

I suppose one could still use less-flexy wires if you can loop them in a manner so there's less bending strain. What characteristics do "flexy" wires have that's better (i.e. metallurgy, strain relief, geometry)?
...

"Nothing more permanent than a temporary fix" is the phrase that comes to mind.

The flexy wires I know of are usually high strand count. Ethernet cable is solid, really sucks for lots of bending. In real thin stuff that doesn't need to carry any sort of current, you may even end up with Litz wire, which is basically copper foil strands around thread which is all braided together. Headphone wire was sometimes this kind of construction. Litz is absolutely miserable stuff to work with though.

Thanks for sharing. Depending on Windows 98 must be a little scary.

The flexy wires I know of are usually high strand count.

Early in my career I designed test equipment that had a lot of repetitive motion and like you said we used wire with a high strand count; the individual strands were very fine, not much bigger than a human hair. The insulation was also different (if I remember correctly it was Teflon instead of PVC). The stuff was like wet noodles when compared to regular machine tool wiring.

Replying to a few comments: the major differences in flexibility are individual strand diameter (proportional to strand count for a given diameter) and insulation. Ethernet cables have multiple layers of shielding since it's 8 wires in one and are usually designed for resilience against getting stepped on, which leads to stiffer insulation. NASA has an insulation selection guide here, the general idea relevant here is you can have either abrasive resistance or flexibility but not both. The wiring I'm currently shipping with the fuse kits is insulated with cross-linked polyalkene; I was looking for abrasive resistance and don't need great flexibility.

High flexibility ethernet cables designed for constant motion equipment are available, I may end up using those to fix George. If I do individual wires it'll most likely be Teflon.

Litz wire is most commonly used for radio frequency signaling (kHz to ~1MHz). Without going way too far into the weeds here (unless someone wants the technical explanation), all the very thin individual strands are electrically isolated from each other which allows for higher power transfer in the same diameter. The downside is now you have a zillion tiny wires you need to connect at each end. Using this stuff for audio wiring could make sense since the high frequency end is 10s of kHz. Foil wrapped wiring sucks too, horrible time getting a good connection to the shielding if you're doing anything weird.

I'm not super pleased it's running win98, but thankfully I haven't actually needed to do anything windows-specific yet. I backed up the original hard drive and copied it to a camera CF card so it doesn't take a century to boot, it's not all that bad.

...
How do you program the sequences? Something like CNC g-code?

In a round-about way. I think the motion controller talks in g-code, but you don't see it as a machine programmer. Modern machines base their programs on a position file that completely abstracts out the hardware - it has the part ID, xy position, and absolute rotation. The machine keeps track of what's loaded into each feeder separately (which allows you to hot-swap feeders when one is getting low) and any required nozzle changes. George uses a custom file format that is closer to actual machine commands, so it has specific instructions for changing vacuum nozzles and each part is defined as a xy location, feeder number, and relative rotation angle. Since part feeders can be loaded on both sides, you need to flip the rotation angle by 180 if you change which side a feeder is on. It's harder to work with - I'm slowly writing a script to turn the standard position files into the George-specific format but it's tedious. On the positive side, it is much easier to jump into the middle of a program with this setup which is nice if a part got dropped or placed wrong.

[jc72]

I work with a lot of motion control systems in my day to day job, there is lots of specialized cabling for this type of application. If the wires are going to be in a drag chain and only move along a single axis flat flex cable is commonly used, if it is going to be moving along more than one axis continuous flex cable is frequently used. Both of these are available from digikey with various numbers of conductors.

[Galahad]

jc72 thanks for the input, I may ask you for further details when I get to speccing wiring.

Let's take a look at the PCBs - gotta start somewhere. Not a whole lot of discussion to be had but it gives hints to what the engineers were doing which is always fun to figure out.

The gantry board top and bottom (roughly 4"x5.75"):




Comments:

• Not entirely obvious from the photos, but this is twice as thick as standard PCBs. I'm guessing that's because the end has the weight of the cable chain hanging off it and standard thickness would flex.
• I don't know what the oval cutouts are for, possibly airflow. Wires don't go through them.
• The lower 10 pins of J1 connect to J10 and J11 and are clearly for some factory option. They added a silkscreen assembly guide next to J1 for the machines that don't have that option because reworking a misplaced connector would be a big pain.
• Assembled with a wave soldering machine (the three surface mount resistors may have been hand work).

Smaller x-axis control board:




Comments:

• This one was assembled by hand - you can tell from the flux residue on the bottom and the fact that the holes for unused connectors don't have solder in them. Presumably the board was too small to send through the wave machine.
• J1 was built as a full connector and they cut off the unused pins.
• Some traces on this and the main board are thicker than others, maybe the manual will shed light on why.
• Resistors on this and the main board are all labeled "2200" (220 Ohm, the final digit is a magnitude adjustment) and are 1210 size. I did double check this. They're being used to reduce the current through the LEDs for the optical axis limit sensors.

I'm most of the way through cloning the main board - plan is to start with a drop in clone, then modify to suit whatever connectors I decide to use.

[Galahad]

Finished cloning the main board, here's a schematic shot:



Wasn't too bad honestly. It's mostly a direct copy of the original board, but I did tweak the position of the edge row of connectors to space them out more and a couple of the wires had to be moved slightly. Only one wire needed a significant route change - the original uses two different sizes for "thin" wires (0.025" and 0.020"-0.015") and slightly different alignment grids. I didn't bother replicating the smaller size and just used 0.025" for everything, but that took up the last wire's original space. I also shrank the resistors slightly to a size I use for most of my other work.

For higher frequency signaling you really shouldn't run a gazillion wires all next to each other in a straight line for multiple inches at a time. This is a very low frequency application so I wasn't particularly worried and, more directly, the original worked so this should too.

All the spacing on the original board is done in freedom units which, as an American, I truly appreciate - however my circuit layout software was set to communism units because I pretty much only work with German car electronics. A lot of circuit board work is done in mm now, which leads to some things not lining up completely when the two systems get mixed. For example, both my copy and the original have a silkscreen offset between the J1 outline and the assembly guide, shown below (and somewhat visible in the pic of the original in the earlier post). This is even more entertaining because the pins in the pin header itself are on a 0.100" pitch, but the outside silkscreen was done in metric anyway.



Time for the smaller board!

[tig]

This is so awesome. Thanks for sharing it all Gally!

[jayjaya29]

George is on the route to recovery.

Thanks for the lunch time reading content.

[Galahad]

Finished the smaller board:


The only notable part is the original uses three vias but you don't actually need any.

It looks like the manufacturer isn't particularly keen on giving me the manual for a price I'd be happy with - makes things harder but not impossible.

Next step is figuring out what connectors I want to use for the updated version.

[1st 5er]

Finished the smaller board:


The only notable part is the original uses three vias but you don't actually need any.

It looks like the manufacturer isn't particularly keen on giving me the manual for a price I'd be happy with - makes things harder but not impossible.

Next step is figuring out what connectors I want to use for the updated version.

The Holy Grail of motherboards!!!

[gadget73]

ovals maybe intended for zip ties to secure the cables to ?

[1st 5er]

[Galahad]

ovals maybe intended for zip ties to secure the cables to ?

Maybe? I'm certainly going to be doing that either way.

I decided what connectors to use, it basically came down to wanting some sort of positive locking mechanism so I don't need to worry about wires vibrating loose. I was originally looking at push-pull aerospace connectors - they would more than meet the requirements, they look nice, and I'm a sucker for shiny stuff. Downside is they're incredibly expensive and I don't need most of the features they have; I didn't want to completely light my wallet on fire so I looked into other options. Turns out Amphenol makes latching USB connectors so I'm going to use those; I'm much happier paying ~$5 a connector than ~$45 and they're easier to get ahold of anyway.

The spec sheets for the USB connectors have some interesting data points:

• USB 2 socket current rating: 1.5A / pin
• USB 3 socket current rating: 1.8A power & ground pins, 0.25A all the others
• USB 3 socket contact resistance: 30mOhm maximum
• USB 3 plug rating: 1A / pin (different supplier)

USB 3 is split into 2 pin groups, the first 4 are for legacy USB and the last 5 are for faster data transfers. The faster pins are about 80% the size of the legacy ones, but this doesn't explain why there's a factor of >6 difference between the two current ratings or why two of the large pins are rated for 1.8A and two other pins of the same size are rated for 0.25A. Doing the basic napkin math on the contact resistance and guesstimated conductor resistance, you'd need to stuff a few amps through each pin before it would warm up meaningfully. I'm pretty sure those numbers come from the USB specification itself, I'd bet there's something in there saying "x pin has to support y current" and there's no reason to claim something wildly higher on your component spec sheet even if you can do it.

Almost all the motors I'm driving take <1A and only run intermittently so I'm just going to go for it and see what happens - I've done far stupider things with USB ports in the past and none of them caught fire so I'm probably fine. Also, the ribbon cable they used from the factory has such thin wires that pretty much anything I do will perform better.

Here's a photo of the redone x axis board:





I looked at some of the photos I took more closely, turns out the stepper motors George has don't have any position feedback at all other than the end limit sensors. An intermittent connection to one of the phases could cause all kinds of drift issues especially with only infrequent position feedback, and based on the number of wires that broke while I was unplugging things this seems very likely to have been happening.

[Panici]

I looked at some of the photos I took more closely, turns out the stepper motors George has don't have any position feedback at all other than the end limit sensors. An intermittent connection to one of the phases could cause all kinds of drift issues especially with only infrequent position feedback, and based on the number of wires that broke while I was unplugging things this seems very likely to have been happening.

How often does it home on the end limit sensors when in operation?
Agreed it sounds like a recipe for drift.

[gadget73]

so I guess it does position by counting how many rotations the motor should have made based on what it was commanded to do, but with no feedback to confirm it actually made that many rotations? Yeah that could definitely lead to accumulated error between trips back to the end of limit sensor.

[Galahad]

I looked at some of the photos I took more closely, turns out the stepper motors George has don't have any position feedback at all other than the end limit sensors. An intermittent connection to one of the phases could cause all kinds of drift issues especially with only infrequent position feedback, and based on the number of wires that broke while I was unplugging things this seems very likely to have been happening.

How often does it home on the end limit sensors when in operation?
Agreed it sounds like a recipe for drift.

so I guess it does position by counting how many rotations the motor should have made based on what it was commanded to do, but with no feedback to confirm it actually made that many rotations? Yeah that could definitely lead to accumulated error between trips back to the end of limit sensor.

Yes, it does commanded step counting; it only homes when you start the pick and place application best I can tell. There is a "precision" mode you can tell it to use for individual components that uses the absolute position sensors, but I don't know if that updates the internal position variables or not.

[Galahad]

Made some progress on this!

I wasn't able to use USB ports for the cable chain-side connectors, there just wasn't enough board space to fit everything. I switched to DisplayPort connectors for the main cabling, there's much higher pin density and the latches are very solid.
Switching to the DP connectors has a couple downsides, unfortunately.

• If anything goes wrong, I'll have fewer cables with more pins (20 per cable instead of 4) which makes repairs harder and expensiver. However, nothing will go wrong because I'm too cool for that so it's only a theoretical concern.
• It was very hard to find cable connectors I can solder to - almost nobody sells them because DP runs at such high clock rates (500+MHz) you'd never be able to hand-make a cable that's usable for video. Luckily that doesn't matter for my application, I'm operating at a few hundred hertz max.
• Higher pin density means all the pins are smaller - the connector is only rated for 0.5A per conductor (presumably a DP spec, not a hardware one - the contact resistance is <30mOhm). One of the motors pulls 3A, which is slightly more than 0.5A. I'm calling a safety factor of 1/6 close enough, and can add a small fan or something if the magic smoke starts threatening to escape.

Positives!

• Everything fits on the PCBs
• In spite of the 20-wire flexy cable costing nearly $7.50 a foot, it's overall cheaper since I need less of it
• I'll have extra wires going to the gantry ends in case I want to add something later

Now for the PCB engineering!
I had to order all the PCBs with thicker copper than I normally use (2oz/ft^2, "normal" stuff is 1). I could have used thicker than 2oz, but I did the trace width calculation based on the expected motor current and they have a better safety factor than the sockets so there wasn't a huge reason to spend the extra money. PCB costs go way up once you go to 3oz or higher since it's really uncommon to absolutely need it so nobody orders it. All the Chinese PCB suppliers make their money by pooling your boards on the same panel with other orders but if you order something weird you essentially have to pay for an entire panel.

Here's the PCB that does all the wire breakout inside George:

Nothing super interesting, but I'm using 3 of these (3 small ones is a lot cheaper than one large one because the PCB suppliers almost always have minimum 5 units) so they all need to support the highest current motor. All the wiring from the DP socket is doubled up so I can get higher effective trace widths. Also I just realized (after ordering) I forgot to put holes in the corners so I guess I'll figure out some dumb mounting workaround.

This is the new x axis PCB:

The smaller 4 and 9 pin sockets are the USB ports, the rectangle labeled "output" is the DP connector. The 2x20 row of circles is for a pin header in between the USB and DP connectors - I can use little jumper thingies straight across if I didn't mess anything up, but if I made a mistake I can change the wire assignments later without needing to get new boards made. On top of that, I can use it as probe points for debugging. I got the idea from my dad - my best source of "old man wisdom" - who also made the questionable decision of getting an electrical engineering degree and has used this trick multiple times.

The new main gantry board is the largest of the PCBs:

The rounded rectangle in the middle is an interior cutout - interior cutouts are milled so they can't do right angles. I'm sure a high end manufacturer could do it somehow (they have lasers*), but I don't need that. It's better to design the curve in yourself instead of leaving it up to whatever the manufacturer does - I've done all kinds of fighting them to get what I specified but the end result is "don't expect good tolerances or even doing what you actually tell them from discount PCB shops".

The original main board was 3.2mm thick because it was also used as a mount point for the cable chain which is cantilevered off the end. My usual PCB supplier won't go that thick, and also won't sell anything other than 1.6mm (standard) with 2oz copper due to low demand. I found a couple alternate suppliers and ended up going with one I've had good results with - I could only spec 3mm not 3.2, but there was only one semi-budget supplier that would let you buy a 3.2mm 2oz (without emailing for a quote) and it was nearly twice as expensive. If I end up needing a little bit more stiffness I can glue a popsicle stick on the bottom.

The thicker PCB will present soldering problems for the through-hole components, nothing I used is explicitly designed to support thicker than standard PCBs and the USB ports appear to be designed to hold the PCB but that'll only work with 1.6mm. I'll figure it out when I get to assembly.

I got all the parts I'll need ordered and the project should come in under my target budget - I'll post cost totals when I've finished.

* High end shops that'll make anything under the sun do exist but boy do you pay for them. Places like Advanced Circuits can do essentially any board layer setup you want, in any material, mix flexible and rigid layers in the same PCB, 3d contour milling on the outside. They use lasers for micro-vias if regular drilled ones are too big. Turns out you can even get passives and ICs embedded in your circuit board which I only found out about recently - I don't want to think about how expensive that would be.

[Galahad]

Progress update:

All the parts arrived except for the PCBs! The flexy wire seems appropriately flexy, and all the other components appear to be in the box.

Now some complaining: Chinese PCB suppliers suck if you want anything other than green 1oz 1.6mm boards. The company I ordered from has two tiers - "advanced" (because "normal" would be too pedestrian) and "premium". Advanced wouldn't give me a quote for 2oz 3.2mm for the main gantry board and told me to ask in an email, but Premium auto-quoted me about $70 so I ordered it - this seemed cheap but not wildly so. I got an email saying I owed an extra ~$250 because the auto-quoter is always wrong for Premium. Why they don't set that to just say "ask for a quote" when the can clearly have the system say that is beyond me - that's the kind of nonsense that really gets under my skin. I'm getting it switched to 1.6mm which is an exercise in ESL patience; thankfully the customer service people at these places are usually reasonable to work with once you can get your point across.

I'll 3d print a bracket to support everything, I've taught myself enough openSCAD to pull off that much.

We're mostly into the home stretch, doing the assembly work will only take about a day and then it's back to fuse boards!