March 2022: three quotes, one bad assumption
The owner walked into the conference room holding his phone out like a subpoena. On the screen was a standby generator bolted to a concrete pad outside somebody's house. 'I want that,' he said. 'For the shop.'
For the next four months he called it 'the whole house generator installation,' even though the building was a 6,200 square foot fabrication shop with a 400A service and three-phase coming in off the pole. I never corrected him. That sounds like a small thing. Small things compound.
I had three quotes on the table. We went with the middle one — not the cheapest, not the best specced, the one with the shortest lead time and a contractor who actually answered his phone. My assumption at the time: backup power is a generator problem. Pick the right kW, pay the deposit, done.
It is not a generator problem. The generator is the loudest part of the system and the least likely part to let you down.
What we actually bought
The written scope: a 48 kW diesel standby unit, an automatic transfer switch, and an APC Smart-UPS SRT 5000VA RM 208V for the controls rack — the PLCs, the network closet, the two terminals that run the shop floor. We also ordered a replacement battery for the APC Smart-UPS 750 in the front office, because the runtime test had been showing about ninety seconds for a while.
Now notice what's missing from that paragraph. Nobody wrote down which circuits were actually on the backup side. Nobody wrote down the battery date on the second 750. And nobody — including me — asked what else was already wired into the panels we were about to re-feed.
Here's the thing about the SRT 5000VA RM 208V: it's a 208V unit. Our electrician's first draft had it on a 240V single-phase feed, because that's what he'd done on the previous job. We caught it during the site walk instead of during order review. Cost us a week of lead time and a re-pull on the conductor.
Install week: the warning I talked myself out of
Tuesday of install week, the electrician comes out of the mechanical room and says the old manual transfer setup has a contactor in it that 'looks tired.' We were leaving it in the circuit as a junction point between the new ATS and the existing distribution.
I said: it's fine, we're not using it. It's just a splice now.
That's the sentence I'd take back if I could take one back.
It wasn't just a splice. The coil was still energized, still tied to a control circuit that ran off a subpanel nobody had mapped. That subpanel also fed the air compressor's battery charger — the rolling unit the maintenance guys use for tire work and blow-down. The charger had been failing for months, tripping intermittently, and every time it tripped it dragged the control voltage on that subpanel down with it. Nobody connected the two, because nobody had ever drawn the panel schedule.
I did weigh it, briefly. Replacing the contactor was maybe $280 and an hour of the electrician's time. The risk of leaving it, as I understood it at that moment, was close to zero. The upside of not touching it was keeping the schedule. Had I known the coil was live and tied to a failing charger, I would have made a different call. But I didn't know, because I hadn't looked.
How to tell if a contactor is bad
This is the part I've now written into our pre-install checklist, and it's the part that would have saved the project.
A contactor can click and still not carry current. That's the failure mode that gets you, because the click is what everyone listens for. What you actually want to verify:
- Coil voltage under load. Measure at the coil terminals while the circuit is calling for it. Should be within about 10% of nameplate. Low control voltage makes a contactor chatter, and chatter welds contacts.
- Free movement. Power off, locked out, press the armature in by hand. It should move cleanly. Grit or a sticky return spring means it's on the way out.
- Contact faces. Some pitting and arc discoloration is normal. Bridged, welded, or badly cratered contacts are not. If you can't see them, that's a reason to replace, not a reason to skip the check.
- Voltage drop across the closed contacts while under load. A healthy set reads close to zero. Anything measurable is resistance you're paying for in heat.
- Auxiliary contacts. The contactor can be fine while the aux block feeding the ATS control board is dirty. That one presents as a transfer failure rather than a contactor failure, which is exactly how it fooled us.
Test day
September 2022, 4:40 PM. A Thursday, which meant the Friday orders were already staged and half-loaded on the dock.
The generator started in nine seconds. Clean start, good oil pressure, no alarms. The ATS called for transfer. The contactor pulled in — I heard it pull in — and the load never moved.
The APC UPS on the controls rack did exactly what it was rated for: it took the load for four minutes and twelve seconds, then dropped it. I know the number because the log told me. The 750 in the front office lasted about ninety seconds, same as it had on every test nobody had bothered to write down. One block in the string had swollen enough that it wouldn't come out of the tray without a pry bar.
Four minutes of ride-through on a battery that should have been swapped the year before, plus a transfer that never happened, adds up to a shop floor that stops in the middle of a run.
What it cost
The invoice side: $2,340 for the emergency electrician and after-hours rate, $610 for batteries (two blocks plus the 750 pack), $280 for the contactor we should have replaced in March. Call it $3,230.
The other side: six hours of a Friday, roughly $4,100 in orders that shipped Monday instead. That number doesn't show up on a purchase order. It absolutely shows up in a meeting with the owner.
So yeah, I put that one on my own tally. Four documented mistakes, about $11,400 in wasted budget since 2017, and this was the most expensive of them. Also the most avoidable, which is the part that stings.
The checklist I built afterward
I sat down the following Monday and wrote seven items. Not because I like checklists. Because I clearly couldn't hold seven things in my head at once, and pretending otherwise cost the company a Friday.
- Name the loads, not the equipment. 'Whole house generator installation' became a real problem because it described a residential product, not our three-phase critical loads. Write the circuits down.
- Walk the panel schedule physically. Breaker by breaker, with a label maker, before anything gets re-fed. The compressor charger would have surfaced on that walk.
- Verify voltage and phase at the point of connection before the PO goes out, not during install. That's the 208V lesson, and it's cheaper the earlier you learn it.
- Date every battery. Write the install date on the case in marker. APC's own guidance puts sealed lead-acid life at three to five years, and ours was at year five with a swollen block to show for it.
- Test under real load at the worst time. Thursday at 4:40 is a better test than Sunday at 9 AM. NFPA 110 calls for monthly exercising of emergency power supply systems anyway — we were doing it annually, if that.
- Own every device in the power path, including the ones that 'were already there.' If it's energized, it's yours.
- Verify carry, not just click. Coil voltage, contact drop, aux contacts. Every time.
What I actually took away from it
Look, the generator was fine. The UPS was fine once it had a battery that wasn't five years old. The failure was in the boring middle of the system — a contactor nobody owned, a panel nobody mapped, a charger nobody tracked — and every bit of it was visible in advance to anyone willing to spend twenty minutes looking.
Five minutes of verification beats five days of correction. That's on a sticky note above the checklist now. In my experience, the expensive part of backup power isn't the equipment. It's the assumptions you didn't write down.
The other thing I say to every new hire: no UPS guarantees zero downtime. What it guarantees is a window. Ours was four minutes and twelve seconds, and we'd spent five years not measuring it. Measure yours, put a date on it, and check it before the day you need it — because the day you need it is not the day you get to find out.
