Quieter, On Purpose

A car alarm is going off outside as I write this.

I live in San Francisco, so it barely qualifies as an event. Nobody looks out the window. It exists to irritate the block and earn a glare from whichever parent just got their baby down. Car alarms here go off when the wind picks up, when a truck rumbles past, or when Mars and Venus align a certain way.

The alarm might be working perfectly. Nobody believes it.

I built one of those for my company. By accident.

Safety nets exist for a reason. In gymnastics, and in software. When you are one person running something other people depend on, monitoring is the net.

I run twelve checks around the clock. One for the website. One for the API. One for each service that keeps somebody's email or calendar flowing. They are heartbeats, which just means every service phones home every few minutes. If one does not check in on time, my phone buzzes.

I wanted to know whenever something unexpected happened. Errors are bad, right?

Over ten weeks the system declared 21 incidents. Twelve of them came from the same alert.

Every one of those twelve cleared on its own in about nine minutes. Which is roughly how long a car alarm runs before somebody finds the keys.

Then, at 11:54pm on July 15th, a different check went down.

This one was new. I had built it a few weeks earlier, after a stretch I would rather not repeat.

A customer's mailbox sync had gotten stuck on an expired bookmark and spent twenty-seven hours grinding away. Retry, fail, retry, fail. Every check I had stayed green the entire time, because every check I had was watching whether the service was running. It was running…just was not accomplishing anything. The first person to notice was the customer, whose daily summary had quietly stopped updating.

So I built a check that watched progress instead of pulse. Not "is this thing alive" but "has this account actually synced in the last eight hours."

That is the check that went down at 11:54pm. A check I had built to catch silent failure had gone silent itself. It stayed that way until 3:51pm the next day…sixteen hours. A couple of similar gaps had happened in June.

The alert fired exactly as designed. On time, to the right phone, saying the right thing. It just arrived like all the others, in a stream where most things had not mattered.

So I treated it like more of the same.

The alert did exactly what I asked. The system around it had trained me not to care.

Every time one of these fired, an agent picked it up. It pulled the full logs and the history, reconstructed what had happened, and proposed a fix. Quite useful on its own.

But being fast on any single alert was never the point. The discovery came from asking it to read across all of them at once.

Twenty-one incidents collapsed into four problems.

People had disconnected their Gmail accounts, which is a normal thing a person does and not an emergency. Outlook sync bookmarks had expired and the service kept trying to use them anyway. The goal evaluator was logging expected conditions as failures. And a queue connection was getting killed every ten minutes by the cloud provider, then opened again, forever.

Twenty-one pings felt endless. Four causes I could finish in a day.

So the work turned out to be subtraction.

A user disconnects Gmail? Record it, stop syncing, do not page anyone. A sync bookmark expires? Mark the integration and move on. You get the idea.

Then I added a couple of very narrow ones in their place. If an active account has not synced inside the window it should have, that is worth waking up for. That alert can fire at 3am and I will believe it.

And I turned three of the original twelve off entirely. They worked exactly as designed. They just were not useful, which makes them worse than no alarm at all. At least silence does not teach you to ignore things.

Everybody has their own version of the car alarm:

We treat visibility as a virtue. Record everything, notify everyone, make sure nothing slips through.

Recording something and interrupting somebody are two different decisions.

Before an event earns the right to become an alert, it should pass three tests:

If any of those is a no, it belongs in a log or a digest. Keep the information. Skip the interruption.

This gets harder rather than easier as agents make monitoring nearly free. An agent can watch every log, flag every anomaly, and write a convincing incident report every single time. The cost of producing an alert is going to zero.

Human attention did not get the same upgrade.

Evals tell you whether the thing worked.
Telemetry tells you what it did.
Alerting decides when somebody has to stop what they are doing and look.

The first two are engineering problems. The third is a judgment problem wearing an engineering costume, and for a small team it decides whether one person can run something other people depend on without living tethered to a phone.

I have been on vacation this week. The heartbeats are still running. Account health is still being checked. Every event is still being recorded. My phone has not buzzed once.

I do not take the silence as proof that everything is fine. Last week cured me of that. But I can see the services answering, the customer data staying fresh, and the narrow alerts standing by.

The work is still noisy. My phone does not have to be.

What goes off in your week, every week, and changes nothing? That is the one to kill first.

ChatGPT and Claude aren't just answering questions. Employees are connecting them directly to Notion, Linear, Jira, and the rest of your stack — with no security visibility into what data moves or what actions they take.

Harmonic Security gives your team the visibility to control it.

See what's connected