LMS Tech Observatory: Building Our Own Monitoring

This was one of the first things I built at LMS Technology. I built it because we were starting to plan and deploy multiple applications and I was the only engineer. I needed a way to monitor logs across multiple servers, and I did not have the time to be doing that manually all day.

Why not just integrate New Relic

I could have integrated an existing tool. I decided that wasn't necessary.

If I wanted to implement something like New Relic, which has been industry standard for a long time, that's a lot of overhead for me to learn their system. A lot of tools in that framework I don't need. A lot of boilerplate setup, in every app, forever.

So in the age of AI, I thought, screw it. Let's develop our own in-house observability tool, built specifically for our tech stack, which is Next.js, so it works seamlessly with every application we make, period.

Front end first

I started by developing the design and the front end, heavily utilizing Claude Code to do that rapidly. In the first couple of days I had something functional. I could see logs. I could see graphs of uptime. I could configure projects. So I established it as a Next.js app and deployed it.

Nine lines to instrument an app

Because we use one stack, I could build the scaffolding straight into our template project. Every new application we start already contains it. So we get this observability for free, on every app, without anybody having to remember to add it.

It's four files and about nine lines. The package doing the talking is @lmstech/monitor.

// 1 — instrumentation.ts  (project root)
export async function register() {
  const { init } = await import("@lmstech/monitor/server");
  init({ app: "edge-hmi", env: "production", version: process.env.COMMIT_SHA });
}

// 2 — app/api/monitor/route.ts
export { POST } from "@lmstech/monitor/server";

// 3 — app/layout.tsx
import { Monitor } from "@lmstech/monitor/client";
// ...
<Monitor app="edge-hmi" env="production">{children}</Monitor>

// 4 — .env  (production only, never local)
MONITOR_URL=https://observatory.lms-tech.net
MONITOR_KEY=<the app's key from the dashboard>

That's the whole integration. No agent inside the app, no config file, nothing per-route. The dashboard generates a key per app, so every app authenticates as itself and I can revoke one without touching the rest.

And if MONITOR_URL and MONITOR_KEY aren't set, the package is completely inert. No patching, no listeners, no network calls. Local dev doesn't set them, so it stays invisible while you work.

The four files

On the Next side it's pretty convenient, because there's really only two or three places you touch.

Server startup. Next has a startup hook that runs once when the server boots. That's where init() goes, and it's the whole trick. It patches console.error and registers the uncaught exception handlers, so every server error and every crash gets captured without me changing a single existing line of app code. Events batch in memory and flush on a timer, or immediately if the process is going down.

The browser. A provider in the root layout listens for uncaught errors and rejected promises and beacons them out. It posts to a route inside the app itself, which attaches the key and forwards it on, so the key never reaches the browser.

Custom logs. Past the automatic capture, there's a server function and a client hook for anything we specifically want to see.

import { log } from "@lmstech/monitor/server";

log({ level: "info", message: "Batch reconciled", meta: { runId, rows } });

One rule I held to: monitoring code never throws. Every capture path is wrapped, the patched console.error always calls the original first, and every send is fire-and-forget. Instrumenting an app can't break the app, and the dashboard going down can't either.

Every event also carries the app, the environment, and the commit SHA it came from. So when something breaks I know which deploy it came out of.

Three things it watches

The dashboard watches three separate things and puts them on one screen.

Events from inside the app, which is everything above. Errors and fatals get deduplicated on the way in. Each one is fingerprinted, and a repeat bumps a count on the existing group instead of adding another row. A group tracks the commit it was first seen in, the commit it was last seen in, and whether it's been resolved. So a crash loop is one row with a count on it, and I can tell whether a fix actually took.

Health checks from a cron. Each app gets one or more ping targets, labeled, because the same app might be running in more than one place and I want to know which copy is down. That's where uptime comes from.

Pipeline state from GitHub. Each app can be linked to its repo, so the dashboard knows the latest successful build. Compare that to the commit SHA in the most recent startup event and you get version drift. The running deploy is either current or it's some number of commits behind. So I can tell whether a fix that passed CI ever actually made it out.

Catching our own leaks

Another feature we added that I think is pretty cool: the dashboard scans what comes in for sensitive data.

Logging is where sensitive information gets out. Somebody logs a whole request body or a whole user object while debugging, it ships, and now personal data is sitting in a log store nobody is thinking about. On ingest, messages, stack traces, URLs and metadata all get scanned, and anything found is redacted before it's stored. The replacement names the type it pulled out, so I can see that an SSN leaked, and which app and which log line leaked it, without the value itself ever landing in my database.

Scanning and redaction are separate switches per app, because our apps don't all carry the same kind of data. Either way, it catches inappropriate logging as a category. It tells us which app is leaking so we can correct it quickly. This is just an added measure that came out of building the system.

Worth it

This isn't a general-purpose observability tool, and I didn't want it to be. It knows one stack. It assumes Next.js. I gave up generality on purpose and got something I can drop into a new app in about nine lines without thinking about it. I built it in March and haven't had to touch it since.