Our Note Search Kept Ranking the Closest Text Match Over the Most Urgent Note

On the screen, four small boosts sat in a row: 0.30, 0.25, 0.20, and 0.15. I had added them one by one to help a note-finding system make better choices. A note waiting more than five days got 0.30. Something due within a week got 0.25. An overdue follow-up got 0.20. A recently updated note got 0.15. The math was neat. The choices were not.

The system was meant to gather a small stack of useful notes whenever someone asked a question. That stack would then give an AI the background it needed to answer well. I started by putting almost all the trust in cosine similarity, a way of asking whether a question and a saved note are close in meaning. Then I added those four boosts on top.

It sounded sensible at first. If a note matched the words in the question, it should rise. If it was also due soon or had been waiting around, it should rise a little more. The trouble was that the close match got the whole starting score. Everything else was just a sticker added afterward.

That first approach did not crash. It did not produce wild numbers. A test would have happily watched it run. The problem was what the formula quietly believed: that a close text match mattered more than anything else, by default. I had not meant to make that choice, but the formula had made it for me.

The cost was time. The target for this first version was blunt: the bundles of notes had to be relevant more than 80% of the time in daily use. Until that happened, I could not move on to building the more visible parts of the product. A ranking system that repeatedly hands over the wrong background makes the whole thing feel dumb, even when every screen looks polished.

Picture a question about what needs attention now. An old note from a project that has been quiet for a long time might use almost exactly the same words as the question. It could beat a slightly less perfect match that is important, active, and overdue. The old formula could only correct that mistake by adding yet another special bonus. Soon the ranking became a junk drawer full of little exceptions.

The fix was not a fifth sticker. I changed the starting score so it was shared on purpose. Sixty cents of every dollar went to how closely the note matched the question. Twenty cents went to importance. Fifteen cents went to recent activity, called heat in the system. The final five cents went to whether the note was active, waiting, dormant, or something else.

That adds up to a whole dollar. It matters because changing the balance now has an honest cost. If importance should get 25 cents instead of 20, those extra five cents must come from somewhere, usually the close-match portion. Nothing appears from thin air. The choice is visible.

The status of a note also became part of the starting score instead of an afterthought. An active or waiting note got the full share for that part. A dormant note got half. Anything else got none. Two notes with the same words could now land in different places for a reason a person can explain: one is still alive in the work, and one is not.

There was one smaller detail that could have quietly spoiled the rewrite. Many notes did not yet have an importance score or a heat score. It would have been easy to treat a blank as zero. That would punish every note that had not been labeled yet. A missing label is not the same thing as a bad note.

So a blank got the middle value instead: 50 out of 100, or 0.5 after it was converted for the formula. In plain language, the system says, “I do not know enough to push this note up or down.” That is much fairer than hiding it because nobody has filled in a box yet.

The time-based boosts stayed. That was the part of the old idea worth keeping. Waiting more than five days, being due within seven days, having an overdue follow-up, or being updated in the last 48 hours are not permanent facts about a note. They are reasons to move it up the line today. Those bumps still get added after the shared starting score, and the final result is capped at 1.0.

That split made the whole setup easier to defend. Some facts describe what a note usually is: how close it is to the question, how important it is, whether it is active. Those belong in the basic mix. Other facts describe a short-lived squeeze, like a due date closing in. Those deserve a temporary nudge.

On Monday, pick one list that decides what gets your attention: an inbox, a job board, a notebook, or a stack of paper on the counter. Ask one question: what on this list is truly important, and what is only urgent today? If the two are being handled the same way, the old note may be winning there too.

dxdev LESSON · paste into your AI coding agent
LESSON: Stacking urgency boosts on top of a 100 percent similarity base score is an unstated weighting decision
SOURCE: dxdev.com/blog/2026-02-26_old-note-that-kept-winning
WHAT HAPPENED: A retrieval system ranking notes for an AI to read used cosine similarity to the question as nearly 100% of the base score, then added four flat bonuses on top for being overdue, due soon, waiting, or recently updated. Because similarity effectively WAS the whole starting score, an old and inactive note that happened to share wording with the question could outrank a genuinely important, active, overdue one, and the target of relevant results more than 80% of the time was not being met. The fix replaced the implicit all-in-similarity base with an explicit weighted split (60% similarity, 20% importance, 15% recency, 5% status) that sums to a fixed total, defaulted a missing importance or recency score to a neutral midpoint rather than zero (so an unlabeled note is not punished as if it were confirmed unimportant), and kept the original time-based urgency boosts as additive nudges applied on top of that explicit base.
THE RULE: When a score's base is implicitly 100% one signal and everything else is a flat bonus added on top, that IS a weighting decision, just an unstated one; making the split explicit and summed to a fixed total is what makes changing it later have a visible, accountable cost.
CHECK MY CODE, then report PASS or FAIL with file:line for each:
  1. Any ranking or scoring formula whose base score is effectively 100% one signal (similarity, recency, a single metric) with other factors layered on as unexplained flat bonuses.
  2. Any missing or null numeric field defaulted to zero inside a score calculation, where zero would treat not yet labeled the same as confirmed unimportant or inactive.
  3. Any multi-factor score whose component weights are not documented and do not sum to a fixed, known total.
THEN PRINT: a table (check, PASS/FAIL, evidence, fix) + a verdict (applies / partially / OUT_OF_SCOPE / no) + the single most important next action.