Announcing CommitWatch: Every Patch Tells a Story
A patch is not the same as disclosure. CommitWatch reads security-critical commits so the rest of us don't have to — and keeps a public record of what was fixed, how clearly it was explained, and whether anyone told the users.
Date: August 6, 2026 · Author: Karma-X Security Research Team
Type: Launch announcement · Status: Live and free · Coverage at launch: 23 projects across 20 vendors, Bitcoin-first
Links: commitwatch.org · Scoring methodology · Public JSON feed · RSS · Corrections
The gap we kept running into
Open source lets anyone read the code. In practice, almost nobody has the time, context, and incentive to read every commit that touches a security boundary — across dozens of wallets, node implementations, and cryptographic libraries, every day, forever.
That gap is where quiet fixes live. Some security-relevant changes arrive with a clear advisory, a CVE, researcher credit, and actionable guidance for users. Others arrive under a four-word commit title and disappear into the history. Both are "patched." Only one of them lets a user find out that the seed they generated last year might need to be replaced.
We have written about this problem repeatedly. We have also been this problem — on the researcher side of it, which is the part that stings.
The past week made the case better than we could
On July 31, 2026, Coinkite shipped Coldcard firmware v5.6.0 as an urgent hotfix for a "limited entropy bug." A firmware integration error had routed seed generation through a fallback software PRNG instead of the hardware TRNG, collapsing seeds to roughly 40 bits on Mk3 and roughly 72 bits on Mk4/Mk5/Q. Attackers ground the keyspace and swept thousands of wallets. Public loss estimates have since climbed past $100 million, and the attributed address count is still growing.
We covered the technical picture in The $38 Million Random Number and, after the hotfix landed, in Full Disclosure: Coldcard v5.6.0 Post-Hotfix Analysis and 39 Unpatched Findings. What matters here is not the bug. It is the paper trail — or the absence of one.
deltamode timing fix. No CVE. No advisory. No acknowledgment. A user reading the changelog had no way to know a key-recovery vulnerability had existed in their device, or that the update in front of them was the one that closed it.
Ten months later, a different entropy failure in the same codebase cost users nine figures. Meanwhile the actual library-level fix for that failure sat on other branches under messages like
bump to fixed libngu — four words, no security note, invisible to anyone not diffing submodule pointers by hand.
Multiple researchers observed strange behavior in this codebase over a period of years. That observation never accumulated into anything a user could act on, because there was no place for it to accumulate. Every researcher's notes stayed in every researcher's notes.
CommitWatch is our attempt to build that place.
What CommitWatch does
CommitWatch continuously ingests public commits from watched repositories, scores them, analyzes the security-relevant ones with a local model, and publishes the result with a link to the primary evidence. As of launch:
| Metric | At launch |
|---|---|
| Projects watched | 23, across 20 vendors — Bitcoin Core, Trezor, Ledger, Coinkite, Foundation, BitBox, Keystone, SeedSigner, Krux, Sparrow, Specter, Electrum, Blockstream, Elements Project, Rust Bitcoin, btcsuite, electrs, ACINQ, Lightning Labs, Lightning Dev Kit |
| Commits captured | ~16,800 |
| Published analyses | ~7,500 |
| High-risk findings, trailing 30 days | 60 |
| Topics covered | Hardware wallets, software wallets, node implementations, cryptographic libraries, Lightning, indexing infrastructure, hardware integration, supply chain |
Each analysis answers, in plain language: what does this commit change, what security-relevant behavior does it touch, and does the public commit message explain that adequately? Every entry links to the source commit so you can check our work.
Three scores, deliberately kept apart
The most important design decision in CommitWatch is that it refuses to collapse three different questions into one number.
1. Commit risk (0–100, evidence-weighted)
How security-relevant is this specific change — not how safe the project is overall. Every point comes from a visible dimension:
| Weight | Dimension | What it measures |
|---|---|---|
| 30 | Potential impact | Loss of funds, key exposure, code execution, privacy loss, availability |
| 25 | Exploitability | Required access, complexity, user interaction, preconditions, practicality at scale |
| 15 | Stealth signal | Whether abuse could remain invisible, and whether the commit language obscures security relevance |
| 15 | Affected reach | Deployment breadth, and whether vulnerable state persists after a software update |
| 10 | Classification confidence | How strongly diff, tests, issue links, and context support the classification |
| 5 | Evidence quality | Directness and completeness of primary-source evidence |
Those two 15-point dimensions are the ones born directly out of the Coldcard incident. Affected reach exists because "we shipped a fix" and "users are safe" are different statements — a weak seed stays weak after you update the firmware. Stealth signal exists because deltamode timing fix is a real commit message for a real key-recovery bug.
Scores map to bands: Critical 85–100, High 70–84, Moderate 50–69, Low 25–49, Informational 0–24.
2. Message quality (0–100, deterministic)
Separately and mechanically, CommitWatch scores whether the public commit message explains the change. Specific subject and scope earn up to 42 points; an explanatory body with rationale or a described failure mode earns up to 23; verifiability — tests, issue references, explicit statements of security behavior — earns up to 23. Generic placeholders, two-word subjects, and work-in-progress language carry penalties up to 95. Bands: Strong, Adequate, Thin, Opaque.
This score is computed for every captured commit, including the thousands that are security-irrelevant, because the point is to characterize a project's public record rather than to hunt for bugs.
runs is poor public evidence even when the patch is perfectly harmless. Message quality never adds points to the security-risk score. We keep them separate on purpose, and readers should too. A thin message on a refactor is a documentation nit. A thin message on a high-risk change is the thing this whole project exists to surface.
3. Vendor hygiene (0–100, five dimensions)
Observable practice: disclosure quality, researcher acknowledgement, security process, patch clarity, and incident response — each scored and averaged equally. Missing evidence is labeled insufficient data rather than scored as failure.
A strong product can have weak disclosure practices. A transparent vendor can still ship a serious flaw. Anyone who tells you those are the same axis is selling something.
Machines publish. Humans strengthen the record.
The pipeline is four stages, and we are explicit about which parts are automated:
- Capture and score. Fetch the public author string, message, diff, file list, metadata, and verified references. Deterministic triage and message-quality scoring run on every commit. "Lower-priority" is a queue position, never a declaration that a change is safe.
- Analyze and publish. A locally hosted Ollama model produces structured risk dimensions plus plain-language and technical summaries. Output publishes immediately, with unavoidable AI provenance on every item.
- Challenge. Researchers, vendors, and readers submit community notes that add evidence, qualify a claim, or identify an error.
- Validate and revise. A human moderator validates notes before they appear. Approved context stays visibly separate from raw AI analysis, and material corrections update the public record.
We publish AI analysis as generated rather than hand-curating it first, and we want to be direct about the tradeoff: that choice buys coverage and timeliness at the cost of occasional wrongness. The correction mechanism is not a disclaimer bolted on afterward — it is a load-bearing part of the design.
What a commit cannot tell us
A diff is evidence, not omniscience. We would rather say this loudly at launch than get quoted defending an overreach later.
- Git author names are public strings. They may not uniquely identify a person, and CommitWatch does not treat them as identity.
- Message quality measures documentation, not competence. Excellent engineers write terse commit messages every day.
- Context we cannot see may change the conclusion. Private reports, embargoed advisories, unreleased patches, hardware behavior, and operational controls are all invisible in a diff.
- A quiet fix is not automatically misconduct. Sometimes it is responsible caution during an embargo. The point is that the public record should eventually reflect it — and right now, often, it never does.
- This is not a substitute for a vendor advisory or a professional security review. It is a layer that has been missing underneath both.
How to use it
✅ FOR DEVELOPERS AND INTEGRATORS: Subscribe to the RSS feed or pull the public JSON feed into your own tooling. If you vendor any of the watched libraries, high-risk analyses are a cheap early signal that you should look at an upstream bump before it lands in your release.
✅ FOR RESEARCHERS: If you have looked at one of these codebases and seen something the analysis missed — or something it got wrong — submit a community note. This is the accumulation mechanism that did not exist during the ten months our own VULN-109 sat in a private notes file.
✅ FOR VENDORS: The hygiene score is built from observable behavior, and observable behavior is entirely under your control. Ship an advisory. Credit the researcher. Write a commit message that says what the commit does. All three are cheaper than the alternative, and we will happily update the record when you do.
Bitcoin first, then everything else
We started with Bitcoin wallets and infrastructure because the failure mode there is uniquely unforgiving: a tiny mistake becomes an irreversible loss, there is no chargeback, no support line, and no way to reissue a compromised key after the fact. The past week is the argument.
But nothing about the method is Bitcoin-specific. The same gap exists anywhere public code protects people — cryptographic libraries, authentication systems, privacy tools, critical infrastructure, AI systems, and software supply chains. Bitcoin is the proving ground, not the destination.
Our operating principles
- Evidence before inference. Link to the diff. Distinguish known facts from hypotheses.
- Plain language is security work. Users deserve to understand risks without reading firmware.
- Credit is part of hygiene. Acknowledgement is what keeps good-faith researchers engaged. When it disappears, researchers disappear, and users pay for it later.
- Corrections are a feature. New primary evidence should change the record — publicly and visibly.
Q&A anticipated
"Isn't this just naming and shaming?"
No, and we have designed against it in three concrete ways. Vendor hygiene is scored from observable practice rather than sentiment, and missing evidence is labeled insufficient data instead of counted as failure. Message quality is kept strictly out of the security-risk score. And the correction mechanism means any vendor with context we lack can put it into the public record. If a vendor's score is low, the remedy is entirely in their hands and it is not expensive.
"Doesn't publishing analysis of security commits help attackers?"
This is the right question to ask, and the honest answer is: marginally, at most, and much less than it helps defenders. Sophisticated adversaries already monitor these repositories — watching commits for silently patched bugs is a well-established technique with a long history, and it costs a funded actor essentially nothing. What does not currently exist is the defender-side equivalent. CommitWatch analyzes public commits that anyone can already read; it does not publish exploits, and it does not develop them. The asymmetry it corrects runs in users' favor.
"You're using AI to analyze security. How much should I trust it?"
Exactly as much as the linked diff supports, which is why the diff is always linked. AI provenance is marked on every generated analysis, the model runs locally, and human-validated community notes appear visibly separate from raw output. Treat an analysis as a well-informed prompt to go look at the commit, not as a verdict. If we get one wrong, tell us and we will correct it in public — that is the whole point of stage four.
"Are you scoring individual developers?"
There is a developer activity view, and it is deliberately shallow: git author strings are public metadata attached to public commits, nothing more. We do not treat them as identity, we do not score people, and message quality explicitly measures documentation practice rather than skill. If you believe a developer view misrepresents you, the corrections channel is open and we will act on it.
"Why is Karma-X funding this?"
Because we kept doing this work by hand, badly, one vendor at a time — and because our own disclosure history with Coinkite is a case study in what happens when a research feedback loop has no public artifact. CommitWatch is free, has no paywall, and does not gate findings. It is public-interest infrastructure, and we would like it to outlive our interest in it.
"Can I suggest a project to watch?"
Yes. Use the corrections and challenges channel or email commitwatch@karma-x.io. Coverage expands as capacity allows, and requests genuinely influence the order.
The long view
The Coldcard incident did not happen because a bug was hard to find. It happened because the signals were scattered across private notes, terse commit messages, and unpublished branches — and nothing existed to hold them in one place long enough for the pattern to become visible.
A repository's history is already a permanent public record. CommitWatch turns it into a public accountability layer: open evidence, explicit uncertainty, and a record vendors cannot quietly rewrite.
Go look at what changed this week: commitwatch.org.