The symptom
On a Windows Server 2019 host, a service failed after a second cold restore. The service used a manually provisioned certificate with a private key in Local Computer → Personal. The certificate and associated private key had been present for more than six months and had not been intentionally replaced or reprovisioned. The service had been working normally.
Investigation of the failed state showed no explicit Read ACL entry for the service account on the private key. Adding that entry made the service work immediately again.
What made it weird
A first incident led to a cold restore, after which the service worked again. A later second cold restore was followed by the failure described here.
Later clones of two backups from known-good periods showed the expected explicit Read ACL entry was absent. One backup came from before the first incident. The other came from after the first cold restore, when the service was working again. This visible ACL state had therefore coexisted with a working service for a significant period.
Adding the explicit Read ACL entry fixed the failure, but the same visible ACL state without that entry had previously coexisted with a working service. The remediation explains the recovery, not the complete historical sequence.
Evidence
| Observed state | Service behavior | Private-key ACL observation |
|---|---|---|
| Later clone of a known-good backup from before the first incident | Working at the time of the backup | Expected explicit Read entry for the service account absent |
| Later clone of a known-good backup from after the first cold restore | Working at the time of the backup | Expected explicit Read entry still absent |
| After the second cold restore | Failed | Expected explicit Read entry absent |
| After granting Read to the service account | Worked immediately | Explicit Read entry present |
These are observations from different points in time. They do not reveal which identity or process actually opened the key during the earlier working periods.
Diagnostic path
For a service that appears to depend on a Windows certificate, work from the current failure outward:
- Identify the failing operation and its runtime identity. Check the service’s configured logon account and, where evidence is available, the identity and logon context of the process that actually requests the key. A configured service account alone does not establish that every key operation runs under that identity.
- Find the certificate in the expected store. On this host it was in Certificates (Local Computer) → Personal. Confirm that the service is selecting the intended certificate. Presence in the store establishes only that the certificate is there; it does not establish that the service can use its private key. Microsoft documents the distinction between machine and user stores.
- Check the private-key association. Verify that the certificate reports an associated private key. If it does not, investigate the certificate/key association before changing permissions. If it does, continue: association alone does not demonstrate that the service’s runtime identity can open the key.
- Inspect access to the correct key. Use Manage Private Keys for the certificate, or inspect the underlying key object’s ACL where appropriate. Check effective access, including group permissions and denies, rather than treating the absence of one explicit account entry as a complete access test. Windows access checks can include group ACEs.
- Keep the key provider in view. CAPI and CNG can use different key storage mechanisms, so identify the provider before mapping a certificate to an underlying key file. This is diagnostic context, not an established explanation for this incident. Microsoft describes CNG key storage.
- Test the smallest relevant permission change. If the evidence points to key access, grant only the required Read access to the identity that needs it, then retest the failing service operation. Record both the permission change and the observed result. A successful retest establishes a present-day dependency; it does not, by itself, reconstruct how the service worked in the past.
If the certificate, key association, and effective access all check out, continue investigating the actual failing operation instead of assuming an ACL fault.
What we can prove
- The certificate and associated private key had been present for more than six months and had not been intentionally replaced or reprovisioned. The service worked normally before the failures.
- The expected explicit Read entry was absent in later clones of two backups from known-good periods, and it was absent after the second cold restore when the service failed.
- Granting Read to the service account was followed immediately by recovery. Private-key access mattered to the second failure and its remediation.
- The missing explicit entry alone cannot account for every observed service state, because the service had also worked without that entry visible in the earlier backups.
What we cannot prove
We cannot reconstruct the exact runtime state during the earlier working periods. The later backup clones show the visible ACL, but not the runtime access path. No additional backups, snapshots, traces, or sufficiently detailed historical logs are available to determine which identity, process, or key access path allowed the service to use the private key then.
The evidence does not establish who or what changed the ACL, or even that an ACL change caused the transition from working to failed. It also does not establish a CAPI or CNG mechanism as the historical cause. Those are questions, not findings.
The remediation
An explicit Read ACL entry was added for the service account on the private key. The service worked immediately afterward. This was an effective fix for the failure after the second cold restore, without resolving why the earlier known-good states worked with that same entry absent from the visible ACL.
The lesson
Treat certificate presence, private-key association, and private-key usability under the service’s actual runtime identity as separate checks. Compare the failed state with known-good evidence before naming a root cause.
When a permission change restores service, report what the test proves: access mattered for the observed recovery. If the same visible ACL state also existed during known-good operation, preserve that contradiction. A successful remediation is not a complete historical explanation.