CertNanny takes care of all your bratty keystores...
CertNanny FAQ

* CA and SCEP server software requirements

Q: Any special considerations on the CA side?
A: The CA must not modify the requested DN, in particular adding the
   request serial number or other data to the issued certificate DN
   does not work. Disable this in the CA configuration.
   You should allow issuance of multiple certificates with the same DN
   in the CA configuration.

Q: Which CA software (SCEP servers) are supported?
A: The following CA software packages are currently supported:
   - OpenCA 0.9.2.4 (or higher): http://www.openca.info/legacy/index.html


* CA specific issues

Q: OpenCA: How do I set up the SCEP server properly?
A: Make sure your SCEP server is set up properly (SCEP certificate is
   configured, private key saved without password, dummy password specified 
   in the configuration)
   - OpenCA scep.conf:
     ScepAllowRenewal "YES"

Q: OpenCA: How do I enable automatic approval with existing private key?
A: - CertNanny keystore section:
     keystore.