Why did PEM_read_bio_PrivateKey fail to load a key in FIPS mode?

My former manager told me that PEM_read_bio_PrivateKey failed to load a private key generated via OpenSSL commandline in FIPS mode. So I tried to load the key he sent to me using a simple test application. When not in FIPS mode, PEM_read_bio_PrivateKey worked fine. However, when in FIPS mode, I got the error -- "error:0608008D:digital envelope routines:EVP_DigestInit:disabled for fips". The error was consistent to what he had told me. Then I asked him whether he used any non-FIPS algorithm during the key generation. The answer was no.

Then I set a breakpoint at PEM_read_bio_PrivateKey and stepped into the function. After several rounds of code tracing, I had the following stacktrace.

0:000> kb
ChildEBP RetAddr Args to Child
0012f0d8 0fb43d0c 0fbc9ab0 0fbbef38 0012f5ec LIBEAY32!EVP_BytesToKey+0x10
0012f5cc 0fb43503 0012f5e8 0091ee40 0012f610 LIBEAY32!PEM_do_header+0xcc
0012f614 0fb46281 0012fa50 0012fa58 0012fa4c LIBEAY32!PEM_bytes_read_bio+0xe3
0012fa60 0041146d 0091c258 00000000 00000000 LIBEAY32!PEM_read_bio_PrivateKey+0x51
0012ff68 00411b18 00000001 003d4c58 003d59d8 openssl_loadkey!wmain+0x7d
0012ffb8 0041195f 0012fff0 7c817077 0115f6f2 openssl_loadkey!__tmainCRTStartup+0x1a8
0012ffc0 7c817077 0115f6f2 0115f77e 7ffdf000 openssl_loadkey!wmainCRTStartup+0xf
0012fff0 00000000 00411082 00000000 78746341 kernel32!BaseProcessStart+0x23

EVP_BytesToKey would return error. It caused PEM_read_bio_PrivateKey to fail to load the key. The second parameter of EVP_BytesToKey is a pointer to EVP_MD.

0:000> ln 0fbbef38
(0fbbef38) LIBEAY32!md5_md | (0fbbef80) LIBEAY32!sha_md
Exact matches:
LIBEAY32!md5_md = struct env_md_st

For some reason, MD5 was passed into the function. Since MD5 is a non-FIPS algorithm, in FIPS mode, it would be disabled in OpenSSL library. Then the question was where MD5 came from. When looking at the OpenSSL source code, I found that when a key file contains a header, PEM_read_bio_PrivateKey would read the header and retrieve the encryption cipher, then it would try to generate the key using EVP_BytesToKey by passing MD5 as its second parameter. Since MD5 is hard-coded as the second parameter, any key file containing a header which specifies an encryption cipher would fail to be loaded in FIPS mode. It is definitely a defect in the OpenSSL code. Until the fix, the workaround is not to generate a key with a header containing an encryption cipher. The OpenSSL versions we used in testing are openssl-0.9.7m and openssl-0.9.8j.