hashlib: usedforsecurity works the wrong way round in FIPS mode in 3.14.10 #161383

Bug report

Bug description:

In 3.14.10, when OpenSSL runs in FIPS mode, usedforsecurity does the opposite of what it should:

• usedforsecurity=False: MD5 should work, but OpenSSL refuses it.
• usedforsecurity=True: MD5 should be refused, but OpenSSL allows it.

Because hashlib can no longer get MD5 from OpenSSL, it quietly switches to its own built-in MD5 (_md5). The built-in MD5 does not check FIPS, so hashlib.md5() now works in FIPS mode with no error, even for security use.

3.14.9 and earlier work correctly.

Cause

The bug was introduced when the fix for gh-149816 was backported to 3.14 (GH-160006, commit 3d2f1b2). In py_digest_by_name() in Modules/_hashopenssl.c:

int fips = disable_fips_property(py_ht);
if (fips == -1) {
    goto finally;
}

disable_fips_property() returns 1 for usedforsecurity=False. The code after it treats fips == 1 as "use the strict FIPS lookup":

PY_EVP_MD **cached = fips ? &entry->evp : &entry->evp_nosecurity;
...
fresh = PY_EVP_MD_fetch(entry->ossl_name, fips ? NULL : "-fips");

So the value needs to be inverted. main (GH-153019) and 3.15 (GH-159997) do this:

int fips = !disable_fips_property(py_ht);

In 3.14 the backport kept a -1 error return, and the ! was lost when the code was adapted for it.

Affected: 3.14.10 and the current 3.14 branch.
Not affected (the source does not contain the bug): 3.14.9 and earlier, 3.13, 3.15, main.

How to reproduce

You need OpenSSL 3 in FIPS mode as described above, so that _hashlib.get_fips_mode() returns 1.

import hashlib, _hashlib

print("fips mode:", _hashlib.get_fips_mode())

def t(label, f):
    try:
        f()
        print("ok    ", label)
    except Exception as e:
        print("ERROR ", label, "->", type(e).__name__, e)

t("openssl_md5(usedforsecurity=False)  [must work]",
  lambda: _hashlib.openssl_md5(b"x", usedforsecurity=False))
t("openssl_md5(usedforsecurity=True)   [must be refused]",
  lambda: _hashlib.openssl_md5(b"x", usedforsecurity=True))
t("new('md5', usedforsecurity=True)    [must be refused]",
  lambda: _hashlib.new("md5", b"x", usedforsecurity=True))
t("new('md5', usedforsecurity=False)   [must work]",
  lambda: _hashlib.new("md5", b"x", usedforsecurity=False))
print("hashlib.md5 module:", hashlib.md5.__module__)
t("hashlib.md5() (default usedforsecurity=True)  [must be refused]",
  lambda: hashlib.md5(b"x"))

Actual output (unmodified 3.14 branch):

fips mode: 1
ERROR  openssl_md5(usedforsecurity=False)  [must work] -> UnsupportedDigestmodError [digital envelope routines] unsupported
ok     openssl_md5(usedforsecurity=True)   [must be refused]
ok     new('md5', usedforsecurity=True)    [must be refused]
ERROR  new('md5', usedforsecurity=False)   [must work] -> UnsupportedDigestmodError [digital envelope routines] unsupported
hashlib.md5 module: _md5
ok     hashlib.md5() (default usedforsecurity=True)  [must be refused]

Expected output (the same build with the fix; 3.14.8 behaves the same way):

fips mode: 1
ok     openssl_md5(usedforsecurity=False)  [must work]
ERROR  openssl_md5(usedforsecurity=True)   [must be refused] -> UnsupportedDigestmodError [digital envelope routines] unsupported
ERROR  new('md5', usedforsecurity=True)    [must be refused] -> UnsupportedDigestmodError [digital envelope routines] unsupported
ok     new('md5', usedforsecurity=False)   [must work]
hashlib.md5 module: _hashlib
ERROR  hashlib.md5() (default usedforsecurity=True)  [must be refused] -> UnsupportedDigestmodError [digital envelope routines] unsupported

Why the tests did not catch it

CI does not run in FIPS mode. test_usedforsecurity_false would fail in FIPS mode, and test_usedforsecurity_true is skipped in FIPS mode, so no test checks that MD5 is refused.

Proposed fix

Invert fips after the -1 check, and add a test that runs only in FIPS mode. I will open a PR against 3.14.

CPython versions tested on:

3.14

Operating systems tested on:

Linux

Linked PRs

• [3.14] gh-161383: Fix inverted FIPS property query in _hashlib #161385