Fenrir fixes

[aidankeefe2022]

https://fenrir.wolfssl.com/finding/5020

https://fenrir.wolfssl.com/finding/5022

https://fenrir.wolfssl.com/finding/5019

https://fenrir.wolfssl.com/finding/5021

https://fenrir.wolfssl.com/finding/5018

https://fenrir.wolfssl.com/finding/5023

Added stdout configuration to rand when running on windows to make it behave like Posix

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #255

Scan targets checked: wolfclu-bugs, wolfclu-src

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Findings are non-blocking.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #255

Scan targets checked: wolfclu-bugs, wolfclu-src

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Findings are non-blocking.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #255

Scan targets checked: wolfclu-bugs, wolfclu-src

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Findings are non-blocking.

[Copilot]

Pull request overview

This PR addresses multiple Fenrir findings by hardening CLI argument handling and file-length processing, adds regression tests for those cases, and fixes Windows rand stdout behavior to avoid output corruption.

Changes:

• Add regression tests for verify/dgst malformed-argument edge cases (missing trailing positional args) and a verify no-crash case.
• Switch Windows rand stdout to binary mode when emitting to stdout to prevent newline translation from corrupting output.
• Improve robustness around buffer sizing and OCSP index parsing (e.g., safer serial handling, larger size types, and negative-length checks).

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/x509/x509-verify-test.py	Adds regression tests for verify malformed-argument and no-crash scenarios.
tests/dgst/dgst-test.py	Adds regression tests ensuring dgst detects missing trailing data file and doesn’t mis-flag valid invocations.
src/tools/clu_rand.c	Sets stdout to binary mode on Windows when writing to stdout to avoid data corruption.
src/sign-verify/clu_x509_verify.c	Adds malformed-argument detection before loading certs.
src/sign-verify/clu_verify.c	Uses long for file sizes and adds negative-size checks before allocating/reading.
src/sign-verify/clu_dgst_setup.c	Adjusts option parsing to avoid scanning the trailing positional and adds malformed-argument detection.
src/ocsp/clu_ocsp.c	Enlarges serial buffer and adds length validation + explicit NUL termination.
src/dh/clu_dh.c	Uses long for BIO length and casts for read/decode.
src/crypto/clu_evp_crypto.c	Uses long for BIO length and checks for negative length before base64 decode.

Comments suppressed due to low confidence (1)

src/dh/clu_dh.c:488

• wolfSSL_BIO_get_len() can return 0/negative for empty or unreadable input. With the current logic, that case silently skips parsing the input params and continues, which can lead to confusing downstream failures. Treat non-positive lengths as an error before proceeding.

        inSz = wolfSSL_BIO_get_len(bioIn);
        if (inSz > 0) {
            in = (byte*)XMALLOC(inSz, NULL, DYNAMIC_TYPE_TMP_BUFFER);
            if (in == NULL) {
                ret = WOLFCLU_FATAL_ERROR;

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 7 comments.

Comments suppressed due to low confidence (2)

src/sign-verify/clu_verify.c:346

• keyFileSz is now a long, but it is still used in (int)XFREAD(...) comparisons and cast to int when calling wolfCLU_KeyPemToDer. Add an explicit keyFileSz > INT_MAX check and avoid truncating XFREAD's return value to int to prevent confusing failures on large inputs.

    if (ret == 0) {
        XFSEEK(keyPathFile, 0, SEEK_END);
        keyFileSz = XFTELL(keyPathFile);
        if (keyFileSz < 0) {
            wolfCLU_LogError("Unable to Get Size of Key File %s.", keyPath);
            ret = BAD_FUNC_ARG;
        }
    }
    if (ret == 0) {
        keyBuf = (byte*)XMALLOC(keyFileSz+1, HEAP_HINT, DYNAMIC_TYPE_TMP_BUFFER);
        if (keyBuf == NULL) {
            ret = MEMORY_E;
        }
    }
    if (ret == 0) {
        XMEMSET(keyBuf, 0, keyFileSz+1);
        if (XFSEEK(keyPathFile, 0, SEEK_SET) != 0 ||
            (int)XFREAD(keyBuf, 1, keyFileSz, keyPathFile) != keyFileSz) {
            ret = WOLFCLU_FATAL_ERROR;

src/dh/clu_dh.c:485

• inSz is widened to long but is then cast to int for wolfSSL_BIO_read and wc_DhKeyDecode without any upper-bound validation. If inSz exceeds INT_MAX, these casts can truncate and lead to short reads/decodes. Validate that inSz fits in an int before using it in int-based APIs.

        long inSz = 0;
        word32 idx  = 0;

        inSz = wolfSSL_BIO_get_len(bioIn);
        if (inSz > 0) {

[philljj]

Merge conflict in src/sign-verify/clu_dgst_setup.c.

Copilot found a few things, but some are probably too nitpicky and can be ignored and resolved (leave a comment explaining if doing nothing).

[philljj]

What is this trying to detect? Is there a more straightforward way to err check input?

[aidankeefe2022]

There is! and it was already merged in to main