cfengine dev format-docs: Enabled reformatting cfengine3 policy code blocks

[olehermanse]

• cfengine dev format-docs: Stopped adding indentation in front of empty lines
• cfengine dev format-docs: Enabled reformatting cfengine3 policy code blocks
• format_policy_file: Fixed inconsistent trailing newlines
• Made the return codes of formatting functions more consistent
• cfengine dev format-docs: Fixed issues with indented code blocks

[nickanderson]

I am not really qualified to review the python there.

But i spent tokens for you

Findings (most severe first)

1. format.py:1166 — assert check crashes on invalid JSON in non-check mode.
format_json_file returns FormatResult.NEEDS_FORMAT for invalid JSON even when check=False (the except JSONDecodeError branch, format.py:96). That value propagates through _combine_results to format_paths, which does if r == NEEDS_FORMAT: assert check. With check=False the assert fails → AssertionError. Trigger: cfengine format bad.json (no --check), or cfengine dev format-docs over any tree containing a malformed .json file. The old code returned 0 here (silently), so this is a regression to a hard crash. I reproduced the assert firing. (The _combine_results docstring at format.py:1099 asserting "NEEDS_FORMAT and REFORMATTED never occur together" is the mistaken premise — they do, via this path.)

2. format.py:1080 — new_data != original_data compares str to bytes, always True.
In format_policy_fin_fout, original_data = fin.read().encode("utf-8") is bytes but new_data = fmt.buffer + "\n" is str, so the comparison is always true and the function always returns REFORMATTED — the NO_CHANGE branch is unreachable. (format_policy_file at line 1046 gets this right with original_data.decode("utf-8").) Impact is currently masked because format_paths maps both REFORMATTED and NO_CHANGE to exit 0, but the new return value is wrong for the stdin path. Fix: compare against original_data.decode("utf-8"). Reproduced: identical content still compares unequal.

**3. docs.py:65 / docs.py:115 — _remove_indentation's assert line.startswith(" " * indent)can crashextract_inline_code.** extract_inline_codenow calls_remove_indentation(lines_to_extract, indent)wherelines_to_extractincludes *both* fence lines. Per CommonMark, a closing fence (or an under-indented content line) may have fewer leading spaces than the opening fence, sostartswith(" " * indent)isFalse→AssertionError, aborting extraction for the whole run. This bites exactly the "indented code blocks" the PR targets. It's also pure dead work — the resulting "lines"field is never read (the author's own TODO at docs.py:72 says so, and grep confirms line 69 is the only reference). Safe only in the commonindent == 0` case. Consider dropping the dead computation, or replacing the assert with graceful handling.

**4. docs.py:240 — enabling cfengine3 makes the cf autoformat path live, but PolicySyntaxErroris uncaught.** Withlanguages=["json", "cfengine3"](docs.py:460),fn_autoformat's "cf"case now runsformat_policy_file(snippet_path, 80, False), which raises PolicySyntaxErroron unparseable policy._process_markdown_code_blocksonly wrapsfn_check_syntaxin try/except — not the autoformat path — so any cfengine3 code block with a partial/pseudo snippet abortscfengine dev format-docswith an unhandled exception. The comment at docs.py:239 ("Dead code - Not used for CFEngine policy yet") is now stale and misleading. Authors would neednoautoformat/skip` flags on every non-parseable snippet.

**5. docs.py:123 — lines = [x[0:-1] for x in lines]drops the last character of a line without a trailing newline.**fn_extractnow usesreadlines()then strips the final char of every line assuming it's\n. For a file whose relevant line lacks a trailing newline (e.g. a snippet ending at EOF), this removes a real character, corrupting the extracted snippet. The prior content.split("\n")` handled this correctly. Edge case (requires the code block or file to end without a newline), but a silent regression.

Findings 1 and 2 are confirmed by direct reproduction; 3–5 are plausible/mechanism-confirmed with the trigger conditions noted. Items 1 and 4 are the ones I'd block on — both turn ordinary inputs into crashes now that cfengine3 processing is enabled.