fix(export): prevent silent MLP linear collapse during HF export#1997
Conversation
Signed-off-by: Surya P R <suryaravichandran5555@gmail.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. 🗂️ Base branches to auto review (3)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
@AAnoosheh @kevalmorabia97 This PR addresses the silent MLP linear collapse bug during Hugging Face export when It introduces strict validation boundaries and unrolling logic for partial wrappers to ensure the exporter safely halts rather than generating a corrupted model. Daniel requested your review on this fix so it can be merged into the distillation branch (targeting PR #1897). Please let me know if you have any feedback or require further modifications! |
…distill' into fix/hf-export-activation-none
|
|
||
| def _get_hidden_act(act_func) -> str: | ||
| """Returns the name of the hidden activation function based on ACT2FN.""" | ||
| if act_func is None: |
There was a problem hiding this comment.
Let's abandon this fix. We changed approach and will export all hf checkpoints once distillation is finished, see in MR: #1897
What does this PR do?
Type of change: Bug fix
This PR addresses the catastrophic drop in MMLU accuracy that occurs during Hugging Face export when the teacher is detached and the student's config is forced to
activation_func = None(or an un-pickleable lambda/partial) for serialization.Previously, when this sanitized config hit the
_get_hidden_actexport pipeline, it silently returnedNoneor raised a silent failure. This caused the Multi-Layer Perceptron (MLP) to lose its non-linearity entirely, collapsing into a purely linear projection and destroying the model's accuracy.Fixes:
_get_hidden_actto raise an explicitValueErrorifact_func is None, preventing the exporter from silently generating a non-activated linear model.__name__fromfunctools.partialwrappers.<lambda>functions from passing silently.pytestunit tests intest_layer_utils.pyto prevent future regressions.Usage
N/A - This is an internal fix to the export pipeline safeguards.*
Testing
Ran local
pytestunit tests verifying thatValueErrorexceptions are properly raised and halting the export pipeline forNone,<lambda>, and unmapped inputs, while safely unwrappingfunctools.partial.Before your PR is "Ready for review"
Make sure you read and follow Contributor guidelines and your commits are signed (
git commit -s -S).Make sure you read and follow the Security Best Practices (e.g. avoiding hardcoded
trust_remote_code=True,torch.load(..., weights_only=False),pickle, etc.).CONTRIBUTING.md: N/AAdditional Information
Builds on top of the temporary fixes introduced in
dkorzekwa/save_hf_checkpoint_for_every_val_iter_during_distill.