Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update SparseGPT updates to respect base model's sparsity #2222

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

abhinavnmagic
Copy link
Contributor

Motivation

If we run SparseGPT on a base model at some sparsity, the sparsity mask after SparseGPT could be very different from the initial one. In other words, SparseGPT doesn't preserve the initial sparsities of the base model. This PR fixes that.
If preserve_sparsity_mask=True in the recipe, the sparsity mask of the base model being pruned is preserve. This can be useful for multiple scenarios: one-shot from 50->70% sparsity, going from unstructured to structured sparsity or vice versa, etc.

Testing

The implementation was tested by applying 2:4 sparsity on a 70% sparse model. The resulting model was checked if it preserves the 70% sparsity masks (ie. weights which were pruned out in the base model remain pruned), and if it has 2:4 sparsity pattern.

Recipe

test_stage:
  obcq_modifiers:
    SparseGPTModifier:
      sparsity: 0.5
      block_size: 128
      sequential_update: false
      quantize: false
      percdamp: 0.01
      mask_structure: "2:4"
      targets: ["re:model.layers.\\d*$"]
      preserve_sparsity_mask: true

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants