Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix gradient accumulation for Z2+offload #6550

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

tohtana
Copy link
Contributor

@tohtana tohtana commented Sep 18, 2024

The ZeRO 1/2 optimizer performs incorrect gradient accumulation in the path for ZeRO2 + Offloading. This issue is caused by two main reasons:

  1. The micro_step_id in the ZeRO 1/2 optimizer is:
  • Initialized to 0 in the constructor.
  • Reset to -1 during the backward pass.

For example, given a gradient accumulation step of 4, the micro_step_id changes as follows:

  • For the first global step: 1, 2, 3, 4.
  • Subsequently: 0, 1, 2, 3.
  1. Gradients are copied to the buffer on the first micro step and accumulated in the buffer during the following micro steps. However, the current code incorrectly copies gradients at steps that are not at the accumulation boundary.

This PR aligns the micro_step_id initialization in both the constructor and the backward pass, and corrects the condition for copying and accumulating gradients.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants