Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate accuracy issues in working models #739

Open
ayerofieiev-tt opened this issue Feb 4, 2025 · 0 comments
Open

Investigate accuracy issues in working models #739

ayerofieiev-tt opened this issue Feb 4, 2025 · 0 comments
Assignees
Labels

Comments

@ayerofieiev-tt
Copy link
Member

Among the 66 models running end-to-end in TT-NN, the following models exhibit accuracy deviations:

Models with Accuracy Issues:

  1. Bloom: 43.3
  2. albert/albert-base-v2: 69.04
  3. albert/albert-large-v2: 23.63
  4. albert/albert-xlarge-v2: 53.42
  5. mobilenetv1_100.ra4_e3600_r224_in1k: -6.46
  6. regnet_x_16gf: 1.01
  7. regnet_x_32gf: 3.94

Model with Accuracy Marked as N/A:

  1. Unet-brain: N/A

Action Required:

  1. Investigate and diagnose the causes of the accuracy deviations.
  2. If the issue is in TT-NN operation, determine what operation on what input causes an issue. Fire a ticket to tt-metal, link here
  3. Validate expected accuracy baselines for each model.
  4. Assess Unet-brain to determine if accuracy reporting is feasible.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: In Progress
Development

No branches or pull requests

2 participants