Add DAB-DETR Object detection/segmentation model #30803

conditionedstimulus · 2024-05-14T14:19:50Z

What does this PR do?

Add DAB-DETR Object detection model. Paper: https://arxiv.org/abs/2201.12329
Original code repo: https://github.com/IDEA-Research/DAB-DETR

Fixes # (issue)
[WIP] This model is part of how DETR models have evolved, alongside DN DETR (not part of this PR), to pave the way for newer and better models like Dino and Stable Dino in object detection

Who can review?

@amyeroberts

amyeroberts · 2024-05-15T14:36:33Z

Hi @conditionedstimulus, thanks for opening a PR!

Just skimming over the modeling files, it looks like all of the modules are copied from, or can be copied from conditional DETR. Are there any architectural changes this model brings? If not, then all we need to do is convert the checkpoints and upload those to the hub such that they can be loaded in ConditionalDETR directly

conditionedstimulus · 2024-05-15T20:38:01Z

Hi @conditionedstimulus, thanks for opening a PR!

Just skimming over the modeling files, it looks like all of the modules are copied from, or can be copied from conditional DETR. Are there any architectural changes this model brings? If not, then all we need to do is convert the checkpoints and upload those to the hub such that they can be loaded in ConditionalDETR directly

Hi Amy,

I attached a photo comparing the cross-attention of the decoder in DETR, Conditional DETR, and DAB DETR, as this is the main architectural difference. I copied the code from Conditional DETR because this model is an extension/evolved version of Conditional DETR. I believe it would be cool and useful to include this model in the HF object detection collection.

amyeroberts · 2024-05-17T12:00:10Z

@conditionedstimulus Thanks for sharing! OK, seems useful to have this available as an option as part of the DETR family in the library. Feel free to ping me when the PR is ready for review.

cc @qubvel for reference

qubvel

Thanks for fixing tests! While I asking the team to move checkpoints to the org, can you please update the last thing (I hope 😄)

qubvel · 2025-01-28T10:31:27Z

tests/models/dab_detr/test_modeling_dab_detr.py

+        self.assertEqual(len(results["scores"]), 5)
+        self.assertTrue(torch.allclose(results["scores"], expected_scores, atol=1e-4))
+        self.assertSequenceEqual(results["labels"].tolist(), expected_labels)
+        self.assertTrue(torch.allclose(results["boxes"][0, :], expected_boxes, atol=1e-4))


I hope the last thing! Can you please update to use torch.testing.assert_close instead of self.assertTrue(torch.allclose(...

Here and in other places in tests, for example:

https://github.com/huggingface/transformers/pull/35903/files

Hi, no problem I changed and ran the test w my model source. Is it enough to change only in that part of the tests or it should be in the whole file? Also apprx. how much time it's gonna take to move the model cards?:)
thanks

Other tests as well in tests/models/dab_detr folder, thanks!

Transfer should not take more than a few hours, just need review from Arthur once again to get his approval

…t assert_close

qubvel · 2025-01-31T14:41:28Z

Noticed we don't have approval from @ArthurZucker, waiting for his review

ArthurZucker

A few super small comments! Thanks for your patience! 🤗

docs/source/en/model_doc/dab-detr.md

src/transformers/models/dab_detr/modeling_dab_detr.py

ArthurZucker · 2025-01-31T15:24:11Z

src/transformers/models/dab_detr/modeling_dab_detr.py

+        h = [hidden_dim] * (num_layers - 1)
+        self.layers = nn.ModuleList(nn.Linear(n, k) for n, k in zip([input_dim] + h, h + [output_dim]))


no what I mean is we should only create n, k) for n, k in zip([input_dim] + h, h + [output_dim] in the config. then you know exactly in and out that should be used for the linear layers.

src/transformers/models/dab_detr/modeling_dab_detr.py

…/transformers into add_dab_detr

conditionedstimulus · 2025-02-01T14:42:04Z

Hi @ArthurZucker and @qubvel,

I’ve made most of the required modifications. Where I didn’t, I left comments on your feedback.
I also updated the test file where needed and added some additional information to the model card markdown file.

Thanks!

qubvel · 2025-02-03T13:05:58Z

@conditionedstimulus Thanks for the updates! Please update converted weights for other checkpoints on the Hub as well and I will ask for transfer

ArthurZucker

Let's go! 🚀

ArthurZucker · 2025-02-03T16:12:13Z

src/transformers/models/dab_detr/modeling_dab_detr.py

+        hidden_states = self.layernorm(hidden_states)
+        intermediate.pop()
+        intermediate.append(hidden_states)


intermediate_state = self.layernorm(hidden_states) intermediate.append(intermediate_states) ` vs `intermediate.append(self.layernorm(hidden_states))`

will avoid this ugly pop append

I removed the list manipulation entirely. I didn’t revisit the original code, but as I recall, this was part of a conditional section. Since we removed many configurations, the list manipulation remained unchanged—popping the last element and appending the same value back. So, I only kept the hidden states layer normalization.

conditionedstimulus · 2025-02-03T20:28:11Z

Hi @ArthurZucker and @qubvel,

I’ve finalized the last modification—if I understand correctly, this should be the final version, and we’ll roll it out soon.
I also updated the converted weights, so I believe it’s ready to be moved under the new organization.
I merged main too and ofc:

SKIPPED [1] tests/generation/test_utils.py:1458: The decoder-only derived from encoder-decoder models are not expected to support left-padding.
FAILED tests/models/qwen2_5_vl/test_modeling_qwen2_5_vl.py::Qwen2_5_VLModelTest::test_prompt_lookup_decoding_matches_greedy_search - IndexError: index 41 is out of bound

Thanks, for your review, guidance, and support! :)

Looking forward to the merge! 🤗

qubvel · 2025-02-03T22:16:45Z

run-slow: dab_detr

github-actions · 2025-02-03T22:18:04Z

This comment contains run-slow, running the specified jobs: ['models/dab_detr'] ...

qubvel · 2025-02-04T16:33:53Z

run-slow: dab_detr

github-actions · 2025-02-04T16:36:05Z

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/dab_detr']
quantizations: [] ...

ydshieh · 2025-02-04T17:11:12Z

run-slow: dab_detr

github-actions · 2025-02-04T17:12:28Z

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/dab_detr']
quantizations: [] ...

qubvel · 2025-02-04T17:16:13Z

run-slow: dab_detr

github-actions · 2025-02-04T17:17:24Z

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/dab_detr']
quantizations: [] ...

qubvel · 2025-02-04T17:36:03Z

@conditionedstimulus Congratulations on merging the model! 🎉 It was a long journey, and we really appreciate you were able to finish it 💪 . Thank you for your contribution, and sorry for the delays on our side. Great job! 🚀

And feel free to share your achievement on social networks, we’d be happy to amplify it!

conditionedstimulus · 2025-02-04T19:18:22Z

@qubvel, @ArthurZucker

Thank you guys!
It was a long and fun journey, and I truly appreciate your support and guidance. I'm glad I could contribute! :)

initial commit

8adf1bb

encoder+decoder layer changes WIP

8291122

conditionedstimulus added 25 commits May 21, 2024 21:48

architecture checks

09e2516

working version of detection + segmentation

8a004cf

fix modeling outputs

defbc43

fix return dict + output att/hs

5cfbcfc

found the position embedding masking bug

6c7564a

pre-training version

35e056f

added iamge processors

24a9d7a

typo in init.py

d9b7af4

iterupdate set to false

a171339

fixed num_labels in class_output linear layer bias init

b8b2201

multihead attention shape fixes

abe0698

test improvements

e60b555

test update

6dafb79

dab-detr model_doc update

5bbdca1

dab-detr model_doc update2

4a5ac4f

test fix:test_retain_grad_hidden_states_attentions

592796b

config file clean and renaming variables

d76fda2

config file clean and renaming variables fix

ade9720

updated convert_to_hf file

6b58e5f

small fixes

eac19f5

style and qulity checks

460e9d6

Merge branch 'main' into add_dab_detr

0151f65

return_dict fix

97194c7

Merge branch main into add_dab_detr

3fc56b4

Merge branch main into add_dab_detr

ffbb1dc

qubvel approved these changes Jan 28, 2025

View reviewed changes

conditionedstimulus and others added 3 commits January 28, 2025 12:36

changed test_inference_object_detection_head assertTrues to torch tes…

ac8f4cb

…t assert_close

Merge branch 'main' into add_dab_detr

3cf9b99

Merge branch 'main' into add_dab_detr

3931e5c

ArthurZucker approved these changes Jan 31, 2025

View reviewed changes

conditionedstimulus added 7 commits January 31, 2025 22:56

fixes part 1

ed7f8f5

Merge branch 'add_dab_detr' of https://github.com/conditionedstimulus…

c962ef1

…/transformers into add_dab_detr

quality update

e08e6f8

self.bbox_embed in decoder has been restored

3f8981b

Merge branch 'main' into add_dab_detr

52e5131

changed Assert true torch closeall methods to torch testing assertclose

757f413

modelcard markdown file has been updated

f1ba30e

ArthurZucker approved these changes Feb 3, 2025

View reviewed changes

conditionedstimulus added 2 commits February 3, 2025 21:02

deleted intemediate list from decoder module

46710c3

Merge branch 'main' into add_dab_detr

350e6af

qubvel merged commit 8d73a38 into huggingface:main Feb 4, 2025
26 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add DAB-DETR Object detection/segmentation model #30803

Add DAB-DETR Object detection/segmentation model #30803

conditionedstimulus commented May 14, 2024

amyeroberts commented May 15, 2024

conditionedstimulus commented May 15, 2024

amyeroberts commented May 17, 2024

qubvel left a comment •

edited

Loading

qubvel Jan 28, 2025

qubvel Jan 28, 2025

conditionedstimulus Jan 28, 2025

qubvel Jan 31, 2025

qubvel Jan 31, 2025 •

edited

Loading

qubvel commented Jan 31, 2025 •

edited

Loading

ArthurZucker left a comment

ArthurZucker Jan 31, 2025

conditionedstimulus commented Feb 1, 2025

qubvel commented Feb 3, 2025

ArthurZucker left a comment

ArthurZucker Feb 3, 2025

ArthurZucker Feb 3, 2025

conditionedstimulus Feb 3, 2025

conditionedstimulus commented Feb 3, 2025

qubvel commented Feb 3, 2025

github-actions bot commented Feb 3, 2025

qubvel commented Feb 4, 2025

github-actions bot commented Feb 4, 2025

ydshieh commented Feb 4, 2025

github-actions bot commented Feb 4, 2025

qubvel commented Feb 4, 2025

github-actions bot commented Feb 4, 2025

qubvel commented Feb 4, 2025

conditionedstimulus commented Feb 4, 2025 •

edited

Loading

		h = [hidden_dim] * (num_layers - 1)
		self.layers = nn.ModuleList(nn.Linear(n, k) for n, k in zip([input_dim] + h, h + [output_dim]))

Add DAB-DETR Object detection/segmentation model #30803

Add DAB-DETR Object detection/segmentation model #30803

Conversation

conditionedstimulus commented May 14, 2024

What does this PR do?

Who can review?

amyeroberts commented May 15, 2024

conditionedstimulus commented May 15, 2024

amyeroberts commented May 17, 2024

qubvel left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qubvel Jan 31, 2025 • edited Loading

Choose a reason for hiding this comment

qubvel commented Jan 31, 2025 • edited Loading

ArthurZucker left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

conditionedstimulus commented Feb 1, 2025

qubvel commented Feb 3, 2025

ArthurZucker left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

conditionedstimulus commented Feb 3, 2025

qubvel commented Feb 3, 2025

github-actions bot commented Feb 3, 2025

qubvel commented Feb 4, 2025

github-actions bot commented Feb 4, 2025

ydshieh commented Feb 4, 2025

github-actions bot commented Feb 4, 2025

qubvel commented Feb 4, 2025

github-actions bot commented Feb 4, 2025

qubvel commented Feb 4, 2025

conditionedstimulus commented Feb 4, 2025 • edited Loading

qubvel left a comment •

edited

Loading

qubvel Jan 31, 2025 •

edited

Loading

qubvel commented Jan 31, 2025 •

edited

Loading

conditionedstimulus commented Feb 4, 2025 •

edited

Loading