Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Apple's Depth-Pro #34583

Open
wants to merge 169 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
169 commits
Select commit Hold shift + click to select a range
2986dc2
implement config and model building blocks
geetu040 Nov 3, 2024
1728a2f
refactor model architechture
geetu040 Nov 9, 2024
11ce50c
update model outputs
geetu040 Nov 12, 2024
27e9593
update init param to include use_fov_model
geetu040 Nov 16, 2024
e74a7f5
update param name in config
geetu040 Nov 16, 2024
8c2460b
fix hidden_states and attentions outputs for fov
geetu040 Nov 16, 2024
55f6ed3
sort config
geetu040 Nov 16, 2024
b25dffb
complete minor todos
geetu040 Nov 16, 2024
c225deb
update patching
geetu040 Nov 16, 2024
176932d
update config for encoder
geetu040 Nov 16, 2024
dcec522
fix config
geetu040 Nov 16, 2024
0384d2f
use correct defaults in config
geetu040 Nov 16, 2024
85e4f86
update merge for compatibility with different image size
geetu040 Nov 17, 2024
00e4aa3
restructure encoder for custom configuration
geetu040 Nov 21, 2024
6be242c
make fov model compatible with custom config
geetu040 Nov 21, 2024
0189108
replace word "decoder" with "fusion"
geetu040 Nov 21, 2024
7614e1a
weight conversion script
geetu040 Nov 24, 2024
7d323ce
fix fov squeeze
geetu040 Nov 25, 2024
6aaa59e
update conversion script (without test)
geetu040 Nov 25, 2024
263b773
upload ruff image processing
geetu040 Nov 25, 2024
17e5487
create fast image processing
geetu040 Nov 26, 2024
a8dd704
use torch interpolation for image processing
geetu040 Nov 26, 2024
261bbaf
complete post_process_depth_estimation
geetu040 Nov 26, 2024
a4b3556
config: fix imports and sort args
geetu040 Nov 26, 2024
f13c632
apply inference in weight conversion
geetu040 Nov 26, 2024
387ddd8
use mllama script instead for weight conversion
geetu040 Nov 27, 2024
9b67f9d
clean weight conversion script
geetu040 Nov 27, 2024
617c872
add depth-pro status in other files
geetu040 Nov 27, 2024
6e1c512
fill docstring in config
geetu040 Nov 27, 2024
12ee607
formatting
geetu040 Nov 27, 2024
d0a8733
more formatting
geetu040 Nov 27, 2024
e6b385a
formatting with ruff
geetu040 Nov 27, 2024
267e50f
formatting with style
geetu040 Nov 27, 2024
a1ec997
fix copied classes
geetu040 Nov 27, 2024
3c656f2
add examples; update weight convert script
geetu040 Nov 27, 2024
f6f6d3d
fix using check_table.py and isort
geetu040 Nov 29, 2024
b4575d0
fix config docstring
geetu040 Nov 29, 2024
c8d8a9e
add depth pro to sdpa docs
geetu040 Nov 29, 2024
77873de
undo unintentional changes in configuration_gemma.py
geetu040 Nov 29, 2024
5f2378d
minor fixes
geetu040 Nov 30, 2024
d51d0b1
test image processing
geetu040 Nov 30, 2024
082b055
fixes and tests
geetu040 Dec 2, 2024
16a3917
more fixes
geetu040 Dec 2, 2024
2408ec5
use output states from image_encoder instead
geetu040 Dec 3, 2024
be0c2a3
Revert "use output states from image_encoder instead"
geetu040 Dec 4, 2024
efed39f
make embeddings dynamic
geetu040 Dec 4, 2024
c3b14fb
reshape output hidden states and attentions as part of computation graph
geetu040 Dec 4, 2024
7cf2485
fix ruff formating
geetu040 Dec 4, 2024
0aa451d
fix docstring failure
geetu040 Dec 4, 2024
160afbf
use num_fov_head_layers in tests
geetu040 Dec 4, 2024
9d2be26
update doc
geetu040 Dec 4, 2024
e208459
check consistency with config
geetu040 Dec 4, 2024
0415722
ruff formatting
geetu040 Dec 4, 2024
402eedf
merge branch main
geetu040 Dec 4, 2024
f4e7404
update test case
geetu040 Dec 5, 2024
2c1cc10
fix ruff formatting
geetu040 Dec 5, 2024
4d94396
Merge branch 'main' into depth-pro
geetu040 Dec 5, 2024
871b80d
add tests for fov
geetu040 Dec 6, 2024
0ff0655
use interpolation in postprocess
geetu040 Dec 6, 2024
befa6cd
run and fix slow tests locally
geetu040 Dec 6, 2024
db16fe6
Merge branch 'main' into depth-pro
geetu040 Dec 6, 2024
99ac5e8
use scaled_images_features for image and fov encoder
geetu040 Dec 12, 2024
ebb62dd
return fused_hidden_states in fusion stage
geetu040 Dec 12, 2024
46c88e8
fix example
geetu040 Dec 12, 2024
2431358
fix ruff
geetu040 Dec 12, 2024
fd38841
Merge branch 'main' into depth-pro
geetu040 Dec 12, 2024
d9d3a49
fix copyright license for all files
geetu040 Dec 21, 2024
8f4c61f
add __all__ for each file
geetu040 Dec 21, 2024
8960535
minor fixes
geetu040 Dec 21, 2024
1ac1b84
return list in post_process_depth_estimation
geetu040 Dec 21, 2024
27bff69
minor fixes
geetu040 Dec 21, 2024
a69b5af
fix "ruff check"
geetu040 Dec 21, 2024
365a71d
update upsample and projection
geetu040 Dec 21, 2024
c009468
major changes: (image size and merge optimization)
geetu040 Dec 24, 2024
7bed369
Merge branch 'main' into depth-pro
geetu040 Dec 24, 2024
1563f06
fix push_to_hub option in weights conversion
geetu040 Dec 24, 2024
e194ae4
remove image_size in weights conversion
geetu040 Dec 24, 2024
a4889f2
major changes in the architecture
geetu040 Jan 14, 2025
be5087b
Merge branch "main"
geetu040 Jan 14, 2025
9e09a6f
placeholder for unused config attributes
geetu040 Jan 14, 2025
bf159b2
improve docs amid review
geetu040 Jan 14, 2025
fb41687
minor change in docs
geetu040 Jan 14, 2025
7fbb53e
further optimize merge
geetu040 Jan 15, 2025
558836c
fix formatting
geetu040 Jan 15, 2025
ed77f78
remove unused patch/batch convertion functions
geetu040 Jan 24, 2025
5bc4b31
use original F.interpolate
geetu040 Jan 24, 2025
628ff09
improve function naming
geetu040 Jan 24, 2025
e2996b6
minor chages
geetu040 Jan 24, 2025
8cb5c7a
rearchitect upsample block for improved modularity
geetu040 Jan 24, 2025
1ba3a4a
update upsample keys in weight conversion
geetu040 Jan 24, 2025
83706b8
improve padding in merge_patches
geetu040 Jan 25, 2025
004cdc2
use double-loop for merge
geetu040 Jan 25, 2025
922b3de
update comments
geetu040 Jan 25, 2025
0f01b08
create feature_extractor, reduce some forward code
geetu040 Jan 25, 2025
4d871a7
introduce config.use_mask_token in dinov2
geetu040 Jan 26, 2025
85f7e3a
minor fixes
geetu040 Jan 26, 2025
c0127d7
minor fixes for onnx
geetu040 Jan 26, 2025
1898459
update __init__ to latest format
geetu040 Jan 26, 2025
bcf1bf3
remove DepthProConfig.to_dict()
geetu040 Jan 26, 2025
09bffc3
major changes in backbone
geetu040 Jan 26, 2025
0936897
Merge branch 'main' into depth-pro
geetu040 Jan 26, 2025
c26dc99
update config in weight conversion
geetu040 Jan 26, 2025
5fb0bb7
formatting
geetu040 Jan 26, 2025
d741890
converted model is fp32
geetu040 Jan 26, 2025
03f137d
improve naming and docs for feature_extractor->reconstruct_feature_maps
geetu040 Jan 28, 2025
2b8ee8f
minor fixes; amid review
geetu040 Jan 28, 2025
774617a
create intermediate vars in func call
geetu040 Jan 28, 2025
b6d15ff
use torch.testing.assert_close
geetu040 Jan 28, 2025
425d63e
use ModuleList instead of Sequential and ModuleDict
geetu040 Jan 28, 2025
f415ee6
update docs
geetu040 Jan 28, 2025
2777305
Merge branch 'main' into depth-pro
geetu040 Jan 28, 2025
1a2dd3a
include fov in integraiton tests
geetu040 Jan 30, 2025
4cfebae
update docs
geetu040 Jan 30, 2025
9062767
improve initialization of convolution layers
geetu040 Jan 30, 2025
fcba6bd
fix unused fov keys
geetu040 Jan 30, 2025
56cd570
update tests
geetu040 Jan 30, 2025
e64d39a
Merge branch 'main' into depth-pro
geetu040 Jan 30, 2025
26b1391
ruff format
geetu040 Jan 30, 2025
8914549
Merge branch 'main' into depth-pro
geetu040 Jan 31, 2025
01247f8
fix test, amid kaimming initialization
geetu040 Jan 31, 2025
0b7e77f
add depthpro to toctree
geetu040 Jan 31, 2025
20b277d
add residual layer to _no_split_modules
geetu040 Jan 31, 2025
ff0e408
architecture rework
geetu040 Feb 1, 2025
1522c53
Update src/transformers/models/depth_pro/image_processing_depth_pro.py
geetu040 Feb 1, 2025
131817a
Update src/transformers/models/depth_pro/image_processing_depth_pro_f…
geetu040 Feb 1, 2025
72a1f0c
update docs
geetu040 Feb 1, 2025
aed7e3d
improve merge_patches
geetu040 Feb 1, 2025
405bee3
use flatten with fov_output
geetu040 Feb 1, 2025
a8528da
ruff formatting
geetu040 Feb 1, 2025
aed655c
Merge branch 'main' into depth-pro
geetu040 Feb 1, 2025
31383e1
update resources section in docs
geetu040 Feb 3, 2025
641cb84
fix typo "final_kernal_size"
geetu040 Feb 3, 2025
6af8a11
fix output typehint for DepthProDepthEstimator
geetu040 Feb 3, 2025
abd5307
residual operation in 2 steps
geetu040 Feb 3, 2025
8dc2751
use image_size instead of global patch_size in interpolation
geetu040 Feb 3, 2025
2f88694
replace all Sequential with ModuleList
geetu040 Feb 3, 2025
208ee26
update fov
geetu040 Feb 3, 2025
bc63511
update heads
geetu040 Feb 3, 2025
e33a531
fix and update conversion script for heads
geetu040 Feb 3, 2025
8c0e81a
ruff formatting
geetu040 Feb 3, 2025
524dda6
remove float32 conversion
geetu040 Feb 3, 2025
029dd9d
Merge branch 'main' into depth-pro
geetu040 Feb 3, 2025
a87d26a
use "Fov" instead of "FOV" in class names
geetu040 Feb 4, 2025
5fccbff
use "Fov" instead of "FOV" in config docs
geetu040 Feb 4, 2025
24f1413
remove prune_heads
geetu040 Feb 4, 2025
a3dab18
update fusion stage
geetu040 Feb 4, 2025
48eb534
use device in examples
geetu040 Feb 4, 2025
39ea929
Merge branch 'main' into depth-pro
geetu040 Feb 4, 2025
26db9ec
Merge branch 'main' into depth-pro
geetu040 Feb 5, 2025
ba37c91
update processor
geetu040 Feb 5, 2025
949ecb9
ruff fixes
geetu040 Feb 5, 2025
0e2861d
add do_rescale in image_processor_dict
geetu040 Feb 5, 2025
a6efedb
skip test: test_fast_is_faster_than_slow
geetu040 Feb 5, 2025
4d8f927
ruff formatting
geetu040 Feb 5, 2025
dd8de27
DepthProImageProcessorFast in other files
geetu040 Feb 5, 2025
75215ed
Merge branch 'main' into depth-pro
geetu040 Feb 5, 2025
ffb3a82
Merge branch 'main' into depth-pro
geetu040 Feb 5, 2025
5caa0bd
revert antialias removal
geetu040 Feb 5, 2025
3ae1134
add antialias in BaseImageProcessorFast
geetu040 Feb 5, 2025
8372ad9
Revert "revert antialias removal"
geetu040 Feb 5, 2025
666f3b7
Revert "add antialias in BaseImageProcessorFast"
geetu040 Feb 5, 2025
41180e3
update processor for grouping and antialias
geetu040 Feb 5, 2025
1265b12
try test_fast_is_faster_than_slow without "skip" or "flanky"
geetu040 Feb 5, 2025
86c4604
Merge branch 'main' into depth-pro
geetu040 Feb 5, 2025
4dc850f
update checkpoint
geetu040 Feb 5, 2025
b7f32b9
Merge branch 'main' into depth-pro
geetu040 Feb 5, 2025
592648c
update checkpoint
geetu040 Feb 6, 2025
162f141
use @is_flanky for processor test
geetu040 Feb 6, 2025
3a62d63
Merge branch 'main' into depth-pro
geetu040 Feb 6, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions tests/models/depth_pro/test_image_processing_depth_pro.py
Original file line number Diff line number Diff line change
Expand Up @@ -117,8 +117,8 @@ def test_image_processor_from_dict_with_kwargs(self):
image_processor = self.image_processing_class.from_dict(self.image_processor_dict, size=42)
self.assertEqual(image_processor.size, {"height": 42, "width": 42})

@is_flaky(
description="fast and slow, both processors use torch implementation, see: https://github.com/huggingface/transformers/issues/34920",
@unittest.skip(
reason="both processors (fast and slow) use torch for resizing, check: https://github.com/huggingface/transformers/issues/34920",
)
def test_fast_is_faster_than_slow(self):
super().test_fast_is_faster_than_slow()
pass
geetu040 marked this conversation as resolved.
Show resolved Hide resolved