unexpected keys

[3d-illusions]

with the model path set to use the dpt_beit_large_512.pt model I downloaded, I get the following error :

model_loader.py", line 54, in load_model
    model = DPTDepthModel(
            ^^^^^^^^^^^^^^
dpt_depth.py", line 163, in __init__
    self.load(path)
RuntimeError: Error(s) in loading state_dict for DPTDepthModel:
        Unexpected key(s) in state_dict: "pretrained.model.blocks.0.attn.relative_position_index", "pretrained.model.blocks.1.attn.relative_position_index", "pretrained.model.blocks.2.att
n.relative_position_index", "pretrained.model.blocks.3.attn.relative_position_index", "pretrained.model.blocks.4.attn.relative_position_index", "pretrained.model.blocks.5.attn.rela
tive_position_index", "pretrained.model.blocks.6.attn.relative_position_index", "pretrained.model.blocks.7.attn.relative_position_index", "pretrained.model.blocks.8.attn.relative_p
osition_index", "pretrained.model.blocks.9.attn.relative_position_index", "pretrained.model.blocks.10.attn.relative_position_index", "pretrained.model.blocks.11.attn.relative_posit
ion_index", "pretrained.model.blocks.12.attn.relative_position_index", "pretrained.model.blocks.13.attn.relative_position_index", "pretrained.model.blocks.14.attn.relative_position
_index", "pretrained.model.blocks.15.attn.relative_position_index", "pretrained.model.blocks.16.attn.relative_position_index", "pretrained.model.blocks.17.attn.relative_position_in
dex", "pretrained.model.blocks.18.attn.relative_position_index", "pretrained.model.blocks.19.attn.relative_position_index", "pretrained.model.blocks.20.attn.relative_position_index
", "pretrained.model.blocks.21.attn.relative_position_index", "pretrained.model.blocks.22.attn.relative_position_index", "pretrained.model.blocks.23.attn.relative_position_index".

[3d-illusions]

I'm attempting to use it with python 3.11.7. It's not an option to use a lower python version unfortunately.

pip -m install torch torchvision opencv-python==4.6.0.66 timm einops==0.6.0 imutils==0.5.4 --user

If I try and pip install timm==0.6.12 I get

RuntimeError: Error: mutable default <class 'timm.models.maxxvit.MaxxVitConvCfg'> for field conv_cfg is not allowed: use default_factory

[3d-illusions]

Using with python 3.10 has no issues. How can I get this working with python 3.11.7?

[3d-illusions]

It seems that timm needed to be modified to work with python 3.11, huggingface/pytorch-image-models@9ecab46

Now that timm is updated, MiDaS no longer works. Where would I need to modify the Midas code, or is it the model itself that would need updating?

[heyoeyo]

Based on the first error you posted, it looks like maybe the newer versions of the timm library don't include weights for the relative_position_index (since every listed key seems to include that). A simple thing to try is to let the model load the weights without strict mode enabled. You'd have to modify the loading line to something like:

self.load_state_dict(parameters, strict=False)

If that doesn't work, you could try explicitly removing those keys from the loaded parameters variable in that same code block (i.e. in a for loop, delete all the keys of the parameters dictionary that end with attn.relative_position_index before trying to load the model).

If that doesn't work, then (shameless plug) I have a script that can load the beit models and doesn't depend on timm. It also has support for the depth-anything models which tend to give better results if you don't specifically need beit. Also, for really good (but slow) depth results, it's worth checking out the marigold model if you haven't already.

[3d-illusions]

Thanks very much, I'll give those things a try 👍