fix Swin Transformer inplace mutation (#6266)

* fix inplace mutation

* Different attn shouldn't share the same attribute

* a simpler solution

Co-authored-by: YosuaMichael <[email protected]>

Author: ain-soph

torchvision/models/swin_transformer.py

@@ -106,6 +106,7 @@ def shifted_window_attention(
     x = F.pad(input, (0, 0, 0, pad_r, 0, pad_b))
     _, pad_H, pad_W, _ = x.shape
 
+    shift_size = shift_size.copy()
     # If window size is larger than feature size, there is no need to shift window
     if window_size[0] >= pad_H:
         shift_size[0] = 0