NOTE
要复现论文的github：SAMWISE

下载并安装 Miniconda#

1
sudo apt update

1
cd ~

1
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

1
bash Miniconda3-latest-Linux-x86_64.sh

1
Do you wish the installer to initialize Miniconda3? yes

创建干净环境（Python 3.10）#

1
conda create -n samwise python=3.10

1
conda activate samwise

安装 RTX5090 必须的 PyTorch（Nightly cu128）#

1
pip install --pre torch torchvision torchaudio \
2
  --index-url https://download.pytorch.org/whl/nightly/cu128

验证 GPU 是否正确支持#

1
python - <<'PY'
2
import torch
3
print("torch:", torch.__version__)
4
print("cuda runtime:", torch.version.cuda)
5
print("gpu available:", torch.cuda.is_available())
6
print("gpu name:", torch.cuda.get_device_name(0))
7
PY

拉取 SAMWISE 工程#

1
cd ~

1
git clone https://github.com/ClaudiaCuttano/SAMWISE.git

1
cd SAMWISE

安装依赖#

1
pip install -U pip

先删除 requirements.txt 中的 pyav ，之后单独装

1
pip install -r requirements.txt

1
sudo apt install -y ffmpeg libavdevice-dev libavfilter-dev libavformat-dev libavcodec-dev libswscale-dev libswresample-dev

1
pip install av==12.0.0

快速 sanity check#

1
python -c "from models.samwise import build_samwise; print('SAMWISE import OK')"

下载Ref-DAVIS17数据集#

Downlaod the DAVIS2017 dataset from the website. Note that you only need to download the two zip files DAVIS-2017-Unsupervised-trainval-480p.zip and DAVIS-2017_semantics-480p.zip. Download the text annotations from the website. Then, put the zip files in the directory as follows.

1
SAMWISE
2
├── data
3
│   ├── ref-davis
4
│   │   ├── DAVIS-2017_semantics-480p.zip
5
│   │   ├── DAVIS-2017-Unsupervised-trainval-480p.zip
6
│   │   ├── davis_text_annotations.zip

1
wget https://data.vision.ee.ethz.ch/csergi/share/davis/DAVIS-2017-Unsupervised-trainval-480p.zip

1
wget https://data.vision.ee.ethz.ch/csergi/share/davis/DAVIS-2017_semantics-480p.zip

1
wget https://www.mpi-inf.mpg.de/fileadmin/inf/d2/khoreva/davis_text_annotations.zip

Unzip these zip files.

1
unzip -o davis_text_annotations.zip
2
unzip -o DAVIS-2017_semantics-480p.zip
3
unzip -o DAVIS-2017-Unsupervised-trainval-480p.zip

Preprocess the dataset to Ref-Youtube-VOS format. (Make sure you are in the main directory)

1
python tools/data/convert_davis_to_ytvos.py

Finally, unzip the file DAVIS-2017-Unsupervised-trainval-480p.zip again (since we use mv in preprocess for efficiency).

1
unzip -o DAVIS-2017-Unsupervised-trainval-480p.zip

下载 Ref-DAVIS17 Model#

1
SAMWISE
2
├── pretrain
3
│   ├── final_model_mevis.pth
4
│   ├── final_model_ytvos.pth

内网无法访问 google drive，自行下载到电脑上后，上传至服务器

Dataset	Total Parameters	Trainable Params	J&F	Model	Zip
MeViS	210 M	4.9 M	49.5	Weights	Zip
MeViS - valid_u	210 M	4.9 M	57.1	Weights	-
Ref-Youtube-VOS	210 M	4.9 M	69.2	Weights	Zip
Ref-Davis	210 M	4.9 M	70.6	Weights	-

验证 Ref-DAVIS17 实验数据#

由于我用的是5090，所以 cuda 和 torch 版本都不太一样

先修改文件 /SAMWISE/inference_davis.py

我的服务器连接不上 huggingface.co ，所以重定向到国内镜像网站 hf-mirror.com

1
export HF_ENDPOINT=https://hf-mirror.com
2
export HF_HUB_ENABLE_HF_TRANSFER=0

开始验证：

1
python3 inference_davis.py --resume=/root/SAMWISE/pretrain/final_model_ytvos.pth --name_exp Ref-DAVIS17-evaluate  --HSA --use_cme_head