Results reproduction failure.

[ZhemengDong]

Hello, thanks for your great work! I'm encountering an issue with reproducing the paper results. The provided evaluation scripts are producing different metrics than those reported in the paper on the Sintel dataset:

Method | ATE | RTE | RRE
CUT3R | 0.21182 | 0.07148 | 0.63300
TTT3R | 0.20303 | 0.09366 | 0.72005

while the metrics in paper are:

Method | ATE | RTE | RRE
CUT3R | 0.213 | 0.066 | 0.621
TTT3R | 0.201 | 0.063 | 0.617

Are there any additional setup steps required? Perhaps I missed steps important during the evaluation.

[rover-xingyu]

Hi @ZhemengDong, thank you for your interest!

We have double-checked the evaluation and confirmed that we can reproduce the reported results using the provided script: bash eval/relpose/run_sintel.sh.

Could you please confirm if there were any modifications made to the code, model, or the data?

Additionally, I recommend running the evaluation on ScanNet to see if the results align with the paper. Since ScanNet contains significantly more test scenes compared to Sintel, the evaluation metrics tend to be much more stable and less variance.

[ZhemengDong]

Hello, thank you for your reply!
I have tried again to reproduce the camera pose metrics on the Sintel dataset, but got the same metrics:

Average ATE: 0.20303, Average RPE trans: 0.09366, Average RPE rot: 0.72005

To run the model successfully, I modified two locations:

1. File src/dust3r/model.py Line 411:
   From
   ckpt = torch.load(model_path, map_location="cpu")
   To
   ckpt = torch.load(model_path, map_location="cpu", weights_only=False)

2. File src/croco/models/curope/setup.py Line 101:
   From
   AT_DISPATCH_FLOATING_TYPES_AND_HALF(tokens.type(), "rope_2d_cuda", ([&] {
   To
   AT_DISPATCH_FLOATING_TYPES_AND_HALF(tokens.scalar_type(), "rope_2d_cuda", ([&] {

Otherwise I cannot run the code successfully. Maybe the two modifications contribute to the problem?

Thank you again for the amazing work!!

[rover-xingyu]

Thanks for the details!

Judging from the code changes, maybe you are using a much newer PyTorch version (likely 2.4+) than ours. We used conda install pytorch torchvision pytorch-cuda=12.1 ... (which installed an older version back then).

Maybe the version mismatch contributes to the numerical difference in results. Maybe you can try downgrading to pytorch==2.1.0 to see if the results align?

[ZhemengDong]

I will downgrade the pytorch and try it again. Thanks!

[ZhemengDong]

Hi, I downgraded pytorch to 2.1.0 and torchvision to 0.16.0 and canceled the two modifications before. But It still doesn't work...

Average ATE: 0.20424, Average RPE trans: 0.09347, Average RPE rot: 0.72422

The entire metrics are as below:

sintel-alley_2 | ATE: 0.02489, RPE trans: 0.02032, RPE rot: 0.17895
0.02489
0.02032
0.17895
sintel-ambush_4 | ATE: 0.15796, RPE trans: 0.06682, RPE rot: 1.38068
0.15796
0.06682
1.38068
sintel-ambush_5 | ATE: 0.07304, RPE trans: 0.03730, RPE rot: 0.59052
0.07304
0.03730
0.59052
sintel-ambush_6 | ATE: 0.24266, RPE trans: 0.11901, RPE rot: 2.12901
0.24266
0.11901
2.12901
sintel-cave_2 | ATE: 1.03254, RPE trans: 0.34471, RPE rot: 2.23133
1.03254
0.34471
2.23133
sintel-cave_4 | ATE: 0.14937, RPE trans: 0.04591, RPE rot: 0.57011
0.14937
0.04591
0.57011
sintel-market_2 | ATE: 0.03460, RPE trans: 0.02115, RPE rot: 0.02698
0.03460
0.02115
0.02698
sintel-market_5 | ATE: 0.26117, RPE trans: 0.20255, RPE rot: 0.74048
0.26117
0.20255
0.74048
sintel-market_6 | ATE: 0.10436, RPE trans: 0.04364, RPE rot: 0.40252
0.10436
0.04364
0.40252
sintel-shaman_3 | ATE: 0.00880, RPE trans: 0.00266, RPE rot: 0.19707
0.00880
0.00266
0.19707
sintel-sleeping_1 | ATE: 0.01360, RPE trans: 0.00518, RPE rot: 0.25681
0.01360
0.00518
0.25681
sintel-sleeping_2 | ATE: 0.01108, RPE trans: 0.00517, RPE rot: 0.10101
0.01108
0.00517
0.10101
sintel-temple_2 | ATE: 0.07854, RPE trans: 0.05960, RPE rot: 0.25124
0.07854
0.05960
0.25124
sintel-temple_3 | ATE: 0.64983, RPE trans: 0.33725, RPE rot: 1.02402
0.64983
0.33725
1.02402
sintel-alley_2 | ATE: 0.02491, RPE trans: 0.02035, RPE rot: 0.18073
0.02491
0.02035
0.18073
sintel-ambush_4 | ATE: 0.15770, RPE trans: 0.06740, RPE rot: 1.39592
0.15770
0.06740
1.39592
sintel-ambush_5 | ATE: 0.07014, RPE trans: 0.03607, RPE rot: 0.58807
0.07014
0.03607
0.58807
sintel-ambush_6 | ATE: 0.24250, RPE trans: 0.11925, RPE rot: 2.13628
0.24250
0.11925
2.13628
sintel-cave_2 | ATE: 1.03710, RPE trans: 0.34871, RPE rot: 2.23406
1.03710
0.34871
2.23406
sintel-cave_4 | ATE: 0.14149, RPE trans: 0.04392, RPE rot: 0.56767
0.14149
0.04392
0.56767
sintel-market_2 | ATE: 0.03262, RPE trans: 0.02050, RPE rot: 0.02688
0.03262
0.02050
0.02688
sintel-market_5 | ATE: 0.28074, RPE trans: 0.20176, RPE rot: 0.73502
0.28074
0.20176
0.73502
sintel-market_6 | ATE: 0.10353, RPE trans: 0.04350, RPE rot: 0.41800
0.10353
0.04350
0.41800
sintel-shaman_3 | ATE: 0.00882, RPE trans: 0.00264, RPE rot: 0.19566
0.00882
0.00264
0.19566
sintel-sleeping_1 | ATE: 0.01359, RPE trans: 0.00515, RPE rot: 0.25512
0.01359
0.00515
0.25512
sintel-sleeping_2 | ATE: 0.01108, RPE trans: 0.00517, RPE rot: 0.10093
0.01108
0.00517
0.10093
sintel-temple_2 | ATE: 0.07933, RPE trans: 0.05998, RPE rot: 0.25285
0.07933
0.05998
0.25285
sintel-temple_3 | ATE: 0.65585, RPE trans: 0.33424, RPE rot: 1.05196
0.65585
0.33424
1.05196
Average ATE: 0.20424, Average RPE trans: 0.09347, Average RPE rot: 0.72422

[rover-xingyu]

Thanks for the update!

I noticed something unusual in your logs: each scene appears to be listed twice, and the values differ slightly between the two entries (for example, sintel-temple_3 shows 0.64983 in the first block and 0.65585 in the second). This behavior suggests there might be some bugs in the execution loop.

And, could you please run the evaluation on ScanNet?

If ScanNet results align with the paper, then the discrepancy you see on Sintel is likely just random noise/variance.

If ScanNet also shows a significant mismatch, then there might be a more serious systemic issue.

Since Sintel is small and noisy, checking ScanNet is the most effective way to rule out random variance and verify the pipeline.

[ZhemengDong]

Thanks for your reply!

I apologize for the confusion - the unusual log was a copying error (sad).

I will try the evaluation on ScanNet later and here are the correct logs:

sintel-alley_2 | ATE: 0.02491, RPE trans: 0.02035, RPE rot: 0.18073
0.02491
0.02035
0.18073
sintel-ambush_4 | ATE: 0.15770, RPE trans: 0.06740, RPE rot: 1.39592
0.15770
0.06740
1.39592
sintel-ambush_5 | ATE: 0.07014, RPE trans: 0.03607, RPE rot: 0.58807
0.07014
0.03607
0.58807
sintel-ambush_6 | ATE: 0.24250, RPE trans: 0.11925, RPE rot: 2.13628
0.24250
0.11925
2.13628
sintel-cave_2 | ATE: 1.03710, RPE trans: 0.34871, RPE rot: 2.23406
1.03710
0.34871
2.23406
sintel-cave_4 | ATE: 0.14149, RPE trans: 0.04392, RPE rot: 0.56767
0.14149
0.04392
0.56767
sintel-market_2 | ATE: 0.03262, RPE trans: 0.02050, RPE rot: 0.02688
0.03262
0.02050
0.02688
sintel-market_5 | ATE: 0.28074, RPE trans: 0.20176, RPE rot: 0.73502
0.28074
0.20176
0.73502
sintel-market_6 | ATE: 0.10353, RPE trans: 0.04350, RPE rot: 0.41800
0.10353
0.04350
0.41800
sintel-shaman_3 | ATE: 0.00882, RPE trans: 0.00264, RPE rot: 0.19566
0.00882
0.00264
0.19566
sintel-sleeping_1 | ATE: 0.01359, RPE trans: 0.00515, RPE rot: 0.25512
0.01359
0.00515
0.25512
sintel-sleeping_2 | ATE: 0.01108, RPE trans: 0.00517, RPE rot: 0.10093
0.01108
0.00517
0.10093
sintel-temple_2 | ATE: 0.07933, RPE trans: 0.05998, RPE rot: 0.25285
0.07933
0.05998
0.25285
sintel-temple_3 | ATE: 0.65585, RPE trans: 0.33424, RPE rot: 1.05196
0.65585
0.33424
1.05196
Average ATE: 0.20424, Average RPE trans: 0.09347, Average RPE rot: 0.72422

[JSP-ywu]

I’m experiencing the same issue.
I obtained ATE: 0.209, RPE trans: 0.091, and RPE rot: 0.721, which are consistent with the results reported above.

For the evaluation, I used the image sets from the 'final' directory as specified in the provided evaluation code.

[strongBrave]

I'm experiencing the same issue too.
I obtained ATE: 0.210, RPE trans: 0.090, and RPE rot: 0.728, which are consistent with the results reported here and there

[rover-xingyu]

Hi all, thank you for sharing your findings!

As we can see from the different metrics reported in this thread, minor variations in hardware and software environments can lead to different numerical outcomes. Because the Sintel dataset consists of very short sequences and a small number of scenes, the evaluation metrics tend to exhibit high variance and are highly sensitive to these environmental differences.

To verify if your setup and the pipeline are working as expected, please run the evaluation on the ScanNet dataset instead.

ScanNet features much longer sequences and over 10x more test scenes compared to Sintel. Averaging the metrics across this significantly larger dataset reduces random noise, leading to much more stable and reliable conclusions.

Please let us know if your ScanNet results align with the paper! If there is still a significant discrepancy there, it will help us identify if there is a deeper systemic issue.