Unexpected behavior from `detect-content`

[rsomani95]

Hello. Firstly, great project!

When using the detect-content option, I observe some unexpected behavior. While the cuts are detected accurate, the exact timing of the cuts is off. This is very odd, since the frame at which the cut is detected doesn't have much of a content_val differential as compared to the frame where the cut actually happened.

I've documented this here after taking a closer look at the stats file.
This notebook is quite long, but well organised, so head over to

Exploring & Analysing the Data →
      Looking At Detected Cuts And Their Surrounding Frames →
           Correctly Detected (But How?)

[Breakthrough]

Hi @rsomani95;

Thanks for sharing this with me, I'll look into this when I have some time. I haven't analyzed the source video you're using in the notebook yet, but will do so as soon as possible.

Just so you know, the way detect-content works is that any frame delta exceeding the threshold triggers a new scene cut, but after doing so, any new scene cuts are ignored for N frames (default for N is 10). This is to prevent noise/flashes from triggering false positives, while ensuring that the new scenes are at least captured within N frames. It seems this might explain some (but not all) of the behaviour you're describing.

One thing I'm looking into doing for the future to solve this issue is, let's say we have 3 frames, A, B, and C, and the delta between A and B is very large (indicating a new scene). It may also be worth taking the delta of A and C, in this case, to see if frames A and C are related/part of the same scene, in which case this would be considered a false negative.

This is part of flash detection, which seems to be causing some issues in your analysis (very thorough by the way, well done). See #35 for some more discussion on the matter.

Thanks for the submission, and the detailed analysis - this will definitely be useful for improving the detect-content algorithm.

[rsomani95]

Hi @Breakthrough

I've done some work to make your life easier. Clone this directory and follow along with the tdk-bankrobbery/PySceneDetect_Analysis.ipynb notebook to see what exactly the issue is. To give you a glimpse -- this is what the notebook offers:

I had to strip the output of most of the cells to manage filesizes.

Just so you know, the way detect-content works is that any frame delta exceeding the threshold triggers a new scene cut, but after doing so, any new scene cuts are ignored for N frames (default for N is 10). This is to prevent noise/flashes from triggering false positives, while ensuring that the new scenes are at least captured within N frames. It seems this might explain some (but not all) of the behaviour you're describing.

This explains the undetected cuts. Frame #3052 in the notebook might also offer some insight into this as well.

One thing I'm looking into doing for the future to solve this issue is, let's say we have 3 frames, A, B, and C, and the delta between A and B is very large (indicating a new scene). It may also be worth taking the delta of A and C, in this case, to see if frames A and C are related/part of the same scene, in which case this would be considered a false negative.

I don't think this will work out too well -- A and C could have a delta of ~0 but still be completely different scenes -- this is well documented in the notebook.

Thanks for the submission, and the detailed analysis - this will definitely be useful for improving the detect-content algorithm.

Happy to contribute :)

[rsomani95]

Update --> I've experimented with another peak-detection algorithm that worked perfectly! I've added this to a new section in the notebook.

[Breakthrough]

Thanks for your contributions @rsomani95. I'll definitely be looking into this when I have some more time on my hands, very nice analysis.