Implementing & using a detector class template

[MohamedBilelBesbes]

The change implemented is making an abstract class BaseDetector which enables generic alert detection using different methods. Implementating an alert detection methods using a concrete class inheriting from BaseDetector:

• Enables defining a custom alert detection logic per method using calculate_alpha function.
• Enables attrributing custom parameters for the given method (min_back_window, max_back_window, fore_window, alert_threshold, etc.) without overriding them if they are signature-specific.
• Enables the control over including a magnitude-of-change check or not.

The concrete class StudentTMagDetector inheriting from BaseDetector is the implementation of the current process for alert detection.

[gmierz]

Instead of modifying this generate_new_alert_in_series method, can you make a new generate_new_test_alerts_in_series method and add your new code to there? I'd rather not touch the code that's currently being used in production for now.

It would be better to call it here like so (after the current call):

treeherder/treeherder/perf/tasks.py

Line 18 in e6e0200

generate_new_alerts_in_series(signature)

try:
    generate_new_test_alerts_in_series(...)
except Exception:
    logger.warning(...)

[MohamedBilelBesbes]

Done. I also used PerformanceAlertSummaryTesting and PerformanceAlertTesting in the new commit instead of the currently used PerformanceAlertSummary and PerformanceAlert classes.