Fix: Refactor is_comment_line to use Pygments for improved accuracy

[langverse2023]

This PR replaces the previous regex-based, line-by-line comment detection with a more robust solution using the Pygments library.

The Problem:
The previous approach analyzed each line in isolation. This caused issues with multi-line comments (e.g., C-style /* ... */), where middle lines starting with * were often invalidly interpreted as multiplication operators or code syntax.

The Solution:
I have introduced a new helper function get_comment_line_indices that preserves the context of the code block.

Implementation Details:

1. Reconstruct Source: The function strips diff markers (+, -) from the patch to reconstruct the continuous source text.
2. Context-Aware Lexing: Instead of checking line by line, we pass the full text to the Pygments lexer. This allows the lexer to correctly identify multi-line comment blocks (like """ docstrings """ or /* block comments */).
3. Token Mapping: We iterate through the tokens generated by Pygments and map them back to the original line numbers.
4. Filtering: A line is marked as "non-scoreable" only if it contains comment tokens and strictly no code tokens.

Key Changes:

• Removed is_comment_line (regex-based).
• Added get_comment_line_indices (Pygments-based).
• Updated count_non_scoreable_lines to pre-calculate comment indices before iterating.

Fixes #64
Improves #56

---

Contribution by Gittensor, learn more at https://gittensor.io/

[hsparks-codes]

@langverse2023 Close this PR, otherwise will report. You're stealing my effort.

[SmartDever02]

# Test cases
if __name__ == "__main__":
    # Test 1: Comments only
    """
    /*
    * Hello world
    */

    """
    test_string = """
* Hello world
"""
    print(f"Rust comments only: {is_comment_line(test_string, '.rs')}")  
    
    # Test 2: Code
    test_string = """
#include <stdio.h>
"""
    print(f"C program code: {is_comment_line(test_string, '.c')}")  
    
    # Test 3: multi-line comment
    """
    '''
    This is a comment
    '''
    """
    test_string = '''
This is a comment
'''
    print(f"Multi-line comment: {is_comment_line(test_string, '.py')}")

I don't think Pygments library works better than current solution as we check line by line

[thewhaleking]

You need to include pygments as a dependency

[langverse2023]

@langverse2023 Close this PR, otherwise will report. You're stealing my effort.
I believe there is a misunderstanding. My implementation is distinct because it integrates the Pygments library to handle comment detection, whereas your solution did not use this library. This is a different technical approach, not a copy of your work.

[langverse2023]

# Test cases
if __name__ == "__main__":
    # Test 1: Comments only
    """
    /*
    * Hello world
    */

    """
    test_string = """
* Hello world
"""
    print(f"Rust comments only: {is_comment_line(test_string, '.rs')}")  
    
    # Test 2: Code
    test_string = """
#include <stdio.h>
"""
    print(f"C program code: {is_comment_line(test_string, '.c')}")  
    
    # Test 3: multi-line comment
    """
    '''
    This is a comment
    '''
    """
    test_string = '''
This is a comment
'''
    print(f"Multi-line comment: {is_comment_line(test_string, '.py')}")

I don't think Pygments library works better than current solution as we check line by line

Thanks for the feedback! You are absolutely right that checking line-by-line is problematic for multi-line comments (e.g., Pygments might interpret an isolated * Hello world as a multiplication or pointer instead of a comment).
To address this, I have refactored the logic to analyze the entire patch context at once.

[langverse2023]

@LandynDev could u check my pr

[anderdc]

I like the idea of this PR, it will take a couple days to fully vet it and test, expect a full review then

[anderdc]

resolve conflicts, I'm looking into this PR now finally

[anderdc]

double check the dummy filename construction, I don't believe file_extension has the ..

also why not just use the actual filename? seems strange to just make up a dummy filename

[langverse2023]

Agreed. I updated the function signature to pass the full filename directly. It's much cleaner now.

[langverse2023]

@anderdc could u check my pr

[langverse2023]

@LandynDev , @anderdc would you review my PR ?