In some cases, plugins cannot be used

[CanCanZeng]

Hi, I can use this plugin to preview numpy images normally in most cases. However, in some projects the plugin cannot be used. The phenomenon is that when the mouse is placed on the numpy variable, no matter how long it takes, the "view image" option will not pop up. I have found that in this situation, some strange logs will appear, as follows:

Backend TkAgg is interactive backend. Turning interactive mode on. [10/09 13:02:15]
Value(False) [10/09 13:02:16]
Value(False) [10/09 13:02:16]
Value(False) [10/09 13:02:16]
Value(False) [10/09 13:02:17]
Value(False) [10/09 13:02:17]
Value(False) [10/09 13:02:18]

The Backend TkAgg is interactive backend. Turning interactive mode on may also appear in other projects, but plugins can be used normally.

[elazarcoh]

There are few things to check:

1. Please set the log to verbose via the settings and upload it here. It may be found in the VSCode "Output" tab in the panel.

2. The extension as also a tree-view in the debug view, please see if it has variables etc, and whether it can be used.

   

3. There's also a command in the command-palette, View Image, please try it and upload the log afterward.

Thanks

[CanCanZeng]

1. The log
   13-View Image for Python.log

2. There is no available variable to view
   

3. The log after try View Image command
   13-View Image for Python.log

[elazarcoh]

Thanks. I'll start to look into it. I haven't found anything helpful enough in the log (means I need to improve my logs :) ), so I'm going to work toward better logs and also display errors/suggestions to the user. Hopefully to be done in the next week or two.

I have a suspicion which I need to check, but first I have to be able to reproduce it on my side. Does it happen even on a small-scale script? a deep-learning training session? multi-processing/threading?

In the meanwhile, there another command that I forgot about, Debug View Python: Run Setup, which forces the extension to re-rerun the python code it need for working. Please try that as well and see if it helps. If not, please upload the log after running it, might have additional information.

[elazarcoh]

I've uploaded a pre-release version 4.0.3 of the extension with improved error messages etc.
Please try it and see if this can shed new information on the issue.

[CanCanZeng]

Hi, this is the log produced by pre-release 4.0.3
8-View Image for Python.log

[CanCanZeng]

[elazarcoh]

Thanks. That might help further diagnose the problem.
I can see from the logs that the extension setup in the python side is indeed failing. It seems the setup is trying to run (repeatedly) but it seems it doesn't work at all (request to the debugger is emitted, but I got no informative error).

I have 2 questions for now, please:

• Did you run the Debug View Python: Run Setup command after the breakpoint is hit? Please let the debugger settle before trying it, I want to see if the debugger is deciding from some reason to not execute the setup at all.
• I want to try to reproduce it on my side:
  • Does this issue consistent, s.t. it always occurs on some scripts, while on other it works?
  • Does this issue happen during a multi-processing/multi-threading session?
  • Can you define or guess what scenarios it fails? What does change between the scenarios (different environment, script, deep-learning vs other stuff)?

For reference, I'm attaching the code I'm running on the python side that enables the extension. You can run it in the debug console and see if it works when you manually does that.

Setup Code

_python_view_image_mod_setup_result = "Starting setup"
try:
    _python_view_image_mod_setup_result = "Checking if module exists"
    _python_view_image_mod
    _python_view_image_mod_setup_result = "OK"
except:
    from types import ModuleType

    _python_view_image_mod_setup_result = "Creating module"
    _python_view_image_mod = ModuleType("python_view_image_mod", "")
    try:
        _python_view_image_mod_setup_result = "Running common setup"
        exec(
            """def keyvalue(pair):
    key, value = pair
    return f"{stringify(key)}: {stringify(value)}"


def sanitize(s):
    # s.replace(\\"'\\", \\"\\\\'\\").replace('\\"', '\\\\\\"')
    return s.replace("'", "").replace('"', '')


def stringify(value):
    if isinstance(value, list):
        return f"[{','.join(map(stringify, value))}]"
    elif isinstance(value, tuple):
        return f"({','.join(map(stringify, value))})"
    elif isinstance(value, dict):
        return f"{{{ ','.join(map(keyvalue, value.items())) }}}"
    elif isinstance(value, str):
        if value.startswith("Value(") or value.startswith("Error("):
            return value  # keep Value/Error without wrapping quoted
        else:
            return f'"{sanitize(value)}"'
    elif isinstance(value, Exception):
        return f'"{type(value).__name__}: {sanitize(str(value))}"'
    else:
        return str(value)


def eval_into_value(func):
    try:
        return f"Value({stringify(func())})"
    except Exception as e:
        return f"Error({stringify(e)})"


def eval_or_return_exception(func):
    try:
        return func()
    except Exception as e:
        return e


def same_value_multiple_callables(get_value, funcs):
    try:
        val = get_value()
        return [eval_into_value(lambda: f(val)) for f in funcs]
    except Exception as e:
        return [f"Error({stringify(e)})"] * len(funcs)


def object_shape_if_it_has_one(obj):
    if hasattr(obj, "shape"):
        shape = obj.shape
        return tuple(shape)
    elif hasattr(obj, "width") and hasattr(obj, "height") and hasattr(obj, "getbands"):
        bands = "".join(map(str, obj.getbands()))
        return {"width": obj.width, "height": obj.height, "channels": bands}
    else:
        return None
""",
            _python_view_image_mod.__dict__,
        )
    except Exception as e:
        _python_view_image_mod_setup_result = repr(e)
        del _python_view_image_mod
    else:
        _python_view_image_mod_setup_result = "Common setup no exceptions"
        try:
            _python_view_image_mod_setup_result = "Running socket client setup"
            exec(
                """import socket
import sys
import traceback
import struct
import numpy as np

MIN_INT32 = np.iinfo(np.int32).min
MAX_INT32 = np.iinfo(np.int32).max
MIN_UINT32 = np.iinfo(np.uint32).min
MAX_UINT32 = np.iinfo(np.uint32).max
MIN_FLOAT32 = np.finfo(np.float32).min
MAX_FLOAT32 = np.finfo(np.float32).max

DATATYPE_64BIT = ("int64", "uint64", "float64")
CORRESPONDING_32BIT_TYPE = {
    "int64": np.int32,
    "uint64": np.uint32,
    "float64": np.float32,
}

CHUNK_SIZE = 4 * 1024  # 4KB

# See webview/communication/protocol.ts for message format
PythonSender = 0x02
# MessageType 
PythonSendingObject = 0x01
# ObjectType
NumpyArray = 0x01
ExceptionObject = 0xff
# ByteOrderType
LittleEndian = 0x01
BigEndian = 0x02
# ArrayDataType
UndefinedDatatype = 0x00
Float32 = 0x01
Float64 = 0x02
Int8 = 0x03
Int16 = 0x04
Int32 = 0x05
Int64 = 0x06
Uint8 = 0x07
Uint16 = 0x08
Uint32 = 0x09
Uint64 = 0x0a
Bool = 0x0b
# DimensionOrder
HWC = 0x01
CHW = 0x02
# ExceptionType
ExceptionTypes = {
    BaseException: 0x01,
    Exception: 0x02,
    RuntimeError: 0x03,
    TypeError: 0x04,
    ValueError: 0x05,
    None: 0xff,  # Unknown exception
}

MessageLengthType = np.uint32
MessageIdType = np.uint32
SenderType = np.uint8
MessageType = np.uint8
RequestIdType = np.uint32
ChunkCountType = np.uint32
ChunkIndexType = np.uint32
ChunkLengthType = np.uint32

ObjectIdType = np.uint32
ObjectType = np.uint8
StringLengthType = np.uint32
NumDimsType = np.uint8
DimType = np.uint32
ByteOrderType = np.uint8
StatsType = np.float32  

array_dtype_to_array_data_type = {
    "float32": Float32,
    "float64": Float64,
    "int8": Int8,
    "int16": Int16,
    "int32": Int32,
    "int64": Int64,
    "uint8": Uint8,
    "uint16": Uint16,
    "uint32": Uint32,
    "uint64": Uint64,
    "bool": Bool,
    "bool_": Bool,
}

BYTE_ORDER = LittleEndian if sys.byteorder == 'little' else BigEndian

def generate_message_id():
    rng = np.random.default_rng()
    return rng.integers(0, 2**32, dtype=np.uint32)

def chunk_header(
    message_length,
    message_id,
    request_id,
    chunk_count,
    chunk_index,
    chunk_length,
):
    message_length = MessageLengthType(message_length)
    message_id  = MessageIdType(message_id)
    sender = SenderType(PythonSender)
    request_id = RequestIdType(request_id)
    message_type = MessageType(PythonSendingObject)
    chunk_count = ChunkCountType(chunk_count)
    chunk_index = ChunkIndexType(chunk_index)
    chunk_length = ChunkLengthType(chunk_length)

    # Create the message
    header = [
        message_length,
        message_id,
        sender,
        request_id,
        message_type,
        chunk_count,
        chunk_index,
        chunk_length,
    ]
    # print({
    #     'message_length': message_length,
    #     'message_id': message_id,
    #     'sender': sender,
    #     'request_id': request_id,
    #     'message_type': message_type,
    #     'chunk_count': chunk_count,
    #     'chunk_index': chunk_index,
    #     'chunk_length': chunk_length,
    # })
    
    # Create the message format string
    header_format = f'!IIBIBIII'

    return struct.pack(header_format, *header)

def is_64bit(array):
    return str(array.dtype) in DATATYPE_64BIT

def check_can_fit_in_32bit(array_64bit):
    corresponding_32bit_type = CORRESPONDING_32BIT_TYPE.get(str(array_64bit.dtype), None)
    if corresponding_32bit_type is None:
        return False
    array_min = np.min(array_64bit)
    array_max = np.max(array_64bit)
    return array_min >= MIN_INT32 and array_max <= MAX_INT32

def guess_image_dimensions(image):
    if not isinstance(image, np.ndarray):
        return None
    shape = image.shape

    if len(shape) < 2:
        return None

    if len(shape) == 2:
        return { 'width': DimType(shape[1]), 'height': DimType(shape[0]),'channels': DimType(1), 'order': HWC }

    if len(shape) == 4 and shape[0] == 1:
        # A batched image
        image = image[0]

    if len(shape) == 3:
        if shape[2] > 4:
            # Possibly a channel first image
            if shape[0] > 4:
                return None
            return { 'width': DimType(shape[2]), 'height': DimType(shape[1]), 'channels': DimType(shape[0]), 'order': CHW }
        else:
            return { 'width': DimType(shape[1]), 'height': DimType(shape[0]), 'channels': DimType(shape[2]), 'order': HWC }

    return None

def array_stats(array):
    if not isinstance(array, np.ndarray):
        return None
    shape = array.shape
    ndims = len(array.shape)

    if ndims < 2:
        return None
    
    if ndims == 2 or (ndims == 3 and shape[2] == 1):
        return { 'min': [StatsType(np.nanmin(array))], 'max': [StatsType(np.nanmax(array))] }
    
    if ndims == 4 and shape[0] == 1:
        # assume batched image
        image = array[0]

    if ndims == 3:
        # per channel stats
        if shape[2] > 4:
            # Possibly a channel first image
            if shape[0] > 4:
                return None
            return { 'min': StatsType(np.nanmin(array, axis=(1,2))), 'max': StatsType(np.nanmax(array, axis=(1,2))) }
        else:
            return { 'min': StatsType(np.nanmin(array, axis=(0,1))), 'max': StatsType(np.nanmax(array, axis=(0,1))) }

    return None

        

def create_numpy_message(
    array: np.ndarray,
):

    object_type     = ObjectType(NumpyArray)
    array_dtype     = array_dtype_to_array_data_type[str(array.dtype)]
    actual_datatype = UndefinedDatatype
    byte_order      = ByteOrderType(BYTE_ORDER)
    num_dimensions  = NumDimsType(len(array.shape))
    array_shape     = np.array(array.shape, dtype=DimType)

    dimensions = guess_image_dimensions(array) or {}
    w = dimensions.get('width', DimType(0))
    h = dimensions.get('height', DimType(0))
    c = dimensions.get('channels', DimType(0))
    order = dimensions.get('order', None)

    stats = array_stats(array) or {}
    min_stats = stats.get('min', [])
    max_stats = stats.get('max', [])
    assert len(min_stats) == len(max_stats)
    num_stats = len(min_stats)

    # The webview only supports up to 32 bit integers and floats. If the array is 64 bit, try to convert it to 32 bit.
    if is_64bit(array) and check_can_fit_in_32bit(array):
        array = array.astype(CORRESPONDING_32BIT_TYPE[str(array.dtype)])
        actual_datatype = array_dtype_to_array_data_type[str(array.dtype)]

    array_data = array.tobytes('C')

    metadata = [
        object_type,
        array_dtype,
        actual_datatype,
        byte_order,
        num_dimensions,
        *array_shape,
        w,
        h,
        c,
        order,
        num_stats,
        *min_stats,
        *max_stats,
    ]
    
    # add padding before the array data, making sure the offset is a multiple of the element size
    # metadata_size = 1 + 1 + 1 + 1 + num_dimensions * 4
    # bytes_per_element = array.dtype.itemsize
    # message_length_placeholder = 4
    # padding_size =  (bytes_per_element - (header_size + message_length_placeholder) % bytes_per_element) % bytes_per_element
    # padding = bytes(padding_size)

    message = [
        *metadata,
        array_data,
    ]

    # Create the message format string
    message_format = f'!BBBBB{num_dimensions}IIIIBB{num_stats}f{num_stats}f{len(array_data)}s'

    # Pack the message
    message_pack = struct.pack(message_format, *message)

    return message_pack

def create_pillow_message(
    image,
):
    image_np = np.asarray(image)
    return create_numpy_message(image_np)

def string_to_message(s):
    length = StringLengthType(len(s))
    bytes = s.encode()
    return (length, bytes)

def create_exception_message(
    exception: Exception,
):
    exception_type_name = getattr(type(exception), '__name__', 'Unknown')
    exception_message = str(exception)

    object_type = ObjectType(ExceptionObject)
    (exception_type_length, exception_type) = string_to_message(exception_type_name)
    (exception_message_length, exception_message) = string_to_message(exception_message)

    message = [
        object_type,
        exception_type_length,
        exception_type,
        exception_message_length,
        exception_message,
    ]
    
    # Create the message format string
    message_format = f'!BI{exception_type_length}sI{exception_message_length}s'

    # Pack the message
    message_pack = struct.pack(message_format, *message)

    return message_pack


    # Create the message body
    exception_type = ExceptionTypes.get(type(exception), ExceptionTypes[None])
    exception_message = traceback.format_exc().encode()
    
    # Create the message
    message = [
        message_type,
        request_id,
        object_id,
        object_type,
        exception_type,
        exception_message,
    ]

    # Create the message format string
    message_format = f'!BIIBB{len(exception_message)}s'

    # Pack the message
    message_pack = struct.pack(message_format, *message)

    return message_pack


def message_chunks(
    message_length,
    message_id,
    request_id,
    message,
):
    loc = 0
    chunk_count = len(message) // CHUNK_SIZE + 1
    for chunk_index in range(chunk_count):
        data = message[loc:loc+CHUNK_SIZE]
        chunk_length = len(data)
        header = chunk_header(
            message_length,
            message_id,
            request_id,
            chunk_count,
            chunk_index,
            chunk_length,
        )
        loc += CHUNK_SIZE
        yield header + data


def open_send_and_close(port, request_id, obj):
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.connect(('localhost', port))

        try:
            if isinstance(obj, np.ndarray):
                message = create_numpy_message(obj)
            elif is_pillow_image(obj):
                message = create_pillow_message(obj)
            else:
                raise ValueError(f'Cant send object of type {type(obj)}')
        except Exception as e:
            message = create_exception_message(e)
        
        message_length = len(message)

        # Send the message data
        message_id = generate_message_id()
        # print(f'Sending message {message_id}')
        chunks = message_chunks(message_length, message_id, request_id, message)

        all_data = b''

        # Send the message
        for chunk in chunks:
            s.sendall(chunk)
            all_data += chunk

        # Close the socket
        s.close()""",
                _python_view_image_mod.__dict__,
            )
        except Exception as e:
            _python_view_image_mod_setup_result = repr(e)
        else:
            _python_view_image_mod_setup_result = "Socket client setup no exceptions"

            _python_view_image_mod_setup_result = "OK"

finally:
    globals()["_python_view_image_mod_setup_result"] = (
        _python_view_image_mod_setup_result
    )

[CanCanZeng]

Yes, I run Debug View Python: Run Setup command after the breakpoint is hit, then there will be some log on the console

Value(False) [19/09 10:53:02]
Value(False) [19/09 10:53:03]
Value(False) [19/09 10:53:05]
Value(False) [19/09 10:53:09]

And after a while, there will be an error

Does this issue consistent, s.t. it always occurs on some scripts, while on other it works?

YES

Does this issue happen during a multi-processing/multi-threading session?

Not very sure, but I think it is a single-threading code

Can you define or guess what scenarios it fails? What does change between the scenarios (different environment, script, deep-learning vs other stuff)?

It is a deep-learning code. This plugin was working fine in one project at first, but it stopped working one day. I don't know what I did. However, I used the same python environment in another project, the plugin worked fine.

[elazarcoh]

That's a tough one..

Did you see the code I added in the previous message? Can you try to run it? This is the same code I'm running to set up the python side of the extension.

If it's working when you run it manually, then it basically means the issue is not with the code+environment, but something to do with either how I'm calling the debugger, or something related to it.

Also, if it's working, you can actually use it as a workaround to make the extension work. The extension only needs the module named _python_view_image_mod available in the global scope, which I create in the setup code (with all the utilities I'm adding to it).

[ssavary]

I often get the "Not ready"/"Not Viewable" message next to my Expressions, and an empty Variables section. The error message, using the pre-release version is:

Running your code from the above reply in the debug console does not seem to help

edit: running the code snippet in the debug console, sometimes works (like once every 20 tries). I execute Run Setup and Update Frame ID frantically, and sometimes it finally work. Most of the time, however, I only get a png preview, not the interactive one (with colormaps and tooltip).

[AlexandreBrown]

Getting the same issue, it works on the remote for me when I'm connected to my compute cluster but on my home machine the extension does not work for some reason.
I have tried many things (re-installing vscode, the extension, enable-disable).
Nothing seems to work.
When I run the python setup command I get :
Value(False) repeated in my terminal.
My image watch has no variables even after I refresh it and if I manually add expressions they are not shown in the image view window.
The python setup seems to fail and I do not know why :

[AlexandreBrown]

Not sure if this is what solved it but now the extension works (after switching some settings). I'm using version 4.0.6 (new version since last time I tried), it's possible that it now works due to an update as well and that my settings has nothing to do with it.
My settings:

• Svifpd: Custom Save Path : Set custom path to blank (nothing)
  
• Add View Context Entry To VSCode Debug Variables
• Allow Plugins
• Debug (verbose)
• Matplotlib Use Agg
• Normalization Method (normalize)
• Preferred Backend (Standalone)
• Restrict Image Types
• Save Location (tmp)
• Show Diagnostic Info in Tree View
• Use Experimental Data Transfer
• Invert Mouse Wheel Zoom