Cannot pickle a pymatching._cpp_pymatching.MatchingGraph object

[MarzioVallero]

Hi Oscar,
I am trying to parallelise the appication of the matcher.decode method over a very large set of syndrome inputs stored in a pandas dataframe.
Hence, I am using modules such as mapply to distribute this over many cores. This module, like many other parallelisation modules, pickles the objects using the dill module.
Whenever I try to do so, I incur in the error:
TypeError: cannot pickle 'pymatching._cpp_pymatching.MatchingGraph' object
For me it would be enough to be able to pickle the matcher.decode() method, however I believe that in order to do that the whole matcher object must be picklable as well.
Is there a chance to make this work?

Here is an example to reproduce the error:

import stim
import pymatching
import dill

circuit = stim.Circuit.generated("repetition_code:memory", rounds=1, distance=3)
dem = circuit.detector_error_model(decompose_errors=True)
matcher = pymatching.Matching.from_detector_error_model(dem)
with open("./test.pickle", 'wb') as f:
    dill.dump(matcher, f)

output: TypeError: cannot pickle 'pymatching._cpp_pymatching.MatchingGraph' object

[oscarhiggott]

Hi Marzio,

Adding support to pickle the MatchingGraph object is a nice idea. This could be done with pybind11 by adding a `py::pickle' method to serialise/de-serialise.

In the meantime, since it looks like you're using stim anyway, have you considered using sinter to parallelise? Sinter parallelises stim circuit simulations and has pymatching support built in (along with other decoders).

You could also look at sinter for inspiration on how you could get your own code to work using the current version of pymatching. For example, the sinter PyMatchingDecoder class is pickleable since the decode_via_files method takes as input a dem to construct a new pymatching.Matching object (rather than storing it as an attribute). So you could either use this class or create your own wrapper class. Here is an example of a simple wrapper class that exploits the fact that stim.DetectorErrorModel is pickleable:

import stim
import pymatching
import dill
import numpy as np


class PyMatchingDemDecoder:
    def __init__(self, dem: stim.DetectorErrorModel):
        self.dem = dem

    def decode_batch(self, shots: np.ndarray, *args, **kwargs) -> np.ndarray:
        matcher = pymatching.Matching.from_detector_error_model(self.dem)
        return matcher.decode_batch(shots=shots, *args, **kwargs)


circuit = stim.Circuit.generated(
    "repetition_code:memory", 
    rounds=1, 
    distance=3, 
    after_clifford_depolarization=0.2
)

dem = circuit.detector_error_model(decompose_errors=True)
matcher = PyMatchingDemDecoder(dem=dem)

shots = circuit.compile_detector_sampler().sample(shots=100)
print(matcher.decode_batch(shots=shots).T)

with open("./test.pickle", "wb") as f:
    dill.dump(matcher, f)

This does incur an overhead where the pymatching Matching object is loaded every time you call decode_batch, but as long as you call decode_batch with a large enough number of shots this shouldn't be a significant overhead.

[MarzioVallero]

Hi Oscar,
thanks for the quick reply.

As far as unsing sinter goes, I am afraid that the analysis I am carrying out forces me to have more fine grained control over the decoding process, thus sinter would not help me much. In any case, I will make sure to double check this.

Unfortunately, I cannot use decode_batch(), as one of the constraints of the analysis I am carrying out forces me to catch if one of the input syndromes sent to the matcher raises an exception for not being graphlike. As such, I am forced to try/except each single decode() call and flag each syndrome which is not graphlike. It could be nice to have an argument to the decode_batch() method to silently ignore non-graphlike errors and flag them as such in the output correction vector, but that is another matter.

Your idea of a wrapper class to the Matcher class that initialises a separate decoder each time decode() is called works. However, given the use case I described above, it is significantly slower due to these continuos initialisations. For the time being, this should suffice.

I hope you can make a new release on PyPI adding the py::pickle method to serialise/de-serialise the MatchingGraph soon.

[oscarhiggott]

Hi Marzio, do you have an example of the exception being raised? I'm not sure what you mean by the syndrome (rather than the DEM) not being graphlike.

[MarzioVallero]

Hi Oscar,
I owe you some apologies, you were right.
Indeed, the exception was being raised for the DEM not being graphlike because I had forgotten to use the option "decompose_errors=True" when generating it from a STIM circuit.
Turns out that I can indeed call decode_batch() without any issues, regardless of the input syndrome bitstring that is being decoded.

Thank you!

[jfoliveiraramos]

Hi Oscar,
I figured this could come in handy, and I have opened a rather thing PR suggesting some changes to implement this. #182
Not sure if I am following the right contribution guidelines. I'm open to any feedback.