Skip to content

[BUG] Fix segfault in python testing #919

@rgsl888prabhu

Description

@rgsl888prabhu

Describe the bug
While testing wheel python, a test fails with segfault

Steps/Code to reproduce bug

tests/linear_programming/test_cpu_only_execution.py::TestSolutionInterfacePolymorphism::test_lp_solution_values optimization_problem_t constructor: Using GPU backend
cuOpt version: 26.4.0, git hash: 98b6ac6a, host arch: x86_64, device archs: 75-real,80-real,86-real,90a-real,100f-real,120a-real,120
CPU: AMD EPYC 7313P 16-Core Processor, threads (physical/logical): 12/12, RAM: 34.37 GiB
CUDA 13.1, device: NVIDIA H100 PCIe (ID 0), VRAM: 79.18 GiB
CUDA device UUID: ffffffbb4d5dffffff8a-1effffffdc-31ff

Solving a problem with 27 constraints, 32 variables (0 integers), and 83 nonzeros
Problem scaling:
Objective coefficents range:          [3e-01, 1e+01]
Constraint matrix coefficients range: [1e-01, 2e+00]
Constraint rhs / bounds range:        [0e+00, 5e+02]
Variable bounds range:                [0e+00, 0e+00]

Using PSLP presolver
PSLP Presolved problem: 14 constraints, 18 variables, 40 non-zeros
optimization_problem_t constructor: Using GPU backend
PSLP presolve time: 0.07s
Objective offset 0.000000 scaling_factor 1.000000
Running concurrent (showing only PDLP log)

Dual simplex finished in 0.07 seconds
Barrier finished in 0.07 seconds
   Iter    Primal Obj.      Dual Obj.    Gap        Primal Res.  Dual Res.   Time
      0 +0.00000000e+00 +0.00000000e+00  0.00e+00   0.00e+00     9.23e-01   0.121s
PDLP finished
Concurrent time: 0.123s
Solved with dual simplex
Status: Optimal   Objective: -4.64753143e+02  Iterations: 10  Time: 0.123s
Primal residual (abs/rel): 1.16e-13/6.04e-14
Dual   residual (abs/rel): 1.58e-16/1.72e-18
PASSED
tests/linear_programming/test_cpu_only_execution.py::TestSolutionInterfacePolymorphism::test_mip_solution_values Setting parameter time_limit to 6.000000e+01
optimization_problem_t constructor: Using GPU backend
cuOpt version: 26.4.0, git hash: 98b6ac6a, host arch: x86_64, device archs: 75-real,80-real,86-real,90a-real,100f-real,120a-real,120
CPU: AMD EPYC 7313P 16-Core Processor, threads (physical/logical): 12/12, RAM: 34.25 GiB
CUDA 13.1, device: NVIDIA H100 PCIe (ID 0), VRAM: 79.18 GiB
CUDA device UUID: ffffffbb4d5dffffff8a-1effffffdc-31ff

Solving a problem with 1 constraints, 2 variables (2 integers), and 2 nonzeros
Problem scaling:
Objective coefficents range:          [1e+00, 1e+00]
Constraint matrix coefficients range: [1e+00, 2e+00]
Constraint rhs / bounds range:        [0e+00, 3e+00]
Variable bounds range:                [0e+00, 0e+00]

Original problem: 1 constraints, 2 variables, 2 nonzeros
Calling Papilo presolver (git hash 741a2b9c)
Presolve status: reduced the problem
Presolve removed: 0 constraints, 0 variables, 0 nonzeros
Presolved problem: 1 constraints, 2 variables, 2 nonzeros
optimization_problem_t constructor: Using GPU backend
Papilo presolve time: 0.01
Objective offset 0.000000 scaling_factor 1.000000
Model fingerprint: 0xa216fc2d
Running presolve!
After cuOpt presolve: 1 constraints, 2 variables, objective offset 0.000000.
cuOpt presolve time: 0.04
Objective function is integral, scale 1

Solving LP root relaxation in concurrent mode
Skipping column scaling
Dual Simplex Phase 1
Dual feasible solution found.
Dual Simplex Phase 2
 Iter     Objective           Num Inf.  Sum Inf.     Perturb  Time
    0 +0.0000000000000000e+00       1 9.00000000e+00 0.00e+00 0.11
    1 +1.5000000000000000e+00       0 0.00000000e+00 0.00e+00 0.11


Root relaxation solution found in 1 iterations and 0.00s by Dual Simplex
Root relaxation objective +1.50000000e+00


 | Explored | Unexplored |    Objective    |     Bound     | IntInf | Depth | Iter/Node |   Gap    |  Time  |
           0            0    +2.000000e+00    +2.000000e+00        0      0   2.0e+00       0.0%      0.12
Gomory    cuts : 0
MIR       cuts : 1
Knapsack  cuts : 0
Strong CG cuts : 0
Optimal solution found at root node. Objective 2.0000000000000000e+00. Time 0.12.
Explored 0 nodes in 0.12s.
Absolute Gap 0.000000e+00 Objective 2.0000000000000000e+00 Lower Bound 2.0000000000000000e+00
Optimal solution found.
Solution objective: 2.000000 , relative_mip_gap 0.000000 solution_bound 2.000000 presolve_time 0.051241 total_solve_time 0.133984 max constraint violation 0.000000 max int violation 0.000000 max var bounds violation 0.000000 nodes 0 simplex_iterations 2
PASSED
tests/linear_programming/test_incumbent_callbacks.py::test_incumbent_get_callback[/mip/swath1.mps] Setting parameter time_limit to 1.000000e+01
optimization_problem_t constructor: Using GPU backend
cuOpt version: 26.4.0, git hash: 98b6ac6a, host arch: x86_64, device archs: 75-real,80-real,86-real,90a-real,100f-real,120a-real,120
CPU: AMD EPYC 7313P 16-Core Processor, threads (physical/logical): 12/12, RAM: 34.15 GiB
CUDA 13.1, device: NVIDIA H100 PCIe (ID 0), VRAM: 79.18 GiB
CUDA device UUID: ffffffbb4d5dffffff8a-1effffffdc-31ff

Solving a problem with 884 constraints, 6805 variables (2306 integers), and 34965 nonzeros
Problem scaling:
Objective coefficents range:          [1e+00, 1e+00]
Constraint matrix coefficients range: [1e+00, 1e+03]
Constraint rhs / bounds range:        [1e+00, 2e+01]
Variable bounds range:                [0e+00, 1e+00]

Original problem: 884 constraints, 6805 variables, 34965 nonzeros
Calling Papilo presolver (git hash 741a2b9c)
Presolve status: reduced the problem
Presolve removed: 402 constraints, 545 variables, 9485 nonzeros
Presolved problem: 482 constraints, 6260 variables, 25480 nonzeros
optimization_problem_t constructor: Using GPU backend
Papilo presolve time: 0.12
Objective offset 0.000000 scaling_factor 1.000000
Model fingerprint: 0x5af1c290
Running presolve!
After cuOpt presolve: 482 constraints, 6260 variables, objective offset 0.000000.
cuOpt presolve time: 1.00

Solving LP root relaxation in concurrent mode
Skipping column scaling
Dual Simplex Phase 1
Dual feasible solution found.
Dual Simplex Phase 2
 Iter     Objective           Num Inf.  Sum Inf.     Perturb  Time
    0 +0.0000000000000000e+00      22 2.20000000e+01 0.00e+00 1.19
    1 +1.5174951000000000e+01      24 2.40000000e+01 0.00e+00 1.20
Removed perturbation of 1.40e-06.


Root relaxation solution found in 107 iterations and 0.04s by Dual Simplex
Root relaxation objective +3.34496858e+02


terminate called without an active exception
Fatal Python error: Aborted

Current thread 0x00007b955f577740 (most recent call first):
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/cuopt/linear_programming/solver/solver.py", line 98 in Solve
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/cuopt/utilities/exception_handler.py", line 24 in func
  File "/__w/cuopt/cuopt/python/cuopt/cuopt/tests/linear_programming/test_incumbent_callbacks.py", line 87 in _run_incumbent_solver_callback
  File "/__w/cuopt/cuopt/python/cuopt/cuopt/tests/linear_programming/test_incumbent_callbacks.py", line 112 in test_incumbent_get_callback
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/_pytest/python.py", line 157 in pytest_pyfunc_call
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/_pytest/python.py", line 1671 in runtest
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/_pytest/runner.py", line 178 in pytest_runtest_call
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/_pytest/runner.py", line 246 in <lambda>
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/_pytest/runner.py", line 344 in from_call
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/_pytest/runner.py", line 245 in call_and_report
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/_pytest/runner.py", line 136 in runtestprotocol
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/_pytest/runner.py", line 117 in pytest_runtest_protocol
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/_pytest/main.py", line 367 in pytest_runtestloop
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/_pytest/main.py", line 343 in _main
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/_pytest/main.py", line 289 in wrap_session
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/_pytest/main.py", line 336 in pytest_cmdline_main
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/_pytest/config/__init__.py", line 175 in main
  File "/pyenv/versions/3.13.12/lib/python3.13/site-packages/_pytest/config/__init__.py", line 201 in console_main
  File "/pyenv/versions/3.13.12/bin/pytest", line 6 in <module>

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, cuopt.linear_programming.cuopt_mps_parser.parser_wrapper, cuopt.linear_programming.data_model.data_model_wrapper, data_model.data_model_wrapper, cuopt_mps_parser.parser_wrapper, cuopt.linear_programming.internals.internals, _cyutility, scipy._cyutility, scipy._lib._ccallback_c, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._pcg64, numpy.random._generator, numpy.random._mt19937, numpy.random._philox, numpy.random._sfc64, numpy.random.mtrand, charset_normalizer.md, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, cuda.bindings._bindings.cydriver, cuda.bindings._bindings.cyruntime_ptds, cuda.bindings._bindings.cyruntime, cuda.bindings.cyruntime, pylibraft.common.cuda, rmm.pylibrmm.stream, rmm.pylibrmm.utils, rmm.pylibrmm.helper, cuda.bindings.cydriver, cuda.bindings.driver, cuda.bindings.runtime, rmm.pylibrmm.memory_resource._memory_resource, rmm.pylibrmm.memory_resource.experimental, rmm.pylibrmm.cuda_stream_pool, rmm.pylibrmm.device_buffer, rmm.librmm._logger, rmm.pylibrmm.logger, pylibraft.common.handle, cupy_backends.cuda._softlink, cupy_backends.cuda.api._runtime_enum, cupy_backends.cuda.api.runtime, cupy._util, cupy.cuda.device, cupy.cuda.memory_hook, cupy_backends.cuda.stream, cupy.cuda.graph, cupy.cuda.stream, cupy_backends.cuda.api._driver_enum, cupy_backends.cuda.api.driver, cupy.cuda.memory, cupy._core.internal, cupy._core._carray, cupy._core._dtype, cupy._core._scalar, cupy.cuda.texture, cupy.cuda.function, cupy_backends.cuda.libs.nvrtc, cupy.cuda.pinned_memory, cupy.cuda.common, cupy.cuda.cub, cupy_backends.cuda.libs.nvtx, cupy.cuda.thrust, cupy._core._accelerator, cupy._core._memory_range, cupy._core._fusion_thread_local, cupy._core._kernel, cupy._core._routines_manipulation, cupy._core._routines_binary, cupy._core._optimize_config, cupy._core._cub_reduction, cupy._core._reduction, cupy._core._routines_math, cupy._core._routines_indexing, cupy._core._routines_linalg, cupy._core._routines_logic, cupy._core._routines_sorting, cupy._core._routines_statistics, cupy._core.dlpack, cupy._core.flags, cupy._core.core, cupy._core._fusion_variable, cupy._core._fusion_trace, cupy._core._fusion_kernel, cupy._core.new_fusion, cupy._core.fusion, cupy._core.raw, cupy.fft._cache, cupy.fft._callback, cupy.random._generator_api, cupy.random._bit_generator, cupy.lib._polynomial, yaml._yaml, numba.core.typeconv._typeconv, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.runtime._nrt_python, numba.np.ufunc._internal, numba.experimental.jitclass._box, numba.cuda.types.abstract.itertools, numba.cuda.types.functions.itertools, numba.cuda.cext._typeconv, numba.cuda.cext._helperlib, numba.cuda.cext.mviewbuf, cuda.core.system._system, cuda.bindings._bindings.cynvrtc, cuda.bindings.cynvrtc, cuda.bindings._internal.utils, cuda.bindings._internal.nvvm, cuda.bindings.cynvvm, cuda.bindings.nvrtc, cuda.bindings.nvvm, cuda.core._utils.cuda_utils, cuda.core.cu13.system._system, cuda.core._layout, cuda.core._resource_handles, cuda.core._context, cuda.core._event, cuda.core._stream, cuda.core._memory._ipc, cuda.core._memory._memory_pool, cuda.core._memory._device_memory_resource, cuda.core._memory._pinned_memory_resource, cuda.core._dlpack, cuda.core._device, cuda.core._memory._buffer, cuda.core._memory._graph_memory_resource, cuda.core._memory._managed_memory_resource, cuda.core._memoryview, cuda.core._launch_config, cuda.core._kernel_arg_handler, cuda.core._module, cuda.core._launcher, cuda.core._program, numba.cuda.cext._dispatcher, pyarrow.lib, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, nvtx._lib.lib, nvtx._lib.profiler, pylibcudf.libcudf.types, pylibcudf.types, pylibcudf.libcudf.aggregation, pylibcudf.aggregation, pylibcudf.gpumemoryview, pylibcudf.utils, pylibcudf._interop_helpers, pylibcudf.table, pylibcudf.filling, pylibcudf.traits, pylibcudf.column, pylibcudf.scalar, pylibcudf.libcudf.binaryop, pylibcudf.binaryop, pylibcudf.column_factories, pylibcudf.concatenate, pylibcudf.contiguous_split, pylibcudf.libcudf.copying, pylibcudf.copying, pylibcudf.libcudf.datetime, pylibcudf.datetime, pylibcudf.experimental._join_streams, pylibcudf.libcudf.expressions, pylibcudf.expressions, pylibcudf.groupby, pylibcudf.hashing, pylibcudf.interop, pylibcudf.io.datasource, pylibcudf.libcudf.io.json, pylibcudf.libcudf.io.types, pylibcudf.io.types, pylibcudf.io.avro, pylibcudf.io.csv, pylibcudf.io.parquet, pylibcudf.io.text, pylibcudf.libcudf.io.hybrid_scan, pylibcudf.io.experimental.hybrid_scan, pylibcudf.io.json, pylibcudf.io.orc, pylibcudf.io.parquet_metadata, pylibcudf.io.timezone, pylibcudf.libcudf.join, pylibcudf.join, pylibcudf.json, pylibcudf.libcudf.labeling, pylibcudf.labeling, pylibcudf.libcudf.lists.combine, pylibcudf.libcudf.lists.contains, pylibcudf.lists, pylibcudf.merge, pylibcudf.null_mask, pylibcudf.nvtext.byte_pair_encode, pylibcudf.nvtext.deduplicate, pylibcudf.nvtext.edit_distance, pylibcudf.nvtext.generate_ngrams, pylibcudf.nvtext.jaccard, pylibcudf.nvtext.minhash, pylibcudf.nvtext.ngrams_tokenize, pylibcudf.nvtext.normalize, pylibcudf.nvtext.replace, pylibcudf.libcudf.nvtext.stemmer, pylibcudf.nvtext.stemmer, pylibcudf.nvtext.tokenize, pylibcudf.nvtext.wordpiece_tokenize, pylibcudf.partitioning, pylibcudf.prefetch, pylibcudf.quantiles, pylibcudf.libcudf.reduce, pylibcudf.reduce, pylibcudf.libcudf.replace, pylibcudf.replace, pylibcudf.reshape, pylibcudf.rolling, pylibcudf.libcudf.round, pylibcudf.round, pylibcudf.search, pylibcudf.sorting, pylibcudf.libcudf.stream_compaction, pylibcudf.stream_compaction, pylibcudf.strings.attributes, pylibcudf.libcudf.strings.char_types, pylibcudf.strings.capitalize, pylibcudf.strings.case, pylibcudf.strings.char_types, pylibcudf.libcudf.strings.combine, pylibcudf.strings.combine, pylibcudf.strings.regex_program, pylibcudf.strings.contains, pylibcudf.strings.convert.convert_booleans, pylibcudf.strings.convert.convert_datetime, pylibcudf.strings.convert.convert_durations, pylibcudf.strings.convert.convert_fixed_point, pylibcudf.strings.convert.convert_floats, pylibcudf.strings.convert.convert_integers, pylibcudf.strings.convert.convert_ipv4, pylibcudf.strings.convert.convert_lists, pylibcudf.strings.convert.convert_urls, pylibcudf.strings.extract, pylibcudf.strings.find, pylibcudf.strings.find_multiple, pylibcudf.strings.findall, pylibcudf.strings.padding, pylibcudf.libcudf.strings.regex_flags, pylibcudf.strings.regex_flags, pylibcudf.strings.repeat, pylibcudf.strings.replace, pylibcudf.strings.replace_re, pylibcudf.strings.reverse, pylibcudf.libcudf.strings.side_type, pylibcudf.strings.side_type, pylibcudf.strings.slice, pylibcudf.strings.split.partition, pylibcudf.strings.split.split, pylibcudf.strings.strip, pylibcudf.libcudf.strings.translate, pylibcudf.strings.translate, pylibcudf.strings.wrap, pylibcudf.transform, pylibcudf.transpose, pylibcudf.libcudf.unary, pylibcudf.unary, pylibcudf.utilities, numba.types.itertools, cudf._lib.strings_udf, cuda.bindings._internal.nvjitlink, cuda.bindings.cynvjitlink, cuda.bindings.nvjitlink, pyarrow._feather, cuopt.linear_programming.solver.solver_parameters, cuopt.linear_programming.solver.solver_wrapper, cuopt.routing.utils_wrapper, cuopt.routing.vehicle_routing_wrapper, cuopt.distance_engine.waypoint_matrix_wrapper (total: 315)
./ci/run_cuopt_pytests.sh: line 12:  1610 Aborted                 (core dumped) pytest -s --cache-clear "$@" tests

Expected behavior
Should Pass without a problem

Environment details (please complete the following information):
wheel-tests-cuopt / 13.1.1, 3.13, amd64, rockylinux8, h100, latest-driver, latest-deps

Metadata

Metadata

Assignees

Labels

awaiting responseThis expects a response from maintainer or contributor depending on who requested in last comment.bugSomething isn't working

Type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions