very slow and memory problems after updating to v2

[JianghuiDu]

My package was build using v1.0 but after updating to v2.0 the same code has becoming extremely slow and causing memory errors. I have a for loop with 14 iterations where I do symbolic calculations. After updating to v2, Julia exists abruptly after a few iterations, even though I tested that I can manually run each iteration individually. It would appear that SymPy.solveset in the new version cannot handle large equations like the old one. I'm not even sure what the problem is because the loop constantly causes Julia breakdown.

These are some of the errors I got:

1-element ExceptionStack:
ReadOnlyMemoryError()
Stacktrace:
  [1] macro expansion
    @ C:\Users\pkudu\.julia\packages\PyCall\1gn3u\src\exception.jl:108 [inlined]
  [2] #107
    @ C:\Users\pkudu\.julia\packages\PyCall\1gn3u\src\pyfncall.jl:43 [inlined]
  [3] disable_sigint
    @ .\c.jl:473 [inlined]
  [4] __pycall!
    @ C:\Users\pkudu\.julia\packages\PyCall\1gn3u\src\pyfncall.jl:42 [inlined]
  [5] _pycall!(ret::PyCall.PyObject, o::PyCall.PyObject, args::Tuple{PyCall.PyObject, PyCall.PyObject}, nargs::Int64, kw::Ptr{Nothing})
    @ PyCall C:\Users\pkudu\.julia\packages\PyCall\1gn3u\src\pyfncall.jl:29
  [6] _pycall!(ret::PyCall.PyObject, o::PyCall.PyObject, args::Tuple{PyCall.PyObject, PyCall.PyObject}, kwargs::@Kwargs{})
    @ PyCall C:\Users\pkudu\.julia\packages\PyCall\1gn3u\src\pyfncall.jl:11
  [7] (::PyCall.PyObject)(::PyCall.PyObject, ::Vararg{Any}; kwargs::@Kwargs{})
    @ PyCall C:\Users\pkudu\.julia\packages\PyCall\1gn3u\src\pyfncall.jl:86
  [8] solveset(x::SymPyCore.Sym{PyCall.PyObject}, args::SymPyCore.Sym{PyCall.PyObject}; kwargs::@Kwargs{})
    @ SymPy C:\Users\pkudu\.julia\packages\SymPyCore\tOpxk\src\SymPy\gen_methods_sympy.jl:33
  [9] speciation_model(substance_spec::String, speciation::DataFrame, adsorption::DataFrame)
    @ Main c:\users\pkudu\documents\sedtrace\src\speciation_code.jl:183
 [10] top-level scope
    @ REPL[374]:1
 [11] eval
    @ .\boot.jl:385 [inlined]
 [12] eval
    @ .\Base.jl:88 [inlined]
 [13] repleval(m::Module, code::Expr, ::String)
    @ VSCodeServer c:\Users\pkudu\.vscode\extensions\julialang.language-julia-1.79.2\scripts\packages\VSCodeServer\src\repl.jl:229
 [14] (::VSCodeServer.var"#112#114"{Module, Expr, REPL.LineEditREPL, REPL.LineEdit.Prompt})()
    @ VSCodeServer c:\Users\pkudu\.vscode\extensions\julialang.language-julia-1.79.2\scripts\packages\VSCodeServer\src\repl.jl:192
 [15] with_logstate(f::Function, logstate::Any)
    @ Base.CoreLogging .\logging.jl:515
 [16] with_logger
    @ .\logging.jl:627 [inlined]
 [17] (::VSCodeServer.var"#111#113"{Module, Expr, REPL.LineEditREPL, REPL.LineEdit.Prompt})()
    @ VSCodeServer c:\Users\pkudu\.vscode\extensions\julialang.language-julia-1.79.2\scripts\packages\VSCodeServer\src\repl.jl:193
 [18] #invokelatest#2
    @ .\essentials.jl:892 [inlined]
 [19] invokelatest(::Any)
    @ Base .\essentials.jl:889
 [20] (::VSCodeServer.var"#64#65")()
    @ VSCodeServer c:\Users\pkudu\.vscode\extensions\julialang.language-julia-1.79.2\scripts\packages\VSCodeServer\src\eval.jl:34

or

1-element ExceptionStack:
PyError ($(Expr(:escape, :(ccall(#= C:\Users\pkudu\.julia\packages\PyCall\1gn3u\src\pyfncall.jl:43 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'TypeError'>
TypeError("unhashable type: '��ɚ�\x01'")
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\solvers\solveset.py", line 2252, in solveset
    rv = solveset(f.xreplace({symbol: x}), x, domain)
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\solvers\solveset.py", line 2276, in solveset
    return _solveset(f, symbol, domain, _check=True)
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\solvers\solveset.py", line 1112, in _solveset
    result_rational = _solve_as_rational(equation, symbol, domain)
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\solvers\solveset.py", line 590, in _solve_as_rational
    return _solve_as_poly(g, symbol, domain)
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\solvers\solveset.py", line 801, in _solve_as_poly
    solns = roots(f, symbol, cubics=True, quartics=True,
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\polys\polyroots.py", line 1089, in roots
    _update_dict(result, zeros, roots_linear(f)[0], 1)
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\polys\polyroots.py", line 43, in roots_linear
    r = factor(r)
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\polys\polytools.py", line 6542, in factor
    return _generic_factor(f, gens, args, method='factor')
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\polys\polytools.py", line 6223, in _generic_factor
    return _symbolic_factor(sympify(expr), opt, method)
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\polys\polytools.py", line 6163, in _symbolic_factor
    coeff, factors = _symbolic_factor_list(together(expr, fraction=opt['fraction']), opt, method)
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\polys\polytools.py", line 6128, in _symbolic_factor_list
    _coeff, _factors = func()
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\polys\polytools.py", line 3350, in factor_list
    coeff, factors = f.rep.factor_list()
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\polys\polyclasses.py", line 823, in factor_list
    coeff, factors = dmp_factor_list(f.rep, f.lev, f.dom)
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\polys\factortools.py", line 1423, in dmp_factor_list
    coeff, factors = dmp_zz_factor(f, v, K)
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\polys\factortools.py", line 1159, in dmp_zz_factor
    g = dmp_sqf_part(g, u, K)
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\polys\sqfreetools.py", line 256, in dmp_sqf_part
    gcd = dmp_gcd(gcd, dmp_diff_in(f, 1, i, u, K), u, K)
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\polys\euclidtools.py", line 1625, in dmp_gcd
    return dmp_inner_gcd(f, g, u, K)[0]
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\polys\euclidtools.py", line 1584, in dmp_inner_gcd
    h, cff, cfg = _dmp_inner_gcd(f, g, u, K)
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\polys\euclidtools.py", line 1553, in _dmp_inner_gcd
    return dmp_zz_heu_gcd(f, g, u, K)
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\polys\euclidtools.py", line 1342, in dmp_zz_heu_gcd
    cfg_, r = dmp_div(g, h, u, K)
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\polys\densearith.py", line 1629, in dmp_div
    return dmp_rr_div(f, g, u, K)
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\polys\densearith.py", line 1397, in dmp_rr_div
    h = dmp_mul_term(g, c, j, u, K)
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\polys\densearith.py", line 184, in dmp_mul_term
    return [ dmp_mul(cf, c, v, K) for cf in f ] + dmp_zeros(i, v, K)
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\polys\densearith.py", line 184, in <listcomp>
    return [ dmp_mul(cf, c, v, K) for cf in f ] + dmp_zeros(i, v, K)
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\polys\densearith.py", line 828, in dmp_mul
    coeff = dmp_add(coeff, dmp_mul(f[j], g[i - j], v, K), v, K)
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\polys\densearith.py", line 828, in dmp_mul
    coeff = dmp_add(coeff, dmp_mul(f[j], g[i - j], v, K), v, K)
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\polys\densearith.py", line 828, in dmp_mul
    coeff = dmp_add(coeff, dmp_mul(f[j], g[i - j], v, K), v, K)
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\polys\densearith.py", line 567, in dmp_add
    if df < 0:
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\core\decorators.py", line 231, in _func
    other = sympify(other, strict=True)
  File "C:\Users\pkudu\.julia\conda\3\x86_64\lib\site-packages\sympy\core\sympify.py", line 383, in sympify
    return conv(a)

Stacktrace:
  [1] pyerr_check
    @ C:\Users\pkudu\.julia\packages\PyCall\1gn3u\src\exception.jl:75 [inlined]
  [2] pyerr_check
    @ C:\Users\pkudu\.julia\packages\PyCall\1gn3u\src\exception.jl:79 [inlined]
  [3] _handle_error(msg::String)
    @ PyCall C:\Users\pkudu\.julia\packages\PyCall\1gn3u\src\exception.jl:96
  [4] macro expansion
    @ C:\Users\pkudu\.julia\packages\PyCall\1gn3u\src\exception.jl:110 [inlined]
  [5] #107
    @ C:\Users\pkudu\.julia\packages\PyCall\1gn3u\src\pyfncall.jl:43 [inlined]
  [6] disable_sigint
    @ .\c.jl:473 [inlined]
  [7] __pycall!
    @ C:\Users\pkudu\.julia\packages\PyCall\1gn3u\src\pyfncall.jl:42 [inlined]
  [8] _pycall!(ret::PyCall.PyObject, o::PyCall.PyObject, args::Tuple{PyCall.PyObject, PyCall.PyObject}, nargs::Int64, kw::Ptr{Nothing})
    @ PyCall C:\Users\pkudu\.julia\packages\PyCall\1gn3u\src\pyfncall.jl:29
  [9] _pycall!(ret::PyCall.PyObject, o::PyCall.PyObject, args::Tuple{PyCall.PyObject, PyCall.PyObject}, kwargs::@Kwargs{})
    @ PyCall C:\Users\pkudu\.julia\packages\PyCall\1gn3u\src\pyfncall.jl:11
 [10] (::PyCall.PyObject)(::PyCall.PyObject, ::Vararg{PyCall.PyObject}; kwargs::@Kwargs{})
    @ PyCall C:\Users\pkudu\.julia\packages\PyCall\1gn3u\src\pyfncall.jl:86
 [11] solveset(x::SymPyCore.Sym{PyCall.PyObject}, args::SymPyCore.Sym{PyCall.PyObject}; kwargs::@Kwargs{})
    @ SymPy C:\Users\pkudu\.julia\packages\SymPyCore\tOpxk\src\SymPy\gen_methods_sympy.jl:33
 [12] speciation_model(parameters::DataFrame, substance_spec::String, speciation::DataFrame, adsorption::DataFrame)
    @ Main c:\users\pkudu\documents\sedtrace\src\speciation_code.jl:193
 [13] top-level scope
    @ .\REPL[349]:2
 [14] eval
    @ .\boot.jl:385 [inlined]
 [15] eval
    @ .\Base.jl:88 [inlined]
 [16] repleval(m::Module, code::Expr, ::String)
    @ VSCodeServer c:\Users\pkudu\.vscode\extensions\julialang.language-julia-1.79.2\scripts\packages\VSCodeServer\src\repl.jl:229
 [17] (::VSCodeServer.var"#112#114"{Module, Expr, REPL.LineEditREPL, REPL.LineEdit.Prompt})()
    @ VSCodeServer c:\Users\pkudu\.vscode\extensions\julialang.language-julia-1.79.2\scripts\packages\VSCodeServer\src\repl.jl:192
 [18] with_logstate(f::Function, logstate::Any)
    @ Base.CoreLogging .\logging.jl:515
 [19] with_logger
    @ .\logging.jl:627 [inlined]
 [20] (::VSCodeServer.var"#111#113"{Module, Expr, REPL.LineEditREPL, REPL.LineEdit.Prompt})()
    @ VSCodeServer c:\Users\pkudu\.vscode\extensions\julialang.language-julia-1.79.2\scripts\packages\VSCodeServer\src\repl.jl:193
 [21] #invokelatest#2
    @ .\essentials.jl:892 [inlined]
 [22] invokelatest(::Any)
    @ Base .\essentials.jl:889
 [23] (::VSCodeServer.var"#64#65")()
    @ VSCodeServer c:\Users\pkudu\.vscode\extensions\julialang.language-julia-1.79.2\scripts\packages\VSCodeServer\src\eval.jl:34

[JianghuiDu]

This seems to only happen to Julia 1.10, but not 1.9.

[jverzani]

Hmm, there are a lot of moving parts.

First, I'd try to see if SymPyPythonCall works.

That should be a drop-in replacement and might just be the solution.

Otherwise, is the code in a package? If so, I could run downstream tests with it so we can see a grid of OS and julia versions.

[JianghuiDu]

I have move my development to WSL+Docker and there is no problem there with the new SymPy version, but on Windows itself this problem persists and SymPyPythonCall is even worse as it kills my computer instantly. Very strange...

[jverzani]

Yes, very strange. I don't know if you have code to share that I can look at. Maybe there is something I can spot.