nested task error: AssertionError: norm(G * v - curr) < if eltype(curr) == Float64

[peggyn7]

Hello,

I am getting an error message when trying to run Omniscape version 0.5.0 in Julia version 1.6.1.

When we previously used Omniscape version 0.1.4 in Julia version 1.3.1, the processing was completed successfully.

The successful ascii file had the following characteristics:
For my sample ascii file I have:
ncols 1565
nrows 1011
xllcorner 6414438.576614
yllcorner 1847921.0801045
cellsize 30
NODATA_value -9999

The ini file params like used were:
block_size = 11
radius = 88
buffer = 0

Following is the log and error message that I received with the recent run in Julia version 1.6.1 with Omniscape version 0.5.0:

julia> using Omniscape

julia> run_omniscape("C:/Users/user/Documents/ArcGIS/Projects/WildlifeConn/ConfigurationFile/configfile.ini")
┌ Warning: The following unsupported arguments were provided and will be ignored:
│          write_flow_potential write_normalized_currmap
└ @ Omniscape C:\Users\user\.julia\packages\Omniscape\DiYof\src\errors_warnings.jl:99
[ Info: Starting up Omniscape with 18 workers and double precision
[ Info: Using Circuitscape with the CG+AMG solver...
[ Info: Calculating block artifact correction array...
[ Info: Solving moving window targets...
Progress:  48%|████████████████████████                          |  ETA: 0:40:26ERROR: TaskFailedException
Stacktrace:
Progress:  48%|█████████████████████████                         |  ETA: 0:40:25 [1] wait
   @ .\task.jl:322 [inlined]
 [2] threading_run(func::Function)
   @ Base.Threads .\threadingconstructs.jl:34
 [3] macro expansion
   @ .\threadingconstructs.jl:93 [inlined]
 [4] run_omniscape(cfg::Dict{String, String}, resistance::Matrix{Union{Missing, Float64}}; reclass_table::Matrix{Union{Missing, Float64}}, source_strength::Matrix{Union{Missing, Float64}}, condition1::Matrix{Union{Missing, Float64}}, condition2::Matrix{Union{Missing, Float64}}, condition1_future::Matrix{Union{Missing, Float64}}, condition2_future::Matrix{Union{Missing, Float64}}, wkt::String, geotransform::Vector{Float64}, write_outputs::Bool)
   @ Omniscape C:\Users\user\.julia\packages\Omniscape\DiYof\src\main.jl:268
 [5] run_omniscape(path::String)
   @ Omniscape C:\Users\user\.julia\packages\Omniscape\DiYof\src\main.jl:560
 [6] top-level scope
   @ REPL[2]:1

    nested task error: AssertionError: norm(G * v - curr) < if eltype(curr) == Float64
            TOL_DOUBLE
        else
            TOL_SINGLE
        end
    Stacktrace:
     [1] solve_linear_system(G::SparseArrays.SparseMatrixCSC{Float64, Int64}, curr::Vector{Float64}, M::AlgebraicMultigrid.Preconditioner{AlgebraicMultigrid.MultiLevel{AlgebraicMultigrid.Pinv{Float64}, AlgebraicMultigrid.GaussSeidel{AlgebraicMultigrid.SymmetricSweep}, AlgebraicMultigrid.GaussSeidel{AlgebraicMultigrid.SymmetricSweep}, SparseArrays.SparseMatrixCSC{Float64, Int64}, SparseArrays.SparseMatrixCSC{Float64, Int64}, LinearAlgebra.Adjoint{Float64, SparseArrays.SparseMatrixCSC{Float64, Int64}}, AlgebraicMultigrid.MultiLevelWorkspace{Vector{Float64}, 1}}})
       @ Circuitscape C:\Users\user\.julia\packages\Circuitscape\Qr6wW\src\core.jl:616
     [2] macro expansion
       @ .\timing.jl:287 [inlined]
     [3] multiple_solve(s::Circuitscape.AMGSolver, matrix::SparseArrays.SparseMatrixCSC{Float64, Int64}, sources::Vector{Float64}, suppress_info::Bool)
       @ Circuitscape C:\Users\user\.julia\packages\Circuitscape\Qr6wW\src\raster\advanced.jl:311
     [4] multiple_solver(cfg::Dict{String, String}, solver::Circuitscape.AMGSolver, a::SparseArrays.SparseMatrixCSC{Float64, Int64}, sources::Vector{Float64}, grounds::Vector{Float64}, finitegrounds::Vector{Float64})
       @ Circuitscape C:\Users\user\.julia\packages\Circuitscape\Qr6wW\src\raster\advanced.jl:291
     [5] calculate_current(conductance::Matrix{Union{Missing, Float64}}, source::Matrix{Union{Missing, Float64}}, ground::Matrix{Float64}, cs_flags::Circuitscape.RasterFlags, cs_cfg::Dict{String, String}, T::DataType)
       @ Omniscape C:\Users\user\.julia\packages\Omniscape\DiYof\src\utils.jl:410
     [6] solve_target!(i::Int64, n_targets::Int64, int_arguments::Dict{String, Int64}, targets::Matrix{Float64}, source_strength::Matrix{Union{Missing, Float64}}, resistance::Matrix{Union{Missing, Float64}}, os_flags::Omniscape.OmniscapeFlags, cs_cfg::Dict{String, String}, cs_flags::Circuitscape.RasterFlags, o::Circuitscape.OutputFlags, condition1_present::Matrix{Union{Missing, Float64}}, condition1_future::Matrix{Union{Missing, Float64}}, condition2_present::Matrix{Union{Missing, Float64}}, condition2_future::Matrix{Union{Missing, Float64}}, comparison1::String, comparison2::String, condition1_lower::Float64, condition1_upper::Float64, condition2_lower::Float64, condition2_upper::Float64, correction_array::Matrix{Float64}, cum_currmap::Array{Float64, 3}, fp_cum_currmap::Array{Float64, 3}, precision::DataType)
       @ Omniscape C:\Users\user\.julia\packages\Omniscape\DiYof\src\utils.jl:497
     [7] macro expansion
       @ C:\Users\user\.julia\packages\Omniscape\DiYof\src\main.jl:273 [inlined]
     [8] (::Omniscape.var"#185#threadsfor_fun#15"{Int64, ProgressMeter.Progress, Circuitscape.RasterFlags, Circuitscape.OutputFlags, Int64, Dict{String, String}, Float64, Float64, Float64, Float64, String, String, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}})(onethread::Bool)
       @ Omniscape .\threadingconstructs.jl:81
     [9] (::Omniscape.var"#185#threadsfor_fun#15"{Int64, ProgressMeter.Progress, Circuitscape.RasterFlags, Circuitscape.OutputFlags, Int64, Dict{String, String}, Float64, Float64, Float64, Float64, String, String, Omniscape.OmniscapeFlags, DataType, Dict{String, Int64}, UnitRange{Int64}})()
       @ Omniscape .\threadingconstructs.jl:48

Thank you,

Peggy

[vlandau]

@peggyn7, this is caused by an issue with Circuitscape. One thing you might try is changing the solver to CHOLMOD, as it is less likely to cause an AssertError because it has a higher error tolerance.

@ranjanan I'm not sure what the best way forward is. Maybe a new argument to the Circuitscape INI that specifies a higher tolerance (obviously to be used with great caution)? I'm also not sure if we should be creating an issue in the packages with the solvers themselves. Shouldn't a direct solver always produce a solution where the norm is 0?

Finally, cc @blindjesse, this error is related to the one you got, and subsequently solved by using CHOLMOD because there is a higher error tolerance. Wanted to tag you in case you want to stay in the loop.

[jessjaco]

In my (particular) case I had to switch to CHOLMOD and single precision. Haven't had a chance to look any deeper to see if it's related in any way to the source / resistance, ini etc.

[peggyn7]

Thanks so much for your help! We made the following change to the configuration file and it worked.

precision = single
solver = cholmod

[vlandau]

Reopening until this can get fixed in Circuitscape.

[ranjanan]

@peggyn7 could you attach your files so I can reproduce this?

[trantom]

@ranjanan Here are the files I used that produced the same error. It is a clipped version of @peggyn7's files. I also attached the config file. It worked with the addition of the solver and precision parameters on the bottom on Omniscape version 0.5.0 in Julia version 1.6.1. Previously it had worked without those parameters on Omniscape version 0.1.4 Julia version 1.3.1. This is all referenced in the original post up above so we used the same environments.

working_config.zip

[ranjanan]

Test with new Circuitscape version? JuliaRegistries/General#42367. Will need a new version of Omniscape

[vlandau]

I'll release a new version of Omniscape with an updated Circuitscape dep today.

[vlandau]

Confirmed to be fixed in Omniscape 0.5.3 (Circuitscape 5.8.3). I tested this with cholmod and cg+amg and double precision (though note that precision doesn't influence the matrix norm check in Circuitscape as of version 5.8.3). I'm going to close #100 as well.

Thanks @ranjanan!

[trantom]

@ranjanan @vlandau Thank you so much for your work. I ran the updated 0.5.3 version and it is able to fully process with double precision and cg+amg on the previous dataset I shared and an entirely new dataset I put together as well.

[tgelmi-candusso]

Omniscape crashing with AssertionError: norm(G * v .- curr) / norm(curr) < 0.0001