heap-buffer-overflow while adding an invalid triangle face
I came across a pathological mesh example that triggered a heap corruption in version %OpenMesh 8.0.
The following is a minimal failing example, which will fail when run with AddressSanitizer (heap-buffer-overflow in OpenMesh::PolyConnectivity::add_face(OpenMesh::VertexHandle const*, unsigned long)
).
#include <OpenMesh/Core/Mesh/PolyMesh_ArrayKernelT.hh>
typedef OpenMesh::PolyMesh_ArrayKernelT<> MyMesh;
int main()
{
MyMesh mesh;
const auto a = mesh.add_vertex(MyMesh::Point{-6.95757 -0.418557 -7.25885});
const auto b = mesh.add_vertex(MyMesh::Point{-7.03569 -0.128014 -7.26395});
mesh.add_face(a,b,a);
return 0;
}
When run with assertions enabled, this triggers:
OpenMesh/Core/Mesh/PolyConnectivity.cc:272: OpenMesh::ArrayKernel::FaceHandle OpenMesh::PolyConnectivity
::add_face(const VertexHandle*, size_t): Assertion `boundary_prev.is_valid()' failed.