those look a bit old, but probably a chemical fixing in the holes.
ide say the same, quick easy way is an angle grinder. however… the grinder will cut into the brick and leave very noticeable damage.
Try create yourself a protection plate. use a bit of 2-3mm flat aluminum or steel plate. just 6 inch square or there about.
create a tight curve or drill a large hole in it. then place over or to the side of the rod.
grind against the locator now, and as your zipping through it the blade will be hitting your protection plate which is between the grinder and flat against the wall.
you may want to paint the stubb matt black or something not noticeable.
hope that makes sense! 😀