at hull speed, if the fins are centered you're not loosing much, if anything.
at higher speed, i've heard that 1 knot is probably in the ballpark, and it's even more critical to make sure that they are centered.
like eveything, if they are not used regularly they will probably have issues, considerign how expensive they are, i'd keep them on and make sure they are used regularly, even if not needed.