The foam was placed there for a reason: to provide adequate flotation in the event that the boat takes on water. If you remove, and do not replace the foam, then IF your boat takes on water it might sink vs flood, but remain near the surface ( this is a big deal, actually. You really want the boat to stay on the surface ).
SO, yes, you do need to put the foam back in. THe length of time the boat stays in the water is not relevant to the question. If the boat is in the water ( for whatever length of time ) you want it to function properly. One of those functions is to stay near the surface, even if filled with water and people. Every time, all of the time.
----------
The two-part pourable foam available at marine stores is a good product to use to replace the removed foam. Definitely read the instructions carefully and follow them exactly. It is not hard to work with, as long as you follow the directions. It gets complicated if you do not. ( The stuff is a bit pricey, however ). It also forms a closed cell foam structure that is resistant to saturation and/or waterlogging. ( Unless you shape it with a knife or tool... )