As you get to know your boat's handling characteristics and gain more experience as a skipper, your judgement of such things will become more reliable. I used to be extra conservative about rough, windy waters with my first boat. Then one time the wind came up unexpectedly and I chose to continue on with the plan to "duck into" to a sheltered anchorage if things got worse. They did not get worse, but didn't improve either. I learned to have more confidence in both my vessel and my capabilities.
But I would also listen to my dock buddies. If they said it was too rough, I would stay at the dock too. One thing is that going through rough water is easier then docking or anchoring, etc. One of the most important things to remember is what will you do if your engines cut out for ANY reason???? Have a plan in mind before you leave and run through it in your mind as you progress.
This planning saved my tush one time when my [single] engine cut out and the wind was quickly taking my boat towards the rocks! We immediately deployed a stern anchor and kept the boat off the rocks. That allowed us the time to deal with the cause of the engine problem and then continue on our way back to the berth.
From that experience, I learned to always have my stern anchor easily accessible---especially when single handling my boat. Different boats have different issues, and your solutions may be different. But you still need a plan for the "what ifs" on your boat.