I grew up around really small boats. Sure, I got aboard larger boats, sailboats, ferry boats, lake mail boats, etc. But what we owned were small. My dad built a nifty "pram" sail boat with a bright blue hull and a bright red sail, and we spent years sailing and rowing it. In my own right, I owned a handsomely designed Old Town Penobscot canoe for years, and enjoyed paddling the lakes and rivers in N.C. So I love boats and boating. The downside is, even small boats need regular attention and maintenance.
For years, I had a garage that stored the canoe easily, but later moved to a house where it had to be stored outside, and pine resin on the hull became a problem. My dad's little sailboat finally got stoved by careless parking of a car in the garage. (I'm glad to say, it wasn't me.)
Living on or near a lake or other body of water is ideal, to factor out travel time, especially if the boat can be securely stored on the water for easy quick use. I've never had that luxury. Larger boats require serious discretionary spending, so it is not only a matter of budget, but of family agreement on wanting to go boating and enjoying the experience.
So my suggestion would be, start with something modest, see how the boat fits into your schedule and family life, and keep an eye on opportunities to sell the boat if it becomes little used or too much of a burden. I finally sold my canoe for a church auction. I was glad to have owned it, and it was a relief to sell it for a good cause.