James Vassar sold his farm and settled in the center of Poughkeepsie to brew ale. He built a brewery, in a part of which the family lived while a house was built for them nearby. 

Matthew Vassar, with his mother's aid, ran away from home to avoid being apprenticed to a tanner. "Set off to seek my fortune with 6/ in my pockett, two corse East India Muslin Shirts, a pair of woolen Socks, Scow Skin Shoes, all tied up in a cotton bandana Handkerchief."   

He found employment at Butterworth's store at Balm Town, near Newburgh.  “…they pointed out my work if I did Stay, viz, to Measure Wood Weight Iron, measure Salt, &c, in fact do all kinds of drudging even to taking care of the Old Gentlemans horse. All of which I consented and found a home for 3Years, Begining with Labour for a living and ending on a Sallary of $300 pr annum.”     Elizabeth Hazelton Haight, ed., Autobiography and Letters of Matthew Vassar