James Wheaton Converse married Emeline Coolidge, daughter of Nathan Coolidge of Boston, on 5 September, 1833 in Boston, Massachusetts. The 1860 census shows the family listed with 3 children, James Wheaton, Jr, age 16, Costello Coolidge, age 11, and Emma Maria, age 9.

Though he never lived in Grand Rapids, James Converse was very involved in business dealings there. When he visited in 1850 he took note of the gypsum deposits in the land he had purchased from the Baptist Mission and went into business with William Hovey to manufacture plaster. He put up most of the money to build the Pearl Street Bridge in 1858. He helped construct the first railroad from Kalamazoo to Grand Rapids, and on to Newaygo. He was President of the Phoenix Furniture Company, and President and owner of the Converse Manufacturing Company in Newaygo. He also furnished the funds to build the Second Baptist Church on the corner of California and Gold Streets.
He was given two telephones by his friend Alexander Graham Bell and brought them to Grand Rapids to set up the first telephone line in the city, from his office on Monroe Street to the plaster mine across the river, on October 30, 1877.
In 1884, James and Emeline built a new house at 347 Beacon Street in Boston's Back Bay neighborhood. The house is still standing.

James, James, Jr and Emeline are buried in Mount Auburn Cemetery in Cambridge, Massachusetts.