Yes, the US would be better off if it had adopted the metric system in Jefferson's era. This would put the country in line with Europe. The US would not need to convert things because the measuring system would be in line with the rest of the world. This would make things easier.
We would be on the same page as scientists around the world. Also students would not have to relearn how to measure things when it came time for them to study science. This is a big problem in our educational system and should have be solved a long time ago.
Although I think it is incredibly unlikely that the U.S. will ever make the change now, I do think it would have been better if we had adopted the metric system early on. Most of the rest of the world uses the metric system so this would make conversions easier between countries. The metric system is also a more scientific system. The smaller units fit logically into larger units, making it easier to teach and to learn.
Whether we use metric or imperial does not really matter in terms of how successful the country is. For most of the population, measurements do not really have much of an impact on their daily lives, so it definitely does not have an impact on the overall well-being of the country.