It’s all about performance and development speed. Of course, if you are a good programmer and design something that is really tailored to your needs, you might achieve better performance than if you had used a Javascript framework. But do you have the time to do it all by yourself?
My personal opinion is that Javascript is incredibly useful and overused, but that if you really need it, a framework is the way to go.
Now comes the choice of the framework. For what benchmarks are worth, you can find one at http://ejohn.org/files/142/ . It also depends on which plugins are available and what you intend to do with them. I started using jQuery because it seemed to be maintained and well featured, even though it wasn’t the fastest at that moment. I do not regret it but I didn’t test anything else since then.