Drupal isn't known for being the platform around. In fact it's got a bit of a reputation as a resource hog and many developers and site owners struggle to make Drupal performant. Plenty of options exists to aid in speeding up a Drupal site but most are are band-aids that provide limited usefulness or are fairly extensive processes that require significant development or sysadmin chops.
Let's take a quick look at two easy to implement solutions that will not only improve the performance of your site, but will also legitimately improve the scalability of your server and allow it to handle much larger volumes of traffic.
PHP Opcode Cache
Before your webserver can do anything with a PHP script, it must first load the source file and interpret the PHP into machine language. Under general circumstances this may not seem like a big deal, but if you're serving hundreds of requests each second and the code required to serve each request is spread across hundreds of files--looking at you Drupal--the resources required to perform these operations begins to add up. The smart thing to do here is so use an opcode cache.
An opcode cache lets the server load and interpret the PHP source code and caches the resulting machine code. Each subsequent request then uses the machine code from the cache. APC is the most widely-used opcode cache and a popular alternative is eAccelerator, both are extensions installed to PHP. Also, PHP 5.5 comes with a built-in opcode cache ready to run.
Alternative Cache Backend
A wide variety of data is computed and cached by Drupal; variables, structured data, rendered chunks of markup, full HTML pages, you name it. One way to keep Drupal running quickly is to compute this information one time and save it, then keep using the same computed data each time after. This works well and is all part of Drupal's core system, but by default it's stored in the database.
What you'll end up seeing is that many of your database queries are this database caching system looking up and saving data, and as your. Instead of saving the cached data in the database, Drupal can be configured to save it somewhere else which frees-up resources from the database as well as performing the cache look-ups and saving much quicker. The most widely-used mechanism in the Drupal community seems to be Memcached, which has been incredibly easy to configure in my experience. There are also folks in the community successfully utilizing Redis in the same manner.
One more bit, the functionality of being able to swap Drupal's cache backend is not available prior to Drupal 7. For Drupal 6 sites there is project called Pressflow that is a drop-in replacement for Drupal core that allows you to utilize an alternative cache backend. There is also the Cache Backport project which allows you to use pieces of D7's functionality. Pressflow has been around for a while and has a fairly large user base while Cache Backport's usage numbers suggest it's not widely used. I've used neither.
Considerations
I recommend using an opcode cache and an alternative cache backend for nearly every site, and generally they can be used without significant changes to your environment, but there are a few things to keep in mind. The primary concern is exhausting RAM the server's RAM. Both the opcode cache as well as Memcache hold their cache in RAM. A Drupal site generally consumes a fair amount of RAM to serve a response so allocating too much RAM to these systems can choke off precious resources from Drupal. If you're going to use these be sure to fine-tune the configurations and if necessary look into a more-robust topology utilizing multiple servers.
Final Thoughts
To me, it's a no-brainier. If you've got a Drupal site you should be utilizing both an opcode cache and a alternative cache backend. APC and Memcached are a part of my build spec and I recommend them to everyone. Even if you don't think your site needs it, it's more than worthwhile to spend a few minutes to install and configure these upfront as both have a low initial commitment and provide a real increase to the performance of your server and the Drupal site it hosts.