WordPress is a popular blogging software. As a design decision it stores the blog content in database and uses php to fetch and display the pages. This consumes more resources (cpu and memory) than blogging softwares generating static pages. Here are 5 tried and tested power tips to reduce memory & cpu resources and speed up your wordpress based sites.

Speed up PHP code execution.
There are several php code accelerators available. I recommend eAccelerator based on performance and stability. Use it with at least 128MB RAM ( eaccelerator.shm_size = “128″ ). You will observe significant performance benefit from this. Note the average cpu utilization before and after.

Optimize MySQL server installation
MySQL on Linux get installed by default with minimal configuration which is suitable for low end machine. It doesn't make use of the memory of your system you may have. There are several online guide to configure mysql for better memory utilization and overall optimum performance. I used the following settings:
key_buffer = 256M
max_allowed_packet = 1M
table_cache = 256
sort_buffer_size = 1M
read_buffer_size = 1M
read_rnd_buffer_size = 4M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size= 16M

Read the fine manual for explanations on each of these settings :)

You may also want to enable slow query log and identify the slow queries. Progressively reduce the time as you better optimize the queries.

Slim & trim WordPress.
WordPress is supported by tons of free and paid plugins. However many of them are not optimized. The slow query log step above should help you to identify the slow queries. Search the plugin source code to identify the plugins responsible. Work, politely, with the authors to resolve the issue. If that doesn't work, look for alternatives or disable the plugin. Plugins should be used in moderation. Prefer plugins for functionality rather than cool / wow factor. Many plugins imposes a significant cost in terms of resource and performance.

Staticize your pages.
wp-cache 2 is a WordPress plugin which will cache your page output in a file (download). For future requests of the same page it serves the content directly from the file thereby bypassing the need for database queires and further php code execution. It recognizes when any page content has changed. Unfortunately it then invalidates the whole cache instead of the page itself and other pages which are likely to change.

It will also invalidate the whole cache it a comment is added to any post. This is a problem for highly commented blogs. I had posted a solution to this problem: How To Improve Performance of Highly Commented WordPress Blogs.

At some point you still will have to move to a better server - from shared hosting to VPS (virtual private server) and then from VPS to dedicated server(s). And then you may have to use a caching reverse proxy server for load balancing between multiple apache servers. Trust me you will want to face this type of challenges for now in you are in the big league :)