<html>
 <head>
  <title>PHP Test</title>
 </head>
 <body>
 <?php echo '<p>Hello World</p>'; ?>
 </body>
</html>

this works well.

ubuntu installation

root@girl:/etc/apache2# apt-get install php5

... libapache2-mod-php5 ...
there is not need to change any conf, apt-get do this for me, still you need to restart apache2