After booting from SD or installed on to internal flash, configure your network, then:

apt-get install xserver-xorg-video-fbdev xfonts-base xinit

You should then either replace your /etc/X11/xorg.conf with the one from the 'useful' directory here, or add the line:

Driver	"fbdev"

to the "Device" section of your xorg.conf.