Close your virtual machine, navigate to x:\Program Files\Oracle\VirtualBox\. Hold Shift and right click an open area of the window. Select “Open Command Window Here”. A command prompt window will open in exactly the correct directory to do the job
Type:
VBoxManage.exe setextradata "virtualmachinename" CustomVideoMode1 1550x768x32
You can edit the name in quotes to your Windows 8 virtual machine name, as well as the custom resolution