HOWTO: Ubuntu VNC Encoding Server


In this howto, I’m going to run through how to install Ubuntu Server (10.10 Maverick Meerkat) and set it up as an Encoding Server, controlled via VNC and SSH.

Obviously, encoding performance will depend greatly on your hardware. For this setup, I’ll be using an Intel Celeron 2.4Ghz with 256Mb of RAM. This is on the underpowered side, and I wouldn’t really recommend setting up anything less powerful as an encoding server, seeing as this setup takes 14hours to encode a DVD. Dual and Quad core computers will be able to complete such tasks in a fraction of the time.

Decent storage would be nice as well, but that depends on what you plan to do with your server. Mine has only 100GB of storage. I’ll let that side of things up to you.

Similarly, how you get your video you wish to encode to your server is up to you. The server you’ll create in this howto will have NFS to share files with Linux boxes, setup Samba yourself if you want to share files with Windows.

Now, the howto;

Firstly, install Ubuntu Server edition. There are plenty of howtos on this, and even Youtube videos to help you.

One thing you will have to do is to setup your server with a static IP address. To do so execute, giving your password when asked,

 sudo nano /etc/network/interfaces

You’ll see a line like “iface eth1 inet dhcp”

Replace that line with the following, altering what I have to suit your network.

iface eth1 inet static





When your done, press Ctrl+X, followed by Y, then Enter to close and save the edited file. This will apply to all files you edit using Nano. (Nano for beginners here)

Startup your networking with

 sudo /etc/init.d/networking restart

When you have your base server installed, and networking set up, update your apt list and get any upgrades by performing the following;

sudo apt-get update


sudo apt-get upgrade -y

Yay, now you’re all up to date 😉 One of the most important pieces of software for this build is Handbrake, a multi-platform video encoder, a brilliant program. Unfortuntely the latest stable release 0.9.4 isn’t available for Ubuntu, due to issues with newer versions of Gnome. So we’ll use the latest nightly build.


sudo nano /etc/apt/sources.list

and at the end add the following

#Handbrake Nightly

deb maverick main
deb-src maverick main 

Install the signing key with

sudo apt-key adv --keyserver --recv-keys 816950D8 

Now, refresh your apt-list, and install the programs we need for the build with,

sudo apt-get update


sudo apt-get install -y nfs-kernel-server ssh gnome-core htop hddtemp lm-sensors devede handbrake-gtk xorg gnome-core vnc4server xinetd

htophddtemp and lm-sensors are not vital here, but are applications I include in every build. Click on their links to see what they are. If you wanted this to also act as a torrent server, you could include “transmission-gtk” (without the “s) in the above command.As these install, hddtemp will ask whether you wish it to run at boot. Press enter to select yes, and hit ok. If you want to use SSH, don’t forget to start it up with

sudo /etc/init.d/ssh restart

It’s easiest to follow the rest of this howto using ssh, so as to be able to copy and paste (using Shift+Ctrl+Ins). If running Linux, you can connect to the server using

ssh user@server-ip

substituting the correct username and IP, and giving your password.

Now that we have a graphical environment installed, you can start it up with


But for our server, no graphical environment will be running on a local screen.  Our VNC environment is completely separate to the local screen, unlike other methods that simply tap into the running session.

Setting up VNC, we need to let it create some default files. Run


It’ll ask you for a password, and once you give it one, it’ll start the VNC server. It’ll tell you what display it’s starting on, usually :1. Make note of which number it is, and if necessary, change the display numbers used bellow to your screen number. If you want to test your connection to your server, head to IP:DisplayNumber (eg, and you’ll be presented with a grey screen with a terminal, if everything’s going ok.

Now, lets stop the server and edit some config files, so that we get more than just an xterm.

vncserver -kill :1
nano .vnc/xstartup

Edit it so that it looks like this,

# Uncomment the following two lines for normal desktop:
exec /etc/X11/xinit/xinitrc

[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
#xsetroot -solid grey
#vncconfig -iconic &
#x-terminal-emulator -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &
#gnome-session &

Now, restart your vncserver with,


Now, try connecting to IP:Display ( with your VNC Viewer, and you should, after entering your password, see your Gnome desktop.
If you find the resolution too big, you can set the resolution you’d like by replacing the above command with,

vncserver -geometry WIDTHxHEIGHT

In my case, I use “vncserver -geometry 900×600” due to my small laptop screen.

Finally, all we need to do is to make Ubuntu run the command to startup the vncserver when it boots up. To do that run

sudo nano /etc/init.d/vncserver

and copy the following,

#!/bin/sh -e
# Provides:          vncserver
# Required-Start:    networking
# Default-Start:     S
# Default-Stop:      0 6


# The Username:Group that will run VNC
export USER="my-user-name"

# The display that VNC will use

# Color depth (between 8 and 32)

# The Desktop geometry to use.

# The name that the VNC Desktop will have.

OPTIONS="-name ${NAME} -depth ${DEPTH} -geometry ${GEOMETRY} :${DISPLAY}"

. /lib/lsb/init-functions

case "$1" in
log_action_begin_msg "Starting vncserver for user '${USER}' on localhost:${DISPLAY}"
su ${USER} -c "/usr/bin/vncserver ${OPTIONS}"

log_action_begin_msg "Stoping vncserver for user '${USER}' on localhost:${DISPLAY}"
su ${USER} -c "/usr/bin/vncserver -kill :${DISPLAY}"

$0 stop
$0 start

exit 0

You’ll need to change line 12 to your own user name, and the resolution you desire on line 24. You can also change the name on line 28 if you wish.

Make the script executable with

sudo chmod +x /etc/init.d/vncserver

Run the follow so create the necessary symlinks,

sudo update-rc.d vncserver defaults

Now your VNC server will run on boot. If you’re brave enough reboot with

sudo reboot

and, after a few minutes, try to connect with your VNC viewer, like before. If you’ve done everything correctly, you should see something like this,

One other important thing, for some bizarre reason, the d key defaults to closing all windows open on the desktop. Strange yes, but easily fixed. Login to your VNC server, click System>Preferences>Keyboard Shortcuts and scroll down to “Hide all normal windows and set focus to the desktop”. Click the D entry, and hit backspace to disable it.

And that’s it. I’ll do blog posts on how to use Handbrake and DeVeDe sometime in the near future, but until then, have a look here, and here.

Until next time.

EDIT: Forgot just to mention, I decided to use Mac4Lin to change the appearance of my Gnome desktop. It’s not quite as snappy over VNC as the defaults, but still quick enough.


10 Responses to “HOWTO: Ubuntu VNC Encoding Server”

  1. 1 Mark Mountford

    This exactly what I have been looking for.
    The only issue I’m having is not having permisson to run xinitrc.

    the log shows
    X: user not authorized to run the X server, aborting.

    I have tried adding my self to the root group and I can run startx from the console no problems.

    • 2 evidex

      You shouldn’t have to actually manually start up xinitrc. The startup script runs that as root, when the vncserver starts. In this setup, no graphical environment runs on the local setup, so if you connect a monitor, all you’ll be left with is a CLI login. To access your server, you’ll need to connect your VNC viewer to serverip:display number (usually 1)

  2. 3 Ariel Balter

    The “d” key! Was driving me nuts. Thanks!

    • 4 Ariel Balter

      Hmmm. Still having “d” key problem. Shortcut disabled, but not behavior.

      • 5 Stefan

        Was the same on my system. I killed the vncserver and disabled the shortcut on the normal desktop as well (I think there it’s mapped to “MOD4+D”, maybe that get’s bastardized to “D” on systems which don’t have MOD4?). Started the vncserver again and now it works fine.

        Not sure if you need to do both steps (new vncserver + change shortcut on normal desktop) but at least one of these seem to be required.

  1. 1 GIZMO» Blog Archive » My Ubuntu LAMP Server 10.10, T.E.D., Deluge, journey…
  2. 2 TightVNC to an Ubuntu desktop on EC2: the letter d doesn’t appear when typed in Terminal - Admins Goodies
  3. 3 HOWTO: Gnome CentOS VNC Server | Linux Expresso
  4. 4 HOWTO: Gnome CentOS VNC Server - Programming helper
  5. 5 TightVNC to an Ubuntu desktop on EC2: the letter d doesn't appear when typed in Terminal - Just just easy answers

%d bloggers like this: