banner Debian GNU/Linux

Apt Proxy

Your Local Debian Mirror


apt-proxy

When you're experimenting with Linux, or when you're maintaining a network with lots of Linux hosts, you may be downloading the same packages over and over again, for each machine. It would make sense to keep them on your local netwswork (not wasting bandwidth, and as a fall back fgor when your internet connection is slow or down).

apt-proxy mimics the file hierarchy of 'real' debian mirror, so it can be used in sources.list, and functions as a caching proxy : it will serve the desired packages from its cache if possible. It will get packages from public mirrors of the desired packeage is not in the cache or if newer versions exist, and serve those to the clients, updating its cache along the way. So if you're maintaining multiple Debian hosts (e.g. for hands-on learning projects), you have a Linux server handy and/or considering automatic installations, apt-proxy is worth having a look at.

For a quick intro about what it is and what it does : see the apt-proxy home page

Installing apt-proxy

apt-proxy used to be in the unstable branch because of dependencies to packages that have not yet made it to testing or stable, but is now available in both stable and testing, so you can just install whatever version matches your server : apt-get install apt-proxy .

Setting up apt-proxy

This is fairly straightforward. All configuration is in /etc/apt-proxy/apt-proxy-v2.conf (for version2). There are 2 types of entries :

Network configuration and cache housekeeping :
This allows using a dedicated directory, partition or disk for the cache, specify ip address and port number the (proxy-)server should be listening on (default port: 9999), indicate how long packages should remain in cache, etc.
back-ends :
the servers where apt-proxy will download from. You can specify multiple servers, like ftp://ftp.belnet.be/debian, or http://archive.ubuntu.com/ubuntu - so all in the format protocol://server/top-level-dir. The three under the toplevel directory will be taken care of automaticaly.It is advised to use sections such as [Debian] and [Ubuntu], each with their own list of sources.

If you've moved the location of the cache, eg to /srv/cache/apt-proxy, you will have to check that the user 'apt-proxy' can write to this directory and its subdirectories. The default /var/cache/apt-proxy is owned by root:root and has drwxr-xr-x permissions.

Here's a list of Official Debian Mirrors you can use in the apt-proxy config. Alternative and additional sources are already listed in apt-proxy-v2.conf, and you can easily add your own.

That's all. Al that remains is to let your clients use your apt-proxy as apt source.

Setting up the clients :
Matching sources/list to apt-proxy sections and back-end servers

sources.list

replace the sources.lists of your clients with something like

	deb http://myproxy.dom.com:9999/debian stable main contrib
	deb http://myproxy.dom.com:9999/debian testing main

	deb http://myproxy.dom.com:9999/security stable/updates main
	

http://APTPROXY:9999/debian in sources.list refers to [debian] section in apt-proxy.conf; For sources that follow the Debian repository structure, you'd add the archive next (stable, testing, unstable for Debian, or Sarge, Etch, or Dapper for Ubuntu, and so on), then the desired components (main, contrib, non-free, restricted, multiverse, and so on). These indications are resolved automatically by apt-proxy, which will also create the corresponding file hierarchy in its cache so the whole system is transparant for the clients.

Of course, you can stil add additional sources to the client's sources.list, but that does not make much sense : they can also be added to the apt-proxy back-end lists - so you won't need to maintain sources/lists on your debian hosts. Nicely centrally managed, that is.

"unofficial" servers usually do not have this standard subdivisin. For those, you can add "./" (ie current directory) as "directory" following the URL

/etc/apt-proxy/apt-proxy-v2.conf

create sections

in sources.list, download paths take the form
deb protocol://server[:port]/path division [component [component [component [...]]]]

when using apt-proxy, path in sources.list should refer to a [section] in /etc/apt-proxy/apt-proxy-v2.conf (and apt-proxy will create a directory with that name in its cache).

That means you can have a sources list with
"http://myproxy.dom.com:9999/debian" and a apt-proxy section "debian',
but it might as well be
"http://myproxy.dom.com:9999/my_favorite_os" and a apt-proxy section "my_favorite_os" , which can serve as a debian mirror if the backend servers in that section are real debian mirrors

define backend servers

In the sections of apt-proxy-v2.conf, you list the server(s) that this section should query. So for a Debian client, whether the section is called "debian" or "my_favorite_os", the servers could be

ftp://ftp.latnet.lv/linux/debian
http://ftp.belnet.be/debian
ftp://ftp.ccc.uba.ar/pub/linux/debian/debian
(and so on)

Note that you need to define the URL down to the directory where the Release (or Packages ?) file is kept. That also goes for non-debian-like archives. They'd go in their own section(s), but again the URL is defined down to the location of Realease, or down to the package itself :

http://debian-knoppix.alioth.debian.org
ftp://ftp.freenet.de/pub/debian-openoffice
http://ftp.sh.cvut.cz/MIRRORS/OpenOffice.deb

And remember that the sources list of the clients just needs to point to proxy_server[:port]/section [repository components]

Ubuntu Dapper Drake (6.06 LTS) example

sources/list

		deb http://my_apt-proxy:9999/ubuntu dapper 		main restricted universe multiverse
		deb http://my_apt-proxy:9999/ubuntu dapper-updates 	main restricted 

		deb http://my_apt-proxy:9999/ubuntu-security dapper-security 	main restricted
	
apt-proxy-v2.conf
	
	[ubuntu]
	backends =
        	ftp://ftp.belnet.be/mirror/ubuntu.com/ubuntu
        	http://be.archive.ubuntu.com/ubuntu
		http://archive.ubuntu.com/ubuntu

	[ubuntu-security]
	backends =
		http://ftp.belnet.be/linux/ubuntu/ubuntu/
		http://security.ubuntu.com/ubuntu
	

apparently there are some issues with apt and apt-proxy in Ubuntu Dapper Drake, and they do not always work well together.

http://www.debianplanet.org/node.php?id=1319
https://launchpad.net/distros/ubuntu/+source/apt-proxy/+bug/46776

preferences and pinning

Apt-proxy doesn't do pinning : it handles requests from apt on the client system. It is apt that uses pinning to decide where a package should come from before it issues a request. There is an order of preference in the apt-proxy back-ends lists, but that is specific to the section it is under, so you still need to maintain a preferences file to have control over what gets installed when your running a mixed system (stable - testing, ubuntu - debian ...).

You can have a collection of backend servers under the same section, but they only come in to play when a server is unavailable / unreachable / ... If the first server can't be reached, the second one is queried. However, if the first backend server [of a particular section] is available but does not contain the requested package, the other backend servers [of that section] are ignored and you get a "package not found" error. Therefore, mixing mirrors (eg main + backports + 3th party) in 1 section does not always work well.

The only alternative to that, to my current knowledge, is to create a real debian mirror server and organise it so that you can make that mix. This also means you will have to maintain a Packages file and so on. Look at apt-move, apt-ftparchive, dpkg-scanpackages, ... You can also have a look at this page about creating packages and running a partial debian mirror.

and the server ...

the server itself is also an apt client, so its sources.list could also point to the apt-proxy instead of directly to external sources. It will have at least some packages (kernel, base system, libraries, ...) in common with the rest of your hosts so it could benefit from caching.


Koen Noens
December 2005