02.02

I recently read an article written about whether Gentoo was fit for the server environment. As many of you may know, I am an avid Gentoo user myself and advocate its use whenever freedom is desired. So in response to this article, I’d like to make a point/counter-point assessment of the original author’s article.
Before I begin, let me mention that I have been using Gentoo since the 1.0 beta days. I had tried every other distribution and grew disgruntled with the package manager in every instance. Then a friend burned me a mini-cd of Gentoo, and I never looked back. The straw that broke the camel’s back was Debian insisting that I needed a mail server and a log rotator even though I knew that I did not want a mail server (sendmail was the default at the time). I was tired of RPMs not correctly updating my system, of having to build my own Slackware packages from scratch just to stay up to date, and of deb’s wanting to pull every possible application that might be used no matter how remote the possibility. I am by no means a newbie with Gentoo, I’ve done many things throughout the years that other distributions just cannot match (that doesn’t mean I haven’t had my fair share of troubles).
1. Gentoo is Time Consuming to Install
The liveCD comes with a graphical installer and the instructions can be followed from the command-line as well. I have done every install from stage1 on the terminal only (that means no X and no framebuffer). Check your documentation pall.
2. Gentoo is Even More Time Consuming to Install
This may be true. `emerge -uDp system && emerge -uDp world` is all that is needed, but if you’re running on old hardware that can barely handle Windows 2000, then it will take a long time to emerge everything (ignoring stage3 installs). Of course, you’re bringing this on yourself by compiling all of your software on a machine that can barely handle running the software, let alone compile it.
Then again, anyone with enough sense will know to install software from source while you are away from the machine. For my server and workstation at home, I do upgrades and installs at night or during the day (`at` is great to delay the emerge of something). Then again, you have less lee-way in the enterprise environment to do so.
3. Gentoo’s Stability Strategy: Update Everything
Nothing about Gentoo forces you to update, ever. Period. If you choose not to update either portage or the installed software, that is your choice. However, as a sysadmin, it would be wise to keep your eyes on profile updates as those occur for a reason. The last profile update I did was because the new Java management system that was implemented alongside gcc and binutils.
In fact, my server runs x86 and hasn’t sync’d portage in 2 or 3 months. I haven’t updated anything, but I have installed a few packages since. This is flat-out a misunderstanding of Gentoo.
“There is no ’stable’ version of Gentoo.”
This is flat out wrong and proves the author did little to no investigation on this. Check out the ACCEPT_KEYWORDS documentation to see how to run stable packages:
# Advanced Masking
# ================
# Gentoo is using a new masking system to allow for easier stability testing
# on packages. KEYWORDS are used in ebuilds to mask and unmask packages based
# on the platform they are set for. A special form has been added that
# indicates packages and revisions that are expected to work, but have not yet
# been approved for the stable set. ‘~arch’ is a superset of ‘arch’ which
# includes the unstable, in testing, packages. Users of the ‘x86′ architecture
# would add ‘~x86′ to ACCEPT_KEYWORDS to enable unstable/testing packages.
# ‘~ppc’, ‘~sparc’ are the unstable KEYWORDS for their respective platforms.
#
# Please note that this is not for development, alpha, beta, nor cvs release
# packages. “Broken” packages will not be added to testing and should not be
# requested to be added. Alternative routes are available to developers
# for experimental packages, and it is at their discretion to use them.
#
# DO NOT PUT ANYTHING BUT YOUR SPECIFIC ~ARCHITECTURE IN THE LIST.
# IF YOU ARE UNSURE OF YOUR ARCH, OR THE IMPLICATIONS, DO NOT MODIFY THIS.
#
Then there was this comment about the profiles that’s debatable:
A profile update will touch a very large number of configuration files, and it may even alter your startup process. Obviously this is not something you want to do to any server. It would be very difficult to verify that everything works as it used to afterwards, and you’d be fairly likely to end up with broken configuration files that may stop working the next time you reboot. This is in fact exactly what happened to me, despite a substantial time spent updating /etc files. The end result: the machine had to be resuscitated on-site with associated downtime.
Last I checked, dispatch.conf takes care of merging the changes you have made so that none of your configured servers are interfered with. Learn to love it because I’ve only hosed a file once and it was my own fault.
4. Gentoo’s Security Strategy: Update Everything
The author starts complaining that there is no official portaudit substitute, even though there is GLSA tools. I checked the FreeBSD ports database and found that portaudit is at version 0.5.11 at the time of writing. In fact, digging through some CVS logs, you’ll find that the Gentoo GLSA concept was first envisioned back in 2003, and glsa-check was initially checked into the gentoolkit module in March of 2004. However the FreeBSD tool only beat glsa-check by 2 months, being checked-in in January 2004. So FreeBSD can’t really claim to steal Gentoo’s thunder on that.
I particularly liked these two paragraphs for their ability to display the author’s lack of experience:
I firmly believe in updating server software only when you need to. If you don’t need new features, and things are working, why change anything? If you update anything you will doubtlessly need to update configuration files. You will need to fix things that break in the upgrade process. This is exactly what happened to me with Gentoo during its test year. I had nearly no idea of what I was updating as I ran the dreaded but most needed “emerge world” update. And once I was done I still no idea. I spent a long time working my way through updates in the /etc folder, using the built in ‘etc-update’ command. I tried to read the enormous emerge log file and take appropriate actions. And still things broke.
The best way to keep a system stable is to get it working and then not changing anything. This is hard with Gentoo. Gentoo wants you to change a lot of stuff. It wants to be bleeding edge.
First, use the -p option, which can be explained fairly well in the man page. You can also interactively emerge packages with the -a option so you can prevent certain packages from installing. This can be dangerous though, as some necessary dependencies will be skipped (if you don’t understand why that’s bad, you shouldn’t be a sysadmin). You can even mask out entire packages or even specific versions of packages.
The author never mentioned the wonder that is the overlay system, or the stage tarball concept. These features allow people to create highly customized installations that meet the exact need for everyone. Time is money, so invest some time and improve your bottom line.
At this point, it’s pretty clear this guy doesn’t maintain a large setup, probably one or two machines. In a real enterprise environment, you NEVER update software on a live server without thoroughly testing in a sandbox first. In fact, when I interned as a sysadmin, I was once told that installing anything without putting it through its paces on the sandbox machines first was a fire-able offense. This just makes sense. What you should do is bang on it left and right on the test environment. Then when everything is okay, make the same changes to an image backup on a separate drive and machine. Then make a swap and let it run on the new machine before re-imaging over the old machine. Hell, you don’t even need to dump the old drive; you could keep it in the event of system failure to fall back to a last known working system.
If that wasn’t enough, then I read this article today about this other nit-wit who claims to have given up on Linux (the OS, not the kernel) simply because StarOffice sucks (yeah, and everyone knows this) and Evolution doesn’t seem to work well with an Exchange server. News flash moron, these are closed specification systems. Without Microsoft’s assistance, open source developers are left to reverse-engineer the protocols. This leaves any proprietary format creator holding all of the cards and leaving the general public to play catch up. Bottom-line: don’t switch to Linux thinking that everything will work without a hitch from the start.
If you are a system administrator and don’t want to put in the time to manage any system, quit your job before you’re fired.
s1n,
Thanks for reading my article and for your comments. If you take a look at the comment thread of my post you will find a couple of notes about the points you are making in this post.
Both the manual and user comments expand on the ‘Gentoo wants updates’ statement in my article. Certainly there is no Godly power forcing you to update, but the manual tells you to update often, and as many have pointed out, if you don’t you will end up with an unusable version of Gentoo within a year’s time span. Perhaps you have been lucky and have been able to avoid this somehow, but I was not.
I am afraid you misunderstand what I mean by a stable version. Certainly Gentoo has stable packages. This is however different from a stable version of the base system. In fact one of the distinguishing features of Gentoo is its progressive update, as opposed to discrete major releases. This is very nice on the desktop but not entirely suitable for a server setup.
Finally, I have talked about the ‘sandbox necessary’ argument before too. Depending on your parameters, you might be able to have a sandbox. But ultimately, no matter how many layers of redundancy you have, it is preferable to have less trouble than more trouble with updates. In my case I was testing Gentoo on only one server. Others run Gentoo on 5, 10 or 15 servers and for them the experience with downtime will be rather different. In particular, it will always be limited to at most 1/5th, 1/10th and 1/15th of the impact of the downtime in my experiment. This doesn’t affect the discussion about the underlying OS though.
Either way I’m glad to hear that Gentoo is working out for you.
Well, the manual can say one thing, but it’s merely a suggestion. Technically, most distributions will recommend to update your entire system once they’ve released a new stable version. This is not limited to Linux, other companies such as Microsoft does the same. They even harass you during normal use to do so. The Gentoo developers simply made a suggestion.
You are right, the only discrete major releases are the liveCD type of releases. Those do come with stable packages, so it is technically possible to download one of the live/install CDs that is complete and install only the ‘x86′ family of packages.
If you have only one machine to act as a live server, then Gentoo is definately NOT the distribution for you if you ever plan on doing incremental updates (preferred method to update). Most companies will be able to afford a few machines to act as a server farm. In my experience (only 1 sysadmin job), selling the idea of a sandbox was always easy:
If an update becomes available that will improve the security/stability/capability of the companies servers, then it is best to have at least 1 additional machine to test all changes before making them live.
I’ve seen that sales-pitch work before.
Don’t be disheartened with your experience with Gentoo. Gentoo is a developer’s (meta) distribution that is geared around maximum flexibility and control. If that is not the goal of the machine you install it on, reconsider your distribution.
By the way, I had never heard of your blog, but I found myself reading it the other day and will probably subscribe to your feed. Thanks for the comment as well.