Delimiter SysAdminTips

This page is a collection point for system administration tips. I've been doing sysadmin since '95.

Logfiles

• Always name logfiles ending in .log or _log. It keeps it clear what the file is even if moved out of /var/log/.
• Use logrotate's dateext option to ensure logs get rotated using a sane naming scheme (YYYYMMDD)
• Compress logs as they are archived (compress), or a day later (delaycompress)
• You can get better compression using bzip2 (see compresscmd and compressext)
• Use the copytruncate option when it's unclear how to restart the process writing to a log (hack).

Cron jobs

• I've found the best cronjobs are the ones that notify you on problems. The trick is to capture stdout to a logfile but leave stderr alone. This way, you'll get e-mail if something is wrong, but otherwise can safely assume life is good.

Example: echo "nothing is wrong" >> /var/log/cronjob.log #This will be quiet unless something goes wrong i.e. the exit value is not 0 in which case it will pipe stderr into an e-mail to $MAILTO which defaults to the user running the cronjob.

DNS

• Use DNS, /etc/hosts does not scale and will quickly become a nightmare.
• Keep your hostnames short as possible (so they're easy to type) and abstracted (see service-name abstraction)
  • In other words, if you build a host that will be a nameserver, don't assign it a hostname of ns1 because ns1 is a service-name
• Use a consistent naming scheme

Inventory/Asset Tracking

• Maintain a system catalog once you get beyond managing a handful of systems
  • Categorize into Server, Switch, Router, Desktop, Laptop, Printer and so on
  • RT Asset Tracker works nice for this if you add the custom fields for RAM, CPU, Make, Model and so on
  • A spreadsheet will work too on a smaller scale

Authentication & Authorization

• Use LDAP to centralize system accounts, mailboxes and so forth. Utilize phpLDAPadmin or gq to manage this data.

Monitoring

• Use monit but judiciously & in moderation
• Monitor using combination of nagios (alerting) & cacti (trending) or something the integrates the two like zabbix or opennms.

Do & Don't

• Don't assume everyone is as smart as you. Explain things politely and respectfully and they will reply in kind.