Syslog: an archaic concept with real utility

The basic syslog daemon included in most modern UNIX like operating systems (Linux, *BSD, to some extent Sun Solaris) is quite simple and is designed to get important information from running processes. Even the most basic implementations of syslogd now allow the system administrator to fine tune the system to the extent of which logs or log levels end up in which log file.

The various implementations of the stock syslogd was always designed to be internetworkable and log messages were transferred via UDP on port 514.

There are modernized alternative syslog daemons to choose from including (but not limited to) the excellent syslog-ng which has a number of quite essential improvements that may not be critical for hosts sending their logs but are crucial for the log server itself. That said, if you can run syslog-ng on your hosts too there are certain benefits to this configuration. The most important one is that it allows transmission of log data via TCP. This of course means that you can wrap it around a SSLized tunnel, not to mention the obvious benefits of using TCP – especially where the log host and log server are geographically dispersed and unfortunately the slightest connection delay or issue will cause potentially critical log information to be lost.

Modernized syslogd alternatives like syslog-ng allow for the log to be kept in an alternative format to a flat file and instead in a SQL database. This makes a lot of sense especially on large networks where log volume can be so large.

Many well meaning system administrators  log to a centralized server and keep this in a database as a way of coping with the deluge of log information (I should note that disabling debugging and otherwise useless log entries goes a hell of a long way to making sense of it all), however this results in unencrypted UDP (and with alternatives like syslog-ng unencrypted TCP) packets flying around your network and you may be surprised what they contain.

If you have a spare fifteen minutes find a spare switch port and configure it as a “monitor” port (most managed switches will let you define a port as a monitor or mirror port and will mirror all packets from the switch into your port. Obviously this single port may exceed the throughput of all the switch users and you may get some dropping) and plug a network analyzer into it. Back in the days of hubs this kind of eavesdropping was easy as every port saw the traffic of the other ports. In fact this was often used as a metric to improve performance and reduce collisions.

If you don’t have a network analyzer grab yourself a laptop and acquire the excellent wireshark (formerly known as ethereal) which does a darn good job at converting a cheap PC into something that can replace a piece of kit that costs thousands. Refine your search to UDP/514 and just sit and wait. If you are in a typical large network you will see all sorts of goodies being logged, from imapd reporting logins of users to a radius daemon reporting successful authentication but includes the username, challenge and response instead. You could potentially build a nice bit of intelligence from log data alone.

Perhaps the crux of this article is in this single paragraph. If you have any sort of operation where you have a heap of systems then logging to an external host is a great idea, but consider the security implications and ensure that the logging traffic is tunnelled or otherwise corralled to prevent trivial recovery of log data.

If you use an IDS, then you have likely established that you are a “target”. Whatever the reason these IDS systems can push their findings via syslog to a remote host. What is important is that the integrity of your logs are maintained. I recall a survey of self confessed script kiddies and almost all of them said that machines that were in an almost default state and only logging locally were an easy target – what’s better, they can rootkit the machine and sanitize the logs so it appears that they were never there. When these folks discover external logging on a host they have obtained a root shell on they direct their focus on the log server and generally the log server is the same OS and the exploit that gained then access to the first host was just effective there.

The earliest machines had paper log printers. While incredibly wasteful any tampering with the log would be obvious.

At the very minimum acquire some 1GB WORM SD cards on eBay or Amazon. These were made by Sandisk at the request of a police department in Asia who was concerned about photographic evidence from cameras taken to the crime scene being altered. Unfortunately the program appears to have been unsuccessful and these SD cards are popping up everywhere for sale. How you use the SD is entirely up to you. Obviously you could just start writing to the block device without even using a filesystem and just continue appending until you ran out of space. By queuing two SD cards in the host an email could be sent to the admin notifying him the card is full while the secondary card is now used until full when hopefully the administrator has placed a blank device in the slot. The advantage of this method is speed – the data is very quickly appended to the WORM device and should an exploit be found that allowed access to the log server then there would be nothing they could do remotely to sanitize the logs.

Instead of this approach others may wish to put a basic filesystem on the media (such as FAT32) and instead export log data at an interval they are comfortable, compress it and then add it to the WORM. While you could argue that this is much more economical than the former suggestion, remember that there would be nothing stopping them from piping their data through gzip or a similar archiver before committing it to the volume.

I have spoken in detail on about my solution for key storage and this actual real world implemented solution would also be ideal for logging. My device uses an AVR with two FIFOs to implement a crude RS232 data diode. By building something similar so that data may only pass through the serial port. We implemented a proxy on each side in the AVR so that data could be resent with out the host seeing the other host. Imagine a hardened down BSD machine running as a log “aggregator” that collects logs, builds the database (that can be replicated elsewhere). Logs of a level above a defined threshold are shot down the serial port to a second machine that is isolated and kept offline deliberately. The machine would only have utility after an attack where log modification is suspected.

Indeed a much cheaper way of doing what is described above involves setting up a host that has the sole purpose of archiving log data sent to it. By simply cutting the TX lines of the Ethernet cable you can make a primitive data diode. The host you connect it to, let’s pretend it is the second network interface on your main log server, will not negotiate ARP or anything fancy due to the lack of a response from the host with the cut TX lines. The easiest way to get your data into the quarantined box is to send it as a broadcast. On the box that has been quarantined you could use a number of methods to capture your data but the easiest way is to run tcpdump on the interface and pipe its results into a FIFO which is where your script listens, converts the received broadcast frames back into log file entries and archives them in an internal database. Some basic watchdog script would ensure everything continued to run. Given this machine doesn’t have access to the network by design it would be unable to tell us of its performance. Perhaps my RS232 data diode could be put to use, pushing out only statistics that could be graphed with cacti or nagios and information on its internal state (HDD SMART status, etc) into perhaps a Raspberry Pi or a similar cheap tiny embedded board which can then contain code to listen to the serial input, process it and then shoot it out as SNMP.

Anyway I have not slept in over thirty hours as a consequence of a long flight and the insomnia that followed, so please excuse me if this article isn’t in my normal style and is perhaps too verbose and wordy.

Ultimately remote logging is a very worthwhile feature. Ensure that any machines that are spread about geographically are not just sending UDP log packets across the notoriously insecure Internet. Even if you have a VPN between locations or you have a single office you should consider moving to a syslogd that allows sending with TCP and is configurable enough to allow you to use stunnel to encapsulate your connections. The overhead on modern hardware is negligible.

So, perhaps it’s time to start taking syslog seriously. There is even a piece of Windows software that pegged the Windows Event Log and send updates using syslog. Something like this would be ideal for a shop with many *NIX machines and only a few Windows boxes. I recall the software had a functioning SNMP client they claimed was superior to the vendor offering. “Why” you may ask.

“Because it actually works and doesn’t crash on its first trap”



8 thoughts on “Syslog: an archaic concept with real utility

  1. Good article. I want to implement a similar setup when I finally get out of the “nest”…My little blog I set up (hopefully doesn’t get attacked and deleted) is purely tutorials and projects, trying to make it easy for people.

    Maybe you should stay up for 30 hours more often. :p

    • Hey Mr. Out,

      Thanks. I try and keep things fresh and original. It is kind of important to me to keep the blog going as it will eventually be a place for me to release my own work (which is ongoing at present; hell, I can’t even talk about one particular multi colored word product’s surveillance program as I will breach an injunction) in due time.

      By the way I had to manually approve your post again as you used a different email address. Can you please stick with one? It doesn’t bother me but it delays your posts for moderation.

  2. Alright sorry had a couple typos, picture checks out atleast. Yeah same here, I don’t care about the comments, more of documenting my projects for employers and making the info easy to find. Nice always interested. I’m working on restoring an old computer.

    Oh hey, random question. Have you ever had a hard drive that wouldn’t read? I have a harddrive that I want to see what’s on it and initial searching looked like it would be kind of hard to do; maybe too expensive and not worth it?

      • Nope it’s an external one too, it won’t even read on Ubuntu and Windoze gets an “I/O error”. It’s an older Seagate 60GB Barracuda. So my suspicion is the USB-SATA controller board (MS-ME198407) but I can’t find the data sheet or much of any talk about it online; I could buy one but I want to be sure that’s the problem before I buy. I suppose I “could” try to figure out the circuit but that reverse-engineering is a bit beyond my skill set. Bet your friend w/ that cold-boot tool could get in lol. I’ll have to look more into it. In the meantime I’m trying to rescue another computer that a family member got a virus on with “Rescatux”. Luckily not all the I/O was destroyed so I at least got in w/ Puppy Linux last night. Nasty virus won’t let me mount and wipe this crude.

        Anyways just asking. You need a “squid thread” for off topic stuff lol. But I definitely want to set up some sort of network analyser/guard for my internet; I think the connection got pwnded almost instantaneously through the modems we got. Thinking that’s definitely going to be a use for the RaspPi or Beaglebone.

      • Well first thing I would do is break open the caddy to get at the HDD inside and then plug the HDD in via SATA (or PATA given its age is more likely) to your PC. Only then can you be sure it isn’t the USB caddy that is causing issues

      • The damn problem is we don’t have trust in anything anymore. I am not sure where this will lead us – I can’t imagine that this big Pandora’s box that the NSA et al opened up will just cease to be an issue in due course. We need to somehow manage to have truly trustworthy computing… I am not sure exactly how this can be implemented or even if it can be done at this point in time.

  3. Yeah I see where I’d have to remove the board and insert the S/PATA strips, it’s screwed in. Luckily I have some as they’re becoming obsolete, but I’ll probably need some sort of converter.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s