Sanford Whitehouse - Floating By

17 June 2008

Why standards?

Posted by sanford under: Log Standards .

I’ve often wondered about the viability of broad vendor adoption of a log standard. There are missing pieces. An organic aspect of adoption that isn’t in the make up of logging. There’s hardly a person dealing with systems and software that hasn’t imagined how logs could be used, or wished the logs they’re dealing with were useful. What’s missing?

I won’t go into the behavioral profiles relevant to selling security. Security is out. Business needs win, anyway. If it doesn’t help make money, why spend money on it?

Compliance? Most governmental regulations are so many characters on a page. The rest are so many sentences. None seem to state that a product vendor must log in a useable form. Many companies and auditors have found ways to make them work without reliance on standards.

Industry mandates? We’re getting closer. Still, they press what a user of their service must do to be up to speed. Nothing about log standards, vendor responsibility, etc.

Maybe general policy enforcment. We’ve lived a long time with “thou shalt and thou shan’t” followed by “and we better not catch you.” Doesn’t seem to be a big winner.

Operational monitoring? A lot of companies working on that. Plenty of products that will help watch the egg basket, as long as the basket is bought from them or a close partner. No push there. They have the solution. Just ask.

Administrators? Sure. Log standards will help them to pull off miracles at 3:00 AM in their 32nd hour, sitting in a dark closet, with no documentation, the tin support plan, and are either doing it from 12 timezones away or are afraid they’ll have to move that far to keep their job. Wake them up first before asking if a log standard would be good.

The vendors creating the logs? Is that one worth discussion? The folks that sell reporting options based on their log formats? That use messages as inter-system communications and monitoring, or make 20% of revenue from support plans?

On top of all this is a sense log standards are too big of a problem. Too many pieces, to many complexities. If one existed today, it would take years, perhaps a decade or so to come up to a broad enough implementation to mean anything. That’s too long. I’m taking vacation in a few months. Can I have something before then?

Let me know if anyone was missed.

So … why? There’s a practical reason, and a why-can’t-everyone-see-it-reason.

Compliance and industry mandates have amplifed a scenario played out in companies big and small. They have placed the company computing based business infrastructure in one person’s lap. Traditional attempts to maintain policy had to span administrative, business, and product domains. Each executive responsible for such an area was directly responsible for understanding and implementation, and communicating policy, risk, and violations. Today those areas belong to a company officer.

This position doesn’t particularly care who made the firewall, or the brand of database. The state of the company is inherently product agnostic. He will ask one person the risk of failing the next audit. That person, or group, cannot afford to worry about how this OS or that OS reports logins. There’s too much disparity and too much information. Bottom line, only. This will never be achieved without standards.

From the company view, a standard is assurance compliance management can reach a reasonable level. Time, effort, money, fines, and bad quarterly reports can be saved or avoided. How well a product can integrate into the compliance/policy infrastrcuture will become part of the purchase decision matrix. If product does not integrate, and the company must purchase equally costly professional services to integrate, how far down the list of pros and cons will the buyer go? Eventually, product vendors must comply or risk market loss.

Now for the fun side.

A favorite analogy for logs is aluminum.   Aluminum is the most abundant metal in the Earth’s crust. It’s the third most common element, after oxygen and silicon. Like aluminum, logs are everywhere. It can be argued that even the smallest change to data or a system can create a message many times the size of the data. Give an appropriate level of silliness one can generate more logs about an event then the sum of elements of the event. Even to the point of bringing a system to the ground. Logs can be common and abundant.

Bring the silliness down a bit.

At one time aluminum was a precious metal, more valuable than silver, or even gold. The value came from the difficulty to extract it. At the end of the 19th century processes were developed that drastically lowered the cost. Ask an aluminum vendor and you will receive a list of uses that add up to how the world as we know it would not exist without that metal.

A cost effective method for extraction of the data contained in logs, implicit and explicit, would enable any number of capabilities in a company. Take your pick. Instead of 30 different console applications for a security product infrastructure, there can be one primary console that works at the event level. Cross domain and product activities can be tracked. IT and similar costs could be justified. The person given company responsibility for compliance/policy conformance could work within that domain, and not the domains of a hundred tribes.

There are dozens of others. If you’ve read this far, my guess is it wouldn’t take long to create a list of issues that might be solved, or capabilities that might be useful. People who take advantage of the normalization and APIs available in some log management products are working these areas, today. If you’re someone more into research and analysis, there is a rich set of behavioral information that today is guessed at.

Standardization is one of the steps to make this happen. Standards must address presentation, content, classification, the broad range of products and product types, and vendor logging efficiency, ease of transition, and maintainability. It must consider or include transport, use by legacy systems, community support, multiple involved log consumer types, perspective structure, some level of future growth.

Standards will move forward if customers press vendors and organizations to develop them. While it may not look like it, product vendors and vendors that use logs are taking interest. This is not a time to watch what happens and accept what comes.

2 Comments so far...