“The concept of a ‘University for Industry’ led to the creation of Ufi in 1998. The organisation then set-up learndirect, a nationally recognised brand for learning. In six years learndirect has become the largest e-learning network of its kind in the world, and has individualised the delivery of learning to a mass audience through a unique combination of flexibility, accessibility and support.”

In this piece I plan to talk a bit about our e-learning platform and the part that open source tools and systems have played in our success.

The learndirect learner management system (LMS) like most learning management systems is more than a website with lots of content.

Content sites like the BBC or CNN while they have some personalisation, typically present their consumers with a collection of web pages. If they are personalised at all they present their consumers with a sub-set of content according to preferences or tracked activity. Critically, the content itself does not change from consumer to consumer and as a result can be load-balanced across a number of serves or caches and requires relatively little tracking.

Learner management systems such as the learndirect system track a learner’s progress through a piece of learning and adapt in response to on-programme formative assessment. Such systems do expect to modify content according to consumer behaviour and as a result the use of multiple content servers only works to an extent. Such systems require a single authoritative data source for each course.

Additionally, consumers visiting a news or similar site have plenty of choice. If the BBC site is slow or not there for whatever reason, there are plenty of other such sites for a consumer to visit.

With web delivered learning, the consumer is intending to engage in a formal learning activity that they have formally enrolled in and in many cases have traveled to one of our learning centres to take their course. There is no other site for them to go to. If the site is slow or closed, then their journey was a waste of time.

For this reason the system must be both available and perform well. It is not enough that a system is available and returns content. If e-learning is to be effective, the medium needs to be as un-intrusive as possible; content has to render without the consumer becoming aware of any wait.

This presents us with a double bind; each user’s content is customised and there is a service expectation of 100% availability and responsiveness. In addition, we have issues of large scale and 24 x 7 availability we can see that constructing such a service is a serious web engineering exercise.

If you are not monitoring the service, then you are just running software.

It’s never good when the first person to tell you that your service has a problem is one of your consumers. Without appropriate monitoring software this will inevitably be the case, and in all probability they won’t tell you immediately.

So, the first key differentiator between a service and a system is Monitoring.

When our service was first constructed a very expensive piece of software was purchased to perform availability monitoring, however, Mr. Heisenberg was forgotten and the load associated with that particular tool was sufficient to detrimentally impact the system. The tool itself was sold as the usual universal panacea, however, in implementation it was clear that its forte was component monitoring and not service monitoring.

Running a live system with this tool gave us all sorts of problems. The tool required agents on all machines and was really only designed around component availability and even then this was often measured from the wrong place (inside the firewall).

We took a look at the open source offerings available at that time and selected two.

Nagios has won lots of awards. We use it to monitor events from two locations.

Nagios is used to provide event monitoring. Implementing such a tool is not to be undertaken lightly. Getting the sensitivity correct so as not to cry wolf, and embedding the culture such that when an alert is sent out, the operational staff respond rapidly is, in my opinion, more difficult than installing the system in the first place.

The second open source monitoring tool we use provides trend monitoring, After looking around we found Cacti.

While Nagios tells us when we have a specific issue/problem, Cacti provides us with the information to understand or diagnose the root cause. In measuring volumes and their trends, Cacti allows us to look across the whole application stack at any point in time and examine critical volumes.

Cacti is used to measure volumes. If a system can return a number, Cacti can capture, store and trend it. These volumes can be business or technical volumes examples of which might include the number of users logged into the system over time or critical system volumes such as bandwidth, disk space, CPU, or Memory usage.

When you want to compare historical volumes or activity at a particular moment in time, Cacti can provide it.

As you might expect from an open source tool set, both of these tools are highly extensible. We have been able to write and adapt agents to interface with them, with the exception of our database monitoring, and we have been able to monitor and trend all our services.

I spoke above about getting the culture right, putting these critical volumes onto big flat screens, making them obvious to everyone in your operations and service team. This was the single most important cultural change we made next to implementing an ITIL service culture.

The real question here is how we’ve been allowed to put all this instrumentation all over our application. Most government contracts are outsourced, but we chose to in-source our operations and development teams.

To understand this we need to talk about technology in a business context. Most organisations have either an implicit or explicit technology strategy. Within our organisation our Technology Strategy provides us with a framework that allows an organisation to make ‘good’, strategic choices, i.e. Hardware, software, monitoring systems, hosting providers. These choices are deployed within a governance framework to ensure that the business and service models that are dependant on technology can be delivered now and the future.

At the risk of stating the obvious, the selection of technology and service model an organisation chooses can mean the difference between a successful business and one that fails. As a consequence, organisations and IT directors tend to be conservative in there decision making.

At a simplistic level, technology is used for three things within an organisation:

Unless you are a start-up, the bulk of investment and cost is already sunk in running your company. Changing the company IS usually occurs incrementally and takes the form of modifying the status quo. We are left with the shinny innovation tip of the cost iceberg to introduce new ways of doing things.

If we accept some of the above, we can see that technology strategies have considerable inertia, and unless there are some strong external pressures (failure to meet Service levels, company financial pressure, loss of market share), the adoption of new technologies is going to be slow. There is still a lot of COBOL out there!

So if you already don’t have a lot of open source in use, introducing it requires overcoming quite a lot of inertia.

As a company we have mandated the use of specific open-source operating systems and applications within our technology strategy where we can see cost and risk reduction. It’s worth saying that if our service was totally outsourced then these would not be our choices to make, other than at contracting and its very dangerous form to tell a supplier both what you want and how to do it.

The last ten years has seen the trend to out-source IT services and development continue to increase. This should not be a surprise when we consider the risk and cost of getting it wrong. Out-source companies come with the allure of having solved all problems previously and having a large pool of experienced staff and many organisations have significantly reduced the cost and risk of running their IT systems as a result.

Central to a successful out-source contract is a contract and a service description and underpinning set of requirements that are well defined. Good example candidates for out-sourcing are Payroll or Desktop management. In both cases, an organisation can describe what it is that it wants and the amount of change required going into the future can be estimated accurately.

It’s in the nature of our-sourced contracts that you describe to the supplier what you want but refrain from telling them how to do it.

If your IT application is the core of what your organisation does (such as the learndirect LMS) and you know you are going to undergo an annual cycle of change then in-sourcing your operations should be considered.

Having in-sourced the learndirect operations, we have seen a significant reduction in cost and have increased our service availability to > 99.9%

If you have in-sourced your application development or hosting then you have the opportunity to exploit open source tools and applications for competitive or service advantage (are they the same thing?)

Having in-sourced the operations and now the development of our core application, we have put open source technology at the core of our technology strategy.

While we retain Oracle as our database of choice we have adopted a wide range of open source tools, Apache, SQUID, JBOSS, Hybernate, MySQL, Linux, to name but a few.

The advantages are obvious:

7 Responses to “Running a Service Not a System”