It’s not often that an emergency call after 5PM on a Friday evening leads directly to a success story. When it does happen, it’s worth reflecting to see what went right. Because as much as we can all learn from failure, there’s also a lot to be learned from success.
Recently, we took just such a call from one of our clients. The employee time tracking system had experienced a technical mishap which resulted in a ‘system down’. As anyone who has ever been responsible for a similar system like payroll or billing knows, it’s the type of outage that can lead to someone needing to find a new job.
The Challenge
Right off the bat, things weren’t looking very good.
A particular piece of the technology stack was installed by the client’s staff who had since retired. This technology that was installed by the client predated our software development phase. Also, Oracle no longer supported the technology.
To top things off, due to various technical and operational issues, there was no easy way to restore the application to it’s pre-outage state from a backup.
It didn’t look like this was going to be a quick fix, so we immediately went to work getting people in place to solve the problem. And because we understood how important the time tracking system is to our client and how fast they needed it back up and running, we skipped over a lot of the normal paperwork.
Before we could get really get started on solving the problem though, we had to do some research, and this is where we learned a lot, both about the actual problem and our strengths as a company. The first thing we did was ask our staff, “Who knows anything about this?”
While we didn’t install the web server caching software, one of our administrators had performed maintenance work on it in 2008. While the client had cycled through several different administrators since the software was installed, that database administrator (DBA) was still with us.
Paired with that, we also found that we had internal documentation concerning the client’s software. So even when the client themselves didn’t have any knowledge of the configuration, we did. Together with an LSG DBA and some additional brainstorming with technical leaders, we were able to put together a plan to get things back up and running.
Once we got the green light from the client, we went to work executing our plan that weekend. Although there were difficulties reengineering the technology stack, LSG persisted over the next few days. Concerns and doubts began to set in after the initial failed attempt; however, our persistence and confidence soon paid off. The time tracking system was back in running order without any work-arounds. We were happy, the client was happy, and everyone kept his or her job.
Lessons learned
It’s hard for many businesses to keep long-term database and software development staff. Working on the same projects all the time, these employees tend to move on to other companies, looking for new challenges.
Because our staff is challenged by a variety of projects, they tend to stay with us. That in turn, adds to our value to our clients. In this case, that continuity and tribal knowledge saved both us and the client from a potentially expensive, time-consuming fix. Something to consider before trying to hire in-house is “Can we keep this person challenged and happy? Or are we better off getting help from a vendor?”
This project was also a great reminder of the importance of documentation, something that needs to be constantly reinforced. Good documentation can save the day when things go south. Even though we didn’t install and configure the Oracle web server cache technology, the experience, confidence, and knowledge we had of the currently unsupported technology along with documentation of the client’s environment went a long way towards helping us solve the problem.
Being reminded of these things helps us stay focused on making sure our staff is happy and fulfilled, and that we do our best at retaining past client documentation via wiki technology. It helps us provide great customer support and deliver quick solutions when our clients need them the most.
Leave A Comment