It’s upon us!! The long awaited much anticipated FIFA 2010 World Cup is here. Everyone’s focus has moved away from the stresses and pressures of making a living … to a little football being kicked about by 22 men on a grassy field. For the next month we will join billions of people around the world in thinking about little more than the 64 games, the 32 teams and the events and incidents that happen on and off the field.
Faced with a choice of either putting “The Software Engineer” blog into hibernation until ATWC (= After the World Cup), or finding a way to join the fun – I’ve opted for the latter. Over the next 6 weeks I will blog as often as I can. I plan to participate in the festival of football wearing my software engineering hat and my Bafana Bafana shirt. How will this work? Will I have anything interesting to write about? Who knows? It’s an experiment with an unknown outcome.
All I ask is for anyone reading this to join in the spirit of the only 2010 World Cup blog with a software engineering flavour! Comment, share your views and support your team – whichever it is – with passion and fun! May the best team win!
Ninety thousand “new” World Cup ticket went on sale at 9:00am on Friday 28th May at FIFA ticketing centres around South Africa. Such is the passion and excitement around the upcoming World Cup that hundreds of people started forming queues on Thursday morning at the ticketing centres. There was a festive atmosphere as supporters braved winter weather spending the night in the queue as they waited for the centres to open.
By 9:30am the mood had changed to one of anger …. the IT systems supporting the sale of tickets had “crashed”. By late on Friday morning police had to be called to bring things under control. By 6pm on Friday many people, after queuing for nearly two days, had walked away with no tickets. FIFA Secretary General, Jérôme Valcke, apologised for the inconvenience. He blamed “technical problems”, promised to get a detailed report from FIFA’s “service provider”, and then went on to his next meeting.
Frankly I think that this is simply not acceptable!!
Let’s unpack what happened. More than 2 months ago FIFA announced that nearly half a million unsold tickets for the World Cup would be sold “over the counter”. On 15th April the first batch of these tickets went on sale at FIFA ticketing centres, some selected branches of Shoprite Checkers supermarkets and at branches of FNB. It was a disaster! The system “crashed” and – as we saw last Friday – people who had queued for hours had to endure frustration and chaos. Jérôme Valcke promised to sort things out.
How is it possible then that 6 weeks later the same thing happened? What are these “technical problems” and why weren’t they solved?
I’m willing to concede that the 15th April crash may have taken FIFA’s IT service provider by surprise. It was never envisaged that “over the counter” ticket sales would be provided for. This represented a new requirement and the system would have needed to be changed. In implementing these changes, any number of assumptions could have been incorrect. Server hardware, communications channels or transactional database systems may have been inadequately sized to cope with the volumes of traffic. There also is evidence that the systems were not tested properly.
But what about the “crash” on May 28th? I would expect that information collected from the 15th April would have been analysed and appropriate changes made to the system. There would also have been enough knowledge available to set up valid test cases. In software engineering terms I cannot accept that the system “crashed” on May 28th .
I have two theories that may explain this recent “crash”:
Theory 1 :FIFA’s IT Service Provider is incompetent
Who is this Service Provider? My answer to this question is based purely on a few hours of surfing the web. If I have my facts wrong maybe someone can correct me.
The first name that comes up is “MATCH Services AG”. It is a company based in Zurich Switzerland and is run by the FIFA President’s son-in-law. It provides FIFA with “ticketing, accommodation and event information technology services” [http://www.match-ag.com] . MATCH has a strategic partnership with the professional services company “Eurotech Global Sports AG”, also based in Switzerland.
It seems to me that Eurotech Global Sports [http://www.eurotech-gs.com] is the “Service Provider” ultimately responsible for the IT system that crashed on 15th April and again on 28th May. On the Eurotech GS website one reads that it’s “staff has event experience in 15 countries in the past 11 years”, and that it is “dedicated to providing high value-added planning, project management and operations support for the implementation and deployment of IT services in large-scale events.” We are told that Eurotech GS “specializes in assisting … international sports federations [amongst others] … with unique IT requirements”.
I’ve even been able to put some names to the key IT experts on the Eurotech Team. We have Jason Anderson [responsible for development and quality assurance when he worked for EDS], Tim Cromie [quality assurance and applications manager during the 2006 World Cup in Germany], Paul Leckie [ a “proven track record of success in delivering IT solutions in complex, high visibility, high risk environments”] and Dave Schoonaert [ with 13 years of “designing, developing and implementing scalable, flexible, and reusable systems”.]
So – do Jason, Tim, Paul and Dave have the experience and competence to spec, build and test a system that can cope with “over the counter ticket sales”? Based on the information I’ve found on each of them, I’m confident that they would be able to deal with the technical challenges quite easily.
I therefore don’t subscribe to Theory 1.
Theory 2: FIFA, or MATCH, are not willing to pay the cost of solving the problems
Assuming that there is a technical solution to the problem that caused the “crash” of 15th April, and assuming that the Eurotech GS Team of Jason, Tim, Paul and Dave, were able to design a fix to the problem, why wasn’t it implemented in time for the 28th May release of 90,000 additional tickets?
My theory is that the solution came with a price tag, and MATCH and/or FIFA looked at the price and decided it wasn’t worth fixing.
Why should they make such a decision? Well, firstly the tickets have still been sold. The hype and negative publicity surrounding some poor unfortunate soccer fanatics coming to blows at ticketing centres certainly hit the headlines around the world. The net result was that people went on-line and bought tickets.
Secondly, do FIFA really care about people queuing for tickets in South Africa? There is ample evidence that we South Africans are not all that important in FIFA’s global vision. Jérôme Valcke’s apologies and promises to sort things out cost FIFA nothing. Investing in technical solutions to solve the problems with the IT systems comes off someone’s bottom line.
“You must be crazy” I can hear some FIFA executive saying to the Eurotech GS team, “why should we spend a million dollars on a fix to the IT systems. When fights break out at the ticketing centres we’ll just call the cops and then roll out Jérôme Valcke to look concerned and apologetic. That’s a much more cost-effective solution.”
Am I being unfair to FIFA and MATCH. From an ICT point of view what do you think? Whatever it is, having a highly visible system crash so dramatically is bad news for software engineers and ICT professionals anywhere in the world.
About 2 weeks ago I asked readers of the blog if they knew of any examples of AMAZING stuff that the South African ICT Sector would be show-casing at the upcoming FIFA World Cup.
I’ve just found a great 2010 ICT story: Yesterday the National Minister of Science and Technology, Naledi Pandor, launched a new information security system, based on “quantum crytography”. The Quantum Security system, developed by researchers at the University of Zwazulu Natal (UKZN), will be used to secure communication between the Moses Mabhida Stadium (Durban’s FIFA World Cup venue) and the Joint Operations Centre [ for details click here ]
In 2006 the 2010 FIFA World Cup was more than 1,500 days away. I set my 4th year Engineering students a challenge for their final year project. “Imagine hundreds of thousands of football fans visiting our country. While in South Africa, enjoying the World Cup, they find some amazing, innovative, exciting ICT applications. They see things being done in software and hardware that really impresses them.” I asked my students to develop a proof-of-concept prototype of some such innovative application.
Two of my students rose to the challenge. They developed a concept where players on the field wore minature cameras. Spectators in the stands could use their cell phones to select a player of their choice, and then watch the play from the viewpoint of that player’s wearable camera. The students had great fun developing a relatively simple prototype that demonstrated their concept.
At the end of 2006 the students left Wits and there was no-one with the resources or time to develop their concept further. Quite a pity, now that I think about it!!
With the World Cup now a few weeks away: What will the South African ICT sector show that will thrill, amaze … or even vaguely interest … the foreign fans who are about to visit us. Will any of them say, when they get back home “I saw this really amazing ICT application that was developed by a South African company”.
Has the local ICT sector missed a chance to bathe in the spotlight that is about to fall on us? Maybe I just don’t know! Maybe there are amazing applications that I’ve just missed. Is anyone able to enlighten me?
I’ve been writing computer programs for close to 40 years. I wrote my first program – in Fortran IV – in 1971. I wrote my most recent programs at the end of last year – in C++. In the years between I’ve written many, many programs. Some large, some small, some serious, some just for fun. I love programming!
But am I a ‘good’ programmer? I guess there are two ways to think about this question: “Do I write ‘good’ programs?” or “Am I ‘good’ at writing programs?”
“Do I write ‘good’ programs?” – the answer has to be “sometimes”. One of my programs, called CART, written single-handed while I was on sabbatical in 1997, is still being used by engineers around the country to support the design of electrification networks. I’m proud of CART and think that it is a really ‘good’ program.
“Am I ‘good’ at writing programs?” – the answer to this depends on how one defines ‘good’ in this context. Words like ‘predictable’ and ‘bug-free’ come to mind. So let us change the question to the following: “Can I predict how long it will take me to write a program?” and “How ‘buggy’ are the programs that I write?’. Answering these questions is quite difficult. The truth is that, even after 40 years of programming, I simply don’t have the required data available to answer these questions.
In October and December last year, I attended a course on the Personal Software Process (PSP). I was required to write 7 small-ish programs (a few hundred lines of code each). I collected data on how I performed.
For the second program in the course I predicted that it would take me 240 minutes to complete. It ended up taking me 530 minutes – an underestimation of about 120%. The program had 361 lines. I injected 31 defects while writing the program and removed them while compiling and unit-testing. That’s about 1 defect for every 10 lines. Not very good quality by any standard!
Therefore, based on how I performed in the PSP training and an interpretation of ‘good’ that relates to predictability and the number of defects, I’m not a ‘good’ programmer. I have to admit this!
It was also extremely interesting to examine how I write programs. The process of writing a program always includes a number of distinct steps:
Planning: getting the required information, tools and resources together;
Designing: both high level (strategic) and detailed design during which we analyse the problem and work out how to solve it;
Coding: actually typing in the lines of code;
Compiling (if we are using a compiled language): running the compiler and correcting any errors until we get the program to run;
Unit Testing: running one or more tests to check if the program is correct. While testing we sometimes have to correct errors to the design or the code and re-compile the program.
During the PSP training I did something I had never done before – in 40 years of programming! I measured the amount of time that I spent doing each of the above steps. The results of this measurement shocked me! I found that over 90% of the time I spent working on a program was used for Coding, Compiling and Unit Testing. I hardly did any design. Not only that – I cycled through these steps (Code->Compile->Test) repeatedly in a very tight loop. The PSP course taught me that working like this was wasteful of my time, unpredictable and frustrating.
By the end of the PSP course I had learnt a new way to program, based on my own (new) personal software process. The 7th program I wrote was completed within 30% of my estimated time (still not brilliant, but much better). It also had far fewer defects.
Although I still can’t claim to be a ‘good’ programmer, I think that the PSP course gave me really deep insights into how I might go about becoming a ‘good’ programmer.
If anyone reading this is a programmer, my question to you is “Are you a ‘good’ programmer?” If the answer is “yes” – my next question is “How do you know?”