“If you could do it all over…”

I got a seemingly innocuous question from a colleague today¹ :

If you could do SAGrid all over again, what would you have done differently ?

wow… but then,

What were the biggest failures ?

Ouch ! Talking about failure, especially to yourself and about yourself, is a hard thing to do. My personal belief though is that there is a time and place for this and when it’s done with the right perspective and context, it can be the only way to actually learn. Perhaps this is one of those times.

To answer the first question, I would probably have quit earlier and moved on to other things, handing over coordination and development of the initiative to others, once the project had been formally started. However, there’s that context thing : what did it mean for the project to formally start ? One day, when the full, unabridged “Olde Historee of Ye Grid of the Southe Africa” is written², perhaps that question will have an answer, but for me the start of the project was a meeting I organised at the iThemba Laboratory for Accelerator-Based Science on the 31st of July 2008 - the very first meeting of the SAGrid Joint Research Unit. We discussed the founding document, which was supposed to have been signed soon thereafter, and create an entity that oversaw the activities of the national grid. Without a doubt, this is the thing I would do over the hardest. My notes said :

Ideally JRU should be recognised by national authority to represent ZA and to create an national grid infrastructure. Would become precursor of SA National Grid Initiative – approved by DST. JRU needs official recognition to work with the Europeans.

This document was treated as a formality, and at several of the institutes participating was never seen by a lawyer. It never made it to the eyes in the DST, and most of all, the funding agencies had no chance to provide input or comment on it. In a way, this was by design - we had intended to build a community that was not driven by any particular authority, nor dominated by ay particular partner, but rather owned by the partners themselves and co-ordinated by a neutral third party, the Meraka Institute. I believed that this was a relevant model to build a federation in South Africa, but boy was I wrong. That was almost 10 years ago now, and looking back on the failure to reach that goal stings.

Perhaps things would sting less if I had not been so personally invested in the ideas, and if there had been a more robust participation by the members of the JRU. Somewhere along the line “coordination” turned to “command” and the direct link between the owners of the sites and the coordinator of the infrastructure broke down. Call it a lack of agency if you will, call it disengagement, call it introspection - the trend to take care of the issues at home (institutionally speaking), at the expense of the greater good was unmistakeable.

A lot went “wrong” with SAGrid as a project. I consider it a failure in several senses, but not in the most important one : we actually built it and it actually worked. Not reaching full potential is a kind of failure, sure, but one has to moderate one’s expectations every now and then. I have no doubt in my mind that the work we did forged principles of collaboration and transparency, as well as providing useful foundational experience to projects which followed. I wouldn’t go so far as to say that we informed the majority’s opinion, or even that without SAGrid things wouldn’t have gotten done anyway, but we made a dent in the the universe some peoples’ research.

Perhaps SAGrid should have been an experiment first. Perhaps it should have been started later. Perhaps we should have ignored EU projects and training networks and gone insular. Perhaps I should have had a magic wand to deal with all of the lack of support and infrastructure in the early days of the project

Biggest regrets

1. not having a review to define the change from project to infrastructure.
2. not doing the CA right first
3. trying to do everything myself
4. having to argue with XXXX (redacted)³
5. being dependent on technology providers

I could have written a litany here, but that would have been even less informative. In the front of my mind, writing this is not necessarily answering my colleague’s question, but rather to learn from failure. You can’t learn from an infinity of regret, but you can learn from atomic failures.

So, a lot of things are left out. A lot of other regrets arose from things that came later. Changes - some of them tectonic changes brought on by cheap and ubiquitous access to high-speed networking, the maturing cloud stacks, institutional priorities, etc arose and perhaps the model we and the reality of who controlled the narrative of computing , network and data infrastructure.

These regrets, these failures are things that I could have changed. If the point of this exercise is to learn, perhaps the lesson to anyone reading this is : “Don’t do that”. Also : try to remember the past, it’s the best way to learn, even if it’s not always pretty.

For what it’s worth, anyway.

---

References and footnotes

 Tagged with SAGrid • failure • thesituation • history