I don't think I said clearly in class today that the purpose of the last blog post was to connect it with what we are doing in Excel in our study of decision making under risk and uncertainty. But unlike how we will model this in Excel, in reality there is much more complexity, the decisions and the random environment both unfold over time, and sometimes very seemingly small choices end up mattering a lot while things that you once assumed were very important might prove to be only incidental.
Since all of you wrote about your choices coming to the U of I and while you were enrolled, here is a brief illustration about my years in college. If you have never been to Boston - it is a town with many different colleges. MIT actually is across the Charles River in Cambridge, but you can walk over a bridge and be in Boston. Boston University is quite close. When considering attending there, I was aware that MIT was a haven for nerds, which might be okay academically, but I wanted contact with other people, just as I had in high school. I assumed (incorrectly) that I'd have such contact because of all the colleges in Boston, especially since some of my high school classmates were enrolled there. I ended up hanging out almost exclusively with folks in my dorm - we all had a lot inertia and that was the easy way out. At the beginning that was okay, but too narrow for me. Eventually the narrowness started to get to me. Rationally, I could have moved elsewhere on campus and starting over again with a new cohort on campus and then make a deliberate effort to engage with people from other campuses. But I didn't know people at MIT other than those in my dorm. I did know many people at Cornell, so it wasn't such a jolt going there, and then I found the variety of people I had wanted.
* * * * *
In a few weeks, when we do the Excel homework on bargaining, we will discuss procurement. The model of bargaining will be very simple. The discussion of procurement, in contrast, will try to illustrate the complexity and the goals in the process of choosing a vendor and then negotiating with the vendor. The discussion below is not meant so much to foreshadow what we will do in a few weeks as it is meant to tie into the issue about making mistakes in decision making when operating under uncertainty. The issue of mistakes might sometimes be ascribed to novice decision making. But the time period I am talking about I was in my late 40s and had made many important life decisions by then. There was, however, novelty to the situation. In the presence of novelty you tend to make assumptions to aid in the decision making. (This is exactly what Herbert Simon argues in his Nobel lecture.) When some of the assumptions prove unsound, which happened in this case, you can get egg on your face; that also happened and things became quite unpleasant for a while.
So with that here is an incomplete description of the choices made - why they were made at the time - and what should have been done in retrospect.
When my little center of educational technologies started in summer 1999, we had 5 products. Two were discussion boards - FirstClass and WebBoard - one was a homegrown quiz tool - Mallard - and we had two fledgling learning management systems - Blackboard and WebCT. It was my decision then to have two LMS. Most other campuses at the time went to one of those or the other. (Iowa also went to both.) I understood things reasonably well from the user/instructor perspective. So I favored offering different alternatives for different users to better match their preferences. I didn't then understand, nor did I gain much better understanding later, the issue of supporting all the services, both at the back end (server) level and at the front end (my staff interfacing with instructors) level. What I did understand is that there are diseconomies of scope here, meaning it it more efficient to provide one comprehensive service and do that reasonably well then to provide a variety of niched services, even when on a function by function basis the niched services beat the comprehensive service. I should also add the the budget of my center was quite limited, so that reality needed to be confronted. But, at the time, I thought our general enthusiasm would overcome these other difficulties. That proved wishful thinking and living in the present without thinking forward. (I am otherwise usually good at thinking forward. The prior three years where I cut my teeth as an administrator were full of that, but all on a much smaller scale.)
Now let's fast forward about 3 years. Three things changed to make the situation quite different. First, usage in our services had grown substantially. At the time, most students on campus would have had at least one course that used our services. We needed to find a way to support all that growth; consolidation in our offerings made sense for just that reason. Second, the university had done the deal to get Banner as its ERP application, which included the Student Information System (what you use to register now and what I use to see who is enrolled in the class). The old SIS would be retired. The systems we supported drew data from the SIS. There was a need then to consider a new way to populate the systems with SIS information. This too favored having one system where this pipeline was well constructed (and where if trouble shooting were necessary attention could be placed). The third thing was that what we were experiencing was happening on most campuses. So the market responded by delivering enterprise versions of the LMS that promised to address these issues.
Now let's add a layer of complexity. At the time my center was located in the Armory. We had some tech staff there and four out of the five services were hosted right out of our office. The fifth service, WebCT, was hosted in DCL by the then campus IT unit. We contracted with them for this hosting. It was clear that if we were to go to an enterprise LMS that they would host it. While I had a relationship with the guy who ran the division that hosted the service which dated back to 1996, and we were on reasonably good terms, I thought that from our end we needed a tech guy who would talk their language and better communicate our needs. Plus, we needed somebody on our side of the tracks who would really understand an enterprise LMS under the hood. At the time there were two other instances of WebCT, one in Chemistry, the other in ACE. One of the ACE staff had previously interviewed with us for a different job (supporting our other servers). I hired him to be the needed tech guy. This got several of my other staff upset, but I think it was the right call. The skill set was limited and he was the right guy for the job.
The other part of this that made life complex was that my boss, the CIO, wanted us to merge with the campus IT unit (which was much bigger than we were) and with his CIO office. On the one hand, this made some sense, for the back end communication issues discussed in the previous paragraph. On the other hand, my little center had a reasonably good reputation while the big IT organization was regarded in a much more mixed way. I thought our way of doing things would get lost in the merger. So I pushed back a little on this with the CIO, but not a lot. In retrospect, that was a mistake. I should have pushed a lot harder because the "getting along" approach contributed to what came next.
We originally conceived of the choice of enterprise LMS as a "bakeoff" between Blackboard and WebCT. When Campus Purchasing got wind of this, they said we couldn't do that and had to go through a formal RFP process, which entailed going out for bid and awarding the contract to the best bidder. At first, this annoyed me. Later I came around to their view, although the RFP process definitely is clunky. A first step in that is the writing of a detailed requirements document. As an enterprise LMS was a comparatively new piece of software at the time, there was the question of just how one determines the requirements for it. In this case, we obtained a couple of RFP documents that had been written previously at other universities. One was from Berkeley, the other from the U Wisconsin system. These are the sort of documents where plagiarism isn't just permissible, but is actually desirable. There is no need to reinvent the wheel. But there were some local circumstances that should have been accounted for. I made another significant error here. I completely trusted our tech guy to come up with the right requirements, without challenging whether they should be requirements at all. I will illustrate this in the next paragraph.
One of the big requirements is that the software had to be supported by Unix-Oracle. (It would run on servers from Sun Microsystems and rely on an Oracle database.) There were actually four enterprise LMS vendors on the market then, but two of them operated in a Windows only environment. (Blackboard and WebCT ran in both environments.) In the bakeoff we originally conceived, some of our users would be upset by the decision, as the other platform would be chosen and they'd be forced to switch. If we had actually chosen one of those Windows only vendors, then everybody would have had to switch, but then it would have been because a majority thought it was the best environment. That might have been better for us politically. (Several other Big Ten Campuses went to one of those Windows only vendors.) By writing the requirements as we had we precluded this possibility. The other big issue issue with this requirement is that while the campus IT organization had quite a lot of experience with Unix servers, it had no prior experience with Oracle. It wanted to develop such a capability, not just for the LMS but for other services as well. (The University IT, which runs the SIS, did have Oracle expertise but at the time they were entirely separate from the campus.) I should have been skeptical about the ability to develop sufficient Oracle expertise (which mainly meant having redundancy at the DBA position) in the same timeframe that we were rolling out the enterprise LMS. But I was not.
A significant reason for being so accepting is that I let my ego get the better of me and I got caught up with "empire building." The enterprise LMS was going to be my baby and it would expand my visibility on campus as well as increase my presence within the newly merged IT organization. My personality, in general, is not very imperial, so you'd might think I'd be immune from these sort of ego rewards as temptation. Here are two reasons why I didn't have such immunity. When my little center was still a concept, nothing more, I was being mentored by the then head of of the campus IT organization. He was very good to me and I thought he was a pretty sharp cookie. He told me how big the center would be (number of staff) at the outset and what it was likely to grow into over time. The number of reports overseen by an administrator was a measure of importance, in his view. If I had more people under me, I'd have more clout and more esteem by my peers, or so it seemed. When everyone else plays by that metric, it is impossible to not to get caught up in it, even if makes little or no sense for you personally.
Then there is the matter of taking knocks on the job, which in my case was getting criticized by our faculty users for making decisions that needed to be made but that were not to their liking and/or for me again being too accepting of the wishes in the IT organization and not pushing back harder at them in favor of delaying the decisions. Some of the more overt of these decisions were about cancelling service offerings. This will happen with IT. Services reach end of life and/or become too costly to continue to maintain. But we didn't have a good model for how to do that at the time (and we may still not have such a model now). Such a model would include some interim period, where the service continued to run after the shutdown had been announced. Users would then learn the model and grudgingly come to accept it. Absent the model, I was left to be the messenger of doom and took flak for doing that. (There was centrally supported ed tech services that were not in my unit. One was called Campus Gradebook. It was cancelled when we adopted the enterprise LMS.) The ego rewards then serve as some sort of compensation for taking the flak. It really is silly, in retrospect, because the ego rewards proved not to be reward at all. But while it was going on it seemed appropriate. This is and example of how you lose focus on the job about wha you are really trying to accomplish.
Here is one last mistake I made, which was about budgeting for the back end of the LMS. Given our earlier discussion about transfer pricing, it is an interesting question in the IT organization about which services get well funded and which other services get poorly funded. As it turns out, the funding approach to services varied quite a bit by service. The network, which was then recently supported by a solid transfer pricing formula, was well funded. Services like email (at the time Gmail didn't exist and the campus hosted student email) were funded off the top. Often, that funding was less than adequate to offer a high quality service. The enterprise LMS seemed to fit in with this second category.
But I should have insisted that we support it at the back end with every recommendation for support that the vendor gave us and that we make adequate investments in personnel, hardware, and complementary software. What needed to be done really was outside my expertise, but ignorance is not an excuse and I should have insisted on understanding that we were doing things the right way, arguing with my boss that he need to supply the requisite revenue to achieve this outcome. I didn't understand the urgent need to do that at the time.
Eventually that did happen, but only after the service failed. We were embarrassed into making the right choice rather than be proactive about it in the first place. I took quite a lot of heat for that, deservedly so. (I will add here that some of this may have been the software itself and that the U of I was a larger instance than the software was really designed for. So the vendor had to share some of the responsibility for the failure, at least in my eyes.)
* * * * *
The above is what I'd call a post mortem on a rather extended set of experiences that I was wrapped up in. We all make mistakes. The purpose of a post mortem is to avoid making the same mistake a second time and to consider how things might be done better when moving forward.
I will close with the following observation. Making mistakes is not a matter of intelligence. I hope this narrative will convince you that I was thoughtful throughout the process. Mistakes that are conceptual in nature happen in situations that are novel so you can't just "play it by the book" and be assured that things will turn out reasonably well. You have to make a call. Hindsight is 20-20. Foresight is not nearly as good.
No comments:
Post a Comment