Category Archives: Enterprise

How a DOMO Fits

In my previous post, I discussed the notion of a DevOps Management Organization or DOMO.  As I said there, this is and idea that is showing up under different names at shops of varying sizes.  I thought I would share a drawing of one to serve as an example.  The basic structure is, of course, a matrix organization with the ability to have each key role present within the project.  It also provides for shared infrastructure services such as support and data.  You could reasonably easily replace the Business Analyst (BA) role with a Product Owner / Product Manager role and change “Project” to “Product” and have a variant of this structure that I have seen implemented at a couple of SaaS providers around Austin.

This structure does assume a level of maturity in the organization as well as the underlying infrastructure.  It is useful to note that the platform is designated as a “DevOps Platform”.  It would probably be better to phrase that as a cloud-type platform – public, private or hybrid – where the permanence of a particular image is low, but the consistency and automation is high.  To be sure, not all environments have built such an infrastructure, but many, if not most, are building them aggressively.  The best time to look at the organization is while those infrastructures are being built and companies are looking for the best ways to exploit them.

Organization with DOMO

Rise of the DOMO

A lot of the DevOps conversations I have had lately have been around organization issues and what to do about the artificial barriers and silos that exist in most shops.  Interestingly there is a pattern emerging among those discussing or even implementing changes to deal with this discussion.  The pattern involves matrix organizations and the rise of what I call a “DevOps Management Organization” (DOMO).  The actual names vary, but the role of the organization is consistent.

Most software delivery organizations end up with some kind of matrix where product managers, project managers, engineers, architects, QA are all tied to the success of a given project / product while maintaining a discipline-specific tie.   In the case of an ISV, you can add some other disciplines around fulfillment, support, etc.  The variable is whether the project/ product is their direct reporting organization or they report to a discipline.  And the answer can depend on the role.  For example, a Project Manager might be part of the engineering organisation and be a participant in the company’s PMO.  Or they might be part of the PMO and simply assigned to (and funded by) the project / product they are working on.

When you add Agile to the mix, the project dimension tends to take on a much higher level of primacy over the disciplines since the focus becomes much tighter on getting working software produced on a much tighter iteration timeline.  This behavior leads to the DevOps discussion as the project/product team discovers that there is no direct alignment of Operations to the project efforts.  Additionally, Operations is a varied / multi-disciplinary space in and of itself.  Thus it becomes extremely difficult for the project/product team to drive focused activity through Operations to deliver a particular iteration/release.  The classic DevOps problem.

The recent solution trend to this problem has been the creation of what I call “Ops Sherpa” roles in the project/product teams.  This role is a cross-disciplinary Ops generalist who is charged with understanding the state of the organization’s operations environment and making sure that the development effort is aligned with operational realities.  That includes full lifecycle responsibility – from ensuring that Dev and QA environments are relevant equivalent configurations to production in order make sure deliverables are properly qualified to making sure that various Operations disciplines are aware of (and understand) any changes that will be required to support a particular release deliverable.  In more mature shops, this may grow out of an existing Release Management role or, if a particularly large

Critically, though, this role gets matrixed back into the Operations organization at a high enough level to sponsor cross-operational silo action.  This point is the head of what I call the Head of the DOMO and provides the point of leverage to deal with tactical problem as well as the strategic guidance role to drive cross-project continuous improvement into the operations platform space in support of faster execution (aka DevOps speed releases).

Whatever the name,  the fact that large scale companies are recognizing the value of deliberately investing in this space is a validation that being good at release execution is strategic to cost-effectively shortening release cycles.

Leaders Should Make Their Teams Teach

Teaching something will make you a better practitioner of that thing.  It is an adjunct to the old adage that true mastery of a subject is the ability to teach it to someone else.  The act of educating someone on something forces you to organize your thoughts on that subject.  This, in turn, gives you new insights on the subject and/or makes you more efficient at processing the subject.  Therefore, there is a lot of value to the teacher in the act of educating.

There is a business value as well.   But a lot of managers, however are not comfortable pushing their team members to do so – particularly if they have team members who resist doing it.  I have spoken to a number of managers who really have no idea how to break down that resistance.  There is obviously the ‘stick’ side of it where it should be a part of a senior person’s job description and if  they don’t do it, their review will not be as good as it could be.  But the stick side is a pretty weak instrument and it just breeds problems over the long haul.  The more important aspect is the carrot side of the discussion.

Look at it this way:   no matter how good a coder someone is, their value is intrinsically limited by the amount of code they can physically produce in a certain amount of time.  That means that once they reach their personal peak productivity, they are basically plateaued in terms of their career opportunity due to laws of physics and biology (i.e. only so many hours in a day and humans need to sleep).  Contrast that with someone who has reached their productivity peak and uses their skills to help make others better.  That person is now leveraging their skills through a larger group and enhancing the produtivity of that overall group by passing on lessons and learnings.  They are multiplying themselves through that group.  That person’s value has not plateaued.  This does not mean they are on a ‘management’ track, either – though it can lead there.  I have found that engineering organizations that have been successful over time all have a non-management ‘technical leader’ career path for folks who are both strong technically and effective leader/teacher/mentors.  Does your organization think like that?  It should.

Even if it doesn’t, it is irresponsible of the manager to not at least have a frank conversation about this to their team members.  That a team lead does a massive disservice to a team member’s career if they just encourage them to be ‘super techie’.  It will fundamentally limit their team members’ value and the value of the team in general.

Start Collaboration with Teaching

Every technology organization should force everyone in the group to regularly educate the group on what they are doing.  This should be a cross-discipline activity – not a departmental activity.  There are three reasons to do this.  The first is obvious – there is an intrinsic value in sharing the knowledge.  The second is that the teachers themselves get better at what they are teaching about for the reasons described above.  The third is that it serves to create relationships among the groups that will open channels of collaboration as the organization grows.

This will create more opportunities for someone to have a critical insight on a situation and invent something valuable as a result.  It may be as basic as the fact that the team is faster at solving problems because they know who to call and have a relationship with that person.  It also means that you have a better chance of keeping your ‘bus number at healthier levels thereby making your organization more resilient overall.  Of course, it will also make your overall organization more cohesive meaning people will be somewhat more likely to stay and ensuring that you have fewer ‘bus number’ situations in the first place – or at least fewer that were not caused by a bus

Agility Comes from Knowledge

One of the members at Agile Austin is fond of saying that ‘the only true source of Agility is knowledge’.  I think that is very true in a lot of situations.  The more you know and understand, the more adaptable you can be.  It might be  a geek-spun buzzword version of the old aphorism that “knowledge is power”, but that doesn’t mean that it is in any way bad.  Indeed, old aphorisms, rephrased or not, stand the test of time because they speak to human nature.  For all of our technology, we’re still pretty much the same.

So, where does this cultural comment hit DevOps?  Many places, really, but today I am going to pick on the fact that Agile and DevOps require participants to know and understand more than what would have been present in their traditional job role.  It is no longer OK to just be the best coder or sysadmin or architect.  You have to maintain a much higher level of generalization to be effective in your specialized job role.

This can hurt people’s heads a bit.  Particularly in larger organizations where the message has been to specialize and be the best [technical role] that you can be.  The irony is that larger organizations tend to have large and complex application systems.  So, they have traditionally compensated by having teams of people who specialize in fitting things together across the specialties.  While this certainly works, there is often a lot of time spent on rework and polish to get the pieces to all fit together.  That also implies time, which directly impacts the responsiveness (agility) of the development organization to the business.

Now those organizations are faced with needing to retool their very culture (and the management structures entwined within it) to place some amount of value on generalization for their people.  That means deliberately encouraging staff to learn more and more about the “big picture” from all aspects – not just technical.  That means deliberately DIS-couraging isolationism in specific disciplines.  It also means deliberately blowing up organizational fiefdoms before they take hold.  And it means rewarding behaviors that focus on achieving the larger goals of the organization while rooting out incentives on very parochial behaviors

The funny thing is that this is not new.  When I was first a manager, I worked for a company obsessed with this sort of thing.  We were very high on the notion of ‘lifetime learning’ and organizational development in general.  It was a way that the company encouraged/taught/focused people to aggressively adapt to the changes that came with fast growth.  That company returned more to its investors than any other tech startup I have seen in a long time.  We never worried about solving problems -we all understood a lot about the business and had a common understanding of how it worked.  It was easy and fast to get people working on a problem because we did not have to waste time bringing people ‘up to speed’.  We knew how it fit together and understood the value of proactively pushing it into newbies’ heads.  They wouldn’t be newbies for long, after all.

This month’s book club selection  at Agile Austin is focusing on Peter Senge’s keystone work in this area – “The Fifth Discipline”.  I have not read it in a while, but it is damn good to hear people focusing on this stuff again.  I really liked a lot of the concepts in that book; probably because they are relatively timeless as it relates to human nature / behavior.

New Toy!!! IBM Workload Deployer

The company I work for serves many large corporations in our customer base, many of whom are IBM shops with the commensurately large WebSphere installed bases.  So, as you might imagine, it behooves us to keep abreast of the latest stuff IBM delivers.

We are fortunate enough to be pretty good at what we do and are in the premiere tier of IBM’s partner hierarchy and were recently able to get an IBM Workload Deployer (IWD) appliance in as an evaluation unit.  If you are not familiar, the IWD is really the third revision of the appliance formerly known as the IBM WebSphere Cloudburst appliance.  I do not know, but I would presume the rebrand is related to the fact that the IWD is handling more generic workloads than simply those related to WebSphere and therefore deserved a more general name.

You can read the full marketing rundown on the IBM website here:  IBM Workload Deployer

This is a “cloud management in a box” solution that you drop onto your network, point at one one or more of the supported hypervisors, and it handles images, load allocation, provisioning etc.  You can give it simple images to manage, but the thing really lights up when you give it “Patterns” – a term which translates to a full application infrastructure (balancing webservers, middleware, DB, etc.).  If you use this setup, the IWD will let you manage your application as a single entity and maintain the connections for you.

I am not an expert on the thing – at least not yet, but a couple of other points that immediately jump out at me are:

  • The thing also has a pretty rich Python-based command line client that should allow us to do some smart script stuff and maintain those in a proper source repository.
  • The patterns and resources also have intelligence in them where you can’t break dependencies of a published configuration
  • There are a number of pre-cooked template images that don’t seem very locked down that you can use as starter points for customization or you can roll your own.
  • The Rational Automation Framework tool is here, too, so that brings up some migration possibilities for folks looking to bring apps either into a ‘cloud’ or a better managed virtual situation
I do get to be one of the first folks to play with the thing, so I’ll be drilling into as many of these these and other things as time permits.  More on it as it becomes available.

“Enterprise” DevOps

Anyone in the IT industry today will note that much of the DevOps discussion is focused on small companies with large websites – often tech companies providing SaaS solutions, consumer web services, or some other solution content.  There is another set of large websites, supported by large technology organizations that have a need for DevOps.  These are large commerce sites for established retailers, banks, insurance companies, etc.  Many of these companies have had large-scale online presences and massive software delivery organizations behind them for well over a decade now.  Some of these enterprises would, in fact, qualify among the largest software companies on the planet dwarfing much more ‘buzz-worthy’ startups.  It also turns out that they are pretty good at delivering to their online presence predictably and reliably – if not as agilely – as they would like.

Addressing the agility challenge in an enterprise takes a different mindset than it does in a tech startup.  This has always been the case, of course.  Common sense dictates that solving a problem for 100 people is intrinsically different than for 10,000.  And yet so many discussions focus on something done for a ‘hot’ website or maybe a large ‘maverick’ team in a large organization.  And those maverick team solutions more often than not do not scale to the enterprise and have to be replaced.  Of course, that is rarely discussed or hyped.  They just sort of fade away.

It is not that these faded solutions are bad or wrong, either.  A lot of times, the issue is simply that they only looked at part of the problem and did not consider the impact of improving that part on the other parts of the organization.  In large synchronized systems, you can only successfully accelerate or decelerate if the whole context does so together.  There are many over-used analogies for this scenario, so let’s use the one about rowers on a boat to make this point and move on.

Let’s face it, large organizations can appear to be the “poster children” for silos in their organizational structure.  You have to remember, though that those silos often exist because the organization learned hard lessons about the value of NOT having someone maniacally focused on one narrow, specialized activity.  Think about this as your organization grows, runs into problems or failures, and puts infrastructure in place to make sure it doesn’t happen again.

One of the main points of this blog is going to be to look at the issues confronted by organizations that are or are becoming ‘enterprises’ and how they can balance the need for the Agile flexibility of DevOps with the pragmatic need to synchronize large numbers of people.