Wednesday, January 30, 2013

Software Architecture and Design in Agile World - Part 4

How can architects and designers add value in agile projects? Architects and designers can apply their experience to create an architectural vision and roadmap. Forward vision and roadmap will provide adequate clarity to team members. Architects can add value by contributing to architecture planning and sharing their ideas, suggestions and recommendations with team members. Their architecture-driven approach will help agile teams discover risks and problems early. Architects can create a learning environment and improve the productivity and job satisfaction of individuals. Above all, architects and designers can become a role model for the values in agile manifesto.

Architecture approach in software projects depends on four key factors. The first one is the architecture or design competency of the team members who perform the job. The second one is project complexity. The third factor is the familiarity of technology – the most likely technology to be used in the project, and the fourth one is the familiarity of problem domain. When your project has all these factors on the positive side, you can define and implement architecture with ease. You are going to deliver features early. Otherwise, it is wise to spend one or two iterations on architecture (architecture definition, architecture prototyping and creating the building blocks of the architecture). While doing this, it is important to keep in mind the ‘bare minimum’ or ‘just enough’ to avoid waste.

Let me give you two examples. When a corporate IT team wants to deliver an attendance management application on an existing technology platform (on which several other applications are functional) it is most likely that these factors are on the positive side. Why would a team on this project spend weeks or months in defining and constructing the architecture? I am sure you agree that a team on this project could start delivering features early as there must be adequate clarity on the architecture and there must be reusable architectural elements unless the CIO decides to revamp what is available. Later when the application is developed and delivered, they can write or specify the architecture and designs, if such artifacts are going to add value to downstream life cycle activities.

The next example is from the other extreme. It is a highly complex project. This is about developing a product that involves integration of multiple sources of intelligence to help commodity traders across countries. Let us assume that this project involves emerging technologies not known to team members and the problem domain is new to team members. Where do we start? What features do we deliver? Is it a good idea to follow pure agile? Or should we rely on BUFD? I don’t think pure agile or traditional approach (BUFD) will help. It is good to spend one or two or three iterations in defining the architecture and delivering the essential basic building blocks before we attempt to deliver user stories or features. Also, it is good to revisit the architecture at regular intervals and deliver technical stories.

<<<<<< Part 3                                     Software Architecture and Design in Agile World - Part 1

Software Architecture and Design in Agile World - Part 3

Part 3 – Let us inspect and adapt

Our expectations

What is the role of an architect? What do we expect from an architect? Obviously, technical skills and experience are the building blocks to become an architect because the primary role of an architect is to understand requirements, suggest architectural alternatives, analyze the pros and cons and arrive at an optimal or feasible architectural recommendation. To do this, architects need to be avid learners. They have to be hands-on in performing design activities. They have to be expert programmers and participating in coding. Architects have to have mentoring and coaching skills. They have to be great influencers and negotiators. They have to be great leaders.

What do architects deliver? Do they deliver a presentation or a document with block diagrams or guidelines or all these? Do they go beyond architecture definition and contribute to architecture implementation? Do they write code or deliver various components that constitute the architecture framework? These questions make us reflect on the ground realities on what is delivered by architects in projects that do not follow agile methods.

The impact

When a product or application is delivered to customer or rolled out in production, production support, sustenance, technical support and consulting teams play a significant role ensuring customer satisfaction. How do architects contribute to these teams? Do the activities or contributions of architects add value to the work life of someone who is involved in production support, sustenance or technical support or consulting? In order to debug or to understand the regression issues, do we get an up-to-date schematic or map that improves the efficiency of such activities? Or have we let these teams wade through thousands and thousands of lines of code and find a solution?

Agility and agile methods

Agility is about the ability to both create and respond to change in order to profit in a turbulent business environment. It means adaptability rather than predictability. Agile methods focus more on people rather than detailed development processes. They are based on collaborative values and principles. They are light-weight and barely sufficient methodologies. They adhere to Agile Manifesto and Agile Principles. Team members in agile teams believe and experience term such as, ‘deliver early’, ‘delivery in short intervals or time-boxes or iterations’, ‘work in sustainable pace’, ‘deliver business value’, ‘self-enabled or led teams’, ‘customer collaboration’, ‘responding to change’, ‘fail early, fail fast’, ‘inspect, learn, adapt’, etc. For them working code is the true measure of progress.

Two schools of thought

Architects from the traditional school of thought live in the world of ‘architecture thinking’. The script that plays in their mind is, “I am the expert. I will convene meeting with some of the senior members. I will decide and suggest the right things. I believe in Big-Up-Front-Design (BUFD). Architecture is a complex, esoteric entity. I am autonomous. I am up in the org hierarchy. Coding is what I used to do several years ago.”

Agile practitioners believe in ‘agile thinking’. The script that plays in their mind is, “We do ‘just enough’ architectural activities. We don’t do BUFD. Often, BUFD results in waste and rigidity. I am here to work with my team members. I am their mentor. We believe in emerging architecture and design. Our approach is to create systems that can accommodate changes in a cost effective manner. For this we will work as a team to evolve the right architecture or design. We will provide the right diagrams and documents as and when necessary to help our teams.”

Architects in agile teams are not confined to their office or cabin. They work in open space with team members throughout the project. They nurture young architects through mentoring and working together. In mature agile teams everyone plays the role of architect or designer. In agile world, architecture and design are not dead! Agile projects need architects and designers who follow agile principles. Architects and designers can add value in agile projects.

How can they add value? What are the key factors that influence the architecture approach? Let us discuss this in the next post.

<<<<< Part 2                                    Software Architecture and Design in Agile World - Part 4

Software Architecture and Design in Agile World - Part 2

Part 2 - The window seat

We come across the term ‘architecture’ in many situations. We have heard of application architecture, business architecture, cloud architecture, data architecture, enterprise architecture, fault tolerant architecture, hardware architecture, information architecture, knowledge architecture, network architecture, performance architecture, security architecture, software architecture, solution architecture, technical architecture, web architecture, and so on.

Software Architecture
Our focus is on software architecture. In order to maintain this focus let me share a definition of software architecture from the book ‘Software Architecture in Practice (2nd edition)’ by Bass, Clemens and Kazman.

“The software architecture of a program or computing system is the structure or structures of the system, which comprise software elements, the externally visible properties of those elements, and the relationships among them. Architecture is concerned with the public side of interfaces; private details of elements—details having to do solely with internal implementation—are not architectural.”

Here is another definition by Grady Booch, Philippe Kruchten, Rich Reitman, and Kurt Bittner.

Software architecture encompasses the significant decisions about

  • the organization of a software system,
  • the selection of the structural elements and their interfaces by which the system is composed together with their behavior as specified in the collaboration among those elements,
  • the composition of these elements into progressively larger subsystems, the architectural style that guides this organization, these elements and their interfaces, their collaborations, and their composition.
Software architecture is not only concerned with structure and behavior, but also with usage, functionality, performance, resilience, reuse, comprehensibility, economic and technological constraints and tradeoffs, and aesthetics.

Which one have you worked with?

With this background, it worth discussing about software architects. Software architects come in different categories. There are software architects who limit their profession to architecture definition either by creating PowerPoint slide decks or short documents with block diagrams. There are software architects who leave the team before the development starts. They appear for a short duration, play their role with authority and move on to other projects. They belong to the second category and treat architecture as relay sport. The third category is architects who play part-time role in multiple projects. Architects in this category are not accessible to team members. Architects in the fourth category are the so called glorified coders - they get the title because of their perseverance on getting the title! They are passionate but lack necessary skills. The fifth category of architect consists of ‘tech savvy’ managers who are terrible in project management. Probably they were given this title because of their technical knowledge. Architects in this category no longer design or code - they live with the premise that they are too busy to code. However they know multiple technologies and solutions. They can talk in meetings and voice their opinion.

The next category is the ‘had been there and done that’ architects. They are experts. They nurture architecture and design competency in teams. They believe in empowerment. They stay with the team and participate in engineering activities such as coding, deployment, performance testing, etc. They are valuable and they are rare to find.

The window seat

The dream of tech savvy software engineers is to become an architect. They have seen architects with authority and power. They want to be one and perform that role in style. They are lost in this dream. They fantasize on occupying the ‘window seat’ and want to move very fast in each and every step of their career from getting on board to moving closer to that seat. In this rush they miss the opportunity to write beautiful code and appreciate good designs. They want to become architects. Is that going to work? Is this what we expect?

How do you connect with this discussion? What is your point of view?

<<<< Part 1                                   Software Architecture and Design in Agile World - Part 3

Software Architecture and Design in Agile World - Part 1

Part 1 - On a lighter note

You are an IT professional, more specifically, a software engineer. You and your teammates develop, test and maintain software products and applications. Iterative and incremental way of delivering software at a sustainable pace, collective decision making, , group learning through short feedback loops, daily standups, retrospectives are not new to you because you have practiced agile methods. Sometimes you feel that your team members rush to deliver features – as many features as possible. There is no software architect who proposes an optimal architecture. There is no designer who delivers design artifacts. All you have is an Agile team that delivers working software at regular intervals.  Think! What happens to software architecture and design in agile world? This blog post is to share my thoughts on this topic and answer your questions.

What is your view?

What comes to your mind when you stand in front of a monument like this and wonder its gigantic view? Terms such as ‘structure’, ‘architecture’, ‘design’, ‘aesthetics’, etc. come to your mind. I am sure you observe the thick walls and huge doors, their precision and aesthetics, etc. You want to know about the history. You go closer and look at the tall gates, and the iron spikes on the gates. You appreciate the designs of various parts of this monument. There is a place and purpose for every component. The entire structure has been architected and planned well.

Let us not ignore the takeaways!

It is not wise to compare civil engineering with software engineering. Taking a deep dive in such discussions has not been beneficial. However, it is wise to view such monuments and appreciate the takeaways from such alternate sources. There can be many takeaways. The number of takeaways is left to your imagination.

Obviously, I am not going to stretch this beyond limits. This is because our IT world is different. It is dynamic. There are moving parts. Products and applications witness many upgrades of operating systems and databases. Interoperability is essential in many cases. That is not it. Scalability, performance, security, fault tolerance and many such non-functional requirements cannot be given a slipshod attention.

Points to ponder

Do you think ‘Big Up Front Design’ (BUFD) done by architects or designers is set on stone and does not change? Is change cost-effective with BUFD?

Do you think the architects of this monument (see picture) handed over their specifications to the next rung of workers and moved on to other projects? Does the initial architecture of this monument emerged or underwent changes when the construction was in progress? Were new ideas and design elements accepted and put into practice?

                                          Software Architecture and Design in Agile World - Part 2  >>>>>>>>>>

Thursday, January 10, 2013

Methodologies, Standards, Models, and Project Success - Part 2

Let me continue from where we left in my previous post. Is there a silver bullet? If yes, what it is? If no, how can we ensure project success? Before answering these questions, let us look at how we have evolved.

1980’s were the years when different operating systems (such as Windows and Unix) and 3rd generation languages (such as COBOL, C) were popular. This is when computer education started spreading many institutions several countries.

1990’s were the years when Relation Database Management Systems (RDBMS), Structured Query Language (SQL), 4th generation languages, Graphical User Interface, DW/BI, Internet or WWW, Extended Markup Language (XML), Unified Modeling Language (UML), and several other things became popular. Process compliance based on ISO or CMM got initiated in several companies across countries.

In 2000’s, after the historic internet boom and crash, we have seen widespread adoption of e-commerce, mobile applications, Web 2.0, Service Orientation (SOA, SaaS), Social Networking, Open Source, etc.

These days (2010 and after), we are hearing about ‘The Internet of Things’, ‘Green Computing’ , ‘Cloud Computing’ etc., In my opinion, we need better decision support systems! We come across articles about failed projects - large and significant projects meant for national security and crime investigation belong to this list. There are challenged projects in all countries. This leads to billions of dollars of waste or loss every year.

For software development, maintenance, testing and production support (known as software lifecycle), we have seen several methodologies such as waterfall, iterative, spiral, SSAD (Structured System Analysis and Design), RAD (Rapid Application Development), RUP (Rational Unified Process), and Agile Methods (XP, Scrum, FDD (Feature Driven Development), DSDM (Dynamic Systems Development Method), Lean, Kanban, etc.).

There are several standards in our industry. ISO standards and IEEE standards are well known. There are usability standards, coding standards, integration standards, architecture/design standards, security standards, and performance/scalability standards. Also, there are standards pertaining to business domains such as finance, healthcare etc. Year after year, standards are revised and new standards are added to the list too.

We have models too. We have software design models (Entity-Relationship model or ER model is a good example). We have estimation models (Function Point Estimation for example). There are process maturity models and so on.

In addition to these, there are other things such as techniques, tools, etc. How do project teams do the right things in the right way to deliver results? In other words, how do projects teams select the right methodology or approach and do necessary course correction while executing the project to deliver success?

In his paper ‘The use of systems development methodologies in practice: a field study’, B.Fitzgerald observes that the ability to select the right methodology, processes, and practices comes with experience as show in this graph.

Experienced team members understand that it is very important to customize methodologies and make them context-driven. Also, they need to understand that collaboration, focus on continuous improvement and balancing discipline and evolutionary paradigms (such as agile) are necessary. Is there a way to bring in this realization at an early stage?

In 2008, a celebratory panel took place at the 22nd International Conference on Object-Oriented Programming, Systems, Languages, and Applications in Montreal. The topic of this panel discuss was “No Silver Bullet” Reloaded – A Retrospective on “Essence and Accidents of Software Engineering”. An interesting observation by one of the panelists (Ricardo Lopez, Principal Engineer, Qualcomm) said,  "Striving for excellence is the real silver bullet that will deliver an order-of-magnitude improvement through growth, both personal and professional. The silver bullet must come within, rather than without. WE are THE Silver Bullet – which we achieve by professional excellence.”

Striving for excellence is required at personal level, team level and organizational level.

With this background, where do we start? How do we ensure success?

We need to start from colleges and universities. Timely revision of syllabus is required. There has to be more than one stream (for example, one for those who want to pursue research and another for those who want to work in the industry, etc.). Students need to experience how methodologies, standards and models apply in group projects. These are few among the list of 15 suggestions to us.

How about a positive change at an early stage? Let us make engineers at all levels understand that it is very important to customize methodologies and make them context-driven. Let us nurture the culture of collaboration. Let us improve the focus on continuous improvement. Let us spread the realization that discipline and team work are essential.

It is hard to build software applications or products, which can stay long and satisfy customers. However, creating such applications or products requires excellence in terms of individuals, collaboration, team work, focus on continuous improvement, understanding customer needs and several such factors.

Methodologies, Standards, Models, and Project Success - Part 1

My invited talk at ICSEMA 2012 was on ‘Methodologies, Standards, Maturity Models, and Project Success’. In the complex and evolving world of information technology how can we ensure successful projects? This was the open ended question in the abstract of my session.

The weather in Chennai was pleasant during that week and the conference venue was very good. At the end of my session and discussions with the participants, I wanted to post this blog so that it can benefit the participants as well as other readers.  I have done it in two parts.

Enterprise systems cover key corporate functions, store data shared by corporate users in one or more data stores, integrate within as well as across (with external systems), and contain enriched multi-media. These systems are web-enabled and often provide mobile, wireless, disconnected access. Enterprise systems include entities such as ERP (Enterprise Resource Planning), CRM (Customer Relationship Management), SCM (Supply Chain Management), HRMS (Human Resources Management System), DW/BI (Data warehousing/Business Intelligence), Email, IM (Messenger), B2B (Business to Business) applications, and B2C (Business to Customer) applications. All these are offered to end users through a corporate intranet preferably with an EAI (Enterprise Application Integration) backbone. All these are governed by the CIO organizations. Process compliance, regulatory compliance, licensing and upgrades, customer satisfaction etc., are critical to the success of such IT implementations. It is a complex world!

With more than three of four decades of computerization software systems are required to enable interaction at different levels - person-to-person, person-to-machine, and machine-to-machine. Integrated systems, interoperability and portability are crucial.

Proliferation of devices (smart phones, tablets, ..) and new paradigms (such as cloud enablement) are catching up in the corporate world. These emerging trends increase the complexity of IT systems.

With the global economic trends, CIOs and CTOs attempt to stretch the existing budget and infrastructure to do more. They have started exploring the benefits of open-source software. They want to reduce TCO (Total Cost of Ownership) and increase ROI (Return on Investment).

Fred Brooks, the author of the popular book ‘The Mythical Man Month’, wrote a paper titled ‘There is No Silver Bullet: Essence and Accidents of Software Engineering’. This paper was published in IEEE Computer in April 1987. In this paper he says, “There is no single development, in either technology or management technique, which by itself promises even one order-of-magnitude improvement within a decade in productivity, in reliability, in simplicity.”

This paper has been discussed and debated in various forums. Is there a silver bullet? If yes, what it is? If no, how can we ensure project success?

Part-2 of this post has answers to these questions.