...by Daniel Szego
"On a long enough timeline we will all become Satoshi Nakamoto.."
Daniel Szego

Sunday, January 25, 2015

Limits of Agile development in SharePoint Field

Agile development has been increasingly popular during the last couple of years.
Independently if we speak about a strict SCRUM or extreme programming methodology or something more flexible agile style development, common characteristic of the development is to have:

-          Strong contact with the customer.

-          Frequent and multiply delivery (like sprints at SCRUM).

-          Flexibility of changing requirements even in late development phase.

-          Lightly formulised specifications, like with the help of user stories.

-          Certainly agile projects have some other characteristics as well like self-motivated self-developed teams however they are less important in the scope of this article.

Certainly Agile development has got a lot of advantages and provide a much higher customer satisfaction as opposed to the classical waterfall V or W model in application development. Despite we think that there are several practical facts drawbacks that should be considered.

-          Complexity of change: As it is pretty simple and fast to develop a use case at the beginning of the project, it is usually much more difficult to do it later. The problem is that at the beginning you can develop everything from scratch however later on consider the existing code and the existing architecture of the system. As an example let we imagine a .NET, SQL MVC development that has got some integration with SharePoint as well. At the beginning of the development it is pretty simple to implement user stories as rules in the business logic, however if you get the requirement only at the end of the development that you should do everything in a transactional way that means that all of your business logic should be reconstructed which might mean rewriting a couple of thousands of line of code retesting the modules and so on. (Just from the deep technical side it is not necessarily so simple that you rewrite everything with TransactionSoce as SharePoint does not provide transactions, so you have to probably manually implement a 2 phase or 3 phase transaction protocol and use overall).

-          Project budget: Project budget is always final. Even if you can argue that a new sprint a new round is necessary and flexibility had its price as well, our experience that you should deliver functionality in a meaningfully budget. Even if it is accepted that the new functionalities cost more they should be in a meaningfully limit. As an example it is usually a not accepted that a implementing a new requirement or user story costs 2 or 3 times more as the whole exiting development. So as a conclusion, even if a little bit more flexible project budget has been accepted, you have to deliver at least a core functionality within a certain budget.  

-          Time for delivery: Same is true for the delivery time. The project should have some functionalities to be delivered that usually have some time constraint. Even if there is always a chance to negotiate one more sprint or one more agile cycle delivery time can usually not extended without limits.  

-          Technological limits: In most software development scenarios you do not develop everything from scratch, despite you use one or more frameworks to make the development faster. All of these frameworks have some technological limits that should be considered at the choosing of the framework. Identifying these limits are sometimes challenging as they are not necessarily strict and documented values; they are sometimes soft limits, best based on experience. Best example the well-known SharePoint strict and soft boundaries. Another example can be .NET entity framework for reaching standard relational databases. Entity framework is pretty much liked from .NET developer to access an SQL database because practically the whole database is mapped by C# objects. The drawback however that at each command part of the database has been queried, it is brought to the client side and after making some transformation usually written back to the database. Although it is easy to use for a C# developer it is much less performant as direct SQL commands or stored procedures. As a consequence technological limits of entity framework are reached if complex and fast database operations should be carried out or one has to move and transform a huge amount of SQL data.  

Our experience that most agile project in SharePoint field follows the following pattern. We analysed the cost of the development in the complexity of specification (Figure 1).

Figure 1

As cost of development we mean the following two factors:

-          Licence costs of the used technologies or software frameworks.

-          The amount of cost of the development itself.

At complexity of specification, we practically mean the number of user stories. We suppose that the project is agile so the complexity of specification is highly correlated with the time axis of the project.  

As examples carried out, we consider three different use cases in which similar user stories are covered by three different kind of software architecture. As a bottom line of all use cases let we try to realise a small customer relationship managing software. We are going to have several user cases from the domain like, I want to store somewhere my customers, I want to store some detailed information about my customers, I want to manage contact persons at each of my customers, I would like to see like as a reports what I for each of my customers sold, and so on and so on….

-          Use Case 1: Let we imagine that the target software is Office 365, we try to develop the specification as Lists, Document Libraries and some JavaScript extension, everything is covered by the SharePoint online platform.

-          Use Case 2: In use case 2 we try to develop everything with the help of the K2 platform. K2 black pearl SmartObjects for the business entities and K2 SmartForms for the user interface components. Storage of SmartObjects are either realised by SharaPoint lists or by simple tables of a Microsoft SQL server.  

-          Use Case 3: In use 3 we build up everything with the help of ASP.NET MVC Framework using SQL server for storage and Entity Framework for communicating with the database.  

Each project has got a set up time, at this period infrastructural components are installed, licenses are paid and other kick off and project start special tasks are carried out. During this set up period the cost usually increased pretty strong however relative small part of the specification is delivered. At the example of Use cases the set up time covers the following cost factors:

-          Use Case 1: in the Office 365 example the set up time and cost is pretty much reduced. It basically means buying some licences and setting up the first site collections through the administration portal.

-          Use Case 2: in the K2 example the set up time and cost is much higher. On the one hand license  cost are much higher as at the office 365 example, on the other hand installing a full K2 environment requires time and expertise, especially if it is should be high performant or high available.

-          Use Case 3: in the MVC.NET example set up time is somehow in between the two previous ones. It usually requires setting up some windows server with IIS and SQL database and some classical developer tools like source control.  

Having set up the development project there is usually a pretty efficient development phase. In this phase user stories and requirements can be carried out very efficiently. This period gives the chance to do a really effective agile development: flexible changes or add user stories, deliver in short iterations, have a lot of interaction with the end customer and use any kind of agile template that you want, like SCRUM, extreme programming….

-          Use Case 1: in the Office 365 example this period is typical to create content types lists and document libraries, sites or pages, implementing some design, changing the item or edit item page of some elements and so on. Some these changes can be carried by information workers in minutes or hours others require some development knowledge take days to realise.

-          Use Case 2: in the K2 example the effective period means defining a business logical layer by SmartObjects and creating user interface elements by SmartForms. As only information worker with high K2 knowledge is required the development of new user stories can be very fast in this period.

-          Use Case 3: in the MVC.NET the effective period means setting up database elements integrating with Entity Framework, setting up business rules and realising user interface components by views and the related UI logic by controllers. Realising user stories in this example requires developers, sometimes not just one but more, like one for the database side one on the business logic side, one on the controller part and perhaps a designer as well. As more than one people taking part in the development you can not avoid some kind of a project management either. As a result carrying out a new functionality always takes a lot of time in comparison with the two previous approaches.    

Unfortunately each technological or architecture choice has got its limits. If you reach that point than new functionalities to deliver will take a lot of effort. Certainly you can change your architecture or technology or extend in some sense, however that basically means that even with the best integration you should cover once again the set up time, perhaps recruit new people with new technological competence. The scenario is worst if you do not clearly extend or modify the architecture but you realise some fast hacks. That usually bring some instability into the system resulting on a long run a lot of efforts in system stabilisation and hunting bugs that only sometime occur. At any case reaching the architecture limit means that the costs are going to explode; you deliver very small amount of user stories or requirement change very slow and expensive.

-          Use Case 1: Office 365 is a pretty good example for architecture limits as it has got a lot of limit. Things like number of elements in a list limited to 5000 or the number of possible lookup fields on a 13 basis very fast result that you have problem with the architecture. In some cases if you just slightly exceed the limits there is some possibility to making some small hacking like storing items in two lists, or reconstructing your data model to minimise the number of lookup fields but in most situation you can not avoid to rethink your architecture. As an example there is a chance to set up an on-premise environment and migrate everything, or you can store some items in Azure and integrate somehow, but at any case both will cost much more and will take more time as realising the requirements staying within the architecture limits.    

-          Use Case 2: The K2 framework is much more flexible in a lot of senses, despite you can reach soft and hard product limits if you integrate with special technologies. Like let us suppose that you build up a lot of K2 SmartForms and Processes to create a lot of self-service possibilities to your customer like ordering a product, giving feedback, setting payment options and so on. After that you want to provide the same functionality with the same automatic processes on a telephony system as well. As there are some possibility to extend the architecture to integrate a third party telephony system you will not really be able to use the workflow engine with a telephony system as the engine itself does not guarantee time constrained response time. You probably need to buy a totally different framework and on that rebuild all of your processes that you want provide through a telephony as well. As it is certainly possibly it surely implies a boom both in the costs and in the delivery delay.   

-          Use Case 3: in the MVC.NET let we imagine that we reach the limits of the entity framework. Due to the complexity of business logic and data oriented computation the application will perform very slowly. As a result the whole data access layer and business logic layer should be reconstructed and put everything that is possible into the database as stored procedure. As it is certainly possibly, it will again result a strong increase in the development cost and prevent the fast delivery of the next release.  

Figure 2 demonstrates 3 different technology curves. They may vary how much time takes the setting up, how effective is the middle part and of course where are the technological limits.

Figure 2,

As a conclusion we propose to use agile methods with caution. At our experience:

-          Agile methodology works well only in a certain architectural framework.
-          Exceeding the architectural limits results a boom in development cost and in delivery time. Surely architecture change can be realised in an agile way as well, however due to its cost and time nature it is usually not accepted by the customers. Plus consider the psychological aspects as well, like you deliver the first 20 user stories as 2 days each, however for the 21 th you should say it is 100 days – to effectively communicate something like that is not simple.  

-          Even if it is agile, more user stories and more requirement analysis should be carried out at the start phase as at the later phases. Especially user stories and requirement elements that influence architecture decision should be specified as fast and as early as possible.

-          It is sometimes a good idea to demonstrate to the customer the chosen architecture and the limits and drawbacks as well. You can formulate some negative examples as well, like requirements or user stories that will reach the architecture limits.

-          It is always a good idea to have a good architect.

I think actually the summarised ideas are pretty much trivial, I was personally not sure if it worth time to write them down. Actually software development has got some similarities with the building industry. It is always possible to paint a living room in an agile way and of course if it is not liked by the customer it can be repainted each day with a different colour or perhaps more than with one colour. However it is not possible to do agile development (or agile house building) if it is about the number of floors. You have to exactly specify at the beginning if you want to have 1 floor, 10 floors or 100 floors. A house planned to have 1 floor will never have 10. It has to totally destroyed and start building from scratch.