Agile
development has been increasingly popular during the last couple of years.
Independently
if we speak about a strict SCRUM or extreme programming methodology or
something more flexible agile style development, common characteristic of the
development is to have:
-
Strong
contact with the customer.
-
Frequent
and multiply delivery (like sprints at SCRUM).
-
Flexibility
of changing requirements even in late development phase.
-
Lightly
formulised specifications, like with the help of user stories.
-
Certainly
agile projects have some other characteristics as well like self-motivated
self-developed teams however they are less important in the scope of this
article.
Certainly
Agile development has got a lot of advantages and provide a much higher
customer satisfaction as opposed to the classical waterfall V or W model in
application development. Despite we think that there are several practical facts
drawbacks that should be considered.
-
Complexity of change: As it is pretty simple and fast to develop a use case at the beginning
of the project, it is usually much more difficult to do it later. The problem
is that at the beginning you can develop everything from scratch however later
on consider the existing code and the existing architecture of the system. As
an example let we imagine a .NET, SQL MVC development that has got some
integration with SharePoint as well. At the beginning of the development it is
pretty simple to implement user stories as rules in the business logic, however
if you get the requirement only at the end of the development that you should
do everything in a transactional way that means that all of your business logic
should be reconstructed which might mean rewriting a couple of thousands of
line of code retesting the modules and so on. (Just from the deep technical
side it is not necessarily so simple that you rewrite everything with
TransactionSoce as SharePoint does not provide transactions, so you have to
probably manually implement a 2 phase or 3 phase transaction protocol and use
overall).
-
Project budget: Project budget is always final. Even if you can argue that a new sprint
a new round is necessary and flexibility had its price as well, our experience
that you should deliver functionality in a meaningfully budget. Even if it is
accepted that the new functionalities cost more they should be in a meaningfully
limit. As an example it is usually a not accepted that a implementing a new
requirement or user story costs 2 or 3 times more as the whole exiting
development. So as a conclusion, even if a little bit more flexible project
budget has been accepted, you have to deliver at least a core functionality
within a certain budget.
-
Time for delivery: Same is true for the delivery time. The project should have some
functionalities to be delivered that usually have some time constraint. Even if
there is always a chance to negotiate one more sprint or one more agile cycle
delivery time can usually not extended without limits.
-
Technological limits: In most software development scenarios you do not develop everything
from scratch, despite you use one or more frameworks to make the development faster.
All of these frameworks have some technological limits that should be
considered at the choosing of the framework. Identifying these limits are
sometimes challenging as they are not necessarily strict and documented values;
they are sometimes soft limits, best based on experience. Best example the well-known
SharePoint strict and soft boundaries. Another example can be .NET entity
framework for reaching standard relational databases. Entity framework is
pretty much liked from .NET developer to access an SQL database because practically
the whole database is mapped by C# objects. The drawback however that at each
command part of the database has been queried, it is brought to the client side
and after making some transformation usually written back to the database. Although
it is easy to use for a C# developer it is much less performant as direct SQL
commands or stored procedures. As a consequence technological limits of entity
framework are reached if complex and fast database operations should be carried
out or one has to move and transform a huge amount of SQL data.
Our experience
that most agile project in SharePoint field follows the following pattern. We
analysed the cost of the development in the complexity of specification (Figure
1).
Figure 1
As cost of
development we mean the following two factors:
-
Licence
costs of the used technologies or software frameworks.
-
The
amount of cost of the development itself.
At
complexity of specification, we practically mean the number of user stories. We
suppose that the project is agile so the complexity of specification is highly correlated
with the time axis of the project.
As examples
carried out, we consider three different use cases in which similar user
stories are covered by three different kind of software architecture. As a bottom
line of all use cases let we try to realise a small customer relationship
managing software. We are going to have several user cases from the domain like,
I want to store somewhere my customers, I want to store some detailed information
about my customers, I want to manage contact persons at each of my customers, I
would like to see like as a reports what I for each of my customers sold, and
so on and so on….
-
Use Case 1: Let we imagine that the target software is Office 365, we try to
develop the specification as Lists, Document Libraries and some JavaScript extension,
everything is covered by the SharePoint online platform.
-
Use Case 2: In use case 2 we try to develop everything with the help of the K2
platform. K2 black pearl SmartObjects for the business entities and K2 SmartForms
for the user interface components. Storage of SmartObjects are either realised
by SharaPoint lists or by simple tables of a Microsoft SQL server.
-
Use Case 3: In use 3 we build up everything with the help of ASP.NET MVC Framework
using SQL server for storage and Entity Framework for communicating with the
database.
Each
project has got a set up time, at this period infrastructural components are
installed, licenses are paid and other kick off and project start special tasks
are carried out. During this set up period the cost usually increased pretty
strong however relative small part of the specification is delivered. At the
example of Use cases the set up time covers the following cost factors:
-
Use
Case 1: in the Office 365 example the set up time and cost is pretty much
reduced. It basically means buying some licences and setting up the first site
collections through the administration portal.
-
Use
Case 2: in the K2 example the set up time and cost is much higher. On the one
hand license cost are much higher as at
the office 365 example, on the other hand installing a full K2 environment
requires time and expertise, especially if it is should be high performant or high
available.
-
Use
Case 3: in the MVC.NET example set up time is somehow in between the two
previous ones. It usually requires setting up some windows server with IIS and
SQL database and some classical developer tools like source control.
Having set
up the development project there is usually a pretty efficient development phase.
In this phase user stories and requirements can be carried out very efficiently.
This period gives the chance to do a really effective agile development:
flexible changes or add user stories, deliver in short iterations, have a lot
of interaction with the end customer and use any kind of agile template that
you want, like SCRUM, extreme programming….
-
Use
Case 1: in the Office 365 example this period is typical to create content
types lists and document libraries, sites or pages, implementing some design,
changing the item or edit item page of some elements and so on. Some these
changes can be carried by information workers in minutes or hours others
require some development knowledge take days to realise.
-
Use
Case 2: in the K2 example the effective period means defining a business logical
layer by SmartObjects and creating user interface elements by SmartForms. As
only information worker with high K2 knowledge is required the development of
new user stories can be very fast in this period.
-
Use
Case 3: in the MVC.NET the effective period means setting up database elements
integrating with Entity Framework, setting up business rules and realising user
interface components by views and the related UI logic by controllers. Realising
user stories in this example requires developers, sometimes not just one but
more, like one for the database side one on the business logic side, one on the
controller part and perhaps a designer as well. As more than one people taking
part in the development you can not avoid some kind of a project management
either. As a result carrying out a new functionality always takes a lot of time
in comparison with the two previous approaches.
Unfortunately
each technological or architecture choice has got its limits. If you reach that
point than new functionalities to deliver will take a lot of effort. Certainly
you can change your architecture or technology or extend in some sense, however
that basically means that even with the best integration you should cover once
again the set up time, perhaps recruit new people with new technological competence.
The scenario is worst if you do not clearly extend or modify the architecture
but you realise some fast hacks. That usually bring some instability into the
system resulting on a long run a lot of efforts in system stabilisation and
hunting bugs that only sometime occur. At any case reaching the architecture
limit means that the costs are going to explode; you deliver very small
amount of user stories or requirement change very slow and expensive.
-
Use
Case 1: Office 365 is a pretty good example for architecture limits as it has
got a lot of limit. Things like number of elements in a list limited to 5000 or
the number of possible lookup fields on a 13 basis very fast result that you
have problem with the architecture. In some cases if you just slightly exceed
the limits there is some possibility to making some small hacking like storing
items in two lists, or reconstructing your data model to minimise the number of
lookup fields but in most situation you can not avoid to rethink your
architecture. As an example there is a chance to set up an on-premise environment
and migrate everything, or you can store some items in Azure and integrate
somehow, but at any case both will cost much more and will take more time as
realising the requirements staying within the architecture limits.
-
Use
Case 2: The K2 framework is much more flexible in a lot of senses, despite you
can reach soft and hard product limits if you integrate with special
technologies. Like let us suppose that you build up a lot of K2 SmartForms and
Processes to create a lot of self-service possibilities to your customer like
ordering a product, giving feedback, setting payment options and so on. After
that you want to provide the same functionality with the same automatic
processes on a telephony system as well. As there are some possibility to
extend the architecture to integrate a third party telephony system you will
not really be able to use the workflow engine with a telephony system as the
engine itself does not guarantee time constrained response time. You probably
need to buy a totally different framework and on that rebuild all of your
processes that you want provide through a telephony as well. As it is certainly
possibly it surely implies a boom both in the costs and in the delivery delay.
-
Use
Case 3: in the MVC.NET let we imagine that we reach the limits of the entity
framework. Due to the complexity of business logic and data oriented
computation the application will perform very slowly. As a result the whole
data access layer and business logic layer should be reconstructed and put
everything that is possible into the database as stored procedure. As it is
certainly possibly, it will again result a strong increase in the development
cost and prevent the fast delivery of the next release.
Figure 2
demonstrates 3 different technology curves. They may vary how much time takes
the setting up, how effective is the middle part and of course where are the technological
limits.
Figure 2,
As a conclusion
we propose to use agile methods with caution. At our experience:
-
Agile
methodology works well only in a certain architectural framework.
-
Exceeding
the architectural limits results a boom in development cost and in delivery
time. Surely architecture change can be realised in an agile way as well,
however due to its cost and time nature it is usually not accepted by the
customers. Plus consider the psychological aspects as well, like you deliver the
first 20 user stories as 2 days each, however for the 21 th you should say it
is 100 days – to effectively communicate something like that is not simple.
-
Even
if it is agile, more user stories and more requirement analysis should be
carried out at the start phase as at the later phases. Especially user stories
and requirement elements that influence architecture decision should be specified
as fast and as early as possible.
-
It
is sometimes a good idea to demonstrate to the customer the chosen architecture
and the limits and drawbacks as well. You can formulate some negative examples
as well, like requirements or user stories that will reach the architecture
limits.
-
It
is always a good idea to have a good architect.
I think
actually the summarised ideas are pretty much trivial, I was personally not
sure if it worth time to write them down. Actually software development has got
some similarities with the building industry. It is always possible to paint a
living room in an agile way and of course if it is not liked by the customer it
can be repainted each day with a different colour or perhaps more than with one
colour. However it is not possible to do agile development (or agile house
building) if it is about the number of floors. You have to exactly specify at
the beginning if you want to have 1 floor, 10 floors or 100 floors. A house
planned to have 1 floor will never have 10. It has to totally destroyed and
start building from scratch.
