Notes on semantic software development...

If we take a closer look, the log service from blogspot, from wordpress, twitter and SharePoint log provide pretty much the same services. You find the same datamodel, similar services like publishing mechanism and similar user interface structure, like views for list of blogs or blog details.Certainly the implementation of the each solution like implementing language, hosting model and user interface are pretty much different.

Similarly, Facebook Linkedin or Yammer provide pretty much the same datamodell, functionalities and user interface structure even if implementing technologies are far from being the same.

Sorces of efficency and inefficiency on the SharePoint software market.

As I described in the previous article, for a basic modelling we suppose that there are companies that want to buy software solutions that match for a given set of R1, R2, .. RN requirements.We suppose that there is such a software soltuion that covers the requirements and which can be delivered in a certain setup time and for a certain price.

Figure 0. Ideal solution.

Efficiency of the market means if in certain projects the actual cost of the whole project is about the software product price and the time of the whole roll out of the project is about the setup time. 

Inefficiency means however if the whole total cost of the project will be much bigger as the cost of the product itself and the total time of the project is much bigger as the setup time of the software solution. As exactly measure the efficiency and inefficiency on market is certainly difficult, it is more simpler to identify several typical situations that surely result in inefficiency, like increased total cost or increased roll-out time.

In the following I try to summarize several typical situations.

1. Requirements are not clear

Figure 1. Requirements are not clear.

As the whole IT field is pretty fast changing, summarizing the requirements are sometimes not simple. Giving examples from SharePoint, usually one requirement is that the pages should be mobile compatible. It is usually not sure however what this requirement exactly mean. 

The following situations can be imagined: 

- The pages should be somehow showed on mobile devices.

- There should be a responsive master page solutions and the layout should be dependent on the screen resolution.

- The downloadable content and javascript libraries should be optimized for different network circumstances. 

- There should be an App for all major mobile operation systems that shows the content.

- ...

Realizing these requirements usually imply pretty different cost and delivery time.

2. Where to find

Figure 2. Where to find.

This is typical a market information attribute. Supposing that a project management software is needed to a company simple collectiong the companies that can provide such a solution can take time and energy. In this sense the so called App stores and Solution stores provide the advantage that all provider is easy to find. Although it is important to note that App and solution stores are usually technology dependent, as soon as solutions between two technologies have to compared, the situation can be difficult. 

3. Which fits 

Figure 3. Which fits.

Even if we find the possibly solutions and providers they are not necessarily too simple to compare. As objective product sheets are not always tipical to provide one should probably install all of the solutions try them out and compare them. It can be carried out if the technological circumstances are pretty similar, like at the example of a SharePoint App store there is always a possibility install a trial version. As soon as however more than one technological solutions should be compared, the situation can be fast complicated. Certainly there is a possibility to order pre-sales demonstrations, however they are not necessarily faster and the do not necessarily provide an objective view.

4. Too big

Figure 4. Too big.

Sometimes the solution is too big for the requirements, meaning that there are a lot of additional requirements that are not needed, despite provided by the solution. As an example consider that we need a simple document management software, as SharePoint can be a natural choice we might as well choose SAP as even in SAP there are some document management functionality. If we do so, we get a big dinosaurs that have a lot of additional functionality like full modules of human resources or finance. It implies of course an extreme increase both in cost and in prices.

5. Too small

Figure 5. Too small.

Most of the software solutions are not just simple products but they contain the possibility to extensions. This can be however dangerous : if the extension is too big than usually it is worth better to implement everything from scratch or get another product. As an example it is certainly possible to set up in SharePoint a full CRM system. Instead of a deep extension of SharePoint however, it might be a better idea to a buy a dedicated CRM solution, like Dynamics CRM. As a thumb of rule, if more than 30% of a software product have to be customized it is a good idea to evaluate alternative solutions as well.  

6. Delivered in parts

Figure 6. Delivered in parts.

It is always a drawback if the solution is delivered in parts. Like infrastructure, application layer, application on the top of an application layer and perhaps consulting and training on the top.The different parts are usually delivered by different experts, even by different companies so they surely implies a increase both in time and in cost. As an example, cloud solutions, like Offie365 for SharePoint, have both the advantage that they are cost efficient and that the delivery time is null as the infrastructure is practically ready.  

7. Lack of extensibility

Figure 7. Lack of extensibity.

This is probably the most difficult aspect. Most software solution should not only be used today but in the future as well. It is usually not clear which requirements should be covered in the future, as the technology might not even exist. On the other hand most software solutions can be extended only in special directions. As an example if we want to have a portal solutions that we want to use in the future for internet of things than Office365 is not necessarily the best solution.

Notes on software market efficiency

Well I am sure that there is a lot of articles about general market efficiency theory analyzing in details offer and demand. However I would like to summarize some thoughts in a little more practical way focusing on the specialties of the software development market. 

Let we assume that we have a company that want to buy a certain software solution. The required software solution probably has got a some requirements or specification, so let we assume that software solution must have the following requirements : R1, R2, ... RN. We can furthermore assume that there is are certain solutions that covers all of the requirements. It sounds probably a strange assumption however in the software industry there are a lot of more or less general application development framework and even if they are not enough you can make a custom development from scratch. So, even if it sounds surprising we can say that in the software industry for all requirements there is a software solution. It is certainly a question how much a solution cost and if the cost is acceptable for the company. Let we assume however that price is good enough, probably not the cheapest but among the solutions that are cheap enough. 

This ideal combination is demonstrated by the following picture :

Figure 1. Ideal Software solution

So let we assume that such good enough solution has a price and a time in which it can be set up.

- Prise(Solution [R1, R2, ... RN]) : price of a good enough solution that covers requirements R1... RN

- SetupTime(Solution [R1, R2, ... RN]) : setup time of a good enough solution that covers the requirements R1... RN.

These are certainly ideal values, in certain projects both the setup time and the solution cost are heavily increased due to the following facts:

- Collection internal requirements in a company usually takes time and money. 

- Evaluation different solutions of different providers is not always straightforward. 

- Setting up and rolling out a solution usually requires internal organisations resources.

As a conclusion the real cost and setup time of a software solution might be significantly higher:

- TotalCost(Solution [R1, R2, ... RN]) - the total cost starting from the requirement analysis until the software is really used from end users in the company.

- RolloutTime(Solution [R1, R2, ... RN]) - the total time starting from the requirement analysis until the software is really used from end users in the company.

From the definitions above we can define two efficiency measures :   

- CostIncrease[R1, R2, .. RN] = TotalCost(Solution [R1, R2, ... RN]) / Prise(Solution [R1, R2, ... RN])

- TimeIncrease[R1, R2, .. RN] = RolloutTime(Solution [R1, R2, ... RN]) / SetupTime(Solution [R1, R2, ... RN])

The ideal case CostInrease and TimeIncrease around 1 means that the software solution is almost buoght as ready product from the catalog. The requirements are well known, the product is well known as well; setting up is practically increased by the delivery time. The opposite case if CostIncrease and TimeIncrease are bigger then 10. In this situation probably the requirement analysis itslef is a difficult tasks, rolling out the solution in the company is probably more difficult.    

In this sense efficiency of a specific M software market can be characterized by the following two measures:

- ACM = Average (CostIncrease)  

- ATM = Average (TimeIncrease)  

As exactly calculating these values on a certain market would require a significant amount of research time, despite they can be used intuitively making predictions on different situations:

- If the market is well informed then ACM and ATM should be tendentiel smaller as opposed to a market where the information is difficult to find. 

- On a well-known software market like word-processing softwares, ACM and ATM should be smaller as on a relative new and voluntile market as big data.

- Cloud market should have smaller ACM and ATM as on-premisse softwares.

Stock market bubble versus technology hype cycle

Are they really two different things ?

Bull market strategy on IT markets

Well, perhaps it is a funny thing to regard the whole information technology market similar to the stock market, however in certain situations a lot of technology works as stocks : the technology is going up, the technology is going down, the technology creates bubbles overvalued and so on.
In this sense, it certainly makes sense to make an analysis in analogy with a classical stock market.

It it important to note however that there are certain differences, I would say the market itself is not so perfectly informed, much less perfect, the changing them self are as well happening on 1-3 yearly basis and not on a daily basis (certainly exception if you only invest only on stocks of the different it companies that are on the market).

So let we see some of the IT fields that can be regarded at the moment as Bull Markets:
- Industry 4.0
- Internet of things
- Big data

So how it is possible to play on a bull market. If I say play, I actually mean to invest, so you actually want to make money with the technology:

1. buy stocks of public companies that deal with the technology

2. build up the competences for your own company (and sell, like for instance project basis):
    2/a. get yourself competence on the upcoming technologies
    2/b. recruit people who have the competence
    2/c. find consultants who have the competence
    2/d. find partners who have the competence
    2/e. buy companies that have the competence

3. Create product
   3/a. create products that focus on some aspect of the technology
   3/b. create framework, related to the technology, that help for other companies to do something faster, better, cheaper,....

4. Recharge old technology
   4/a. create migration tools that upgrade somehow the old technology to the new one

From a timeline financial perceptive, the proposed solutions are not the same, As No. 1 can be very fast staring from one day, typically No 3 and 4 has got at least a year horizon. The financial perspective is pretty much similar, as for instance buying a stock or finding a partner with a competence can be pretty cheap, buying a whole company usually means more money.

The following table summarizes the possibilities:

Rapid application development versus product development

Rapid application development versus product development from the customer perspective:

A social method for software development

If you consider software development there are basicaly different models of doing it like, 

the types of classical waterfall, V or W models and of course the different agile methodologies, ranging from extreme programming to SCRUM. 

All of these methodologies suppose however that there are some kind of a more or less explicite spefication what should be develped. 

Considering the waterfall model the specifications should be explicitely given as a set of requirements to start the development itself. 

Having developed the reqirements there might be one more round to extend both the specification and the covered the changed requirements. 

Agile development focuses much more on the changing of the specification, having more iterations and more check and validation of the requirements. 

Agile development usualy does not suppose very strongly that the reqirements covered in the form of a precisely described specification, 

but it is supposed that there is at least someone like a product owner who has got an explicetly view what should be developed. We can say that with agile methodologies, 

even if it is not written the product owner knows exactly the specification. 

What happens however if collection the specification in an organisation is problematic, 

if noone is able to act as an efficient product owner because getting information from the members of the company would require too much time and effort. 

Not to speak about writing a precise specification that would be much more complicated. 

Is there a way in such a situation to carry out an efficient software development ? 

Is there a way to have a software development methodology that actually supports collecting pieces of the specification itself ?

I think the answer is yes. 

Let we imaging that we do not want to have a development in which everything has to be developed from scratch instead we are dealing with components that are not necessarily ready but they are at least in a prototype that can be somehow demonstrated. 

Let we imagine that we have (C1, C2, C3, ... CN) prototype or demonstrable components in our system and we provide user interface called metamodel in which the users or customers can configure demo applications and like them or dislike them based on different like mesaures. 

Like supposing one customer wants to have only the (C1, C3) components in a certain configurations another customer wants to have only (C2) components and the third one only (C2 and C4). 

The system that should we realised is the combination of (CN...CM) components that are liked the most and makes sense to build it up. Certainly they are fuzzy concepts, how the components can be congigured and how the system can be build are depending on the exact domain and technology.

In this way creating the specification is more like a social network: people like on software components and configurations instead of posts or pictures (see the following picture). 

Having this model we should not sctrictly follow the classical principle : developing one software at the end of the process. 

Several other outcomes can be imagined as well :

- One software at the end (one size fits for all): based on the likes and techniqual restrictions we still choose one model to implement. 

- Several variations: we choose some of the most liked solutions and implement as several variations, from which the userers might choose one on their own. 

- Self service software creation (mass customisastion): if the components are not just prototype but practicaly 100 percent ready solutions, there might be a possibility to provide everything and the users can configure everything on their own.

Let have an example, realised in SharePoint.    

We want to develop personal records in a company for storing and showing information for each person. We are however not very much sure exactly which information should be stored. 

Let we say, first name and last names are for sure but othwervise noone knows which metainformation should be used. 

Let we have the following simple meta factory with the following metainformations: 

- ObjectName - name of the object that we want to store, in our example there is only one object called PersonalRecord.

- PropertyName - the name of a property of an object.

- PropertyType - the type of a property of an object.

- ProeprtyLike - information who wants to realise which property, for example in our example there is only one property that wants to be seen only by "test" user, there are only two properties that wanted to be implemented by "test2" user : "FirstName" and "LastName" and so on.

In this sense the components are practicaly the objects and property elements (see the following picture).

Pressing a preview button each customer can see how the choosen configuration could look like, in this sense they do not just choose blindly, but there is a certain view, a certain idea how the choice might look like. As an example having the administrator choice, the given configuration might look like as follows. 

It is important to note that the preview should not necessarily be the final version; it can be regarded as a sketch as well. Like in our example the concrete design is pretty poor, however in a final implementation it can be much more fancy.

​

Based on the likes and users' preference there should be a review and a decision, like: 

- implementing some of properties, mostly that are liked by the most and create one common profile

- implementing more property sets, so the users can choose which one they want to apply

- or simply implementing everything 

8 Years SharePoint development summarised

Thoughts on economics

Limits of Agile development in SharePoint Field

Agile development has been increasingly popular during the last couple of years.

Independently if we speak about a strict SCRUM or extreme programming methodology or something more flexible agile style development, common characteristic of the development is to have:

-          Strong contact with the customer.

-          Frequent and multiply delivery (like sprints at SCRUM).

-          Flexibility of changing requirements even in late development phase.

-          Lightly formulised specifications, like with the help of user stories.

-          Certainly agile projects have some other characteristics as well like self-motivated self-developed teams however they are less important in the scope of this article.

Certainly Agile development has got a lot of advantages and provide a much higher customer satisfaction as opposed to the classical waterfall V or W model in application development. Despite we think that there are several practical facts drawbacks that should be considered.

-          Complexity of change: As it is pretty simple and fast to develop a use case at the beginning of the project, it is usually much more difficult to do it later. The problem is that at the beginning you can develop everything from scratch however later on consider the existing code and the existing architecture of the system. As an example let we imagine a .NET, SQL MVC development that has got some integration with SharePoint as well. At the beginning of the development it is pretty simple to implement user stories as rules in the business logic, however if you get the requirement only at the end of the development that you should do everything in a transactional way that means that all of your business logic should be reconstructed which might mean rewriting a couple of thousands of line of code retesting the modules and so on. (Just from the deep technical side it is not necessarily so simple that you rewrite everything with TransactionSoce as SharePoint does not provide transactions, so you have to probably manually implement a 2 phase or 3 phase transaction protocol and use overall).

-          Project budget: Project budget is always final. Even if you can argue that a new sprint a new round is necessary and flexibility had its price as well, our experience that you should deliver functionality in a meaningfully budget. Even if it is accepted that the new functionalities cost more they should be in a meaningfully limit. As an example it is usually a not accepted that a implementing a new requirement or user story costs 2 or 3 times more as the whole exiting development. So as a conclusion, even if a little bit more flexible project budget has been accepted, you have to deliver at least a core functionality within a certain budget.  

-          Time for delivery: Same is true for the delivery time. The project should have some functionalities to be delivered that usually have some time constraint. Even if there is always a chance to negotiate one more sprint or one more agile cycle delivery time can usually not extended without limits.  

-          Technological limits: In most software development scenarios you do not develop everything from scratch, despite you use one or more frameworks to make the development faster. All of these frameworks have some technological limits that should be considered at the choosing of the framework. Identifying these limits are sometimes challenging as they are not necessarily strict and documented values; they are sometimes soft limits, best based on experience. Best example the well-known SharePoint strict and soft boundaries. Another example can be .NET entity framework for reaching standard relational databases. Entity framework is pretty much liked from .NET developer to access an SQL database because practically the whole database is mapped by C# objects. The drawback however that at each command part of the database has been queried, it is brought to the client side and after making some transformation usually written back to the database. Although it is easy to use for a C# developer it is much less performant as direct SQL commands or stored procedures. As a consequence technological limits of entity framework are reached if complex and fast database operations should be carried out or one has to move and transform a huge amount of SQL data.  

Our experience that most agile project in SharePoint field follows the following pattern. We analysed the cost of the development in the complexity of specification (Figure 1).

Figure 1

As cost of development we mean the following two factors:

-          Licence costs of the used technologies or software frameworks.

-          The amount of cost of the development itself.

At complexity of specification, we practically mean the number of user stories. We suppose that the project is agile so the complexity of specification is highly correlated with the time axis of the project.  

As examples carried out, we consider three different use cases in which similar user stories are covered by three different kind of software architecture. As a bottom line of all use cases let we try to realise a small customer relationship managing software. We are going to have several user cases from the domain like, I want to store somewhere my customers, I want to store some detailed information about my customers, I want to manage contact persons at each of my customers, I would like to see like as a reports what I for each of my customers sold, and so on and so on….

-          Use Case 1: Let we imagine that the target software is Office 365, we try to develop the specification as Lists, Document Libraries and some JavaScript extension, everything is covered by the SharePoint online platform.

-          Use Case 2: In use case 2 we try to develop everything with the help of the K2 platform. K2 black pearl SmartObjects for the business entities and K2 SmartForms for the user interface components. Storage of SmartObjects are either realised by SharaPoint lists or by simple tables of a Microsoft SQL server.  

-          Use Case 3: In use 3 we build up everything with the help of ASP.NET MVC Framework using SQL server for storage and Entity Framework for communicating with the database.  

Each project has got a set up time, at this period infrastructural components are installed, licenses are paid and other kick off and project start special tasks are carried out. During this set up period the cost usually increased pretty strong however relative small part of the specification is delivered. At the example of Use cases the set up time covers the following cost factors:

-          Use Case 1: in the Office 365 example the set up time and cost is pretty much reduced. It basically means buying some licences and setting up the first site collections through the administration portal.

-          Use Case 2: in the K2 example the set up time and cost is much higher. On the one hand license  cost are much higher as at the office 365 example, on the other hand installing a full K2 environment requires time and expertise, especially if it is should be high performant or high available.

-          Use Case 3: in the MVC.NET example set up time is somehow in between the two previous ones. It usually requires setting up some windows server with IIS and SQL database and some classical developer tools like source control.  

Having set up the development project there is usually a pretty efficient development phase. In this phase user stories and requirements can be carried out very efficiently. This period gives the chance to do a really effective agile development: flexible changes or add user stories, deliver in short iterations, have a lot of interaction with the end customer and use any kind of agile template that you want, like SCRUM, extreme programming….

-          Use Case 1: in the Office 365 example this period is typical to create content types lists and document libraries, sites or pages, implementing some design, changing the item or edit item page of some elements and so on. Some these changes can be carried by information workers in minutes or hours others require some development knowledge take days to realise.

-          Use Case 2: in the K2 example the effective period means defining a business logical layer by SmartObjects and creating user interface elements by SmartForms. As only information worker with high K2 knowledge is required the development of new user stories can be very fast in this period.

-          Use Case 3: in the MVC.NET the effective period means setting up database elements integrating with Entity Framework, setting up business rules and realising user interface components by views and the related UI logic by controllers. Realising user stories in this example requires developers, sometimes not just one but more, like one for the database side one on the business logic side, one on the controller part and perhaps a designer as well. As more than one people taking part in the development you can not avoid some kind of a project management either. As a result carrying out a new functionality always takes a lot of time in comparison with the two previous approaches.    

Unfortunately each technological or architecture choice has got its limits. If you reach that point than new functionalities to deliver will take a lot of effort. Certainly you can change your architecture or technology or extend in some sense, however that basically means that even with the best integration you should cover once again the set up time, perhaps recruit new people with new technological competence. The scenario is worst if you do not clearly extend or modify the architecture but you realise some fast hacks. That usually bring some instability into the system resulting on a long run a lot of efforts in system stabilisation and hunting bugs that only sometime occur. At any case reaching the architecture limit means that the costs are going to explode; you deliver very small amount of user stories or requirement change very slow and expensive.

-          Use Case 1: Office 365 is a pretty good example for architecture limits as it has got a lot of limit. Things like number of elements in a list limited to 5000 or the number of possible lookup fields on a 13 basis very fast result that you have problem with the architecture. In some cases if you just slightly exceed the limits there is some possibility to making some small hacking like storing items in two lists, or reconstructing your data model to minimise the number of lookup fields but in most situation you can not avoid to rethink your architecture. As an example there is a chance to set up an on-premise environment and migrate everything, or you can store some items in Azure and integrate somehow, but at any case both will cost much more and will take more time as realising the requirements staying within the architecture limits.    

-          Use Case 2: The K2 framework is much more flexible in a lot of senses, despite you can reach soft and hard product limits if you integrate with special technologies. Like let us suppose that you build up a lot of K2 SmartForms and Processes to create a lot of self-service possibilities to your customer like ordering a product, giving feedback, setting payment options and so on. After that you want to provide the same functionality with the same automatic processes on a telephony system as well. As there are some possibility to extend the architecture to integrate a third party telephony system you will not really be able to use the workflow engine with a telephony system as the engine itself does not guarantee time constrained response time. You probably need to buy a totally different framework and on that rebuild all of your processes that you want provide through a telephony as well. As it is certainly possibly it surely implies a boom both in the costs and in the delivery delay.   

-          Use Case 3: in the MVC.NET let we imagine that we reach the limits of the entity framework. Due to the complexity of business logic and data oriented computation the application will perform very slowly. As a result the whole data access layer and business logic layer should be reconstructed and put everything that is possible into the database as stored procedure. As it is certainly possibly, it will again result a strong increase in the development cost and prevent the fast delivery of the next release.  

Figure 2 demonstrates 3 different technology curves. They may vary how much time takes the setting up, how effective is the middle part and of course where are the technological limits.

Figure 2,

As a conclusion we propose to use agile methods with caution. At our experience:

-          Agile methodology works well only in a certain architectural framework.

-          Exceeding the architectural limits results a boom in development cost and in delivery time. Surely architecture change can be realised in an agile way as well, however due to its cost and time nature it is usually not accepted by the customers. Plus consider the psychological aspects as well, like you deliver the first 20 user stories as 2 days each, however for the 21 th you should say it is 100 days – to effectively communicate something like that is not simple.  

-          Even if it is agile, more user stories and more requirement analysis should be carried out at the start phase as at the later phases. Especially user stories and requirement elements that influence architecture decision should be specified as fast and as early as possible.

-          It is sometimes a good idea to demonstrate to the customer the chosen architecture and the limits and drawbacks as well. You can formulate some negative examples as well, like requirements or user stories that will reach the architecture limits.

-          It is always a good idea to have a good architect.

I think actually the summarised ideas are pretty much trivial, I was personally not sure if it worth time to write them down. Actually software development has got some similarities with the building industry. It is always possible to paint a living room in an agile way and of course if it is not liked by the customer it can be repainted each day with a different colour or perhaps more than with one colour. However it is not possible to do agile development (or agile house building) if it is about the number of floors. You have to exactly specify at the beginning if you want to have 1 floor, 10 floors or 100 floors. A house planned to have 1 floor will never have 10. It has to totally destroyed and start building from scratch.