The future of SharePoint == Cloud Microservice Integration and Development for Business Productivity

The future of SharePoint == Cloud Microservice Integration and Development for Business Productivity

...

The future of SharePoint == Cloud Microservice Integration and Development for Business Productivity

...

The future of SharePoint == Cloud Microservice Integration and Development for Business Productivity

...

The future of SharePoint == Cloud Microservice Integration and Development for Business Productivity

...

8 months MatchPoint / Snow experience summarized...

8 months MatchPoint / Snow experience summarized...

Is it time to "sell short" on the on-premise SharePoint market ?

Well current trends show that on-premise SharePoint market is going down. Microsoft has been being communicating for years that there is only Office 365 planned on the long run. As still there is a version for SharePoint 2016 on premise, but a lot of integrated functionalities like MySite, OneDrive or search are provided rather (or only) in different hybrid cloud scenarios. Hence developer certifications are cancelled as well; there is only Office 365 in the future. It basically means that the SharePoint on-premise market will go down, not because there is no demand for the technology, just because Microsoft communicates so strongly and consequent: Firstly the experts and the partner companies will change expertise, concentrating rather on other technologies than SharePoint. Secondly, the end customers will recognise as well, even if there is a demand for the technology, there is not enough expertise on the market, there is not enough support from Microsoft and last but not least, most of the cool new features are only available in the cloud. So the market is falling...

So, what is gonna happen ? Certainly it is possible that everyone changes to the cloud, although I think there are gonna be companies that still want to make sure to have the data privacy. Perhaps there is gonna be sometimes a so called "black" or highly secured cloud, where really nobody is capable to get the data apart from the owner, but until only the standard scenarios exist, the only option will be for companies who want to store high privacy data is to build and host a private cloud, that is gonna be pretty much expensive.    

So is it possible not just quitting from the market, but really "sell short" in a stock exchange sense, meaning betting on the falling of the market and trying to make capital gain from the fall ? Well it is not typical to interpret "sell short" for a technology cycle, but I think it is not impossible. I can imagine two scenarios, focusing on the customers who still want to use internal collaboration and intranet solution with high data privacy and they are having SharePoint: 

a. Creating regional private hosting solutions, to host Office 365 private: well as it is certainly a solution, it is actually not sure how much and which scenarios will be from Microsoft supported. As it is surely possible to host in a private cloud Office 365, I am not sure if all functionalities or integration possibilities are supported. As an example, what about Azure Data Mining services or Cortana analytics ? It is pretty much questionable if something similar will be supported in a private cloud scenario. 

b. Start to rechange SharePoint with another technology, that surely will develop on-premise scenarios in the future as well. The process and the idea can be actually the same that Microsoft uses to attract SharePoint companies in the cloud. 

Let we find such a T technology that:

   - T can be good integrated with SharePoint

   - T provided on-premise solutions on a long run.

   - T provides collaboration, corporate intranet functionalities and possibly rapid application development capabilities a well.

   - T provides useful extensions to SharePoint. 

   - you can scale up and out T.

   - on a small scale T has got a moderated entry price.

   - T should not be from a no-name provider, otherwise nobody buys it.  

   - start sell to T with SharePoint integration in a way that new SharePoint functionalities are fully or partly realized on T.

   - As the SharePoint on-premise market falling start to re change or re implement existing functionalities  on T with or rather without SharePoint integration. That is the reason why rapid application development is important, you will never have the chance to get alive your SharePoint 2010 source code on another platform, you have to re implement the business applications. 

   - At the end, migrate everything to T, forgetting SharePoint.

Does anybody know T ?  

Notes on digital workplace and the visual models behind

There is an ongoing direction both from Microsoft in the direction of SharePoint and from other companies as well to speak about digital workplace. The question that is not hundred percent clear what does digital workplace exactly mean ? I mean I am pretty much sure that there is a lot of definition for that but from a practical point of view I would imagine a classical at least half digital workplace and I would imagine that digital version of that. So than basically the question is what is out model ?  Which workplace do we want to imagine as a digital workplace for the first run ?

Visual Model 1: Is our model something similar than bellow ? Than we speak about a software that supports digital documents, printing perhaps some extended functionalities like approving workflows and if we are very creative some digital brainstorming methods or electronic post-its.  

Figure 1. A workplace that can be imagined as digital.

Visual Model 2: Or is our model rather something bellow ? If so than the requirements are however much higher, than we speak about real time and high availability systems, mission critical system integration probably real time operation systems and basically everything that our state of the art IT can provide...

Figure 2. A workplace that can be imagined as digital.

WHAT IS OUR BASIC MODEL FOR DIGITAL WORKPLACE ?

Amortization of a software technology

Considering the current technological improvement, most of a technological knowledge will be obsolete in a couple of years. One of the best example is SharePoint development. In 2008 mostly ASPX and server side knowledge was required, in 2010 Silverlight, in  2012 Web Service Integration was the focus in 2014 mostly JQuery and Apps and actually who knows what comes in 2016 ? (I guess Angular JS is the new trend but who knows how long),In this sense it makes sense to define the amortization of a technology either in a sense of how much effort does it take to keep the knowledge up to date, or in a sense that how fast will be an existing knowledge obsolete.

Amortization of a software technology: 

Definition 1: the average amount of time until an exiting technological knowledge gets obsolete.

Definition 2: the amount of energy that needs to be invested to keep the technological knowledge up-to-date.

In this sense we can speak about technologies with high amortization rate and technologies with low amortization rate. Typically technologies that are pretty knew or at the beginning of the technological curve, like in introduction or early growth phase, have got high amortization rate. On the contrary, technologies at the maturity or late growth phase usually have got low amortization rate.

Comparing MatchPoint with SharePoint Custom Development based on Agile Technology Curve

If we want to compare a classical SharePoint development with a MatchPoint development, the easiest way is to consider the following diagram on comparing a classical development life-cycle based on the complexity of specification and the cost of development.

Figure 1. Comparing MatchPoint with classical SharePoint Development.

From a technological perspective each project has got at least three stages:
1. Setting up the environment (until Q1)
2. Working within the technological limits (between Q1 and Q2)

3. Reaching the technological limit. 

Comparing MatchPoint with SharePoint custom development based on a agile characteristic curve we can conclude the following things. 

Phase till Q1: Setting up time and cost is higher at MatchPoint as with SharePoint custom development, as MatchPoint has to be licensed and installed on the top of SharePoint. 

Phase between Q1 and Q2: Agile Phase at MatchPoint is much bigger as with SharePoint. A lot of taks can be carried out simply by carrying out XML configuration tasks avoiding the pretty much resource intensive tasks, like JavaScript or C# developments. Typical scenarios that are supported by MatchPoint are for example: search based applications, tagging and ontology based applications, forms, workflows and simple case based applications.   

After Q1; After reaching the technological limit development both for SharePoint and for MatchPoint will be much more difficult and much slower. MatchPoint extends in some sense the technological limit of SharePoint, as an example extended tagging mechanism is feature that is not very much possible to realize with SharePoint. In this sense MatchPoint not only provides an easy configuration possibility to speed up agile phase of the development but it extends the architecture limit as well. 

Notes on self adaptive SharePoint Environment

Recent trends in Data-Mining software solutions are going in the direction that data-mining and intelligent analysis solutions are getting cheaper and cheaper. It actually does not only mean the price of the software, but the ways of integration the different solutions are getting easier and easier as well. This provides new possibilities in the classical software and application development as well.

Having the possibility of collecting data about a running application, like application usage, logs, infrastructure data, and having the possibility to analyse them with data-mining tools, provide eventuelly the way of proposing a application improvmenet. Based on data measured and cheap data-mining possibilities a general SharePoint application development process can be defined as follows:

0. Analyse: This one is a classical requirement engineering step, analysing the requrements to set up an initial application. However, the requirements should not only be collected by interviewing the stackeholders, there is a possibility to analyse existing documentations or data, that might also be supported by data-mining. 

1. Setup: This is the classical step for setting up the system, planning and installing infrastructure components, developing and delivering custom solutions.

2. Collect data and usage: SharePoint collects pretty many data out of the box, like standard log files, search or usage and health data. As most of these pretty much infrastructure oriented it is important to measure data about certain application usage as well.

3. Analyse: The collected data has to be by different data-mining tools analysed.

4. Propose new structure: based on the data and analysis new application structure can be proposed. The improvement might be only infrastructure oriented to achieve a better performance, however completely modified use cases or business processes can be defined as well.

Repeat from step 1: The process can be actually repeated from the beginning, the new structure or can be again set up and the application usage can be again measured and analysed.

Integrating MatchPoint with SAP (a step by step initial approach)

Integrating MatchPoint with SAP is pretty easy as a first approach. You have to do two things: 

- get an SAP connector to read out some information from SAP

- implement a data provider in MatchPoint with the SAP adapter to use the imported data.

So let we have the steps in details:

STEP 1. Let we assume that we want to see the controlling area names from SAP controlling:

Figure 1. List of controlling areas in SAP.

STEP 2. Let we create some integration with SAP, there are many ways of doing that ranging from BAPI calls to web service integration through Netweaver. I m choosing the most simple SAP .NET connector. 

STEP 3. Let we create a Visual Studio project console application and reference the SAP.NET connecor dlls: sapnco.dll and sapnco_utils.dll. I am probably using a little bit old version, because for me it was working only with .NET framework 4.0 and not with 4.5 or higher. 

STEP 4. Let we create a basic connection class named SAPSystemConnectMP to store the SAP connection information, like port, system number, access information...:

    public class SAPSystemConnectMP : IDestinationConfiguration
    {
        bool IDestinationConfiguration.ChangeEventsSupported()
        {
            return true;
        }
        public event RfcDestinationManager.ConfigurationChangeHandler ConfigurationChanged;
        RfcConfigParameters IDestinationConfiguration.GetParameters(string destinationName)
        {
            if ("K47".Equals(destinationName))
            {
                RfcConfigParameters parms = new RfcConfigParameters();
                parms.Add(RfcConfigParameters.AppServerHost, "192.168.137.238");
                parms.Add(RfcConfigParameters.SystemNumber, "00");
                parms.Add(RfcConfigParameters.User, "root");
                parms.Add(RfcConfigParameters.Password, "***");
                parms.Add(RfcConfigParameters.Client, "800");
                parms.Add(RfcConfigParameters.Language, "DE");
                parms.Add(RfcConfigParameters.PoolSize, "50");
                parms.Add(RfcConfigParameters.MaxPoolSize, "100");
                parms.Add(RfcConfigParameters.IdleTimeout, "6000");

                return parms;
            }
            return null;
        }
    }

STEP 5. Let we create a wrapper class for the ControllingArea, with the static setup procedure that calls the BAPI_CONTROLLINGAREA_GETLIST Bapi and from the result takes the CONTROLLINGAREA_LIST table and lists all controlling area name and code.

   public class ControllingArea
    {
        public string ControllingAreaCode;
        public string ControllingAreaName;

        public static List<ControllingArea> getAllControllingAreas(RfcDestination destination)
        {
            List<ControllingArea> ret = new List<ControllingArea>();

            RfcRepository repo = destination.Repository;
            IRfcFunction controllingAreaList = repo.CreateFunction("BAPI_CONTROLLINGAREA_GETLIST");

            controllingAreaList.Invoke(destination);
            IRfcTable controllingAreas = controllingAreaList.GetTable("CONTROLLINGAREA_LIST");

            for (int cuIndex = 0; cuIndex < controllingAreas.RowCount; cuIndex++)
            {
                controllingAreas.CurrentIndex = cuIndex;
                ControllingArea area = new ControllingArea();
                area.ControllingAreaCode = controllingAreas.GetString("CO_AREA");
                area.ControllingAreaName = controllingAreas.GetString("Name");
                ret.Add(area);
            }
            return ret;
        }
    }

STEP 6. Let we implement a MatchPoint Data Provider, of course after referencing the MathcPoint dlls, possibly from the development kit : Colygon.MatchPoint, Colygon.MatchPoint.Server and Colygon.MatchPoint.Snow.

    [Serializable]
    public class SAPDataProvider : BaseDataProvider
    {
        public string[] List;

        public override BaseDataProviderInstance CreateInstance(string cacheKey, IEnumerable<string> columnNames)
        {
            return new SAPDataProviderInstance(cacheKey, this, columnNames);
        }
    }

STEP 7. Last but not least let we implement a data provider instance. Perhaps the most important part of the instance is the GetInternalData function. It opens an SAP connection through the configured connection class, queries all the controlling areas and transforms the names to a simple string list.

   public class SAPDataProviderInstance : BaseDataProviderInstance
    {
 private readonly SAPDataProvider provider;

 public SAPDataProviderInstance(string cacheKey, SAPDataProvider provider, IEnumerable<string> columnNames) : base(cacheKey, columnNames)
 {
  this.provider = provider;
 }

 protected override CachePolicy CachePolicy
 {
  get { return new CachePolicy(CacheGranularity.NoCache, 0); }
 }

 protected override IEnumerable<object> GetInternalData()
 {
            SAPSystemConnectMP sapCfg = new SAPSystemConnectMP();
            try
            {
                RfcDestinationManager.RegisterDestinationConfiguration(sapCfg);
            }
            catch(Exception ex){}
            
            RfcDestination rfcDest = RfcDestinationManager.GetDestination("K47");
            List<ControllingArea> cAreas = ControllingArea.getAllControllingAreas(rfcDest);
            List<string> res = new List<string>();
            foreach (ControllingArea area in cAreas)
            {
                res.Add(area.ControllingAreaName);
            }

            provider.List = res.ToArray<string>();
            return provider.List;
 }

 public override string GetCacheToken()
 {
  return null;
 }
    }

STEP 8. Having compiled,we have to registrate everything in GAC (the SAP connector DLLs as well) and we have to registrate the DataProvider in the matchpoint configuration as an external assembly and of course IISreset at the end.

Figure 2. MatchPoint Configuration.

STEP 9. If everything went well, you have to see the new dataprovider in your system. So as an test we can create a composite WebPart setting the SAPDataProvier and showing with a pattern transformer the DataItem.

Figure 3. Composite WebPart configuration.

STEP 10. If everything wen well, you should see the list of Controlling Areas directly from SAP.

Figure 4. Controlling area information in MatchPoint from SAP.

On IT and SharePoint market segmentation

It is interesting to investigate both the IT and generally SharePoint market as well based on the following two dimensions: Market segment and customer flexibility (Figure 1). 

Figure 1. SharePoint market segmentation based on market segment and customer flexibility.

Market segment is basically the standard dimension that is usually described, we can scope from a simple business to customer market to a different business sizes ranging from small business to the enterprise.

The second dimension is perhaps a little bit less standard. It describes how much flexibility do we want to provide for our customers. The simplest way a basic product, that provides some functionality without a too much customization or extending possibility. It is like, if you buy a packed sandwich in a supermarket, you can choose which one do you want to buy, you can not really pack out your sandwich and fill with mayonnaise instead of ketchup. 

The next stage is products with the possibility to heavily customize. They are like eating sandwich at Mc.Donalds, you have the possibility for some variation or customization, however the possibilities are limited. But you can surely choose if you want to have an extra polonaise or ketchup.

Frameworks are not actually 100% ready products but some general building blocks that help you deliver a lot of different products in a fast and efficient way. It is like Subway, you do not really have ready sandwich,  but rather a coupe of basic foods and processes that help you to get a ready sandwich. 

Of course, the last stage is to have fully customized project, you can precisely describe each part of your product. It is as a 5 star restaurant where you can have a small appointment with the chef, describing what kind of a sandwich do you want to eat.

Certainly the bigger customer flexibility means as well that more preparation, more professional stuff are required and of course the whole process can be less automated. As a consequence more customer flexibility implies an increase in cost, so it is not surprising that IT provider companies offering more flexibility to the customers concentrating rather on the enterprise segment.

The SharePoint market can be characterized this way as well: 

- IT companies having small boxed products selling the product mainly as cloud solutions with self service buying possibility and primarily on the B2C market. Typical examples are SharePoint apps.

- As soon as a product has got a a bigger range of customization possibility a certain amount of consulting and pre-sales activity is practical to be delivered as well. Despite the product can be cloud hosted, but the whole sales process is not based on an online shop on a "buy if you want" basis. Such products are for example special Outlook or Mobile components that communicate somehow with SharePoint. 

- It is difficult to distinguish between a product and a Framework. I would say as you are having more and more possibility to customize, more and more possibility to further develop than it is better rather to speak about a framework than a product. The best examples are BPM frameworks like Nintex and K2, however SharePoint itself can be regarded as a general collaboration framework as well.

-  If everything has to be newly invented and developed, than it is better to speak about a 100% custom project. Certainly such projects are pretty much time and resource intensive: time and resources are required for specification for testing and quality for project management and so on. As an example .NET development from scratch can be regarded as custom project. 

Notes on SharePoint hybrid environments

Considering the new directions of SharePoint from Microsoft, I would note a couple if things:

- Building up hybrid technologies (partly cloud, partly on-premisse) makes SharePoint environment more difficult as ever. You do not only need SharePoint infrastructure, developer and business consultants, but cloud infrastructure and development experts as well.

- I am not sure if it provides a real value for an end-customer. I mean, a standard user needs some IT functionality that runs stable if it is on the cloud, on-premisse or hybrid does not really matter, at least not from end user perspective...

- From an internal CIO perspective having a hybrid environment does not necessarily make sense, actually with a hybrid environment you have both the problems of the on premisse and the new challenges of the cloud. Despite it perhaps make sense having a hybrid environment if there are some motivations to outsource only part of your IT infrastructure, like  for security reasons, or simply because clound is simply too new and perhaps too risky.

Notes on SharePoint Apps versus SharePoint Addins

Recent change from Microsoft that SharePoint Apps will be renamed as SharePoint Plugins. As it is theoretically only just a new name, despite it is interesting to a look which name describes the technology better. 

App Framework for mobile devices means something in the direction that you have a got a basic framework, that is the core mobile device. It is extended in a couple of directions by Apps. The basic Framework provides among the others the following services:

  - Framework and access to the touchscreen
  - Phoning
  - Data Transfer
  - Geo location service
  - Access to other hardware devices
  - ....

The idea is pretty much similar to the standard operation systems and applications. The operation system provides most of the core functionalities and the applications simply extend these functionalities. The core functionalities are for example the followings:

  - Access to the file system

  - Access to IO devices
  - Memory management
  - CPU Resource management
  - ...

As a consequent, if we speak about SharePoint Apps, we would expect something similar, a core framework that is extended by the SharePoint Apps themselves. So the question is what should be regarded as SharePoint basic functionalities. Having more than 6 years SharePoint development experience I would say that at least the following elements should be regarded as SharePoint core services:

  - Integration on Authentication

  - Integration on Authorization

  - Workflow integration

  - Data model integration, like standard Lists and extended data model elements.

  - User interface and design integration

  - Integrated operation model, like integrated logs or monitoring 

  - ...

Some of these functionalities are covered by the SharePoint hosted App model, however almost none of them are realized by a provider hosted App. Provider hosted App model realizes only the integration of the Authentication, everything else has to be developed from scratch, like building up ta SharePoint like user interface, realizing the same authorization model or developing web services and custom activities for workflow integration.

As a conclusion, a SharePoint hosted App might be called really as an App, but for a provider hosted App the name is surely pretty much misleading. One would expect a model and core functionalities that do not exist. In this sense AddIn is probably a much better naming choice.

  

SharePoint evolution: from platform to product

Introduction.

It is pretty interesting to see the current evolution of SharePoint from an agile technology curve perspective. Briefly speaking if we consider a classical software project where we deliver services to a customer in a certain time frame, most project have got three typical phases:

1. Set up: most technology requires some kind of a setup phase, like installing the software, setting up the basics for the environment and usually buying licenses. This phase can be characterized with  increased efforts (setting up effort and license fees), on the contrary relative few direct user requirement are covered. This phase is typically realized by infrastructure teams.

2. Efficient agile phase: in this phase user requirements can be delivered fast and efficient. Use cases that are delivered here can really be regarded in an agile way, like without explicit specification, consulting with the customer on a daily basis, having sprints and delivering use-cases on a weekly basis. This phase is typically realized by small project teams of consultants and developers, usually working on site at the customer.

3. Architecture limit: Once the architecture limit has been reached further use-cases can only be delivered for a huge cost, usually needing to extend the architecture itself. In this phase agile cooperation can not really be efficiently supported, instead classical requirement engineering and product development are required. 

Figure 1. General Agile characteristic.

Platform versus product.

Comparing a classical product with a rapid application development framework from the agile technology curve there are major differences. Firstly setting up a product means delivering a lot of end-user functionalities as opposed with a framework where setting up usually means only a minimum scenario usually without covering too much end-user requirements.

Secondly, a product offers limited possibility for efficient agile development. There is certainly some possibilities to set some parameters but the situation can be very easily reached where the product itself has to be further developed, A rapid application development  framework is however planned to realize a long effective agile phase in which user requirements can be fast and efficient realized. 

From the architecture limit point of view there are not too many differences. If the limits reached than either the product or the framework itself has to be further developed that practically means slow and hard core software development.

Figure 2. Product versus framework compared.

Briefly formulate a framework is usually a general application development framework on which you can realize a lot different user requirements in an agile way with the help of a some IT consultants. A product is practically something ready, if you like it you get from the shelves and pay at the cash desk, if you do not like it, you look for another product. Extending the product itself usually realized only by the product manufacturer.

    

SharePoint evolution.

The last couple of years of SharePoint development was more or less going very strongly to the direction of Office 365. Even if there is still at least one version on premise, this version supposed to be similar with the cloud one. Besides, it seems that on a long run more or less the Office 365 will be the standard perhaps only on the cloud.  

From an infrastructural point of view Office 365 is similar to a normal cloud solution, it is practically a product. You have a couple of well defined services that you can order and buy right away and after some configuration the set up phase has been delivered. As it seems to be a good thing for the first side, however it is more or less the one size fits for all principle.You can build up perfect data models based on lists and document libraries supposing that you did not reach data view limit however as soon as you have reached there is gonna be difficult to extend, you are practically at the architecture limit Certainly provider hosted app and storing everything in an Azure database is a possible way, but then you should consider implementing the provider hosted app, the logging mechanism, the user interface, the integration of standard Office365 lists, search and workflows... it is a lot of work for practically zero functionalities. On the contrary with a classical SharePoint on premise environment there are some possibilities , like extending the data limit on the central administration, integrating BCS with SharePoint Designer and SQL Database, having some Visual Studio Development and developing BCS with like Oracle...

Similar limits are to banning the public pages, constraints on the cross site information, problems on the brand new workflow and so on.   

On the other hand the agile phase is definitive smaller at Office 365. There are smaller number of possibilities that one can click together, like the missing cross side or public page features. Hence the development styles are much more limited as well. Farm solutions are definitive missing, sandbox as well, as a consequent developing fast and efficient backed functionality is already not possible. Customization on the design or javascript side is still possible, however on the one hand only a limited number of functionality can be delivered as on a pure javascript basis, on the the proposed delivery way is to build an App that can be sometimes pretty much resource intensive. As soon as something more is required the only way is to implement a provider hosted App, however implementing a rpovider hosted App from scratch is pretty much resource intensive, one should consider implementing apart from the core functionalities, logging, SharePoint integrated user interface, SharePoint integrated data model, workflow and search integration. Such a development has got very much the characteristic of being outside of the agile phase: the development is slow, the requirements can not be changed too often, it is usually can not be sold in one project but the development has to be financed by several customer.

Conclusion.

Based on the agile characteristics it seems that SharePoint is appearing more or more in the direction of a product and not as a rapid application framework. There are a certain number of predefined components that you can by and configure but if your requirement is coincidentally not supported than you can buy another product or practically finance platform extension. Actually the naming conversion shows this direction as well: it is already Office 365 that implies that SharePoint is a similar product as Word, Excel or PowerPoint and not a Rapid Application Framework. Similarly future names of App are actually AddIn-s that also points into the direction of rather being a Product as a framework.

Comparing SharePoint on premise with Office365 based on agile technology curve

Agile technology curves provide the possibility to compare a SharePoint 2013 on premise scenario with an Office365 one. 

Set up time and cost (before Q1): Set up time with Office365 is pretty fast, one has to order the applications and functionalities and everything is ready in minutes. Supposing an on premise SharePoint scenario, everything has to be installed from scratch that takes of course more time. From the licensing point of view usually Office365 is cheaper than SharePoint.

Effective agile development (between Q1 and Q2): From an agile development point of view the two solutions are pretty much the same at the beginning phase, there is a possibility to deliver very fast standard configuration elements like lists webparts or document libraries, it is a little bit more difficult to configure html, css or javascript elements.   

Architecture limit (after Q2): The main different is the architecture limit. With SharePoint 2013 on premise there is still the possibility to extend the framework or integrate external systems with farm level development, sandbox, business connectivity services or custom service applications. Considering Office365 most of the framework extensions are  simply not possible; the only way to realize some extension is to develop a provider hosted App, which requires a high expertise and can not be carried out very fast. 

Figure 1. Agile technology curve of Sharepoint 2013 on premise and Office 365.

Technology curve for Agile development à la SharePoint

Let we see how the agile characteristic look like, if we use SharePoint as a framework to delivers use cases to our customer. 

Set up time and cost: Installing the infrastructure, setting up the service applications, creating the first web  applications and site collections.

Effective agile development: SharePoint has got several phases for agile development. Firtly standard elements can be configured in which period use cases can be delivered very fast. Secondly HTML and CSS elements can be configured; it requires more time and competence. Last but not least full JavaScript extensions can be developed. Actually the situation is more complicated as there are some extensions, like business connectivity services that help in creating external system integration pretty fast.

Architecture limit: If limits are reached the architecture has to be extended or external systems have to be integrated like with the help of farms solutions or provider hosted apps. There might be a situation as the Architecture limit can be overcome only from Microsoft, which practically means that no delivery time or cost can be foreseen.

Figure 1. Agile technology curve for SharePoint

Technology curve for Agile development à la rapid application development

If we work with a Rapid Application Development framework, like SharePoint or K2, than there is always a long period where use cases can be realized very fast with relative little cost. These frameworks are ideal for Agile development.

Set up time and cost: Setting up the Rapid Application Framework means buying and installing the framework. Setting up can be costly and takes time despite relative few use cases are covered in this period.

Effective agile development: Rapid application framework has got a long effective agile development phase. In this phase use cases can be rolled out very fast by consultants or developers for low cost. 

Architecture limit: As soon as the architecture limits reached the development will be much slower, as practically the framework itself has to be further developed. It usually requires special development knowledge; in some cases only the framework provider is allowed to make such changes.

Figure 1. Technology curve of a Rapid Application Development Framework.

Sorces of efficency and inefficiency on the SharePoint software market.

As I described in the previous article, for a basic modelling we suppose that there are companies that want to buy software solutions that match for a given set of R1, R2, .. RN requirements.We suppose that there is such a software soltuion that covers the requirements and which can be delivered in a certain setup time and for a certain price.

Figure 0. Ideal solution.

Efficiency of the market means if in certain projects the actual cost of the whole project is about the software product price and the time of the whole roll out of the project is about the setup time. 

Inefficiency means however if the whole total cost of the project will be much bigger as the cost of the product itself and the total time of the project is much bigger as the setup time of the software solution. As exactly measure the efficiency and inefficiency on market is certainly difficult, it is more simpler to identify several typical situations that surely result in inefficiency, like increased total cost or increased roll-out time.

In the following I try to summarize several typical situations.

1. Requirements are not clear

Figure 1. Requirements are not clear.

As the whole IT field is pretty fast changing, summarizing the requirements are sometimes not simple. Giving examples from SharePoint, usually one requirement is that the pages should be mobile compatible. It is usually not sure however what this requirement exactly mean. 

The following situations can be imagined: 

- The pages should be somehow showed on mobile devices.

- There should be a responsive master page solutions and the layout should be dependent on the screen resolution.

- The downloadable content and javascript libraries should be optimized for different network circumstances. 

- There should be an App for all major mobile operation systems that shows the content.

- ...

Realizing these requirements usually imply pretty different cost and delivery time.

2. Where to find

Figure 2. Where to find.

This is typical a market information attribute. Supposing that a project management software is needed to a company simple collectiong the companies that can provide such a solution can take time and energy. In this sense the so called App stores and Solution stores provide the advantage that all provider is easy to find. Although it is important to note that App and solution stores are usually technology dependent, as soon as solutions between two technologies have to compared, the situation can be difficult. 

3. Which fits 

Figure 3. Which fits.

Even if we find the possibly solutions and providers they are not necessarily too simple to compare. As objective product sheets are not always tipical to provide one should probably install all of the solutions try them out and compare them. It can be carried out if the technological circumstances are pretty similar, like at the example of a SharePoint App store there is always a possibility install a trial version. As soon as however more than one technological solutions should be compared, the situation can be fast complicated. Certainly there is a possibility to order pre-sales demonstrations, however they are not necessarily faster and the do not necessarily provide an objective view.

4. Too big

Figure 4. Too big.

Sometimes the solution is too big for the requirements, meaning that there are a lot of additional requirements that are not needed, despite provided by the solution. As an example consider that we need a simple document management software, as SharePoint can be a natural choice we might as well choose SAP as even in SAP there are some document management functionality. If we do so, we get a big dinosaurs that have a lot of additional functionality like full modules of human resources or finance. It implies of course an extreme increase both in cost and in prices.

5. Too small

Figure 5. Too small.

Most of the software solutions are not just simple products but they contain the possibility to extensions. This can be however dangerous : if the extension is too big than usually it is worth better to implement everything from scratch or get another product. As an example it is certainly possible to set up in SharePoint a full CRM system. Instead of a deep extension of SharePoint however, it might be a better idea to a buy a dedicated CRM solution, like Dynamics CRM. As a thumb of rule, if more than 30% of a software product have to be customized it is a good idea to evaluate alternative solutions as well.  

6. Delivered in parts

Figure 6. Delivered in parts.

It is always a drawback if the solution is delivered in parts. Like infrastructure, application layer, application on the top of an application layer and perhaps consulting and training on the top.The different parts are usually delivered by different experts, even by different companies so they surely implies a increase both in time and in cost. As an example, cloud solutions, like Offie365 for SharePoint, have both the advantage that they are cost efficient and that the delivery time is null as the infrastructure is practically ready.  

7. Lack of extensibility

Figure 7. Lack of extensibity.

This is probably the most difficult aspect. Most software solution should not only be used today but in the future as well. It is usually not clear which requirements should be covered in the future, as the technology might not even exist. On the other hand most software solutions can be extended only in special directions. As an example if we want to have a portal solutions that we want to use in the future for internet of things than Office365 is not necessarily the best solution.

A social method for software development

If you consider software development there are basicaly different models of doing it like, 

the types of classical waterfall, V or W models and of course the different agile methodologies, ranging from extreme programming to SCRUM. 

All of these methodologies suppose however that there are some kind of a more or less explicite spefication what should be develped. 

Considering the waterfall model the specifications should be explicitely given as a set of requirements to start the development itself. 

Having developed the reqirements there might be one more round to extend both the specification and the covered the changed requirements. 

Agile development focuses much more on the changing of the specification, having more iterations and more check and validation of the requirements. 

Agile development usualy does not suppose very strongly that the reqirements covered in the form of a precisely described specification, 

but it is supposed that there is at least someone like a product owner who has got an explicetly view what should be developed. We can say that with agile methodologies, 

even if it is not written the product owner knows exactly the specification. 

What happens however if collection the specification in an organisation is problematic, 

if noone is able to act as an efficient product owner because getting information from the members of the company would require too much time and effort. 

Not to speak about writing a precise specification that would be much more complicated. 

Is there a way in such a situation to carry out an efficient software development ? 

Is there a way to have a software development methodology that actually supports collecting pieces of the specification itself ?

I think the answer is yes. 

Let we imaging that we do not want to have a development in which everything has to be developed from scratch instead we are dealing with components that are not necessarily ready but they are at least in a prototype that can be somehow demonstrated. 

Let we imagine that we have (C1, C2, C3, ... CN) prototype or demonstrable components in our system and we provide user interface called metamodel in which the users or customers can configure demo applications and like them or dislike them based on different like mesaures. 

Like supposing one customer wants to have only the (C1, C3) components in a certain configurations another customer wants to have only (C2) components and the third one only (C2 and C4). 

The system that should we realised is the combination of (CN...CM) components that are liked the most and makes sense to build it up. Certainly they are fuzzy concepts, how the components can be congigured and how the system can be build are depending on the exact domain and technology.

In this way creating the specification is more like a social network: people like on software components and configurations instead of posts or pictures (see the following picture). 

Having this model we should not sctrictly follow the classical principle : developing one software at the end of the process. 

Several other outcomes can be imagined as well :

- One software at the end (one size fits for all): based on the likes and techniqual restrictions we still choose one model to implement. 

- Several variations: we choose some of the most liked solutions and implement as several variations, from which the userers might choose one on their own. 

- Self service software creation (mass customisastion): if the components are not just prototype but practicaly 100 percent ready solutions, there might be a possibility to provide everything and the users can configure everything on their own.

Let have an example, realised in SharePoint.    

We want to develop personal records in a company for storing and showing information for each person. We are however not very much sure exactly which information should be stored. 

Let we say, first name and last names are for sure but othwervise noone knows which metainformation should be used. 

Let we have the following simple meta factory with the following metainformations: 

- ObjectName - name of the object that we want to store, in our example there is only one object called PersonalRecord.

- PropertyName - the name of a property of an object.

- PropertyType - the type of a property of an object.

- ProeprtyLike - information who wants to realise which property, for example in our example there is only one property that wants to be seen only by "test" user, there are only two properties that wanted to be implemented by "test2" user : "FirstName" and "LastName" and so on.

In this sense the components are practicaly the objects and property elements (see the following picture).

Pressing a preview button each customer can see how the choosen configuration could look like, in this sense they do not just choose blindly, but there is a certain view, a certain idea how the choice might look like. As an example having the administrator choice, the given configuration might look like as follows. 

It is important to note that the preview should not necessarily be the final version; it can be regarded as a sketch as well. Like in our example the concrete design is pretty poor, however in a final implementation it can be much more fancy.

​

Based on the likes and users' preference there should be a review and a decision, like: 

- implementing some of properties, mostly that are liked by the most and create one common profile

- implementing more property sets, so the users can choose which one they want to apply

- or simply implementing everything 

8 Years SharePoint development summarised

Limits of Agile development in SharePoint Field

Agile development has been increasingly popular during the last couple of years.

Independently if we speak about a strict SCRUM or extreme programming methodology or something more flexible agile style development, common characteristic of the development is to have:

-          Strong contact with the customer.

-          Frequent and multiply delivery (like sprints at SCRUM).

-          Flexibility of changing requirements even in late development phase.

-          Lightly formulised specifications, like with the help of user stories.

-          Certainly agile projects have some other characteristics as well like self-motivated self-developed teams however they are less important in the scope of this article.

Certainly Agile development has got a lot of advantages and provide a much higher customer satisfaction as opposed to the classical waterfall V or W model in application development. Despite we think that there are several practical facts drawbacks that should be considered.

-          Complexity of change: As it is pretty simple and fast to develop a use case at the beginning of the project, it is usually much more difficult to do it later. The problem is that at the beginning you can develop everything from scratch however later on consider the existing code and the existing architecture of the system. As an example let we imagine a .NET, SQL MVC development that has got some integration with SharePoint as well. At the beginning of the development it is pretty simple to implement user stories as rules in the business logic, however if you get the requirement only at the end of the development that you should do everything in a transactional way that means that all of your business logic should be reconstructed which might mean rewriting a couple of thousands of line of code retesting the modules and so on. (Just from the deep technical side it is not necessarily so simple that you rewrite everything with TransactionSoce as SharePoint does not provide transactions, so you have to probably manually implement a 2 phase or 3 phase transaction protocol and use overall).

-          Project budget: Project budget is always final. Even if you can argue that a new sprint a new round is necessary and flexibility had its price as well, our experience that you should deliver functionality in a meaningfully budget. Even if it is accepted that the new functionalities cost more they should be in a meaningfully limit. As an example it is usually a not accepted that a implementing a new requirement or user story costs 2 or 3 times more as the whole exiting development. So as a conclusion, even if a little bit more flexible project budget has been accepted, you have to deliver at least a core functionality within a certain budget.  

-          Time for delivery: Same is true for the delivery time. The project should have some functionalities to be delivered that usually have some time constraint. Even if there is always a chance to negotiate one more sprint or one more agile cycle delivery time can usually not extended without limits.  

-          Technological limits: In most software development scenarios you do not develop everything from scratch, despite you use one or more frameworks to make the development faster. All of these frameworks have some technological limits that should be considered at the choosing of the framework. Identifying these limits are sometimes challenging as they are not necessarily strict and documented values; they are sometimes soft limits, best based on experience. Best example the well-known SharePoint strict and soft boundaries. Another example can be .NET entity framework for reaching standard relational databases. Entity framework is pretty much liked from .NET developer to access an SQL database because practically the whole database is mapped by C# objects. The drawback however that at each command part of the database has been queried, it is brought to the client side and after making some transformation usually written back to the database. Although it is easy to use for a C# developer it is much less performant as direct SQL commands or stored procedures. As a consequence technological limits of entity framework are reached if complex and fast database operations should be carried out or one has to move and transform a huge amount of SQL data.  

Our experience that most agile project in SharePoint field follows the following pattern. We analysed the cost of the development in the complexity of specification (Figure 1).

Figure 1

As cost of development we mean the following two factors:

-          Licence costs of the used technologies or software frameworks.

-          The amount of cost of the development itself.

At complexity of specification, we practically mean the number of user stories. We suppose that the project is agile so the complexity of specification is highly correlated with the time axis of the project.  

As examples carried out, we consider three different use cases in which similar user stories are covered by three different kind of software architecture. As a bottom line of all use cases let we try to realise a small customer relationship managing software. We are going to have several user cases from the domain like, I want to store somewhere my customers, I want to store some detailed information about my customers, I want to manage contact persons at each of my customers, I would like to see like as a reports what I for each of my customers sold, and so on and so on….

-          Use Case 1: Let we imagine that the target software is Office 365, we try to develop the specification as Lists, Document Libraries and some JavaScript extension, everything is covered by the SharePoint online platform.

-          Use Case 2: In use case 2 we try to develop everything with the help of the K2 platform. K2 black pearl SmartObjects for the business entities and K2 SmartForms for the user interface components. Storage of SmartObjects are either realised by SharaPoint lists or by simple tables of a Microsoft SQL server.  

-          Use Case 3: In use 3 we build up everything with the help of ASP.NET MVC Framework using SQL server for storage and Entity Framework for communicating with the database.  

Each project has got a set up time, at this period infrastructural components are installed, licenses are paid and other kick off and project start special tasks are carried out. During this set up period the cost usually increased pretty strong however relative small part of the specification is delivered. At the example of Use cases the set up time covers the following cost factors:

-          Use Case 1: in the Office 365 example the set up time and cost is pretty much reduced. It basically means buying some licences and setting up the first site collections through the administration portal.

-          Use Case 2: in the K2 example the set up time and cost is much higher. On the one hand license  cost are much higher as at the office 365 example, on the other hand installing a full K2 environment requires time and expertise, especially if it is should be high performant or high available.

-          Use Case 3: in the MVC.NET example set up time is somehow in between the two previous ones. It usually requires setting up some windows server with IIS and SQL database and some classical developer tools like source control.  

Having set up the development project there is usually a pretty efficient development phase. In this phase user stories and requirements can be carried out very efficiently. This period gives the chance to do a really effective agile development: flexible changes or add user stories, deliver in short iterations, have a lot of interaction with the end customer and use any kind of agile template that you want, like SCRUM, extreme programming….

-          Use Case 1: in the Office 365 example this period is typical to create content types lists and document libraries, sites or pages, implementing some design, changing the item or edit item page of some elements and so on. Some these changes can be carried by information workers in minutes or hours others require some development knowledge take days to realise.

-          Use Case 2: in the K2 example the effective period means defining a business logical layer by SmartObjects and creating user interface elements by SmartForms. As only information worker with high K2 knowledge is required the development of new user stories can be very fast in this period.

-          Use Case 3: in the MVC.NET the effective period means setting up database elements integrating with Entity Framework, setting up business rules and realising user interface components by views and the related UI logic by controllers. Realising user stories in this example requires developers, sometimes not just one but more, like one for the database side one on the business logic side, one on the controller part and perhaps a designer as well. As more than one people taking part in the development you can not avoid some kind of a project management either. As a result carrying out a new functionality always takes a lot of time in comparison with the two previous approaches.    

Unfortunately each technological or architecture choice has got its limits. If you reach that point than new functionalities to deliver will take a lot of effort. Certainly you can change your architecture or technology or extend in some sense, however that basically means that even with the best integration you should cover once again the set up time, perhaps recruit new people with new technological competence. The scenario is worst if you do not clearly extend or modify the architecture but you realise some fast hacks. That usually bring some instability into the system resulting on a long run a lot of efforts in system stabilisation and hunting bugs that only sometime occur. At any case reaching the architecture limit means that the costs are going to explode; you deliver very small amount of user stories or requirement change very slow and expensive.

-          Use Case 1: Office 365 is a pretty good example for architecture limits as it has got a lot of limit. Things like number of elements in a list limited to 5000 or the number of possible lookup fields on a 13 basis very fast result that you have problem with the architecture. In some cases if you just slightly exceed the limits there is some possibility to making some small hacking like storing items in two lists, or reconstructing your data model to minimise the number of lookup fields but in most situation you can not avoid to rethink your architecture. As an example there is a chance to set up an on-premise environment and migrate everything, or you can store some items in Azure and integrate somehow, but at any case both will cost much more and will take more time as realising the requirements staying within the architecture limits.    

-          Use Case 2: The K2 framework is much more flexible in a lot of senses, despite you can reach soft and hard product limits if you integrate with special technologies. Like let us suppose that you build up a lot of K2 SmartForms and Processes to create a lot of self-service possibilities to your customer like ordering a product, giving feedback, setting payment options and so on. After that you want to provide the same functionality with the same automatic processes on a telephony system as well. As there are some possibility to extend the architecture to integrate a third party telephony system you will not really be able to use the workflow engine with a telephony system as the engine itself does not guarantee time constrained response time. You probably need to buy a totally different framework and on that rebuild all of your processes that you want provide through a telephony as well. As it is certainly possibly it surely implies a boom both in the costs and in the delivery delay.   

-          Use Case 3: in the MVC.NET let we imagine that we reach the limits of the entity framework. Due to the complexity of business logic and data oriented computation the application will perform very slowly. As a result the whole data access layer and business logic layer should be reconstructed and put everything that is possible into the database as stored procedure. As it is certainly possibly, it will again result a strong increase in the development cost and prevent the fast delivery of the next release.  

Figure 2 demonstrates 3 different technology curves. They may vary how much time takes the setting up, how effective is the middle part and of course where are the technological limits.

Figure 2,

As a conclusion we propose to use agile methods with caution. At our experience:

-          Agile methodology works well only in a certain architectural framework.

-          Exceeding the architectural limits results a boom in development cost and in delivery time. Surely architecture change can be realised in an agile way as well, however due to its cost and time nature it is usually not accepted by the customers. Plus consider the psychological aspects as well, like you deliver the first 20 user stories as 2 days each, however for the 21 th you should say it is 100 days – to effectively communicate something like that is not simple.  

-          Even if it is agile, more user stories and more requirement analysis should be carried out at the start phase as at the later phases. Especially user stories and requirement elements that influence architecture decision should be specified as fast and as early as possible.

-          It is sometimes a good idea to demonstrate to the customer the chosen architecture and the limits and drawbacks as well. You can formulate some negative examples as well, like requirements or user stories that will reach the architecture limits.

-          It is always a good idea to have a good architect.

I think actually the summarised ideas are pretty much trivial, I was personally not sure if it worth time to write them down. Actually software development has got some similarities with the building industry. It is always possible to paint a living room in an agile way and of course if it is not liked by the customer it can be repainted each day with a different colour or perhaps more than with one colour. However it is not possible to do agile development (or agile house building) if it is about the number of floors. You have to exactly specify at the beginning if you want to have 1 floor, 10 floors or 100 floors. A house planned to have 1 floor will never have 10. It has to totally destroyed and start building from scratch.