Connected architecture: the end of information chaos
— written by Marnix Dalebout and Martijn ten Napel
We live in an age where data technology is developing at an accelerated pace. The massive increase in information and its possible applications have led to an increase in complexity and information chaos.
We see a lot of companies struggling with their data use. Nothing seems certain except for change itself. Over the years we have developed an architecture which can deal with these developments ‘by design’. This is an architecture which offers ways of finding not the best but the most appropriate solutions.
We call this ‘connected architecture’. It is a means of setting up data implementation in a structured and future proof way. It’s a handy set of recipes for a step by step implementation of sustainable data landscapes; a handbook by which an organisation can continually adapt itself to the future.
We would like to share this vision with you. We are going to do this by publishing a book early next year. We will also be doing this in online publications. In this article we will be telling you about the main points of connected architecture.
The chaos is increasing
Organisations are groaning under the weight of their unending urge for information. The demand for information is greater than can ever be met. At the same time, the expectation is that new information can be made available more quickly. These expectations are hyped up by technology suppliers promising us the moon during unending series congresses, seminars, webinars and publications, all about the promise of data.
There is often a lack of coherence and synergy between information solutions. A sub-solution is thought out for each sub-problem. So, a jumble of solutions emerges which deliver almost the same, but not quite identical information. The landscape becomes more and more complex and the maintenance load becomes heavier and less transparent.
In this way, data implementation is becoming more complex by the day and the chaos is increasing. This has a negative influence on the ability to deliver new information effectively. In addition, increasingly strict legislation often makes it unclear whether you are unwittingly breaking the law.
Managers and architects want to reclaim control. They are looking for solutions which can keep up with all the developments in data usage. And they are looking for guidelines to make correct choices out of the overwhelming amount of technological options.
The most appropriate solution
We are often asked: “Tell me, what do I need to do to put our information chaos in order?” People react amazed when we react by telling them that we don’t look for the best solution, but for the most appropriate one. Each organisation is set up differently and requires its own approach.
The common feature in our approach is however always reducing complexity. We know that not just today’s challenges, but also those of tomorrow will make themselves known. Change will happen, be prepared.
Our answer to increasing complexity, chaos and uncertainty can be referred to by the name ‘connected architecture’, a tool kit with instruments with which organisations can take informed decisions about designing data.
What is connected architecture?
- Connected architecture is an architecture aimed at not just the use of information but also its processing. It is an architecture which structures both the organisation of information requests as well as the implementation of information solutions in the data landscape.
- Connected architecture isn’t just one solution in which all the data is collected, but an architecture in which data is ‘connected’ between different data sources. We call this the data landscape.
- Connected architecture offers design patterns and implementations to create an information eco-system of coherent data environments. Information products can be delivered without the required data having to be physically integrated first.
But it is also more than this.
- Architecture also needs to be anchored in the organisation. Working with another way of thinking means that new information requests can be absorbed into the data landscape with as few issues as possible. As a result, the data landscape can evolve in parallel with the development of the organisation.
How did we arrive at connected architecture?
Our search for most appropriate solutions has brought several elements together:
- Discovering similarities in the challenges in the area of information by combining insights which we have gained in our many, diverse projects,
- Career long working with the principles behind Agile working methods,
- New types of data and the technological diversity with which these can be processed,
- The book ‘The Business Unintelligence’ by Barry Devlin in which the author offers a new reference architecture where new data types are given a place,
- Gartner’s logical data warehouse concept and what that means for being prepared for future information requests in organisations.
We have combined these into a handy, tried and tested set of recipes. It is an architecture with proven, practical merit.
What are the basic assumptions of connected architecture?
Connected architecture operates on the following basic assumptions:
- Development of requests and insights are asynchronous. You need to be able to deal with different speeds of change within one organisation and with different levels of knowledge about working with information. You need a vision for coherence in your information services and how you are going to manage this.
- The only certainty is constant change:
- Requirements change: People find it difficult to work out and put into words which data they need and what for, now and definitely in the future. The more that information is worked with, the faster change takes place.
- Organisations change: New people arrive with new skills, important people leave. Processes, aims, budgets and priorities change. Organisations grow and shrink, are taken over, broaden or deepen their scope.
- Technologies change: The innovations of today are the legacy of the future. New technologies, definitely in the area of big data and analytics, appear and disappear at a high speed. What should you invest in?
- Legislation changes: There is a discrepancy between what is possible and what is permissible. The laws which determine what is permissible, change on a regular basis.
Controlled change in the data landscape.
So, how should you deal with constant change? Our answer is always: work pro-actively to reduce complexity. That means that in everything you do, at every decision you take, you should be asking the question: ‘Will this remain simple and does everyone understand what we are going to do?’ By not losing sight of human aspects, it becomes easier to make the consequences of choices transparent to all involved parties. This means that your people, processes and information are brought into, and kept in, balance, appropriate for what is feasible in your organisation.
This has resulted in our guiding principle of ‘just enough’. If you take a small step each time in a dynamic environment, ‘just enough’ to answer the request at hand, you can let the data landscape evolve iteratively, without losing the support of all those involved.
No extension of or change in the data landscape is random. We offer a method for drawing up a road map which clarifies the direction and framework in which the data landscape is developed and how you can continue to take each next step.
What does connected architecture look like?
The diagram below visualises connected architecture:
An update of this image has been made. Read all about it in the successive article.
- Information valorisation: The aim of information usage is to harvest as much value as possible contained in the information. Connected architecture is aimed at information valorisation.
- The application of information isn’t automatic, you need to organise it.
- People only know what information they need when they have seen it. The principle of ‘just enough’ also applies to the demand side of an architecture in the information valorisation layer.
- The information demand is translated into solutions which are made available in the information provisioning layer and for which the infrastructure layer provides the technological foundation.
We have worked out the framework on three planes. We elaborate a fitting version of the model for an organisation in collaboration with our clients.
These three planes are:
- A collaboration model, based on the principles of agile working. The ‘just enough’ principle is encapsulated in this.
- An architecture decision making process to determine what ‘just enough’ is to guarantee the flexibility of the data landscape.
- A data landscape design to implement the landscape in a sustainable way without leading to chaos.
The realisation of the three planes should be managed by a part of the organisation which has been assigned this task, so that coherence and direction for the development of the landscape can be maintained. In the market this is known as the BI competence centre (BICC). We have an opinion about how the BICC should be designed if it is to operate effectively, however that is a subject outside the scope of this article. We use the label BICC, but the name itself isn’t important.
The collaboration model
We could write an entire book about the collaboration model, but for now we will suffice by referring to the ‘agile with architecture’ series for the vision behind our connected architecture.
The architecture decision-making process
Most organisations we work with, are struggling with the question ‘Where should I start?’. The architecture decision-making process helps by making a start by bundling the information requests into generic use cases and by determining which use cases should be prioritised in the short term.
There is always the risk that you devise specific solutions for specific challenges. To avoid this, the architecture decision making process is divided into two processes. These ensure a framework in which to determine what the right decisions are, without losing sight of the continued development of the data landscape as a whole.
- The domain architecture process. A process in which the road map is drawn up. This process answers the question which solution architectures need to be implemented to realise the most important use cases within an organisation.
- The solution architecture process. A process by which the most appropriate ( ‘just enough’) implementation decisions are taken, based on the limits of accountability as imposed by the domain architecture.
The data landscape design
You can realise data landscapes in lots of ways. To develop the landscape sustainably and to maintain coherence, a reference architecture is needed which is used as a blueprint for the design.
We have made a blueprint based on Barry Devlin’s REAL architecture, to which we have added elements from Gartner’s logical data warehouse to give form to the connectivity.
How the blueprint is translated into an implementation plan and solution architectures is a subject all of itself. In our book we will going into this in detail, but we would like to present the blueprint briefly here.
At the top, the five use patterns of information which have a place in the landscape are named. For more information about the use patterns we refer you to the article ‘The aim of information requests: information valorisation’.
The three pillars represent the information categories which each require a different form of processing, but which the users also want to link and use at the same time.
This coherence doesn’t just happen by itself: pre-conditions are needed to guarantee the links between the various solutions. These are pre-conditions in the context of information, consistency of information and in managing access to the information in the data landscape.
The three elements which are set up for this are:
- Master data for integration, which provides the pattern by which information is connected over different solutions. Connecting means that you can query the information integrally over different solutions without having to first combine the information into one solution. This is also known as ‘loosely coupled information’.
- Master data for authorisation, which provides the patterns by which access to data over the various solutions is maintained from one point. This makes it possible to check authorised usage.
- The data model in the various solutions that connects to the master data patterns for integration and authorisation.
The information delivery layer makes use of these various elements to make information available in a format or form as desired by the users.
The way in which the blueprint is realised in the implementation determines the flexibility of the data landscape. The architecture decision making processes are set up to guard for keeping both the agile collaboration model to supply the necessary development speed of the data landscape as well as flexibility of the information delivered.
I would like to know more about connected architecture. Where can I find more information?
We are aware that we may be giving rise to more questions than we are answering by introducing our connected architecture in this way. A book which we are publishing next year will provide a more detailed description. Please feel free to contact us if you would like any more information.
Reactions through Twitter @MartijntenNapel or e-mail firstname.lastname@example.org