Our PoV

fluid-integration, Uncategorized

Fluid Integrations – The Network Effect

The impact on communications as it evolved from individual email messaging to social networks, both in business world as well as in personal life’s is pretty obvious. Take a pause to digest this: every 60 seconds in 2016, 40+ years old workhorse of communication aka email had ~150k messages sent in 2017, against 5 years old newbie known as WhatsApp’s ~29,000k messages.

While it is debatable what % of either of them carried any business value or even any meaningful content, there is no doubt that the communication approach has evolved from point to point messaging to networks based publishing /subscription based on relevant topics communication.

As business technology follows the evolution path of personal technology, the interest is growing on the topic of integrating organizations to networks (eg, Ariba / Coupa etc networks & online marketplaces like Alibaba.com) & how is that different from using B2B / EDI communications. The answer we feel lies in the Metcalfe’s law, which projects that impact is not additive but exponential to the number of connected participants.

As an important integration pattern, we evaluated the same using the Fluid-Integrations evaluation model. Obviously, the traditional sender / receiver based messaging is being mostly being evolved to the network based models though latter is characterized by lower volume / higher frequency of interchanges.


Enabling the pub/sub model using traditional EAI tools like SAP PO / CPI is doing a catch up act & cloud native applications like AWS SNS are showing the path ahead, especially when it comes to integrating on a network model.

Here’s a sample of settings needed to send data from on-premise EAI tool to a JMS based pub/sub system on premise. Key difference form regular communication is the Queue/ topic support.


Main capabilities for pub/sub model are topic support, Push Delivery, support for diverse protocols including REST, replication to multiple receivers, topic filtering, message durability, security / encryption, scalability & HA and ability to support microservices.

A hybrid integration of on premise EAI with AWS cloud based SNS / SQS is supported using custom adapters provisioning connectivity and topic support.


Why should we care? Future of Fluid-Integrations will probably follow very similar path of personal communications ie, network driven, topic based, hybrid infra, short lived messages at high velocity and happening in realtime. Who says millennials are not showing the way for progress?

fluid-integration, Uncategorized

Fluid Integrations for Applications: Task Vs Tool?

If there is one thing most organizations & consultants will agree on, it is : Integrations are tough, ranging from mildly frustrating to downright painful.

Given a choice between re-platforming a functioning EDI setup with their vendors for something “newer” or getting their teeth pulled, probably only debatable point will be the strength of the painkiller for the latter .

This is further complicated by the increasing vocabulary of TLA’s, technical jargon spewing architects & integration subject matter advisors who have depth of know-how in their platforms / tool capabilities but conspicuously low attention on the task at hand. This situation is amply demonstrated by Prof Clay Christensen in this blog about provisioning something as mundane as milkshakes & how easy it is to miss the forest for the trees.

In our experience, most organizations are looking for ways to deliver products / service with a differentiated customer experience, reliably receive the goods / services they procured across their supply chain and timely settlement of their cashflow. That, at a very broad level is their “quarter inch hole” & the underlying force for the ask of Fluid-Integrations across applications, organization & their partners ecosystems. Accomplished integrators get a good handle on these “forces of integration” early on in their engagement with the organization leadership and relentlessly focus on delivering on the same. As they walk down this road, the realization that while tools evolve, the goals pretty much remain the same over long periods.

We believe that a tool agnostic framework for identification, measurement and implementation of the forces of integration is an activity any organization can benefit from in the long run as well as upfront.

We had seen this framework in an earlier article and applied it to the evolving pattern of API integration’s. we leveraged further on this framework for evaluating the challenge of applications integration (including A2A / B2B / EDI / API and data integrations including unstructured as well as structured) to visually identify how these patterns differ from each other while still retaining the inherent commonalities.

Here are the highlights :

  1. Principal challenge for organizations as of today / near future remains data integration especially on the variety, volume and velocity dimensions. Consider the challenges of an SAP centric manufacturing organization as they try to grapple with challenge of integrating their material master data along with its dependencies on their ERP platform, keeping mind this is just their structured data.


    As the unstructured information, engineering drawings, product images et al are factored in for hundreds of thousands (probably millions) of material data records, few solutions are able to scale upto the challenges inherent in such an integration.


  1. The API integrations pattern is the newest one & is evaluated here.
  2. For the applications integration pattern, the key dimension seems to be increasing volume and need for higher velocity.

    As a mature pattern, it is reasonably well supported by the tools available for on-premise / cloud based EAI platforms with increasing trend of using marketplaces for B2B integration.

 

fluid-integration, fluid-manufacturing

Fluid-Integration for API’s – Getting high with new wine in the old bottle?

There is something to be said about working in a customer facing retail organization: Never a dull moment. If it is not about procurement logistics, store issues with PoS or delivery logistics, you can bet that the ever present challenge of “Delighting the consumer better” is always on. While doing Digital Customer Experience work for a B2C retailer who wanted to provide the customer a delivery date & time upfront for home delivery using ERP based API’s & external cloud services in real time, the challenge boiled down to:

How to Enable people facing cloud applications from on premise backend ERP with API microservices?

From many perspectives, this debate seems to be trivial with proponents of SOA projecting that not only the coarse grained services provisioned by standard package (aka BAPI’ / RFC’s) are fit for purpose, but also that traditional SOA middleware platforms for EAI will be able to cater for these needs. Without a framework to assess the fitment, many organizations have taken the path of trying it and then, finding midway the pitfalls of trying to put the new wine (Microservices API’s) in the old bottle (SOA middleware platforms) very well depicted by this picture.

So what is it that is different with API’s & how they should be enabled?

We evaluated the application integration & API integration side by side on the Fluid-Integrations framework & the results are pretty much on the expected lines:

  1. Traditional application interfaces enabled on SOA platforms are characterized by high volume, low velocity (often asynchronous), comparatively lower variety of data being interchanged and veracity being provisioned by trusted applications setup. These services are often coarse grained (think sales order / PO etc), mostly using XML / SOAP and enabling process chains.
  2. API’s integration are driven by variety of microservices with higher velocity (mostly request-response in synchronous mode), lower volumes and need for veracity (especially with oAuth2.0 / encryption etc). these are often fine grained services (eg, delivery date for an order, PO item approval etc) and may have an end user waiting for a response, using JSON over REST and authorization / authentication both being important.



 

 

 

 

 

 

Summary is: while there are overlapping needs, there are certain clearly identified needs which differentiated one pattern with the other.

So, what is needed to comprehensively cater for API’s for a reasonable scale of enterprise integration?

Here is the API management feature set from SAP, which you will notice has features beyond traditional EAI platforms like caching, analytics & monetization.


From a recent engagement, sharing some of the topics for API management which need specific treatment from traditional EAI platform.

  1. JSON conversion: as a lightweight defacto standard for REST services, JSON-XML conversion & vice versa is a required feature and is provisioned out of the box on PO.
  2. Exposing backend API’s on PO/PI without using integration engine: these API’s can be setup using only the Adapter engine, without having to go through the integration / mapping steps. This improves the performance significantly while still exposing the services with available security / authentication mechanism.

    If AE is setup in the DMZ, this might be especially relevant.

    In the configuration, just leave the operation mapping as blank and use the same interface name / namespace to enable passthrough processing using only the adapter engine. For calling the RFC’s, the name / URN can be directly specified in the receiver adapter configuration.

  1. oAuth2.0: this is the dimension where probably more product enhancement is needed. Currently, the oAuth2.0 support for PO/PI is only for REST receiver and with SEND polling mode. This means that in scenarios where a user facing app needs authentication & authorization from API, PO can provide the basic authentication and authorization will need to be handled by API itself.
  2. Selecting the backend protocol: there are multiple options available for backend API protocol with PO/PI (eg, RFC, SOAP, oData, proxy etc). RFC’s have some known limitations for monitoring, logging and caching (may need to use RFC_CONNECTION_CLOSE). Proxy probably provides for most relevant protocol for this usage. Synchronous logging needs configuration on PO/PI as well as NWA setup.
  3. Analytics: API’s need fine grained management of the requests-response, validating quotas , throttling, spike arrests etc. while PO/PI provides for UI based access to monitoring using PIMON, programmatic access to these statistics is provisioned by following servlets :
  • MessageOverviewQueryServlet
  • PerformanceDataQueryServlet

Usage of these servlets is described by this excellent blog. for API’s context, some development / configuration will be needed to extract, analyze and trigger actions on the PO/PI system (probably using BRM tool inbuilt).

In summary, we evaluated the coverage provided by PO/PI system for API usage without an API management layer as suggested by SAP for on-premise or provisioned on cloud using Apigee tool on CPI.

It will be very interesting to get your views on this evolving topic and how these challenges are being solved in your experience.

 

fluid-integration

Noah’s Ark & Digital Integration challenges

Its not business-as-usual for us refer to biblical stories to understand the challenges being faced for technology integration, but then its not often that such tectonic shifts happen in the data and its integration space to be called as the next Industrial Revolution.

Of course, by now everyone & their aunts have heard about big data and how it is changing everything around us. For those of us who work with organizations in different industries, sizes and use cases (aka the Integration Architects), the challenge is how do we help organizations handle this flood of information and still, make sense by integrating their customers, ecosystems & operations using an ever growing set of 3-letter acronyms that technology providers throw at this challenge. For a business leader working through the exciting realm of Digital Customer Experience using people facing technologies, the challenge at hand is pretty different from someone trying to streamline manufacturing operations using IoT or “Things” which in turn is significantly at variance with a procurement specialist looking to enable on-demand procurement by integrating cloud based procurement exchange applications with on-premise financial system. Obviously as the integration goals are varied, so are the integration patterns and the challenges. Hence the analogy of organizational leaders today feeling about the data deluge and their own “arks” of technology solutions with which they are trying to solve their respective challenges.

To put the issue in context, probably the most useful framework is provided by Gartner and the underlying 4-V’s of big-data ie, Velocity, Variety, Volume and Veracity. Simply put, these define the different dimensions of the data & its integration challenges for modern technology landscapes.

  • Volume dimension describes the evolution from MB / GB sizes to current TB / PB scale of data.
  • Data Variety has just exploded in recent years, growing mostly from table / DB based data to social / mobile / web based photo / video / audio based unstructured data .
  • Velocity is being driven by need for real time / near real time data & analytics from yesteryears periodic / batch based integrations.
  • Probably the one which is most underappreciated but is definitely growing in importance is the Veracity dimension, which describes the data reliability in terms of accuracy, probability, efficiency and exposure.

We used this framework with typical enterprise integration patterns of Application Integration, People Integration and now exploding pattern of IoT / Things integration. Based on conversations with executives, architecture & implementation experiences and good old “crystal ball gazing”, we came up with our own scores for each of the 4-V’s mapped on the integration patterns.

Sharing at a macro level the results (which should not be surprising). What is clearly evident is:

  1. Applications integration (A2A / B2B / Data / API / Structured etc) have their biggest driver as the growing volumes being sought, with velocity being the other key driver
  2. People integration (B2C / B2E / B2B / Social communities / GUI etc) is being dominated by the variety of data (structured / unstructured) and also, by the ability to validate the reliability of the inputs
  3. IoT / Things integration (M2M / Bots like Alexa / personal devices like Fitbit and other increasing items) is driven by sheer velocity of the data and again, need for validating the veracity of it.

While these are very macro level scores which should be refined to be usable, probably most practitioners of integration will agree that broadly the Patterns -Drivers mapping looks rational.

Next logical questions from an executive responsible for such an integration might be: This is good, but how do I use it with my current / near future technology portfolio?

  • Will my EAI hub work well with the API’s model I am looking to monetize for real time management of fleet logistics?
  • What do I need to change so that my B2B portal works for the new B2C / mobile app we are launching for Digital Customer Experience?
  • How will I secure my on-premise financial applications when they start integrating with cloud based procurement exchanges for on-demand buying?

All of these queries and probably many more fall in the major integration need which we see evolving already, ie, need for increasingly Fluid-Integrations.

This need for increasingly Fluid-Integrations is driven by organization imperative integrate across organization boundaries, amorphous formats, different technology protocols, disparate consumption patterns and ever newer data elements.
Please share your own experience & inputs to assess where your organization is within their competitive landscape for these integration priorities / challenges. You will receive a snapshot of survey data instantaeneously:

In the next blog in this series, we will leverage the Fluid-Integrations model to identify specific components which are suitable for each of these challenges and also, refine the integration patterns more granularly for architects to connect the dots better. Keep reading

fluid-integration

Tempted to Join the Mobility Gold rush: Avoid the sensationalism, be “sense”able!

By now, probably every one of the business leaders and their aunts have already heard enough about Mobility and how it is going to change our businesses, lives, et al… I can already hear fatigue setting in. Those who have been around a bit longer will have skeptically heard the pitches from inhouse geeks / highly paid consultants / technology evangelists about the game- changing- gizmo’s available now and how the business paradigm has shifted.

If you have drunk the kool aid and already know how your business will benefit from this mobility gold rush, need not read any further.

For the rest of us, especially those who see how this is different but still scratching their heads about what it means for their day to day business, where their business benefits lie and indeed, where they can start off to grab the low hanging fruits… you know why you are still reading J

As someone who has spent more than a decade helping organizations setup business processes around technology platforms, I was honestly less –than-impressed till recently about the whole Mobility gold rush. So, the geeks have a new technology and churning out buzz words (BYOD, MDM et al) to sound smarter on their twitter accounts, right?

Well, having been off the mark about viability of other technology waves (google is just another search engine! facebook is valued at what? Social is for kids, not businesses), this time reaching for the humble pie before writing off the Mobility Kool aid maybe more fulfilling.

So quest for getting what is this Enterprise Mobility and more specifically, what can it do for my business begins. Lets take a look at this one step at a time. Of course, mobile or not, the business senses have stayed the same :

What has evolved substantially over the years has been the way businesses have tried to connect with individuals. Call it B2B/ B2C/ B2E/ B2G or whatever, the pace of evolution on individuals side has outpaced that of the business side. Lot of it was accelerated by sheer innovation from Apple Inc, who were just disrupting the individual technology landscape as an annual KPI. Increasingly, these devices were catering, even extending the capabilities of 5 basic human senses.

As we speak today, we are able to touch, talk, see, listen thru these devices. iPhone was the first device which probably stepped into intuitively figure out what a user wants, thus expanding into catering to the sixth human sense, ie, intuition . Recently, Google’s Android has also developed capability to intuitively suggest to users things they may wish.

Over a period, the chasm between Business technology and how it connected with individual tech became so wide that something like an arab spring was on cards if that gap was unfulfilled.

Final straw in this Business to Individual gap (lets just call it B2i gap, encompassing all of B2B/ B2C/ B2E) was around 2010, when Apple launched the iPad. Since then the autocratic business organizations realized that to avoid an arab spring of revolt, they will need to support individuality.

By then, not only devices but also UI, way of interaction, collaboration message exchanges and overall experience had gone thru a paradigm shift. From Transaction based systems to interaction oriented experience.

So, what’s next? Technology evolution is kind of risky field to do crystal ball gazing, but we suggest that as long as businesses take a “sense” able approach we all should be on safe ground.

Connect the dots of business drivers / senses with the those on “i” side and ask yourself these basic questions:

  • What is the quantified benefit for my business with Mobility adoption (specify revenue accretion / cost reduction)?
  • What will it cost me over short term (low hanging fruits), medium term and long term?
  • What are the risks in such a fast paced evolutionary market (anyone remember Nokia/ RIM) and how they will be mitigated.