Hide Notes 

Slide 2: CHT073 Integration, Service Orientation and Pattern Driven Design Ron Jacobs Product Manager patterns & practices
Slide 3: Agenda Integration Architecture Requirements and Scenarios Integration Patterns Portal Integration Data Integration Functional Integration Summary Questions
Slide 4: Global Bank Scenario Global Bank is a midsize, traditional bank Acquired a complete range of financial services capabilities through a series of acquisitions The bank currently has a limited online presence that is fragmented across its various divisions. Global Bank has decided to innovate in the online banking market by providing a host of value-added services on top of a fully integrated financial management capability. Global Bank has not seen the synergies it anticipated from offering a complete line of products. This lack of synergy is caused by its inability to effectively cross-sell based upon existing relationships and customer knowledge
Slide 5: The Integration Imperative Accounts Credit Loan
Slide 6: How did Global Bank get into this mess? By building software to meet a need of the business By acquiring existing systems through mergers By allowing business units to build whatever they wanted without any thought for an integration architecture
Slide 7: How will Global Bank get out of it? By formulating an integration architecture for the business By funding the integration architecture work at an enterprise level Perhaps using a shared services "tax" on projects By defining standards for plugging into the integration architecture Similar to the way that a building plugs into the electrical grid By starting small and building on success Retrofit existing apps as needed By using known good solutions to these common integration problems...patterns
Slide 8: Integration Patterns First published in 2004 First phase of the Global Bank project Describes the business and technical scenario in detail ideal application is a thin layer of presentation that "...the consumes shared functionality or data at the enterprise level. Available online at Ideally, much of this functionality already exists and is accessible http://msdn.microsoft.com is meaningful to the business. And if at a level of granularity that new /practices must be built, it is designed not to stand alone, functionality but to be shared with other enterprise applications and services." - Integration Patterns, pg. 3
Slide 9: Phase 2: Baseline Integration Architecture
Slide 10: Using Patterns to Communicate Design Decisions Each pattern clearly limits the scope of its problem and solution to a discrete and comprehensible, or "mindsized," decision point In complex environments, there is often no single right answer for a given problem A series of technical decisions must be made Each design decision involves tradeoffs—both advantages and disadvantages Tradeoffs made at one level constrain the decisions at other levels The sum of these design decisions must result in an architecture that meets both the functional and nonfunctional requirements of the system
Slide 11: Baseline Architecture Risk management technique Build a thin executable slice of the system which implements the most architecturally significant use cases Focus on retiring the most serious technical risks early, not refining business requirements Not a throw away prototype, it becomes the skeletal structure of the system "Thoughtfully and Intentionally Incomplete" The project evolves the baseline through successive iterations until it is complete
Slide 12: Integration Patterns Data must be integrated into a single view for the customer Patterns represent known ways of doing this each with advantages and disadvantages Solutions fall into three main categories Presentation Integration Data Integration Functional Integration
Slide 13: http://patternshare.org/default.aspx/Home.PP.PortalIntegration Customer Portal Integration Portal Integration Loan System Payment System CRM System Mainframe "...a portal is a single view into many back-end systems that are integrated 'at the glass,' or, in other words, at the user presentation level." - Integration Patterns, pg 39
Slide 14: Portal Integration Variants Display Only Read only data displayed in panes or frames Remote systems may return rendered (HTML) or raw (XML) data Simple Post-Processing Portal UI interprets raw data to alter the display based on it's own rules Single Application Interaction Multiple sources display data – user may interact with one application at a time Cross Pane Interaction Panes within the UI permit interaction with other panes Rules for interaction and integration are contained within the UI
Slide 15: SharePoint Portal Integration
Slide 16: Integrated Dell Desktop
Slide 17: Portal Integration Analysis Benefits Non-intrusiveness Speed of implementation Flexibility Allows the user to make integration decisions Liabilities Inefficient Requires manual interaction Error-prone User makes integration decisions – often in error Not suitable for complex business processes
Slide 18: Data Integration Most enterprises contain multiple systems that were never designed to work together Many applications are organized into three logical layers: presentation, business logic, and data. You want to be as noninvasive as possible You want to isolate applications' internal data structures Reading data from a system usually requires little or no business logic or validation Many preexisting applications couple business and presentation logic so that the business logic is not accessible externally Direct access to an application's data store may violate security policies that are frequently implemented in an application's business logic layer
Slide 19: Data Integration http://patternshare.org/default.aspx/Home.PP.DataIntegration Integrate applications at the logical data layer by allowing the data in one application (the source) to be accessed by other applications (the target) • Shared Database. All applications that you are integrating read data directly from the same database. Legacy Application Presentation • Maintain Data Copies. Maintain copies of the application's database so that other applications can read the data (and potentially update it). • File Transfer. Make the data available by transporting a file that is an extract from the application's database so that other applications can load the data from the files. Other Application Business Logic
Slide 20: Shared Database • • • • Eliminate latency by allowing multiple apps to access database directly Intrusive approach Reading generally not a problem Writing can be more difficult because the logic to validate data and business rules may not be contained in the database • Requires strong validation and consistency rules in the database • Stored procedures • Triggers • Constraints Shared Database Application Application Application
Slide 21: Maintain Data Copies http://patternshare.org/default.aspx/Home.PP.MaintainDataCopies • Maintain multiple copies of the data for each application • Synchronization of changes becomes a problem • There are 12 patterns used to handle synchronization issues defined in "Data Patterns" • http://msdn.microsoft.com/architecture/patterns/default.aspx?pull= Data Replication Application Application Application
Slide 22: File Transfer http://patternshare.org/default.aspx/Home.HW.FileTransfer • One application produces a file and transfers it so that other applications can consume it. • Because files are a universal unit of storage for all enterprise operating systems, this method is often simple to implement • Disadvantage: Applications can lose synchronization with each other because each one is changing the file independently File Transfer Application Application
Slide 23: Data Integration Analysis Benefits Non-intrusive High bandwidth Access to raw data Metadata Good tool support Liabilities Unpublished schemas. Bypassed business logic No encapsulation Simplistic semantics Different storage paradigms Semantic dissonance
Slide 24: Functional Integration http://patternshare.org/default.aspx/Home.PP.FunctionalIntegration Integrates application at the business logic layer The business function that you want must be available inside the source application's business logic The API of the source application must be accessible remotely Legacy Application Presentation Business Logic Other Application Data
Slide 25: Functional Integration Styles Distributed Objects Enterprise Services (COM+), RMI (Java), Corba, .NET Remoting Tight integration between server and client Works well within a single application Not recommended for integration of multiple apps Message Oriented Middleware Asynchronous message queues Proprietary technology: MSMQ, MQ Series, TIBCO No model for contracts Recommended for unreliable networks Service Oriented Web Services (SOAP) Transport independent Portable type system (XSD) increases interoperability Loosely coupled technology XML Serialization has performance impact
Slide 26: Service Oriented Integration http://patternshare.org/default.aspx/Home.PP.ServiceOrientedIntegration Service Consumer Service Provider Service Gateway Service Interface Service Implementation Integrate at the business logic layer Can be done with any wire technology Web services provide the most interop-friendly implementation
Slide 27: Service Oriented Integration Analysis Benefits Interoperability between disparate technical architectures Decouple an enterprise's business architecture from its information technology With interoperable systems the cost of process change is dramatically lowered Businesses become more agile as a result of creating interoperable systems and using services that are relevant to business practices Liabilities Performance cost of serializing, deserializing, and parsing XML documents. XML documents are much larger than their binary equivalents because they contain metadata. This increases the size of the payload that must be processed during message exchange.
Slide 28: Entity Aggregation http://patternshare.org/default.aspx/Home.PP.EntityAggregation Provides a unified data view to applications Applications can interact with this representation as if it were a single data source The integration layer must now resolve any Application Application Application instances of semantic dissonance between the individual systems Entity Aggregation Loan System Payment System CRM System Mainframe
Slide 29: Before Entity Integration Layer Semantic Dissonance – what is a "Phone Number?" Incompatible definitions U.S. Database Euro Database Phone Number Area Code Exchange Number Phone Number Country Code City Code Exchange Number U.S. Application Euro Application
Slide 30: After Entity Integration Layer U.S. Database Euro Database Phone Number Area Code Exchange Number Entity Aggregation Phone Number Country Code City Code Exchange Number Phone Number Country Code Area Code Exchange Number U.S. Application Euro Application
Slide 31: Entity Aggregation Approaches Straight-through processing Fetches information from the respective back-end repositories in real time and correlates the information into a single unified view Assumes Entity Aggregation layer has real-time connectivity to the repositories and should be able to associate the disparate instances of the entity Replication Real-time connectivity to the repositories is absent Complicated joins across multiple instances of the same entity across various repositories is required to provide a consistent representation High performance is vital to the solution.
Slide 32: Entity Aggregation Design Considerations Data Representation How will the entity be represented and schemas reconciled? Data Identification What identifier will be used at the enterprise level? Data Operation How will transactional operations be performed? Data Governance Who "owns" the data?
Slide 33: Data Representation Select an existing entity representation Often driven by existing system (CRM, ERP etc.) Could be based on industry standard schema Create a custom entity representation Reaching a consensus on a single representation may not be possible (in this lifetime) Entity Aggregation must Account for the representation of all entities held within the different repositories Define a unified schema for all entities which represents a logical consolidation of all the views Effect transformations between each repository’s schema and the unified schema
Slide 34: Schema Reconciliation Entity Aggregation Layer Customer CRM Mapping Back End Repositories CRM System Customer Customer ID Last Name First Name Address Email Phone Card Num Card Type Expiration Customer ID Last Name First Name Address Email Phone Financial System Customer Financial Mapping Customer ID Card Num Card Type Expiration
Slide 35: Data Identification Entity Aggregation layer uniquely identifies an entity instance by using a reference known as a master reference. A master reference could be: A reference held by one of the repositories. A new reference that the Entity Aggregation layer Enterprise ID creates for the entity instance and maps to other references held by different repositories. Loan # Acct # Credit Card #
Slide 36: Data Operation Writing data to a entity database requires the entity aggregation layer to synchronize the backend systems Failed writes must be accounted for using one of the following solutions Distributed transactions Compensation Tentative Operations Manual Intervention
Slide 37: Data Governance When data elements are duplicated you must decide on an authoritative source for the data You will need to create a change management process to handle changes to Configuration Data Model Reference Data which spans repositories
Slide 38: Entity Aggregation Analysis Benefits Consensus - forces consensus across business and functional units on the manner in which entities are represented at an enterprise-level. Single view - enables a single view of key business entities such as Customer, Account, Product, Patient. Reduced semantic dissonance Central location - supports a central location for validating data that is populated into the repositories. Reduced change impact - the Entity Aggregation layer can continue to serve the needs of the applications while changes are in progress. Liabilities Additional layer - could adversely affect end-to-end performance of the solution. Consensus - requires consensus across business units on the definition of the enterprise-wide representation of entities. Reengineering applications - existing applications that are tightly coupled to a given set of repositories would have to be reengineered to accommodate the new architectural layer. It is not always possible to reengineer some applications — especially packaged solutions.
Slide 39: Summary There is no single right answer for every architecture problem Patterns help us to understand the context, forces, benefits and liabilities of existing solutions Patterns can be composed as elements of design to create an architecture
Slide 40: Resources Patterns and Practices: http:// msdn.microsoft.com/practices My home page : http://www.ronjacobs.com Patterns: http://www.patternshare.org Global Bank Community http://workspaces.gotdotnet.com/gbi Threats and Countermeasures: https://www.threatsandcountermeasures.com
Slide 42: © 2004 Microsoft Corporation. All rights reserved. MICROSOFT CONFIDENTIAL. INTERNAL USE ONLY.