Sunday, September 30, 2012

My response to an expertise request

Solution Key Components:
* Ability to ingest upto 3,000 messages per second from external domain
* Messages can be in XML, EDI, etc format. ~20 kb each.
* Transform into canonical format
* Perform authorization & authentication
* Break message into multiple messages (achieve parallelism)
* Route based on load, content to message queues (JMS, etc)
Function 1:
* Message queue clusters bucket and shoot to Rules engine
* Rules engine devises appropriate action, forwards to event handlers
* Event handler integrates with e-mail service to send an e-mail
Function 2:
* Message hits Data Access layer for storage
* Looking to store Transaction-type data (1bn records per month)
* Need quick retrieval techniques. Ensure Data consistency/quality
* CRUD operation time should be under 0.1 seconds
Web Component:
* Request comes in from an external site for a iFrame request
* Request needs to be authenticated/authorized/load-balanced
* User information should be cached right @ log-in, so cache can retrieve data we expect the user to view, so we don't retrieve when the user starts navigating
* Planning to have an Active/Active multi-site design
* Don't want to do sticky sessions
* Should we have a distributed cache with regions replicated across sites to avoid sticky sessions?
* Web layer needs to handle 500 concurrent requests minimum
* Overall solution primarily designed on-premise (Virtualized environment) with DR-site on public cloud

Solution Architecture
After thinking about the limited problem statement, it seems that security, scalability, data transformation, messaging, declarative rules, HA and persistence are key.  A good architecture style for the solution design would be SEDA See this for reference: http://www.eecs.harvard.edu/~mdw/proj/seda/
The solution should be broken down into a set of stages where a component or a set of components interacts with the data and performs logic. Each stage is connected by queues. An orchestration layer can govern the path depending on the routing logic.
Layer 1: Rules Engine. Fronted with incoming and outgoing queues.
Layer 2: Routing Engine: Choreographs the processes.
Layer 3: Executing Engine: Contains logic and data access.
Layer 4: Presentation: Web Server/App server with logic.
0.1 seconds for data retrieval is enough latency to avoid the complexity of managing a distributed cache. Content caching and Akamai edge cache can be useful instead.
Technologies: Use IBM’s MQ Series to define the queues, clusters etc based on logical names. Install Mule ESB to implement SEDA and routes. Bring in iLog or Aion for the declarative rules engine. Host on internal cloud or EC2.
 

Wednesday, September 26, 2012

How would you solve for these requirements?

Sometimes I get expertise requests, and here is a challenge.

Solution Key Components:
* Ability to ingest upto 3,000 messages per second from external domain
* Messages can be in XML, EDI, etc format. ~20 kb each.
* Transform into canonical format
* Perform authorization & authentication
* Break message into multiple messages (achieve parallelism)
* Route based on load, content to message queues (JMS, etc)
Function 1:
* Message queue clusters bucket and shoot to Rules engine
* Rules engine devises appropriate action, forwards to event handlers
* Event handler integrates with e-mail service to send an e-mail
Function 2:
* Message hits Data Access layer for storage
* Looking to store Transaction-type data (1bn records per month)
* Need quick retrieval techniques. Ensure Data consistency/quality
* CRUD operation time should be under 0.1 seconds
Web Component:
* Request comes in from an external site for a iFrame request
* Request needs to be authenticated/authorized/load-balanced
* User information should be cached right @ log-in, so cache can retrieve data we expect the user to view, so we don't retrieve when the user starts navigating
* Planning to have an Active/Active multi-site design
* Don't want to do sticky sessions
* Should we have a distributed cache with regions replicated across sites to avoid sticky sessions?
* Web layer needs to handle 500 concurrent requests minimum
* Overall solution primarily designed on-premise (Virtualized environment) with DR-site on public cloud

I will post my take based on these needs. Also, it will be fun to look back at it after a while as new technologies & solution options evolve.

Sunday, September 23, 2012

A Culture of Discipline

Those who try to manage change, motivate employees and work to create alignment are negatively correlated with companies that move from good to great. To get a company to top gear – a culture of discipline is critical.

Think about it – a CULTURE of DISCIPLINE. This means that everyone does what they are supposed to without stepping on toes, do what they do best and are diligent about the task at hand. No one has to remind them and they produce excellence for the team. And then you get a team of teams, a culture that pervades the organization. This is what is required for an organization that transforms itself.

I am re-reading this book “Good to Great” by Jim Collins. It’s a classic. I really like books that dwell on evidence and then synthesize that into evidence-based recommendations. I have taken a few of those recommendations and internalized them into traits, I present a few in “Traffic Signals” here.

So here are a few traits of Great companies -

image

Red: Negative Correlation (with Good to Great)

Green: Positive Correlation

Black: No difference.

The culture is an enabler to excellence. It is the HOW.

The 3 circles need to intersect to understand the WHAT.

The hedgehog concept revolves around , in my mind 3 questions

Do you love what you’re doing?

Do you excel at what you’re doing?

Does it pay ?

If the answer is true – the you’re excelling and if its true for the organization – it is poised to be great in the future.

A good pre-read is 7 Habits…because it can link the individual habits of successful individuals to the culture required for greatness in an organization.

Sunday, May 6, 2012

Trade offs

In the 70s scientists noted that people who had hookworms did not have allergies and asthma. Yuck – who wants hookworms? Well do you want asthma instead ? Nature is all about trade offs. The obvious symbiosis is the result of intricate trade offs at every level. This logic applies in various domains including Software Architecture. An analysis of the architecturally significant trade offs is essential to objectively understanding any complex system relative to risk themes.

Comprehensively analyzing software architectures can be simplified by working along significant attributes. ATAM (Architecture Trade off and Analysis Method) from SEI was developed by Carnegie Mellon University. Between 2006-2009 I visited Carnegie Mellon and took several courses and got myself certified as a Software Architect.  The primary benefit was to gain an appreciation of trade-off analysis based on specific attributes across facets of architecture.

It is another tool in the strategic and objective thinking toolbox that every enterprise architect needs. For more information: http://www.sei.cmu.edu/architecture/tools/evaluate/atam.cfm



Monday, March 5, 2012

Building a Stronger Team that YOU work in

Anyone who has worked in a team knows that there is no simple answer for team success. Each employee/team-member offers a unique perspective and comes from a unique background (typically). This is even true in homogeneous teams, or centers of excellence. Building and sustaining team morale can be difficult especially in homogenous teams because direct comparisons can be drawn very quickly.

I recently read a general article, http://www.inc.com/jeff-haden/the-5-qualities-of-remarkable-bosses.html , this outlines a list of things bosses should do to build and sustain remarkable teams. I could not find myself agreeing more.

 

1. Develop every employee.

2. Deal with problems immediately.

3. Rescue your worst employee.

4. Serve others, not yourself.

5. Always remember where you came from.

The list above outlines how to be a good boss. However if you are team member these rules apply as well.


In order to help continue to build a stronger team, there may be a few more that need to be added to the list.

6. Build Trust: this is the nucleus of the team. Without a core sense of trust in one another, the team can breakdown into clusters and tuples.

7. Enable & even enforce team transparency: In order to build trust, establishing a level playing field of opportunities within the team. “The right person on the right seat in the bus” – Collins.

8. Ensure consistent and equal visibility: In an idea economy  it is very important to ensure that the right contributors are cited and given the opportunity to present. 

9. Enable and enforce core value from the ground-up: developing a core values set –a set of team values/principles and operational rules is very critical.

10. Be a follower: over time every team member must learn and demonstrate behaviors of a good follower.

As a leadership team member, each member must play a dual role sometimes – leader and follower. There is so much spotlight on being a good leader today, that people don’t have the skills anymore of being a good follower. Oftentimes a good leader is also a good follower.

Thursday, November 3, 2011

Rolling Back Transactions

While doing a spot check on code and configuration – I noticed that the developer wasn’t throwing a runtime exception nor was he setting rollbackOnly in a CMT EJB.

I was told that any exception will rollback a transaction and it is not possible to setRollbackOnly, and there are only 4 types of transaction attributes on CMT. Wrong, wrong and wrong.

1. If your code throws an application exception – the container expects the bean to handle it. However, if your bean throws a runtime exception (or subclass), like javax.ejb.Exception – the container will rollback the transaction.

2. If you don’t want to throw RTEs all around your code – and/or you have massive catch all exception code blocks, you should context.setRollbackOnly – to rollback transactions.

3. There are various transaction attributes – 6 to be exact: Requires, Requires New, Supports, Not Supported, Never & Mandatory. Never and Mandatory are opposite to one another. Requires starts a transaction if not called with one, Mandatory will throw an exception if called without a transaction, Supports will use one if there is one, but won’t complain….rest are self explanatory.

What’s most important about Transactions are how the transaction propagates in both directions (commit and more importantly rollback)

Thursday, October 20, 2011

GoF: Polymorphism, Encapsulation, Inheritance and Delegation

Polymorphism is when your client has a reference to an interface, and it’s concreate class can be one of many implementations. A different different behavior or value in a subclass can be utilized without a switch or an if. Parent class class defaults can be inherited and overridden where necessary. A method can be declared in a parent class, but each subclass can have a specialized implementation/logic of that method e.g. calculateArea();


Encapsulation is a core principal, it refers to the bundling of data with the methods that operate on that data. Basically, logic and data go together. If you have a class, with properties and logic together – it is encapsulated. Helps separation of concerns and reduces surface areas – also supports immutability – since only the object can control the data.

 
Inheritance is the ability of objects in an Object-Oriented language to inherit properties and methods of other objects e.g. in Java, use extends or implements keywords.

Delegation is when at runtime you can invoke a different object to complete a task dynamically. A useful OO pattern that has seen several implementations in Java EE.

The ability to frequently modify or add functionality, or quickly fix defects (Extensibility & Maintainability) comes from early design decisions.

All in all – these are core design principles – any object oriented analysis and design must respect, consider and apply these principles.

Wednesday, October 19, 2011

Java SE Security APIs and Frameworks

 Java SE has a deep foundation for security – there are a variety of APIs and frameworks that plug on top of various security impls.

  1. JAAS: Java Authentication and Authorization Services
  2. GSS: Generic Security Services. Think Tokens.
  3. JCE: Java Cryptography Extension. Keys and Ciphers.
  4. JSSE: Java Secure Sockets Extensions. SSL and TLS.
  5. SASL: Simple Authentication and Security Layer. Layer between Client and Server – describes the how. RFC 2222

 

TLS (SSL) is a point-to-point, transient only solution which provides no context, discrimination to content. Authentication, confidentiality and integrity is provided.
MLS (Message Layer Security) is an end-to-end security because it stays encrypted at rest and in motion. It is encrypted by the sender and can only be decrypted by the intended recipient. It does not depend on the transport layer.

Realm is the complete database of users and group, a user is an individual, a group is a collection of individuals, each group or individual can be assigned a key to the locks (aka role). In Java EE you can specify whether to propagate a client identity to the bean container or specify a run as. There is no choice either way but to trust the identify – as there is not authentication data propagated just the identity.

The EJB interoperability protocol is based on IIOP/GIOP 1.2 and CSIv2 (Common Secure Interoperability protocol).

Tuesday, October 18, 2011

Gang of Four Patterns for Java EE developers : Cheat Sheet

Here is the list of Gang of Four patterns related with an actual implementation in Java EE. Read on – it will make sense (hopefully).

Strategy: EJB interface.

Defines a family of algorithms, encapsulates each one and makes them interchangeable. Strategy lets the algorithm vary independently from clients who use it.

Decorator: Dependency Injection.

Attach additional responsibilities to an object dynamically. Provide a flexible alternative to sub-classing for extending functionality.

Factory: EJBs .

Define an interface for creating an object, but let the subclasses decide which class to instantiate.

Chain: Filters.

Decoupled requester and handler. Chain the handlers, one of them must handle it.

Singleton: JNDI

Ensure a class only has one instance, and provide a global point of access

Flyweight: JDBC Connection Pooling.

JDN

Adapter : Use the same interface, but adapt to other classes.

Façade: Single point of entry for a sub-system.

Template: Defer implementation to subclass.

Builder: For complex varying structures.

Iterator: Access items in any collection implementation.

Thursday, October 13, 2011

XML Processing in Java EE 5

All of the new Web Services API requires XML processing. Thankfully there have been changes to how Java EE will handle that as well with a fresh batch of updates.


JAXB 2.0: Improves vastly over JAXB 1.0

W3C XML Schema features (fixes missing bindings)

Adds javax.xml.bind.annotation and supports Java-to-XML binding.

Reduction in generated schema-derived classes.

Validation via JAXP 1.3 validation APIs

Smaller runtime binaries.

Schema compiler, Schema generator and Binding runtime framework.

JAXB 1.0 allowed validation: at unmarshall time, and on-demand validation on the content tree. JAXB 2.0 allows validation at marshall time and unmarshall time.


Streaming API for XML (StAX)

StAX is the all new efficient API for XML, it has a lot of great features:

  • Stream-oriented
  • Event-Driven
  • Pull-design
  • Read/WriteYou can create fast, light-weight, bi-directional parsers that is easy on the heap.
    JAXP (Java API for XML Processing) family includes StAX, TrAX, SAX, and DOM. StAX is good for low memory and limited extensibility applications.
    Pull Parser – simpler than SAX, more memory efficient than DOM.
    SAX can’t write – and isn’t bidirectional. DOM is way more powerful and flexible. One would dump SAX for StAX. An iterative pull parser – stax, an event driven push parser – then go for SAX.
    I can’t see anyone using SAX anymore. Why would you? Unless you don’t want a cursor and iterator concept in your code – or you simply hate procedural and believe everything should be read-only events for XML processing. XMLStreamReader or XMLEventReader are the Cursor and Iterator APIs – well, Iterator APIs can do things a Cursor cannot do: Iterator is more extensible and flexible. Cursor is efficient, performant and memory friendly – ideal for small JVMs and JME

        Wednesday, October 12, 2011

        JAX-WS in Java EE 5

        JAX-WS: Java API for XML Web Services. Does message oriented as well as RPC oriented services. Hides complexities of SOAP. No need to generate or parse SOAP messages (or understand the structure or format).

        The JAX-WS endpoints must be annotated with @WebService or @WebServiceProvider. The business method must be annotated @WebMethod – a Service Endpoint Implementation (SEI) will be generated for this. JAXB compatible parameters are required.

        Um, if you think Web Services or Clouds are NOT important, I hope the following stat will convince you.

         

        The Client needs @WebServiceRef – the reference to the service (or wsdlLocation). Get the port from the service and then invoke the exposed method on the service. Yes you need the interface to the service.

        JAX-WS 2.0 Support WS-I Basic Profile Version 1.1, SOAP 1.1 and WSDL 1.1.

        There is support for doc/lit, rpc/lit, static ports, dynamic proxies, and DII.

        All in all JAX-WS seems like a winner!

        Well, can you still use SAAJ? Yes – it gives you direct access to the SOAP protocol and the SAAJ 1.3 API supports SOAP 1.1 and SOAP 1.2 specifications.

        <Message>

        <Part>

        <Envelope>

        <Header>

        </Header>

        <Body>

        </Body>

        </Envelope>

        </Part>

        </Message>

        You can have Attachment Parts as peers to the Part. all Under the soap message but outside the envelope.

        The attachment part will contain MIME headers and the content (any).

        Um – BTW – you can use JAX-B to send SOAP Attachments too – so why would you want to bother with the SAAJ APIs is going to remain a mystery. But it’s there.

        Is WSJF "better" than traditional ROI calculations for Applications?

        I love road trips, and i like analogy.   The Premise: Two couples are planning a road trip. The "Perfection" group: This group spe...