SOA Challenges

Service Oriented Architecture (SOA) places signficant priority on software quality and testing.

I just read a vendor whitepaper about SOA.  The paper emphasised a number of points about SOA and software quality, as indicated by the quote:

"Developing an SOA that guarantees service performance, scalable throughput, high availability, and reliability is both a critical imperative and a huge challenge for today’s large enterprises."

This whitepaper from Oracle confirms for me my experience that SOA is placing greater emphasis on the importance of software quality and testing.

Testing becomes the pivotal role in integration activity where SOA is used to glue together larger systems.  It is at testing where we see all the components and the fabric that binds them come together into a working solution.

Not only must testing support function, end-to-end, and acceptance testing of the SOA integration, but as this paper highlights other quality attributes are critical to the success of SOA:

• Performance
• Scalability
• High availability
• Reliability

These are difficult areas to test and evaluate particularly where SOA is integrating many and large separate applications.

My experience on testing projects involving SOA is:

• Building end-to-end test environments is difficult, and we must use stubbing and harness to contain the scope of testing.
• Performance testing is a must, as SOA communication between applications impact user response times significantly.
• SOA is distributed, which means vulnerability to failure of a single component can bring down other interdependent systems and applications.
• Performance monitoring should run tests of user experience periodically.  Many of these tests can be drawn from automated tests used for performance or functional testing.
• Testing is more technical, as tests are constructed around SOA messages.  Tool support (either commercial or open-source) is essential.  Frequently manual / business testers aren't suitable in this areas, as there is no user interface, and instead tests must be coded to send and retrieve messages.
• Asynchronous behaviour of messaging makes functional testing more complex.  We must deal with messages coming back in different orders, it is not always send then receive, other messages may be interleaved.

When adopting SOA, testing teams will need to evolve their strategy regarding how the evaluate and test.  New methodology and technology will need to be incorporated to address the points above.

7th Australian Test Managers Forum

The 7th Australian Test Managers Forum concluded on Friday 24 April.

Numbers were down on previous years.  Previous years have always been a sellout 5 or 6 weeks before the event.  This was clearly a sign of the current economic conditions, with most frequent attendees citing tight travel, training and conference budgets, as well as reluctance on staff to be out of the office in uncertain times.  However we had pretty close to the same number of representative companies, just less multiple attendees.  Nevertheless, the interaction was the same as in the past, and a rewarding time was had by all.

Particularly enjoyable for me each year is the ideas exchange with people who are equally passionate.  What I like best is the information interchange and sharing of what has and hasn't worked for each other.

The most critical challenges this year (from the participat challenges survey) :

1. Requirements - still number 1 from last year.  We often experience poor quality requirements leading into testing, and no doubt the same is happening for development.   Testing needs to work more in the requirements phase to have clear expectations on what is being tested.  Similar our role in this phase means that we test the requirements specifications and find defects earlier.
   
2. Return-On-Investment: the global economic crisis is putting more pressure on us to demonstrate business value and to justify costs.  With ROI, if we dont show the "R" (the return to business), all they see is the "I" (investment) as a "C" (cost).
   
3. Environments: with the introduction of larger integration and technologies such as SOA, environment management is becoming more complex and downtime wastes testing effort.  While virtualisation (like VMWare and others) is changing test environment management significantly, there is still significant challenges in specifying and controlling testing environments.
   
4. Early involvment: the biggest cost savings that testing will make will come from working earlier in the lifecycle.  Yet it is a challenge for test teams to get engagement in early phases.  We need to use test results to show defects coming out of early phases, such as requirements bugs, then offer more cost effective ways that testing can find these bugs, e.g. through requirements and specification evaluation (such as inspections, modelling, walkthroughs).
   
5. People: while the pressure has come off staffing with the economic downturn, selecting the right candidate, and having succession plans from existing staff leaving remains a critical issue.  Here focus on skills definition, evaluation and staff development techniques are required.
   
6. Scheduling: the schedule squeeze on testing at the end of the lifecycle remains.  Development still slips, yet the release date doesn't.  Many organisations discussed the greater move towards agile, which has other challenges for testing, to help alleviate this.  Also discussed we techniques are sourcing strategies, flexible resourcing, and risk-based prioritisation of testing.

The were other challenges discussed, but the survey showed these as not as critical to the audience as a whole:

• Resourcing
• Estimation
• Automation
• Process Improvement
• Methodology
• Coordination
• Stakeholder Engagement
• Governance
• Agile
• Release Management

Much of the information, surveys and presentations will be collated and progressively added to the website at http://www.kjross.com.au/page/News_and_Events/Australian_Test_Managers_Forum/Test_Managers_Forum_2009_