While the term agile itself is indeed ten years old, the philosophy and approach of iterative and incremental development (IID) has a surprisingly rich and extensive history. The article Iterative and Incremental Development: A Brief History (PDF) by Craig Larman and Victor R. Basili published in IEEE Computer discusses how the ideas of IID actually predate the existence of software, and have been used and promoted in every decade since for over 50 years. This article also discusses a classic 1970 article by Winston Royce well-known for supposedly promoting the use of waterfall, but which actually recommended the development of an initial pilot or preliminary version prior to creating the final version intended for delivery to the client. Larman's and Basili's article also discusses the ongoing evolution of the standard used by the U.S. Department of Defense for software development. The initial standard was document-heavy, gated, single-pass waterfall, but the high rate of project failures led to first the allowance of and eventually the full adoption of IID approaches.

So a limited set of organizations (often governments due to their byzantine contracting restrictions) did experience an evolution from waterfall to agile over time, but notably they started with a misunderstood version of waterfall. The use of IID approaches has always been part of the I.T. industry.

What about the effectiveness of agile? The second article Quality metrics: Software quality attributes and their rankings is an interview with Capers Jones and Olivier Bonsignour regarding their new book The Economics of Software Quality. The authors in this book discuss the effectiveness of over 100 quality factors using research based on over 10,000 software projects. On a scale of -10 for extremely harmful to a scale of +10 for extremely valuable, Agile methods rated a 9 - highly valuable - while waterfall only rated a 1 - barely useful.

These two articles highlight the fact that agile-like methods have been in use since the start of the software field, and are on average far more effective than the waterfall approach. This leads me to conclude that there is really no defensible reason for organizations to mandate or promote the use of waterfall.

Tooling

WebSphere provides the wsadmin command-line interface for interfacing with WebSphere servers. Two scripting languages can be used: JACL (the default) and Jython, a Java-based version of Python. Invoke wsadmin with the argument "-lang jython" to use Jython. I recommend the use of Jython for two reasons. First, it is a more mainstream language. Second, the WebSphere Admin Console shows scripting commands in Jython equivalent to the GUI operations you last performed. See the screenshot below for an example:

You may think that since the Admin Console provides equivalent scripting commands to GUI actions that you could just copy and paste these commands into a script and you would be set. Unfortunately if you want the script to be re-runnable (on a given server) or reusable (on other servers), you will almost always need to make modifications. Certain operations, like creating a data source, will fail if there is an existing data source with the same name. So to have a re-runnable script you will need to add logic to your script to first detect if the data source you want to create already exists. Many commands use a hard-coded configuration id to refer to a particular item to be configured. This id generally consists of the item's name, directory path to the configuration file, name of the XML configuration file, and XML ID of the item within the file. While the first three are constant, the XML ID appears to be a numerically generated ID that will vary between servers. So you need to change the script to look up the configuration id.

The changes required to adapt the script for broader use are not trivial and require use of WebSphere APIs - in particular the AdminConfig and AdminTask objects provided to Jython scripts that are executed within wsadmin. The best tooling I have found for writing these scripts is to use Rational Application Developer which includes a Jython editor that includes wsadmin objects as part of the content assist. See the screenshot below for an example:

Examples

With the tooling in place let us look at some examples. The following scripts all assume a simple WebSphere topology of a single node with a single server that we are setting up - no clusters or multiple servers to worry about.
A number of settings require the configuration id or name of the server or node, so here's how to look them up:

nodeConfigId = AdminConfig.getid('/Node:/')
nodeName = AdminConfig.showAttribute(nodeConfigId, 'name')
serverConfigId = AdminConfig.getid('/Server:/')
serverName = AdminConfig.showAttribute(serverConfigId, 'name')

AdminConfig.getid() is one of the primary methods to use to look up configuration ids. But what is that "/Server:/" syntax used as an argument? That is called a containment path, which is a XPath-like syntax for looking up configuration ids. For more important details on containment paths and configuration ids read this excellent article by Vincent Partington.
As mentioned above, this only works when there is a single server defined on the node - with multiple servers, the call AdminConfig.getid('/Server:/') would return multiple server config ids separated by newlines. In this case if you want to configure a specific server you can look up the server configuration id by name as follows:

serverName = "testserver1"
serverConfigId = AdminConfig.getid('/Server:' + serverName + '/')

Having a configuration id is a start, but we still have not changed any settings. So here's a basic example of setting the maximum heap size to 1 GB:

maxHeapMb = 1024
jvmConfigId=AdminConfig.getid('/JavaVirtualMachine:/')
AdminConfig.modify(jvmConfigId, '[ [maximumHeapSize "' + maxHeapMb + '"]  ]')

This follows the standard pattern of first obtaining the configuration id and then changing the setting. I obtained the code for the AdminConfig.modify call from the Admin Console. But how did I figure out the AdminConfig call, in particular which containment path to use? This was painful initially to figure out. The hard-coded configuration id provided within the AdminConfig.modify call was "(cells/Node1Cell/nodes/Node1/servers/Server1|server.xml#JavaVirtualMachine_1183121908656)". The prefix "JavaVirtualMachine" on the XML id is the key type to use in the containment path.

Not all configuration commands require a configuration id. The AdminTask object typically takes arguments specifying the server and node name. Here's an example that prevents applications from accessing internal WebSphere classes:

AdminTask.setJVMProperties('[-serverName ' + serverName +
' -nodeName ' + nodeName + ' -internalClassAccessMode RESTRICT]')

The last step in any configuration script is to save the configuration changes. This is straightforward:

AdminConfig.save()

Resources

Here are some useful resources for this topic:

• Excellent article on the differences between configuration ids, containment paths and object names for WebSphere scripting.
• IBM article introducing scripting WebSphere using Jython
• Documentation on the WebSphere 7 Express sample scripts library
• List of commands available on the AdminConfig object
• Free online Jython language book
• Book published by IBM called "WebSphere Application Server Administration Using Jython"

Happy scripting!

There are a number of pragmatic practices that address this objective while avoiding the pitfalls afflicting formal traceability. I will start with a simpler form of traceability and then move on to practices seemingly unrelated to traceability.

Specification Highlighting to Track Test Coverage

A tester given a written specification to verify the software against must have some way of keeping track of their progress. One simple method is to highlight each statement in the specification as they test it. This is essentially tracking test coverage of the specification. I credit James Bach for this idea.

Testing Dashboard

A testing dashboard is great at illustrating overall testing progress and can be used as part of the summary in the final test report. To produce the dashboard first divided the system under test into test areas that usually correspond to features, components, or significant non-functional requirements. Then create a grid with the test areas as rows, and the columns report testing progress for each area. Key information to report per area is an assessment of quality (is this area ready to ship?), and the thoroughness of testing (test coverage and test effort). For an example and fuller explanation see James Bach's presentation on testing dashboards.

The testing dashboard is closest in form to a requirements traceability matrix - there's still a grid and something like requirements are listed down the grid. The big difference is that the dashboard shows a summary of testing for each area rather than listing individual tests and showing how they map.

Agile User Stories with Acceptance Tests

A common approach used by Agile methods is to divide requirements into fine-grained increments of business value called user stories and then define with examples or automated tests the criteria by which to accept that the story was correctly implemented. This takes a two-pronged approach to address traceability's objective. First, using fine-grained increments of functionality that are quickly developed minimizes the amount of work-in-progress that needs to be tracked and thus reduces the chance of mistakes. Second, the process produces tests for each user story early, sometimes before coding starts, so missing tests for a requirement is much less likely. Agile basically sufficiently changes how development is done to negate virtually all the sources of problems that would otherwise justify a requirements traceability matrix.

Task Tracking via Task Board

Popularized by Kanban but also used by many Agile teams, a task board is a practice for tracking the status of tasks. The board has multiple columns representing different status: the minimum set is usually "Not Started", "In Progress", and "Done". Tasks are placed on the board according to their status, and the team meets daily to discuss the tasks and update appropriately. I vastly prefer a physical task board on a wall, but for teams not co-located many online versions exist. Tasks are often quite fine-grained - one user story is often decomposed into multiple tasks.

Many teams include a status of "Testing" on the board, or use a symbol on the card to signify the completion of testing. Other quality control procedures such as code reviews can also be tracked. See the image below for an example of a task board with such states.

The combination of fine-grained tasks, daily updates, and tracking quality ensures that each requirement that is tackled by the team is properly developed. This achieves traceability's objective.

Conclusion

Despite formal traceability being too effort-intensive for too little value the underlying objective should not be ignored. I have described a variety of alternative practices that help ensure that requirements are properly developed and tested with a more modest level of effort.

Formal Traceability

The most frequent incarnation of formal traceability I see is the requirements traceability matrix (RTM). This is a two-dimensional table, often represented by a spreadsheet, in which the one dimension lists requirements and the other dimension lists the artifact being traced to - business objectives, test cases, or design documentation. Cells within the table are filled in with a mark when the corresponding artifact and requirement are related.

The example traceability matrix below lists requirements across as columns and test cases down as rows. Notice that requirement 2.2 has no corresponding related test case, which is considered a traceability gap that indicates that this requirement has not been tested.

Traceability Troubles

Below are listed the aspects of traceability that trouble me.

• I have successfully developed and enhanced high quality software applications that have pleased clients and been used in production without needing formal traceability. So does it really provide value? Traceability seems like it addresses the problem of having unimplemented or untested requirements. This has rarely been a problem on the development projects I have been on. We are much more likely to struggle with ambiguous or unclear requirements, missed or new requirements, and incorrect logic in obscure special conditions.
• The tracing of requirements to test cases assumes they can be identified and listed which in turn limits the testing approaches that can be used. The context-driven school of testing would challenge these limitations. Exploratory testing in particular does not fit the confines imposed by a formal requirements traceability matrix.
• While the value of producing the initial RTM is unclear to me at best, I am even more dubious about the value of maintaining a RTM over time. The typical claims is that for for future requirement changes (e.g. enhancements) the RTM can be used to do impact analysis and determine what tests need to be executed and/or changed. Regarding impact analysis, the RTM might get you started, but the analysis must always consider code-level dependencies not reflected in the RTM (through, for example, reuse of common functionality), and must consider other requirements and functionality that are not directly related (and thus not reflected in the RTM) but make assumptions that are no longer true under the proposed change. The idea that the RTM can be used to determine what tests to executes makes several poor assumptions. First, that the tests are manual and not automated. If you have an automated test suite, you can just rerun the whole suite to identify failing tests - you never need to worry about running a subset. Second, that reusing manual tests is a good idea. If a test is worth re-executing often, it is frequently worth automating. For those tests that are manual, much of the benefit of having human testers is lost if they simply re-run the same scripts over and over, since there is a low probability of such re-executions failing and testers are less likely to find defects when they do occur. A better approach is to let the tester come up with their own variations each time to try to creatively break the system.
• Tracing requirements to design is conceptually flawed. It at best very indirectly addresses the objective of traceability - ensuring the customer gets the functionality they require. Doing this mapping is in reality quite hard because the mapping from design to code (and from requirements to design) is not a strict one-to-one and is often a complex many-to-many, especially for holistic non-functional requirements like performance or security. Furthermore, the idea of tracing to design assumes that there are fine-grained pieces of design that can be identified and referenced. This presumes a certain approach to design and its documentation that is at odds with many modern methods of software development.
• For small-scale software applications, the system is simple enough that producing a RTM is a reasonably small effort. The number of requirements and test cases is relatively small. But the chance of errors occurring that would be caught be traceability is likewise low because the system's complexity is low. For very large systems with hundreds or thousands of requirements and tests, it is far more conceivable that requirements may be forgotten to be implemented or tested. But while the benefits of formal traceability are higher, the efforts are much, much higher. For a RTM, the number of cells to complete essentially scales non-linearly with the number of requirements in the system.
• I do not understand the obsession with testing in the context of traceability and especially RTMs. Reviews or inspections are much more effective at finding defects than testing. So why do I never hear of formal traceability from requirements to reviews?

Conclusion

Software engineering researchers and industry leaders have reached the same conclusions about traceability. Andrew Kannenberg and Dr. Hossein Saiedian in Why Software Requirements Traceability Remains a Challenge (PDF) are clearly big proponents of requirements traceability, but look at their conclusions:

• "Unfortunately, manual traceability methods are not suitable for the needs of the software engineering industry."
• "Currently existing COTS traceability tools are not adequate for the needs of the software engineering industry."

Robert Glass in his book Facts and Fallacies of Software Engineering points out the underlying cause of troubles implementing traceability: "when a requirement commonly links to 50 or more design requirements and each of those links to some very much larger number of coding elements and coding elements may be reused to satisfy more than one requirement, we get a burgeoning complexity problem that has resisted manual solution and even tended to thwart most automated solutions" (page 78)

I hope the above discussion has convinced you to tread cautiously when it comes to traceability. In a future article I will look at alternatives to formal traceability that achieve the same underlying objectives.

Shakespeare quotes aside, every software development project has to make decisions on how many defects to fix and which ones to leave alone prior to shipping. While I have seldom seen this question debated within projects, the advice from industry thought leaders varies considerably. The Agile and Lean methods of software development in particular have somewhat opposing perspectives.

I believe that considering both sides of this question provides a fuller understanding of the issues and better equips us to answer appropriately. Therefore in the two sections below I explore the reasons behind both sides of the debate.

To Fix

• Shipping poor quality, defect-ridden code can upset users, turn away customers, and lead to a hard-to-shake bad reputation.
• The decision that a feature is worth developing is made with the expectation that it will work correctly. So any defects found in a feature means that the feature is still incomplete until these issues are fixed.
• Defects provide feedback regarding the development process. Each defect represents an opportunity to do a root cause analysis of what led to the defect and put countermeasures in place to prevent re-occurrence. The Lean mindset of "Stop the line" demands that new development be put on hold to fix newly discovered defects.
• Defects introduce the risk of compounding quality problems. The impact of a defect can be more significant than initially realized. Defects can be inadvertently replicated in other parts of the system. Enhancing components with too many defects can slow progress to a halt, as the system becomes essentially a shifting quicksand that is too unstable to work on. Constantly fixing defects helps maintain a high velocity of development over time.
• To mitigate risks in not fixing defects, each defect needs to be analyzed to understand its impact, cause, and required changes to fix. But after performing this analysis most of the work is usually done - the fix is relatively straightforward. Waiting to decide later to fix the defect (e.g. in a subsequent release) causes all the knowledge gained in the analysis to decay over time which is wasteful (in the Lean sense).

Not To Fix

• Significantly delaying the release of software to fix all defects leads to a loss of immediate revenue and potentially loss of market share due to competitors beating you to market. So you cannot afford to wait to fix all defects.
• The entrepreneurial mindset, especially for startups, is to ship early to get feedback from paying customers. Perfection is the enemy of the good.
• Under at least some versions of Scrum, defects are considered new tasks that are added to the product backlog to be prioritized by the product owner. This prioritization is based on the defect's impact (severity and likelihood of occurrence) and the effort required to fix it. Many minor defects will therefore likely never be fixed as new functionality will typically be of higher value.
• Stopping to analyze and fix defects disrupts developers who are in the middle of working on other functionality and is wasteful.
• Fixing defects in functionality that is already otherwise finished development and testing will require additional regression testing. Not fixing now and waiting until enhancements to this functionality are needed minimizes the extra effort required.

Conclusion

Shakespeare was wrong. There is actually a third perspective regarding whether or not to fix defects: avoid the question as much as possible by focusing on defect prevention. The Lean mindset of building quality in avoids all the waste associated with finding, analyzing, and fixing defects and should be our preferred approach. Only when it fails and the occasional defect is introduced do we then have to answer the question.

• Documentation is more effective than code at communicating high-level or cross-cutting design and operational concerns.
• Larger organizations or distributed organizations cannot rely on face-to-face communications or having everyone co-located in one room so documentation has a role to play in knowledge transfer and communication.
• Documentation can target a non-developer audience, such as business representatives.
• Documentation helps protect against team turnover, which while a bad practice is not uncommon when using a vendor for development or maintenance or using separate development and maintenance teams.
• The act of creating documentation helps clarify thinking and identify gaps, thus functioning as a form of quality control.

This last reason is actually quite significant as it is often under-appreciated. In my experience it has happened quite often when I am working on design documentation for a body of code that I identify things that are sub-optimal, such as a badly named class or an unwanted dependency between components. This form of quality control is most valuable, however, when it uncovers gaps such as missing functionality or flawed design. Such gaps are very difficult for most other forms of quality control like testing or reviews to find.

The use of documentation templates makes it much easier to find these gaps by acting essentially as checklists of items to consider. I recently came across a great set of comprehensive templates covering all aspects of software development by Klariti . For a team or organization the price of these templates is ridiculously low - the full set of software development templates costs far less than a day's salary - and there are an assortment of free templates as well.

There are some potential drawbacks to avoid when using documentation templates. Some people have a tendency to want to fill in every section of a template or to use all available templates. This can waste a lot of time and effort. Focusing on identifying what is relevant and useful to document and doing only that is much more effective. Another drawback is that while the templates are in Office format (Word or Excel), you might be better served using a different medium such as a Wiki or a text-based format that is more friendly to version control. In these cases I would convert the templates to the desired medium.

Even if your team or organization has some templates, I think it would be beneficial to check out Klariti's templates and use them as a checklist to see if there is anything missing or needing revision within your own. That link again is: Klariti's software development templates.

This chain of thought then prompted me to ask myself this same question. What do I find most disturbing when reviewing code? My instinctive reaction was to answer "defects", but upon a little reflection I realized this was not true - I expect to find defects as a fundamental principle of quality. So it usually does not bother me to find a few during a review.

There are times when I am very disappointed when reviewing code - what am I finding at those times? Here are some specific occurrences:

• Code riddled with defects reflecting a fundamental lack of understanding about the requirements.
• Code very difficult to understand due to poor names and overly complicated logic that seemed repetitive or unnecessary.
• A large amount of non-GUI code written without any supporting tests.
• Code with inconsistent formatting and style.

What is the common theme here? After further reflection, I realized that the common element underlying these situations that I find most disturbing is a lack of professionalism / craftsmanship. This can manifest in a number of ways as indicated by the above list. The key criteria is whether a developer is helping achieve or is hindering our mission as software developers, based on what they produce for code. My evaluation of what is most disturbing is at its essence based on my core values and beliefs concerning software development.

Too often I see projects and even entire organizations misaligned with one or more of these principles and in the worst cases taking actions in complete violation of these principles. The result is predictable: poor quality or significantly extra cost to achieve adequate quality, with the complete absence of high quality.

Achieving high quality is difficult. It suffers from what I call the weakest link problem: no matter how many things you do well, it is the area you are poorest at that generally dictates the overall quality. Another way of putting it is this: there are many ways to fail at developing high quality software and only a few ways to succeed.

So to help you achieve high quality here is my list of the top seven quality principles in software development. For each principle think about the implications - how might your organization be acting in ways that contradict it? How might you change to better align with the principle?

People as individuals and teams are the most significant factor affecting quality

Software engineering research has consistently found that variations between people are by a large margin the single largest contributing factor to quality and productivity. While good process is important, good people have a much greater impact.

Achieving high quality requires people capable of doing high quality work, both individually and as teams. While individual excellence is important, most software is produced by a team of people and it is the team's overall quality of workmanship that ultimately determines the quality of the software being produced.

Within organizations, throughout the blogosphere, and even in many software development books too much attention is placed on process and methodology and too little on the quality of the people doing the work.

All software has defects

Any significant piece of software has defects, no matter how carefully it has been produced. Zero defects is an Utopian ideal, but not a realistic goal. With extreme discipline and effort you can get very close to zero defects, as evidenced by the group working on the space shuttle control software, but under more normal conditions defects are unfortunately a fact of life.

It is impossible to fully test software

For even the most trivial of functionality like adding two numbers there are a nearly infinite set of tests that could be performed. So completely testing a significant piece of software is impossible. Testers therefore must appropriately choose and perform an extremely small subset of all possible tests.

Defects are more expensive to fix the later they are found

As time elapses from the point at which a defect is introduced, the effort required to fix the defect grows. There are several reasons for this. The most significant is that as the software moves through subsequent phases of the software development life-cycle - from development into test and then finally into production use - the effort involved to get to that phase increases. Work previously done, like testing and deployments, has to be repeated to some degree. Another reason is that over time, the developer's memory of the problematic functionality in question fades, thus requiring more effort to recover the context.

Build quality in

This principle comes from the lean thinking literature and addresses the issue highlighted by the prior principle. Finding and fixing defects is considered waste - non-value-add work - and thus according to lean should be minimized. This is accomplished in a twofold manner: first, use practices that help prevent the introduction of defects, such as specifications with examples, and second use practices such as test driven development and code reviews that find defects as quickly and cheaply as possible after they are added.

Adopt your approach to quality based on the level of criticality and complexity

Different contexts require different actions. Criticality and complexity are the two most significant factors to consider in order to determine the level of quality you require and the approach you will take to achieve it. Criticality is the measure of how important the application is to the business and/or the users and is generally assessed using categories such as life critical, mission critical, business important, and casual use. Complexity is the measure of how difficult it is to understand and work with the code. A number of factors increase complexity such as complicated algorithms, sheer volume of code, and poor architecture resulting in high coupling.

Higher quality requires more quality-focused activities and better execution

In order to achieve a higher level of quality corresponding to a lower volume of production defects requires that you introduce less defects and/or find and fix more defects throughout your software development process. This process can be modeled as a series of activities acting as feedback loops or filters that each prevent or remove some percentage of defects.

Using this model it becomes obvious that the only way to reduce the defects remaining at the end of the process is to either make existing activities more effective - what I call better execution - or add more activities to filter out additional defects. The choice of which quality-focused activities to use and the effort to put into each requires deliberate planning which I have written more about in my article titled Filter by Failure Mode Matrix: A Method for Planning Quality.

The constraints need to push us out of our comfort zone. We need to bump up against the constraints, repeatedly, in order to generate those learning opportunities. The constraints cannot be so harsh, however, that they prevent us from finding solutions that fit within them.

The nature of the constraint dictates the area in which the growth will occur. Twitter's constraint encourages conciseness and the ability to focus on the core message. Writing assessments in classes tend to set the opposite constraint of a minimum number of pages of content in order to encourage deeper exploration of a topic and more hours of practice in writing.

Thinking about this principle in the context of software development made me realize that the many different types of software are often differentiated by the major constraints they face:

Constraints can also be applied to processes or methodologies to provide learning opportunities. I have heard more than one ScrumMaster say they prefer one week iterations for teams new to scrum as a way of learning how to decompose functionality into the smallest possible increments. The concept of stretch goals is similar: set targets beyond the current capabilities of the team in order to push team members out of their comfort zone and seek innovative ways to meet those targets.

How can you use constraints to improve and grow? I encourage you to reflect on this and leave a comment sharing your thoughts.

Context

This pattern applies when you need to process a significant volume of data but the processing can be done incrementally on small subsets of the data. A typical example is loading a list of entities and then iterating through the list to process each one. While the results (output) of processing can be combined across all the entities, it is important that the input to the processing only requires a small subset of all the data, and not the entire list of entities. A code example illustrating this problem context is shown below:

List<Entity> entities = loadEntities();
List<ProcessingResult> results = new ArrayList<ProcessingResult>();
for (Entity entity : entities) {
  ProcessingResult result = processEntity(entity);
  results.add(entity);
}

Solution

Reducing the memory usage in the above example is based on the observation that loading the entire list of objects to process can consume a large amount of memory and is not necessary since we only use one object at a time. So the solution is to stream - incrementally retrieve - these objects instead of loading them all at once. For the consumer of this data the only change required is to first obtain a reference to the stream such as an Iterable that incrementally fetches data. Updating our prior code example results in the following (changed lines shown in green background):

Iterable<Entity> entities = streamEntities();
List<ProcessingResult> results = new ArrayList<ProcessingResult>();
for (Entity entity : entities) {
  ProcessingResult result = processEntity(entity);
  results.add(entity);
}

Examples

The mechanism to use for streaming objects will depend on the source of the data and may require significant changes compared to a bulk load. Here are some specific examples.

Parsing XML

Parsing XML files using JAXB is a convenient approach for converting the entire file into a tree of Java objects, but it populates the entire tree at once. To instead stream such data use the SAX parser provided as part of the JAXP API. The SAX parser is event-based, which means that it iterates over the entities (and attributes) of your XML and for each item invokes callbacks you define.

Querying Databases using Hibernate

When using Hibernate to query for a collection of entities it is convenient to simply ask Hibernate for the entire collection. A typical example of doing this using the query by criteria API within Hibernate is below:

public List<Entity> queryData() {
  Criteria criteria = session.createCriteria(Entity.class)
  // Add appropriate restrictions
  // ...
  List<Entity> result = criteria.list();
  return result;
}

When the criteria returns a large volume of data, however, this approach will consume a high volume of data. Instead use the scroll method on Criteria to return a ScrollableResults instance that can be used to iterate through the results. If you prefer to not expose the rest of the application to Hibernate classes, you can wrap the ScrollableResults in a special implementation of Iterator (which I leave as an exercise to the reader). The revision of the above example using streaming looks like the following (changed lines shown in green background):

public Iterator<Entity> queryData() {
  Criteria criteria = session.createCriteria(Entity.class)
  // Add appropriate restrictions
  // ...
  ScrollableResults scrollableResults = criteria.scroll();
  Iterator<Entity> result = new ScrollableResultsIterator(scrollableResults);
  return result;
}

This scroll approach only works when all the data can be processed within the same database transaction since the Hibernate session must remain open for the ScrollableResults to be able to continue fetching data. If this is not suitable then another option is to load the data using multiple queries that each return a subset of the data. One common example of this is when displaying search results to an user. Rather than showing all the results (which may number in the hundreds or thousands) show one page at a time and let the user step through the various pages of results. Due to the frequency with which this occurs I refer to this solution as paging. To implement this in Hibernate using the query by criteria API is fairly simple:

1. Start by creating your criteria object and defining its restrictions as you normally would.
2. Apply an ordering to the criteria. It is best if this ordering is consistent, by which I mean that database updates or inserts between queries will not result in invalid or unexpected results being returned. This assumes each query for a page executes in a separate database transaction which provides no guarantees of transactional isolation for the group of queries as a whole. In some contexts, consistency is not required. If it is then I prefer to use an auto-incrementing surogate primary key as the field to sort by in order to achieve the highest level of consistency.
3. Apply restrictions to retrieve only the specific page. This is done using the methods setFirstResult and setMaxResults on the Criteria object.

Consequences

One potential consequence of streaming data is a reduction in performance because data is loaded piece by piece rather than in bulk. To mitigate this, the solution is use what I call loading sets: define subsets of the total data volume that are small enough to not impact memory usage but large enough to minimize performance impacts. Then load the data one set at a time. The consuming API does not need to change: it can still iterate or stream over each loaded set, and then fetch the next set once the current one is exhausted.