Download the Complete DII COE Study Here
See also: CORBA Performance Overview

The Defense Information Systems Agency (DISA) facilitates the collaboration of multiple branches of the U.S. military in setting standards for information systems. One major initiative at DISA is the Defense Information Infrastructure Common Operating Environment (DII COE).

The DII COE Real-Time Integrated Product Team (DII COE RT IPT) focuses on standards for embedded and non-embedded real-time systems. It is important for defense projects using or targeting real-time technologies to participate in the DII COE RT IPT in order to ensure that their requirements are fully represented. Contact Lt Col Lucie Robillard for further information if you are a federal contractor or federal agency working on a defense program that involves real-time command and control technology.

Boeing Phantom Works has been working under the direction of the DII COE RT IPT. One of their tasks is to study the Real-Time CORBA vendors and products. Boeing presented the preliminary results of this real-time ORB trade study at the August 1999 meeting of the RT IPT. The final results were published in December 1999. This trade study includes a thorough benchmark of several aspects of three real-time CORBA products.

The real-time ORB products in the study include:

• HARDPack by Lockheed Martin Federal Systems, Owego
  
• ORBexpress by Objective Interface
  
• TAO by Washington University
  

The study was approved for public release and is available for download. The goals of the study were to:

• Benchmark CORBA ORB middleware products, and identify the products meeting or exceeding the standards for real-time technologies;
  
• Identify the viable RT CORBA products for recommendation into the DII COE infrastructure;
  
• Determine Real-Time standards compliance and interoperability with other ORBs;
  
• Highlight qualitative results (i.e. customer satisfaction, documentation) for each vendor
  

Additional information on the final results of the trade study is available from the DII COE RT IPT web pages:

DII COE RT IPT Web Page - RT CORBA Trade Study

NOTE: For access to this Real-Time (RT) DII COE web page please submit the access form to lucie.robillard@hanscom.af.mil. Access permission is restricted to IP address within the U.S. and to:

• U.S. Federal Agencies; or
  
• Companies and organizations that are working on contract to support the U.S. Federal Government.
  

The detailed data produced from the final benchmark will be provided to each ORB vendor and not disclosed publicly. Boeing has left the further disclosure of this detailed data to the ORB vendors. Objective Interface will make the detailed data for ORBexpress available upon request. To get the detailed data for the other products you will need to contact the other ORB vendors directly.

Selected Performance Graphs
The benchmarks were performed on a variety of hardware and software platforms, including:

• SunOS 2.6 on a Sun Ultra 167MHz, 256M RAM
  
• LynxOS 3.0 on a Motorola PPC, 300MHz, 256M RAM

The following graph shows the benchmark results on a single SunOS machine, using the loop-back adapter. No shared memory or other optimizations were used for any of the tests. The x-axis is the amount of data that was transferred for each benchmark, and for this test, the data type was float. The y-axis shows time for the two-way transfer in milliseconds. The bottom (fastest) line is the time it took to perform the transfer using a simple socket program.

DII COE benchmark results on a single SunOS 2.6 computer

The next graph shows a similar chart for a single LynxOS processor. HARDPack is absent from this chart since it was not available for the platform. Once again, the bottom line is sockets, and the next two lines are ORBexpress and TAO, as read from the bottom up.

DII COE Benchmark Results on a Single LynxOS PPC

Customer Service Satisfaction
In addition to the ORB performance benchmarks, the study also determined the quality and responsiveness of the Technical Support Department compare customer service satisfaction levels. These customer service levels were based on:

• Ease of Use for Software Development
  
• Ease of Use Compared with Other ORB's
  
• Satisfaction with Vendor Response to Critical Problems
  
• Satisfaction with Vendor Response to Routine Problems

The following graph compares these satisfaction levels, and shows the Technical Support Department provided superior customer satisfaction levels as compared to the other competing companies.

The entire DIICOE benchmark study is available here. The study includes the results shown above, as well as results for other platform configurations and other CORBA data types. The DII COE benchmark also includes tests for interoperability, predictability, and a review of the differences in architectures and platform support between the three ORBs.

Benchmark Environment
Boeing ran the benchmarks on both sparc-sun-solaris 2.6 and powerpc-vme-lynxos3.
The Solaris tests were run on an isolated 10 Mb Ethernet segment between two UltraSPARC 1/170 systems each with a single 167 MHz UltraSPARC I CPU.
The LynxOS tests were run on the isolated 10 Mb Ethernet segment between a Motorola MV3600-2 PowerPC (LynxOS 3.0.0) and a Cetia PowerPC (LynxOS 3.0.1).

What the Tests Measured
The tests were run using seven types of IDL operations. Each operation had only one "in" mode parameter as follows:

Description Parameter Type Operation Type OW Primitives A single struct containing a single array of a simple type One way OW Records An array of a struct that containing three small arrays of simple types One way OW Any A single type Any containing the "Records" data One way CR Primitives A single struct containing a single array of a simple type Two way CR Records An array of a struct containing three small arrays of simple types where the struct layout is aligned to a natural memory boundary Two way CR NA Records An array of a struct containing three small arrays of simple types where the struct layout is not aligned to a natural memory boundary Two way CR Any A single type Any containing the "Records" data Two way

Each of the above represents the averaging of multiple tests. For example, "OW Primitives" was the average of sending an array of CORBA octets, shorts, longs, doubles, and other simple types in a single struct.

Each test was run with a single client thread in four different scenarios:

Description Test Span O/S Delay Between Invocations Scenario 1 client/server on same machine Solaris none Scenario 1a client/server on same machine Solaris 70 milliseconds Scenario 2 client and server machines separated by 10 Mb Ethernet Solaris none Scenario 3 client/server on same machine LynxOS none Scenario 3a client/server on same machine LynxOS 70 milliseconds Scenario 4 client and server machines separated by 10 Mb Ethernet Solaris 70 milliseconds Scenario 5 client and server machines separated by 10 Mb Ethernet LynxOS 70 milliseconds Scenario 6 client and server machines separated by 10 Mb Ethernet Solaris & LynxOS 70 milliseconds Scenario 7 client and server machines separated by 10 Mb Ethernet LynxOS & Solaris 70 milliseconds

Only the results of Scenarios 1a, 3a, 4, and 5 were published in the report.

Some of the Results

The results of the average invocation times in the Scenario 1a tests are presented in the table below. All times are in milliseconds.

Scenario 1a Test Number of Argument Bytes Sockets HARDPack ORBexpress TAO CR Primitives 144 0.230 1.090 0.364 0.977 24,016 0.642 2.893 0.958 1.603 CR NA Records 144 0.230 1.092 0.383 1.176 24,016 0.642 5.382 4.957 7.980 CR Any 144 0.230 NP 0.9 3.4 24,016 0.642 NP 24.3 78.9

  * NP: not published

The results of the average invocation times in the Scenario 3a tests are presented in the table below. HARDPack was not measured on LynxOS for the final report. All times are in milliseconds.

Scenario 3a Test Number of Argument Bytes Sockets ORBexpress TAO CR Primitives 144 0.148 0.268 1.148 24,016 1.476 1.886 3.453 CR NA Records 144 0.148 0.285 1.170 24,016 1.476 4.878 21.620 CR Any 144 0.148 0.595 2.771 24,016 1.476 12.950 52.632

Note that the socket tests only sent simple octet arrays not the data types listed in the tables above. Processing the actual data types listed would have introduced additional processing for the socket tests. The socket times are reiterated for each test for ease of comparison only.

Please reference the RT IPT report for further results.

Objective Interface's Analysis of the Results

• The speed of all three products was well beyond the performance of most non-real-time ORB products in our experience.
  
• For all of the tests listed above and in the full RT IPT final trade study ORBexpress performed better than both of the other real-time ORBs in all tests.
  
• The ORBexpress results are consistent with our internal performance tests for the product.
  
• While the Solaris version of ORBexpress GT used by Boeing (V2.1.3a) does not include several assembler optimizations that are currently present in other versions of ORBexpress (LynxOS, Windows NT, VxWorks, et al) the product performed within our expectations.