The Multilateral Interoperability Programme (MIP), a military standardization body comprising 26 member nations and NATO, explains what the MIP Test Reference System (MTRS) is and how it helps achieve command and control interoperability.
The »MIP Test Reference System« (MTRS) allows to test conformance of C2 information systems with regard to Baseline 3.1 of the Multilateral Interoperability Programme (MIP). The purpose of the MTRS is to improve testing efficiency and to ultimately increase the quality and interoperability of C2ISs.
The MTRS is continuously maintained and extended by Fraunhofer FKIE to support the latest specifications available, to enrich the existing test suite, and to provide the best possible user experience. The conformance test framework has been designed in a way that it opens the door for future standards, coming from MIP or other standardization bodies.
Since its debut in September 2007, the MTRS has been used to test 53 systems from 26 nations and NATO. By December 2014, more than 300,000 test runs have been conducted. A tailored version of the MTRS is available to monitor the data exchange during military exercises and identify problems as they occur. It was used in several years at the Coalition Warrior Interoperability Exercise (CWIX) and in the 5 Power Experiment.
Unlike interoperability testing, which checks whether two systems are able to communicate, conformance testing checks whether a system conforms to a given interface specification. When testing with the MTRS, the C2IS under test is considered a black box, i.e., no assumptions are made on its internal structure and processing. The C2IS is tested by sending a message, a stimulus, to the system and observing its response (or the lack thereof). To cover a broad range of scenarios, the system behaviour is checked for valid, valid but unexpected, and invalid input. Load tests check whether the C2IS under test is able to handle a huge amount of data.
The MTRS comes with a formalized test suite, comprising more than 700 test cases that cover all aspects of MIP Baseline 3.1: communication protocols, database replication, and data model mapping (in both directions as sender and receiver).
The test cases are written in domain-specific test languages that include special language constructs for handling concurrency and timers. The formal notation allows automated test execution. This way, the MTRS can be used 24x7 over the Internet without manual intervention by the MTRS administrators. At the end of a test run, a test verdict is assigned by the test system that is not subject to interpretation by human operators.
The MTRS is a client-server application. Its robust and scalable architecture supports parallel testing with multiple systems. The MTRS server is subdivided into a test manager and MIP gateways. The MIP gateways are implementations of the MIP Baseline 3.1 protocols. The test manager is responsible for executing test cases, determining and managing test results, and handling users. The test manager is agnostic of any specific interoperability standard, i.e., it could easily be reused for other interoperability solutions. The framework allows for a non-disruptive testing process. For instance, test cases can be updated on the MTRS server without having to restart the server; current testing activities are not affected.
National test operators use the MTRS client to start test cases and analyse test results. For that purpose, the MTRS client connects to the MTRS server. Once a test case is started, one or more MIP gateways are instantiated. They communicate with the national C2IS. During test execution, the test manager triggers its own gateways and tracks the communication with the national C2IS.
In addition to the graphical user interface, an application programming interface (API) allows to automate the testing process on the test operator’s side. This way, you can run the entire test suite overnight as part of a continuous integration solution. The API is written in Java.
The MTRS has a unique set of features that support testing in a multinational community:
Structured & Formal Test Suite: All test cases are publicly available to support transparency. They can be viewed from within the MTRS client and downloaded from the MTRS website. The MTRS client supports filtering of test cases based on keywords, e.g., to ignore test cases corresponding to optional elements of a specification.
Test Execution: Test operators are able to start and abort test cases at any time. Multiple test cases can be performed in parallel for the same C2IS by different test operators as well as for different C2IS without interference.
Test Evaluation: The test operator is able to perform a detailed error diagnosis. The internal data flow within the MTRS is presented both as a sequential log and graphically as a sequence diagram. Protocol data units are shown in a structured manner. Test operators are able to comment on test cases (or mark them as non-applicable).
Test Reporting: System-specific test reports can be exported by test operators from within the MTRS client. These reports can be used to document the testing progress for the customer. Reports are available in XML and PDF format. Crosssystem reports can be generated by the MTRS administrators to support standardization bodies or program managers. These reports are available in PDF format.
Release Management: Test case development adheres to a strict versioning scheme. This allows to clearly identify those C2ISs that need to re-run an updated test case. Whenever a test operator logs into the MTRS, all test cases that must be repeated are highlighted. The MTRS also supports regression testing by greying out test results that were obtained with a former version of the C2IS.
Development and maintenance of the MTRS has been funded by the Federal Office of Bundeswehr Equipment, Information Technology and In-Service Support (BAAINBw). It is offered free of charge to all nations participating in MIP and their national developers. For further information on the MTRS and on how to request a test account, please see the MTRS website here