Dependable information systems require a stable and responsive means of storing and retrieving data. This paper describes the MandateIP&#reg; database architecture and how that architecture provides the highest possible level of data availability while meeting the transactional demands encountered in real-time applications. The paper is divided into three main topics. The first topic addresses the actual database engine and the architecture for integration with the application programs. The second section addresses the challenges faced by the MandateIP&#reg; database architecture and how those challenges are handled. The last section describes the availability and recoverability of data in abnormal conditions (failure handling and recovery).
Database Engine And Application Process Integration
The MandateIP&#reg; database architecture is based on an “SQL” database engine. The architecture itself does not dictate the use of any particular vendors engine although the various vendor’s engines have their own individual strengths and weaknesses. The selection of the particular engine is primarily based on the user or client’s preference. Initially MySQL had been the preferred engine primarily due to its performance, replication features and its licensing policies. Oracle has been a close second and with recent MySQL licensing changes Oracle and MySQL have equal preference.
As with the base MandateIP&#reg; architecture, the MandateIP&#reg; database architecture is a “distributed” model where processing may be seamlessly spread across multiple computing elements. The MandateIP&#reg; database architecture identifies related datasets and then allows the assignment of an individual dataset to a database engine executing on a particular computer. Application processes closely associated with a particular dataset are assigned to computing elements that have high accessibility to the computer in which the dataset is hosted. This architecture provides a natural scalability allowing the platform to be used to implement an extremely broad range of systems. Systems with only minimal performance requirements to systems with massive performance requirements can be built on one platform architecture.
Challenges Of Dynamically Optimized Systems
Real-time or dynamically optimized systems place some challenging requirements on data availability. Real-time systems are systems that require decisions to be made within a guaranteed timeframe. Dynamic optimization makes decisions “on-the-fly” where the decision is based on current system conditions based on a set of rules for making the decision. This makes it necessary to have availability of the data to make the decisions within the required limited timeframe. Since many VAS systems involve the control of mechanical equipment, many of the required time frames may be very short (milliseconds). An example of these requirements is one in which an “optimized” sorting system continuously sorts 36,000 units per hour (10 sorts per second), and each sort decision involves 4 separate events (144,000 transactions per hour) for the sorting operation only. This system must make individual decisions within a 200 milli-second window decisions.
The database architecture of MandateIP&#reg; meets these challenges through a number of related techniques and procedures. MandateIP&#reg; database architecture uses an “event driven” model where events are communicated through messages. For events that alter a dataset, the application process processing the event is responsible to update or modify the dataset.
Events initiate dataset modification—dataset modification does not initiate events. By isolating datasets to individual computers, data availability is increased since multiple CPUs and disk drives share the responsibility. Additionally, restrictions of database access can be easily made insuring that the real-time processes have timely data availability. The MandateIP&#reg; database architecture provides tools (and reporting means) for accessing data that do not impact time-critical processes. It is not recommended that MandateIP&#reg; datasets be accessed through other means. Likewise, the computing platforms used in a MandateIP&#reg; system have been selected to allow the required processes to function within the real-time required timeframe. It is likewise possible to interfere with system operation CPU and data resources are used outside the architecture provided.
Data Availability And Data Recovery
Data availability and thus the underlying data architecture is never an issue during “normal system operation” in a properly designed system. Data availability only becomes an issue during and following some abnormal or “failure” situation. This section of the white paper discusses data from a “system failure” and “recovery” perspective. The nature of a “system failure” is extremely diverse, and can range from a power failure, a computing hardware crash, a network failure and even a software bug. The MandateIP&#reg; data architecture supports multiple servers. Individual servers may have differing requirements in terms of performance, volatility, and recoverability of data. Some servers with extremely low volatility (very limited changes) may only require backup procedures to insure data availability. The servers that have high data volatility and high transactional demands are normally supported with RAID 5 hardware controllers incorporating battery backed up cache and SCSI drives.
Some systems may have some even greater challenges. In extreme cases the MandateIP&#reg; architecture supports the creation of a “real-time transaction log” for the recording of data changes. The transaction log provides a means to re-build a highly volatile data set from a given point to allow for recovery from some of the more obscure failures (i.e. a failure of a hardware RAID 5 controller could potentially destroy an entire dataset). The method of using a transaction log for such cases provides a completely separate mechanism (CPU, drive controller, drives, power supply…) for maintaining data.
Normally VAS and the customer jointly agree upon the specific computing hardware for system implementation. If the customer has no preference, VAS generally uses Dell equipment. It is interesting to note that some “Computing Industry” terminologies have a somewhat different meaning to VAS as they have to others. In particular, a “server class” machine normally is thought of in the industry as a “high speed machine”. At VAS, performance of a machine is normally not of utmost importance, for our architecture allows us to obtain performance through distribution of work—additional machines. Reliability is of much greater importance to us than performance. This same feature of our architecture (distribution of work across machines) is why we have no real preference to a particular database engine.
With any of the methods mentioned above, the recoverability of operation has a “time restraint” in which the recovery is accomplished. The MandateIP&#reg; database architecture is designed to minimize this recovery period. Recovery periods for a failed RAID 5 disk drive are essentially zero; where as recovery from other failures and with other servers may take from several minutes to nearly an hour. Rebuild of a data set from a transaction log may take longer depending upon how much data must be applied to from the last backup point.
Conclusion
In conclusion, the MandateIP&#reg; database architecture uses third party database engines distributed across multiple computing elements to obtain the necessary performance to achieve the desired system operation. The architecture incorporates standard industry methods to obtain a high degree of data availability and rapid error recovery.