Processor Architecture Patterns

One of the most important steps in architecture design is identifying processors/modules in the system and assigning roles/requirements to them. This article focuses on this very subject. Here we will explore different processor design patterns commonly seen in embedded and distributed systems.

The following processor architecture patterns are seen in many embedded and distributed systems:

• Operations & Maintenance Processor
• Central Manager
• Module Manager
• Terminal Manager
• Device Controller

The figure below shows the relationship between some of the processor architecture patterns listed above. Note that the this list is by no means complete. Embedded systems span a wide spectrum, so these patterns should be treated as being representative only. In simple systems you might design the architecture by assigning multiple processor architecture patterns to a single processor.

Operations and Maintenance (O&M) Processor

In most embedded and distributed systems the operator controls the complete system from a console. When you are designing a distributed system the operations and maintenance processor should provide a unified view about the management of the system. The operator should be given a simple task based view of the system.

Keep in mind that O&M capabilities of the system have a lot of impact on the perceived quality of the system. The operators often judge the quality of the system just from the ease-of-use of the user interface. A good user interface lowers the cost of ownership for your customers and thus increases the chance of getting more business.

Role

• Provides form based screens interface to the operator to control the system.
• Provides statistics screens to ascertain the health of the system
• Provides interface to notify the operator of serious problems that need operator intervention. 
• Provide Periodic and on-demand reports about the health and performance of the system
• Support configuring and dimensioning of the system.

Guidelines

• As mentioned earlier, user interface design should be task based. i.e. make a list of tasks that the operator would perform with the system. Provide direct screens to accomplish the tasks. The operator screens in the system should be more like:
  1. Obtain system loading statistics
  2. Create a subscriber
  3. List major alarms
  4. List subscriber related critical alarms
  5. Add a new device
• Conduct usability studies with the world be operators. See if they feel comfortable with the operator interface. For more information about usability studies and task based user interface design refer to the following external link.
• Do not implement architecture based screens. Stick to task based screens. The operator is just interested in getting the job done. He/she should not be interested in how your architecture works.  Your screens should not read like:
  1. Obtain central manager statistics
  2. Obtain system status as seen by module manager
  3. Check device controller 0 alarms
• Generate alarms only for actions that need operator intervention. Do not use alarms as a field level debugging tool. Thus suggestions like "Lets add a critical alarm to see if the code is entering this error leg" should be shot down and the designers should be asked to use alternative debugging mechanisms.
• Minimize the number of configurable parameters. Restrict configurable parameters to customers needs. Adding configurable parameter for system tuning by engineers almost always turns out to be a bad idea.
• Architect your O&M processor as an HTTP server. This way the operators would be able to use any client with a browser to access the user interface.
• Failure of the O&M processor should not have any impact on the systems operation. The system should be able to go on for days without an O&M processor.

Central Manager

Central Manager pattern should be used to define a central processor that will manage the operations of the complete system. Central Manager maintains a system level view of the system. Typically the Central Manager interacts with Module Managers.

Role

• Maintains the health of the system by checking the sanity of the highest level modules. Global fault handling is performed at this level.
• Allocates resources that are shared at a global level.
• Coordinates system start up and other activities that involve system wide coordination
• Interfaces with the O&M processor for operator interface for system wide commands/statistics.
• Manages the network with highest level module managers

Guidelines

• Only truly global activities should be assigned to the Central Manager. If the Central Manager is involved in too many activities, it will soon become a bottleneck in system performance.
• Care should be taken to reduce Central Manager involvement from complexity point of view. If adequate attention is not given, you might end up with a complex and difficult to debug Central Manager.
• Central Manager should coordinate activities between the highest level Module Managers. This leads to a simpler system design as Module Managers do not need to handle system level coordination.
• As far as possible, keep the Central Manager out of transaction level processing. If this is not possible, minimize the role played by the Central Manager in every transaction. For example, if your system is designed to handle telephone calls, minimize the involvement of the Central Manager in call processing.
• If a IP based architecture is used, the central manager and the highest level module managers should communicate on a single IP network. Nodes belonging to an individual module will not connect to this global network.
• Failure of the Central Manager should have minimal impact on existing transactions. Setup of new transactions may be impacted by the failure of the Central Manager, 

Module Manager

Module can be defined to be collection of processors that implement an important part of the system's functionality. Module Manager is any processor in the system that manages all other processors belonging to the module. This is a hierarchical pattern. A Module Manager might further manage sub-modules. The sub-modules further manage their own processors.

Role

• Handles commands and requests from the Central Manager (applicable to highest level Module Managers)
• Handles commands and requests from a higher level Module Manager (applicable to sub-module managers)
• Highest level Module Managers participate in the global network connecting them with the Central Manager.
• Module level fault handling is carried out by the Module Manager. Intra-module faults are not reported to the Central Manager.
• Manages the state and startup of Terminal Managers and other processors within the module.
• Performs most of the transaction processing. For example, in a switching system, most of the call processing will be handled by the Module Manager.
• Coordinates transaction processing actions within the module. For example, when a call is setup by a Module Manager, it will coordinate the ringer set up, tone generation etc. by talking to specialized processors within the module.
• Module Manager controls the local intra-module network.
• Handles all O&M processor requests for operator initiated commands.

Guidelines

• The interface between the Central Manager and Module Manager should be designed to be simple. All the interfacing complexity should be handled within the Module Manager.
• Module Manager should coordinate most of the transaction processing. Services of other processors in the module should be used as and when required.

Terminal Manager

Terminal Manager is a hardware board that supports a number of entities of the same type (terminals). For example, a digital trunk manager card might manage several digital trunk circuits. The Terminal Manager card has a single microprocessor that controls all the devices. Thus a Module Manager can assign individual terminals. Most of the Module Manager software just deals with individual terminals and is transparent to the number of terminals per Terminal Manager card. The fault handling software keeps track of the health of the Terminal Manager cards. When a Terminal Manager fails, the fault handling module informs rest of the software in terms of individual terminal failures.

Role

• Partitions hardware and software resources on the board into terminals.
• Handles terminal configuration and de-configuration requests from the controlling Module Manager.
• Handles operator status change requests from the operator. (The Terminal Manager does not directly interface with the O&M processor It receives the requests from the controlling Module Manager).
• Monitors the status of individual devices implementing the terminals. Terminal Manager also exchanges health messages with micro-controller and DSP controlled terminals.
• Hides the rest of the system from specific hardware implementation of terminals (e.g. the Terminal Manager might consist of say 8 DSPs implementing four terminals per DSP. All other processors just see the card as a collection of 32 terminals)

Guidelines

• Interface between Terminal Manager and other cards should be designed to scale automatically with hardware changes. One simple way of achieving this is to let the Terminal Manager report the type and count of terminals it can support. Thus if a new board is designed, it can report a higher number of terminals to the other boards. This way new versions of the hardware will work with older software without any hitch.
• Terminal resource management should not be performed on the Terminal Manager as this would break the abstraction provided to the controlling Module Manager. Note that in any case the Module Manager would have to select the Terminal Manager card for allocation. Thus spreading the terminal allocation and de-allocation logic at two places. In other words, keep the Terminal Manager dumb.
• Terminal Manager interfaces should be designed to allow for operation of different types of Terminal Manager cards to work in the same chassis. This minimizes the impact on software of inevitable hardware upgrades to the Terminal Manager.
• If terminals are implemented using micro-controllers/DSPs, the interface between the main processor and terminal processors should be clearly defined.

Device Controller

Device Controller is a processor which interacts with the lowest level of hardware. A few  examples of Device Controllers are:

• DSPs implementing terminals in a Terminal Manager
• A microprocessor controlling a transducer, heat sensor etc.
• Micro-controller operating a stepper motor

A Device Controller can interface with other processors with any of the following interfaces:

• FIFO
• Dual Ported RAM
• Serial link
• Ethernet
• DMA Operations

Role

• Provides a message level device programming interface to higher layer software
• Handles interrupts and device interactions, thus offloading the main processor from device level processing
• Keeps other processors transparent to the actual device operation. 

Guidelines

• Clearly define the message interface between the Device Controller and other parts of the system
• Device Controllers are often implemented by lower end microprocessors. Restrict the Device Controller to just handling the device.
• Keep the Device Controller dumb. It should need very little configuration information. All decision making should be left to the other higher end processors in the system.

Comparison Matrix

Different Processor Patterns are compared for different attributes in the following matrix:

	Operations and Maintenance Processor	Central Manager	Module Manager	Terminal Manager	Device Controller
Software Complexity	Medium Complexity depends upon the following factors: Overall system complexity Total number of processors that need to be maintained Skill level of operators using the O&M Processor	High Complexity depends upon the following factors: Total number and type of Module Managers controlled Degree of involvement in every transaction Redundancy and Availability Requirements Global resources that need to be managed	High Complexity depends upon the following factors: Total number and type of nested Module Managers controlled Total number and type of Terminal Managers controlled Module resources that need to be managed Redundancy and Availability Requirements	Medium Complexity depends upon the following factors: Number and type of Terminals managed Number of terminals that share the same hardware Degree of transparency needed from future hardware changes Impact of Terminal Manager cost on the cost of the complete system	Medium/Low Complexity depends upon the following factors: Programming model of the hardware device Degree of transparency needed from future hardware changes Impact of Device Controller cost on the cost of the complete system
Performance Requirement	Medium Use of HTTP, Java and SNMP increases the performance requirements for a O&M Processor.	High A high performance platform should be used as the Central Manager so that its performance doesn't become a bottleneck when the system is expanded.	High In most cases the top level Module Managers will handle most of the transaction processing load on the system. A high performance platform should be used.	Medium Overall performance requirements for the Terminal Manager might not be very high. However, the Terminal Manager might be executing a lot of time critical code, thus needing a reasonably good performance platform.	High/Medium/Low Device controller performance requirement varies a lot from one device to another. For example, a controller involved in digital signal processing might need a very high MIPS CPU. A device handling protocols might work well with a medium performance CPU. Most devices however would work very well even with low performance CPUs.
Redundancy	Cold Standby Failure of the O&M processor has no impact on system performance. A replacement can be brought in service manually. (This assumes that both O&M processors communicate to an off-the-shelf database server machine.)	Hot Standby Central Manager's failure can result in the complete system going out of control. Thus a hot standby should be maintained for quick recovery from the fault condition.	N + X Redundancy All transactions handled by a failed Module Manager will be released abnormally by the Central Manager. Central Manager would then configure a spare Module Manager to replace the failed module.	N + X Redundancy All transactions handled by a failed Terminal Manager will be released abnormally by the controlling Module Manager. Controlling Module Manager would then configure a spare Terminal Manager to replace the failed board.	None Device controllers are not redundant within a Terminal Manager.
Hardware Cost Sensitivity	Low Only two machines are required for the O&M Processor. The cost of these machines is not a significant portion of the overall hardware cost.	Low Only two machines are required for the Central Manager. The cost of these machines is not a significant portion of the overall hardware cost.	Medium Typically a system will need many Module Managers. Their aggregate cost may be a significant portion of the system cost.	High A large number of Terminal Managers might be needed. Aggregate cost of Terminal Managers may be a large percentage of the total system cost. This might mean that the software developers have to optimize their code to work within the cost limitations.	High A very large number of Device Controllers might be needed. Aggregate cost of Device Controllers may be a very large percentage of the total system cost. This might mean that the software developers have to optimize their code to work within the cost limitations.
Suitable Platform	Unix Servers and Work Stations Linux PC Servers Windows PC Server	Unix Servers and Work Stations Linux PC Servers Windows PC Server	If the Module Manager does not perform time critical operations: Unix Servers and Work Stations Linux PC Servers Windows PC Server If the Module Manager performs time critical operations, use PCs running: VxWorks P-SOS RT-Linux etc.	PCs or proprietary hardware running: VxWorks P-SOS RT-Linux etc.	Micro-controller OS or proprietary.
Operator Interface Support	High O&M Processor completely handles the operator interface.	High Central Manager has a large operator interface component.  It supports system wide statistics, status and alarm reporting.	Medium Module Manager has a considerable operator interface component.  It supports module wide statistics, status and alarm reporting.	Low Terminal Manager does not have a direct operator interface. It reports statistics and alarm conditions to the Module Manager. Module Manager would then directly interact with the O&M processor.	None/Low Device Controller might report error conditions which result in alarms being generated. It might report some statistics to the Terminal Manager.
Developer Skill Set	Graphical user interfaces Relational databases SNMP HTTP/HTML Java	State machine design Message interaction design System level design experience Object oriented design C/C++	State machine design Message interaction design Real-time software development Object oriented design C/C++	Real-time software development State machine design C/C++ Software and hardware interfacing	Device programming Hardware interfacing and hardware design knowledge C/C++ Assembly programming Signal processing knowledge DSP programming
Related Links	O&M Processor Hardware Fault Tolerance Handling Processor Reboot	Central Manager Resource Allocation Patterns Resource Manager Pattern Feature Coordination Patterns Hierarchical State Machine State Machine Inheritance Publish-Subscribe Design Patterns Timer Management Design Patterns STL Design Patterns Software Fault Tolerance Handling Processor Reboot Hardware Fault Tolerance Reliability and Availability Basics System Reliability and Availability Congestion Control	Module Manager Hierarchical State Machine State Machine Inheritance Half Call Design Pattern Collector State Pattern Parallel Wait State Pattern Serial Wait State Pattern Resource Allocation Patterns Resource Manager Pattern Feature Coordination Patterns Timer Management Design Patterns Publish-Subscribe Design Patterns STL Design Patterns Software Fault Tolerance Handling Processor Reboot Hardware Fault Tolerance Fault Handling Techniques Congestion Control	Terminal Manager Manager Design Pattern Transmit Protocol Handler Design Pattern Receive Protocol Handler Design Pattern DMA and Interrupt Handling Processor Bus Cycles	Device Controller Hardware Device Design Pattern Serial Port Design Pattern High Speed Serial Port Hardware Diagnostics DMA and Interrupt Handling Processor Bus Cycles