This short guide is intended to highlight what sort of debugging tools are appropriate to different kinds of project and offers practical advice for avoiding common pitfalls.

Document Overview

Starter Kits

PC Development With No Target Debugger

HITOPWIN/166 And DSCOPE166 Monitor Debuggers

Simulator Debuggers

In-Circuit Emulators

Case Study 1.

Case Study 2.

Case Study 3

Choice Of Tools Supplier

Summary

1. Starter Kits

· Evaluation/Assessment Of CPU Suitability
· Educational Projects
· Laying the groundwork for major projects

These are an excellent way of both assessing the C16x's suitability for your intended application and laying the groundwork for the project proper. All kits are supplied with 4KB restricted versions of the Keil C166 C compiler and DSCOPE166 debugger plus our own HITOP166/WIN debugger, in its monitor and simulator guises. While the compiler is limited in that it cannot generate more than 4KB of program code, the fact that the ANSI library code is not included, programs of up to 8KB can be produced. To put this in perspective, our complete demonstration AC motor drive application fits into this.

The HITOP166/WIN and DSCOPE166 monitor debugger is usually "bootstrap" loaded into the large RAM on the board, removing the need for EPROMs. This is why none of the boards are supplied with EPROM but all have ample RAM. HITOP166/WIN is our preferred debugger as it has a more intuitive operation and is also used on Hitex in-circuit emulators.

The starter kits can be used for complete small applications for hobby projects plus the sort of programs produced during undergraduate projects.

The microcomputer boards in the kits are a useful source of hardware design ideas and they are supplied with full schematics which can be used a starting point for your own board. During the main project development phase, having a working 167 target board can be very useful if things go wrong!

2. PC Development With No Target Debugger

In theory, it is possible to write a C16x program on a PC using Microsoft C and then cross-compile it to the C16x at the last minute. The resulting HEX file is blown into an EPROM and fitted into the target system. Initially, this approach seems very appealing as on the face of it, the tool cost very low.

This route is fraught with dangers and should not be used in commercial projects. It is much favoured by inexperienced engineers, usually with only PC programming experience and is also commonplace in companies where the management are from a non-electronics background. It usually results in a large program which either does not run at all on the C16x target or runs unreliably. Fundamental misuse of the C167 core and ANSI C grey areas are typical problem areas. With no means of checking how the program really works, many projects conducted this way end with major bugs being released, with consequently very high maintenance costs.

Using a PC compiler and no target debugger is almost always a totally false economy and borders on the irresponsible. EPROM emulators are not really debuggers and cannot contribute to program testing (or integrity), especially in on-chip FLASH systems. You are strongly advised not to go this way.

3. HITOPWIN/166 And DSCOPE166 Monitor Debuggers

These are tools which are capable of debugging larger programs, using an unrestricted version of the HITOP166/WIN debugger. They allow program download, single stepping and breakpointing. And are pretty easy to get on with. Although this route has its limitations, it is infinitely preferable to using no debugger at all.

However, like all monitor-type tools, they require significant resources from the C16x target system and cannot be used at all in single-chip designs as an external bus is essential. To allow you to download the program to be tested into the target, you must provide enough RAM to hold the program plus all its run time data. Most monitors require access to the C16x on-chip serial port although HITOP166/WIN can use the "ROMLINK" EPROM to PC-printer port interface which leaves this valuable resource free for the application. Inevitably, the memory map imposed on the application is not identical to that to be used in the final EPROM+RAM system.

Problems are often experienced when programs developed under a monitor are run in EPROM for the first time: as the monitor establishes a friendly environment in which most hardware-related settings are ready-made. A lot of time can be wasted trying to figure out why, unless an in-circuit emulator is used. Like other non-optimal methods, it is very easy to create bugs which do not show up when the monitor is being used - variable accidentally located in code areas is a classic mistake that a monitor will tend to mask. Monitors have no means of detecting READ or WRITE accesses and so memory overwrites cannot usually be detected. Unfortunately monitor debuggers cannot really be used in projects where the code resides in internal FLASH as the external bus must be enabled and a large area of external RAM be present.

Under no circumstances should code developed under a monitor be released without spending some time testing with an in-circuit emulator.

Both HISIM167 and DSCOPE provide accurate simulation of the C16x family and its peripherals. Inevitably, the program run time is several thousand times slower than real time but provided this fact is borne in mind, they can be useful. Unlike a monitor, they do not impose unnatural memory maps on the program being developed which makes them a good complement to proper in-circuit emulators on multi-engineer projects. To some extent, it is possible to write scripts to simulate external real time events to stimulate the simulated C167 program, although this is really only limited to simple digital waveforms and isolated analog values.

Simulators are viewed as being "trendy" as many companies believe that the software should only meet the hardware at the last moment and that everything in the real world can be simulated. In complex real time systems, this is not true. It should also be realised that a simulation of a sophisticated CPU like a C167 is just that - a simulation. In crude terms, it is just one person's interpretation of the silicon manufacturers data book. It is quite common to find that the simulator does not quite match the real CPU's operation. In this respect, the monitor debugger is to be preferred. However despite the foregoing, simulators are a valuable tool, provided that they are not relied on as the primary means of testing.

Monitor and simulator debuggers of any sort are not really adequate for professional use and are best reserved as a secondary debugging tool to a proper in-circuit emulator. They should certainly not be used on the following types of project:

· Safety-critical systems
· High integrity systems
· Medical applications
· Highly-real time applications
· Any project conducted under a quality regime such as TICKIT
· Real time control systems where significant mechanical hardware is involved

5. In-Circuit Emulators

The in-circuit emulator is the only primary debugging tool that should be used in professional projects. It is the only means of testing programs that does not change the operating environment in any way whatsoever. It is true to say that if a program functions correctly under emulation, it will also run in the final system. For developers of single chip XC and ST10 applications, there is really no other way of debugging the code as programs run some 50 to 100% faster from internal FLASH memory.

As a hardware-based tool, the emulator allows measurement of program performance and will allow back-tracing of program execution to discover what led up to program failures. READ and WRITE detection is possible so that memory-overwrites and corruptions are easily identified.

They make the measurement of program performance very straightforward and the more powerful systems allow coverage measurements to be made as part of common software quality regimes.

Emulators can represent a substantial capital investment and for the inexperienced project manager, their merits can be difficult to comprehend and justify. However a good emulator is to the embedded software engineer what the micrometer is to the precision mechanical engineer - it is highly unlikely that an reputable mechanical engineering firm would consider developing a complex mechanical part without a lathe or milling machine yet many electronics companies are happy to let software-based products be developed with the equivalent of a file and tape measure! Management is often unaware how much time engineers waste overcoming problems that proper debugging tools would be enable them to locate in minutes. Engineering time is not taken from capital budgets and so rarely has be accounted for.

In fact most low-cost debugging and testing methods have the fundamental limitation that the environment in which the program is tested is altered in some way from what the end user will use it in. Thus even though maybe 18 months have been spent developing the program, it is only when it is tested in the final hardware and memory map that any results are genuinely valid. The emulator allows the program to run in its final memory map throughout development so that at the end, some parts of the code will have been in use for several years, giving increased confidence.

The only drawback to using emulation on all projects is usually quoted as high initial cost and historically, this has been true in the case of the C16x family. However, with the release of low-cost true emulators such as the Hitex DPROBE range, all the high integrity capabilities of the emulator can be available to all but the smallest projects.

In the final reckoning, much of the resistance by project managers in small to medium sized companies to budgeting for proper development tools is based on the erroneous and dangerous notion that embedded software is cheap and easy to develop. It is not. The following three examples are taken from Hitex's extensive UK technical support archives and illustrate the dangers of using inappropriate development methods.

The program was written on a PC in Borland C, with some sections tested using Turbo debugger. The program was cross-compiled to the C16x using the Keil C166 compiler for blowing directly into an EPROM. Throughout the development phase, it was noticed that occasionally the EPROMmed program failed to execute but this was casually attributed to faulty EPROMs. No C16x target debugger was ever used.

The version one of the board ran acceptably, apart from the occasional EPROM problem. The second generation board was destined for mass production. The entire pick-and place line was configured to run to produce 1500 boards. However, a first trial board was assembled for testing which was found not to work. After 16 man days of intense effort with the production line stalled, the use of a borrowed in-circuit emulator revealed a mistake in the configuration of the chip-selects. Depending on noise, the program would as a result either run or hang-up, giving the appearance of a faulty EPROM.

The fault was quickly found by running with an in-circuit emulator and back-tracing with the trace buffer to see where the program went awry.

A low cost in-circuit emulator would have cost £4000 and would have allowed the original error to have been found just after it had originally been introduced. Thus the net identifiable cost due to inadequate methods was approximately £2300. In fact, further costs resulting from the delay of other work were incurred by this incident. Had the emulator not been used to locate further bugs, the costs of product recall would be very high indeed.

The machine control system software had been developed using Keil C166 and the HITOP166 monitor debugger used to debug programs in the C167 target system. The program had been extensively tested under the monitor and when blown into EPROM for delivery to the customer, appeared to function correctly.

Once the machine had been installed (abroad), erratic operation was noticed every few weeks, which on several occasions resulted in severe mechanical damage, with subsequent downtime.

After investigation with an in-circuit emulator, it was discovered that two variables had been inadvertently placed into the EPROM, causing the memory protection hardware to trigger. These data items were only accessed in the minutes after certain maintenance procedures and resulted in the system failure.

The fault had never been noticed during development and testing under the monitor as the entire program was held in RAM. Thus any writes to the erroneous variables had succeeded.

5.2.1 Cost Analysis

Further intangible costs were incurred through loss of goodwill from customer no further contracts being placed.

A new control system for a laboratory centrifuge employed the C167. The CPU had to perform closed loop speed control of a three-phase induction motor, whose invertor drive was generated by the 167's own PWM unit. There were other closed loop systems for temperature and pressure control within the software.

The first attempt at the program was to write dozens of source files in A166 assembler, without any in-target testing at all. Indeed, the company had no debugging tools of any sort. The development was taken over by another rather inexperienced engineer who had to fight to get the use of a C compiler but was forced by the immediate management to retain the bulk of the untested assembler code.

After two years, the project was still not out of the evaluation stage and numerous major bugs rendered the software worthless. The senior management was from a mechanical background and was totally unaware of what constituted proper development tools. The second engineer left and the project ground to a halt. This was (and is) the most appallingly managed project ever encountered by Hitex.

5.3.1 Cost Analysis

Two years employment of engineer £42000

Intangible costs resulting from the unavailability of the new product range were also incurred.

6. Real Time Operating Systems

Increasingly, project managers are looking to real time operating systems (RTOS) to solve the technical and management problems that can arise on multi-engineer real time projects. The reasons for this have their roots in some or all of the following areas:

(i) Difficulty in recruiting and keeping experienced special embedded software engineers from an electronics background: non-specialist IT and computer science-trained programmers are much easier to find but usually such engineers are only comfortable when working within the confines of an operating system. They do not fully understand the interface of software to external real time hardware signal sources and can only think in terms of BIOS calls for accessing the real world.

(ii) Poorly structured projects where modularity has not been a goal and sourcefiles containing code and external references are used: the breaking-up of functionality and allocation to different engineers is a valid goal but the integration of the new code into a growing existing system can be made hard by undisciplined layout of source files and object definitions.

(iii) CPUs with poor real time and/or task switching performance and limited interrupt levels: older CPUs and popular current devices like the PIC and H8 have very limited interrupt systems and this can make the execution of multiple real time tasks tricky. A real time operating system can mask these shortcomings by allowing task-switching and the simualtion of extra priority levels.

(iv) A desire in managers to do something "better" or "different" from when they were practising software engineers.

(v) The need to speed up software development by removing the problems associated with combining diverse real time functions into a single unit: the communication of data between different real time functions can be awkward. Ensuring that specific functions execute on time can be a major headache.

The C166 family has a multitude of operating systems which claim to be "real time". on paper, the use of an operating system will indeed solve most of the problems listed above. However, there are a number of factors which must be borne in mind when deciding to go this way.

(i) Many RTOS have a runtime licence fee . In the cost-sensitive applications this amounts to an additional component cost.

(ii) The C166 architecture has several features which reduce the need for a kernel to perform multitasking and task switching. The ability to secure a fresh set of registers for a new task in 100ns makes task switching very efficient. The 15 interrupt priority levels, software traps and the ability to change priority level for currently executing code makes the production of a multi-tasking scheduler very easy.

(iii) An RTOS will inevitably increase code size somewhat but almost always, RAM usage will rise considerably. With the wider use of single-chip C166 variants, the latter can be a major problem.

(iv) An RTOS will allow the much more rapid attainment of early project goals by giving a framework into which major functions can be slotted. This gives the illusion of faster progress to managers. However, once the system is complete enough to perform real time control tasks, errors in the RTOS kernel, misunderstandings about RTOS calls and shortcomings in overall software design usually cause development to come to an abrupt halt. With the presence in the system of a large block of alien code, tracking down bugs can be almost impossible.

RTOS offered for embedded use are unlikely to be sufficiently tested to guarantee the complete absence of bugs. It is quite common for race conditions and spurious events to occur in real time kernels and it is virtually impossible for the user to remove these. In fairness to tbe RTOS vendors, it is probably impossible to fully gurantee an RTOS-based software as there are too many varying factors, not least the competence of the user.

A further factor to be considered is that real time systems are not deterministic. It is therefore impossible to fully test such systems under all possible operating conditions which mean that some residual bugs are likely to remain undiscovered. For software systems that use "conventional" barefoot approach, such errors found under realtime conditions are usually fixable as the entire system was written in-house and the working of the real time interaction of functional blocks are under the direct control of the development team. When an RTOS is used, there is little chance of kernel-related problems being solved without the active involvement of the supplier but even then, some doubts must remain. Commonly the result is that no reduction in overall development time is obtained.

Products that must run for very long periods under varying conditions are the most likely to suffer from RTOS-related malfunctions. The chances of a combination of events that will reveal subtle kernel errors or kernel/user software interaction problems are more likely to occur, albeit very infrequently when a system has to run for days or weeks at a time. Software-based systems that are mass-produced are also liable to RTOS issues due to the sheer number of units in use over a given period of time.

An important fundamental point about operating systems is that most of the common ones are PC-based and do not have safety implications if they fail. The real time content of most PC functions is small, reducing the scope for different combinations of inputs. Operating system failures on PCs do not have catastrophic effects on expensive mechanical hardware as would occur if a motor drive glitched.

Overall, when deciding to use an RTOS, these questions must be asked:

• Is there a runtime royalty?
• Can I rely on the kernel to be perfect?
• How am I going to test the final system?
• Is the product safety-critical or high-reliability?
• Are occasional failures acceptable?
• Are the RAM overheads acceptable?

Types Of Application In Which RTOS is a signifcant risk

Motor drives
Hard real time control systems
Engine management systems
Automotive systems in general
High-integrity systems

Typical Applications for which RTOS is suitable

Measuring instruments
EPOS terminals
Non-safetly-critical monitoring systems

7. Choice Of Development Tools Supplier

Unfortunately no single manufacturer produces all the elements required for an embedded project, i.e. the CPU, compiler and emulator are all made by different companies. With a relatively complex CPU like the C166 family engineers will encounter problems from time to time which prove very difficult to solve and may even be almost impossible to characterise. Unfortunately this is where the deficiencies in the majority of C166 tool suppliers in the UK will be brought home, particularly amongst the emulator companies. Most that have a base in the UK do not have any specialist knowledge of the C166 family and are only able to supply a "box" and perhaps install it. They may also be able to supply associated tools like evaluation boards and C compilers but they have no overall view of the CPU, its operation and its interaction with the tools.

This gives rise to the common situation where a problem in the operation of the CPU under emulation could be caused by either one or more of:

(a) a silicon bug
(b) an instance where the bondout chip differs from the final CPU
(c) compiler bug or C language dialect misure
(d) incorrect configuration of the CPU by the compiler and/or emulator.

The latter is becoming more common as newer versions of the C166 such as C161CS have non-standard peripherals and interrupt structures. It is not reasonable to expect the silicon supplier to be able to solve a problem which in fact may be caused by a tool or user-related factor. The result of this is that the problem then passes to non-specialist personnel at the tool supplier who has no deep understanding of all the parts. This leads to the classic "run-around" of buck passing which can take weeks to resolve. A good example is "I cannot run the program from the RESET vector to main()" when working on a standard Infineon starter kit like the SK167. The emulator is from supplier A and the compiler from supplier B.

• Is it a bug in the silicon?
• Is the memory map of the board correctly reflected in the linker/locator configuration?
• Is the /EA pin in the expected state?
• Is the SYSCON correctly configured?
• Is the BUSCON0 correctly configured?
• Is the BUSCON1 correctly configured?
• Is the ADDRSEL! correctly configured?
• Is the P0H.0 pull down resistor fitted?
• Are the memory initialisation tables in the right place?

Supplier B may be the compiler manufacturer's local agent who has one support engineer to cope with perhaps 10 different CPU architectures. He will have no specific knowledge of the CPU and board configuration and will probably only ever have worked with a simulator under which all programs function correctly. The limit of his expertise is making sure that the program will compile. Making it run on real hardware is somebody else's responsibility. Being a junior employee, he will not be able to make a site visit to investigate at first hand what is going on, although in this example, no such visit should be necessary for a tool supplier that knows what he is doing! Supplier B thus cannot provide a solution.

Supplier A only sees his job as supplying the emulator and has little knowledge of either how the CPU should be configured and may even be hazy as to how to set up the emulator itself. This is especially true if the emulator is one of the crude USA-sourced units that has a multiplicity of jumpers to set. He has little knowledge of the compiler and even less about the CPU as he has to deal with perhaps half a dozen other architectures.

The net result is that the two tool suppliers are of little use as neither has a complete understanding of all the elements involved. It is therefore left to the user to figure out what is wrong, taking up valuable time. Indeed, some cases the problem is never properly resolved and only manifests itself thereafter as an occasional production unit that fails end-of-line testing.

To avoid this situation it is essential that that you use a tool supplier who can provide a single point of contact for boards, compilers the emulator and to some extent the CPU itself. Your tools supplier must have a real understanding of the CPU, compiler and emulator. Naturally no tool supplier will admit that they are not equipped to cope with the support of major projects so it is up to the project manager to do a little detective work to see whether the local representative of the supplier really has the necessary experience and knowledge. The stress on "local" is because while in majro markets like the USA and Germany this expertise may exist, the UK is very much the poor relation and often suppliers have no proper support, relying on overseas specialists. It is not too difficult to spot which suppliers have the necessary expertise.

If it is an emulator supplier, check their UK website and note how much material is devoted subjects other than the emulator. Any decent C166 tools supplier will have various application notes, hints and tips pages which indicate that there is some real experience of the architecture available. Much as it might pain you to clutter up your office with tools salesmen, ask in two or three in and really press them on what resources they have in the UK. To try find out the following:

• Have they got a track record of providing training for example?
• Do they really understand how the compiler works?
• Can they provide any consultancy on board design?
• Have they got a verifiable list of at least 20 major UK C166 customers?
• Can they demonstrate expertise in the use of the compiler?
• Perhaps most significantly, what do Infineon's engineers themselves think about the supplier?

The wrong choice of tool supplier (and emulator) can add weeks if not months to your timescales so it pays to do your homework!

When embarking on a commercial embedded systems project, it must be released that software development is not easy and must be approached with the same type of care and planning required for a mechanical development. Failure to provide engineers with adequate development tools is a totally false economy; while it reduces the apparent up-front project cost, it always results in longer development times and hugely increased maintenance costs. Overall, failing to equip engineers with adequate debugging tools, particularly emulators, always increases total project costs.

Producing programs by simply compiling and then blowing EPROMs is not an appropriate method for professional engineers working on commercial projects. The monitor and simulator debugger are useful complements to proper in-circuit emulators but must never be the sole means of testing embedded software.