Performance Comparison of the IMMPDAF Algorithm Using Real and Simulated Radar Measurements

ABSTRACT

Performance indexes obtained in idealized simulated scenarios are the primary source of data for evaluating different target tracking algorithms in most researches presented in the literature. Despite the convenience of simulation, ultimate evaluation of a tracking algorithm must be made in real scenarios. Unfortunately, real radar measurements as well as accurate aircraft position, necessary for calculating tracking errors, are not easily available. In this paper, we present an evaluation of the well-known Interacting Multiple-Model with Probabilistic Data Association Filtering algorithm using data obtained from a flight inspection of a Brazilian Air Force ground-based long-range surveillance radar. The presented results show that, in this scenario the Interacting Multiple-Model with Probabilistic Data Association Filtering algorithm performance using real data is worse compared to simulation. Statistical properties of the real radar measurements are also investigated, and some evidence is found that embedded noise is not well modeled as perfectly white.

Keywords
Radar tracking; State estimation; Data simulation; Radar data

INTRODUCTION

When dealing with simulation for performance evaluation of target tracking algorithms one is faced with the problem of modeling radar measurements and target dynamics. Numerous studies addressing this problem have been published in the literature, and some examples are presented in Bar-Shalom et al. (2001) and Blackman and Popoli (1999).

Radar measurements are often assumed to be corrupted by additive white Gaussian noise in most simulation setups (Blackman and Popoli 1999), whereas target dynamics are generally emulated with simple kinematic models, such as constant or nearly constant velocity or acceleration models, constant angular rate, or variations and combination of these (Bar-Shalom et al. 2001).

Due to the possibility of real targets exhibiting complex dynamics, target state estimation has been tackled by the multiple-model approach, in which it is assumed that the target can switch between several simpler flight models, each one matched to a target mode-of-flight (Bar-Shalom et al. 2001).

The Interacting Multiple-Model (IMM) algorithm (Blom and Bar-Shalom 1988) has been widely considered for target state estimation in this context due to its remarkable costeffectiveness balance (Bar-Shalom et al. 2001). In respect of the origin of the data used for target tracking, it is worth to notice that the IMM algorithm as well as some other multiple-model solutions have been developed assuming unity probability of detection and correct measurement-to-target association.

Some algorithms capable of handling measurement origin uncertainty have been proposed in the literature, such as the Probabilistic Data Association Filter (PDAF) (Bar-Shalom et al. 2009) and its extension to maneuvering targets, the Interacting Multiple-Model with Probabilistic Data Association Filtering (IMMPDAF) (Kirubarajan et al. 1998). Besides the zero-mean white Gaussian measurement noise assumption, these algorithms are also rooted on simplifying assumptions regarding radar performance characteristics and spurious detections, such as constant probability of detection and uniformly-distributed clutter (Bar-Shalom et al. 2011).

Despite being mathematically convenient, such assumptions may be far away from conditions presented by real scenarios of application. Therefore it is reasonable to say that only when an algorithm is deployed and evaluated in these scenarios the designer can truly assess its performance. This question has been addressed in some recent researches such as Hess et al. (2014), where efforts have been made to verify the performance of tracking systems in real conditions of inspection flights.

Following this approach, the current paper uses data obtained in a flight inspection of a Brazilian Air Force ground-based long-range radar to evaluate the performance of the IMMPDAF algorithm. The results obtained with real measurements are compared to those obtained with simulated data, but the generation of simulated measurements differs from previous studies in the literature. Instead of emulating target dynamics with simple kinematic models, radar measurements are generated by adding white Gaussian noise to the actual target trajectory obtained from the inspection aircraft. The main contribution of this paper is to show that the IMMPDAF can yield larger estimation error in a real environment compared to simulation possibly due to different statistical properties of noise in simulated and real environments.

The paper is organized as follows. In the next section, we present basics of target tracking and the IMMPDAF algorithm. Next, data collection and its preparation are described. Afterwards, some numerical results and conclusions are presented.

IMMPDAF AND TARGET TRACKING BASICS

Under the multiple-model approach, the maneuvering target is modeled as a stochastic dynamic system whose state and observation equations are usually given by:

where: x_k, F[M_k], z_k and H[M_k] are, respectively, the state vector, state transition matrix, measurement vector and observation matrix at time k. M_k represents the flight model in effect at time k and can be any element of the model-set M = {ψ_j}^r_j=1, being r the number of models. The random sequences {υ [k, M_k]} and {w [k, M_k]} are Gaussian, zero-mean, white and mutually independent. At time k, υ [k, M_k] and w [k, M_k] are random vectors with covariance matrices Q[M_k] and R[M_k], respectively. Given M_k, the values of F[M_k], H[M_k], Q[M_k] and R[M_k] are assumed to be known. The initial state x₀ is a Gaussian random vector with known mean and covariance matrix. Model transitions are Markovian with probabilities P{M_k = ψ_j | M_k–1 = ψ_i} P_ij . Both the model-set and transition probabilities P_ij are assumed constant and known.

Considering measurement uncertainty for handling targets in clutter, association hypotheses are defined to express the choice of a particular measurement-to-track association (Bar-Shalom et al. 2011). In order to avoid searching the entire measurement space to construct measurement-to-track associations, a multidimensional gate, or validation region, is defined. The set of validated measurements (i.e. those that fall inside the validation region or gate) at time k is denoted as Z_k = {Z_k,i}^nk_i=1, n_k being the number of validations at that time instant. An association hypothesis at time k is defined as Θ_k = θ_q, where θ_q, q ∈ {0 … n_k}, is the event in which the q-th measurement was originated from the target of interest, and q = 0 is used for the hypothesis that no measurement came from the target (the target was not detected or its measurement was not validated). The set of all validated measurements up to time k is denoted as Z^k = {Z_i}^k_i=1.

The posterior density of the target state vector at time k is written considering all model and association sequences (Bar-Shalom et al. 2011), as follows:

where L_kl=1 (n₁ + 1). M_j and Θ_i represent M_k = ψ_j and Θ_k= θ_i, respectively. ∏

The density in Eq. 3 is a Gaussian mixture with exponentially increasing number of terms, and the evaluation of the a posteriori mean of x_k (the Minimum Mean Square Error estimate, referred to as MMSE estimate) cannot be feasibly realized. Numerous sub-optimal algorithms have been proposed in the literature to approximate Eq. 3 and provide good approximations of the MMSE estimate at a feasible computational effort (Blackman and Popoli 1999; Rong Li and Jilkov 2005).

The IMM algorithm has been one of the most successful tools for tracking maneuvering targets without assuming measurement origin uncertainty (Bar-Shalom et al. 2001). On the other hand, the Probabilistic Data Association (PDA) Filter (Bar-Shalom et al. 2009), in particular, handles measurement origin uncertainty for a single flight-model by combining the Gaussian mixture representing all measurement associations at time k into a single Gaussian.

The basic idea of the IMMPDAF algorithm is to combine IMM and PDA to track a single maneuvering target in clutter. It first merges all measurement associations conditioned to the same model at time k into a single Gaussian (PDAF step) and then uses the IMM framework to propagate a Gaussian mixture with r terms.

A detailed description of the IMMPDAF can be found in Bar-Shalom et al. (2011), Kirubarajan et al. (1998) and Sinha et al. (2006). It is worthy to notice that to implement this algorithm in practice some parameters assumed to be known must be chosen by the designer. This is the case of the model-set and the model transition probability matrix. Guidelines for choosing these parameters can be found in Bar-Shalom et al. (2001). Parameter values needed in the measurement-to-track association procedure are the gate probability, i.e. the probability that the true measurement from the target falls into the validation region, and the sensor probability of detection, used in the association probability update. Details about the choice of these parameters can also be found in Bar-Shalom et al. (2011).

DATA COLLECTION

In this section we describe the flight inspection data and simulation experiments conducted for evaluating the IMMPDAF algorithm.

FLIGHT INSPECTION

The Brazilian Air Force usually carries out flight inspections to evaluate the performance of radar and tracking systems by comparing information from aircraft avionics with those displayed by ground systems. One of these flights was selected for the evaluation presented in this paper. The radar in this inspection is a ground-based long-range surveillance radar, with antenna rotation period of 10 s. This radar is suitable for air traffic control as well as for defense applications.

The flight was performed by a Hawker EU-93A aircraft of the Flight Inspection Group (GEIV) of the Brazilian Air Force (DECEA 2015). Its 2-D trajectory obtained onboard from GPS data is depicted in Fig. 1.

Figure 1
Flight inspection 2-D trajectory.

Axis coordinates are given in nautical miles (1 NM = 1,852 m), and the radar was at coordinate (0, 0). The flight started at coordinate (80, 0) and comprised a sequence of maneuvers performed by the aircraft. Longitudinal acceleration/deceleration as well as vertical ascending/descending maneuvers (not explicitly shown in the 2-D plot of Fig. 1) have also been executed.

GPS position points were available every second, and the radar produced target measurements every 10 s (antenna rotation period). Consequently, radar data had to be linked to the corresponding GPS position point. This was achieved by analyzing the timestamp of the radar plot and selecting the two nearest GPS position points. These two points were then linearly interpolated to match the radar measurement timestamp.

Furthermore, when dealing with real data one has to ensure that biases in measurements are properly corrected to produce zero-mean measurement noise, as it is usually assumed in the evaluation of tracking algorithms. The SASS-C software (Zeebroek 2010), developed by EUROCONTROL and licensed to the Brazilian Air Force, was employed to obtain radar measurement bias values that were used in the bias compensation procedure. SASS-C uses trajectory reconstruction algorithms based on Renes et al. (1985) to calculate sensor systematic errors.

Estimates of noise standard deviations used for range and azimuth coordinates were obtained by the Plot Accuracy Analysis tool (PAA) of SASS-C using real data from a recording of approximately 3 h. This tool calculates differences between radar measurements and points of the reconstructed trajectory to produce accuracy measures (Zeebroek 2010). These procedures are summarized in Fig. 2.

Figure 2
Performance evaluation with real data.

SIMULATION SETUP

For simulation, radar measurements were artificially generated by adding zero-mean white Gaussian noise to the trajectory depicted in Fig. 1. Missed detections observed in the real scenario were considered.

Radar measurements were given in polar coordinates, and noise samples were generated independently in range and azimuth (Bar-Shalom et al. 2011).

Figure 3 illustrates the simulation setup. It is similar to Fig. 2, differing only that measurements were artificially generated from aircraft GPS position points. Since no bias was introduced during measurement generation, there was no need for the IMMPDAF to perform bias correction. The estimates of noise standard deviation were obtained by using SASS-C as above described.

Figure 3
Performance evaluation with simulated data.

NUMERICAL RESULTS

SETUP

The IMMPDAF algorithm was implemented in MATLAB using the IMMPDAF equations presented in Bar-Shalom et al. (2011). This implementation had three flight models: a nearly-constant velocity model with process noise standard deviation of 0.1 m/s², a discrete Wiener acceleration model with process noise of 3 g, where g = 9.8 m/s² is the acceleration of gravity, and a constant acceleration model (zero process noise). These are similar to the models implemented in Bar-Shalom et al. (2001), Heidger and Mathias (2008) and Kirubarajan et al. (1998). As the state vector is described in Cartesian coordinates, standard conversion from polar to Cartesian was applied (Bar-Shalom et al. 2011).

The model transition probability matrix was fixed as:

where the lines and columns correspond, respectively, to the nearly-constant velocity, discrete Wiener acceleration and constant acceleration models.

Additional choices were the gate probability P_G = 0.99 and probability of detection P_D = 0.80. This value of gate probability is a common choice (Bar-Shalom et al. 2011), and detection probability of 80% is the minimum required for a primary surveillance radar according to the current Brazilian legislation (DECEA 2014).

The error measure adopted to evaluate the IMMPDAF algorithm was the difference between the aircraft reported position (GPS position) and the position estimate. For radar measurement error analysis, the difference between aircraft GPS position and radar measurements was used.

Since the aircraft GPS generated position reports in geocentric Cartesian coordinates, equations presented in Engel (2005) were applied to transform these reports to a 2-D coordinate system centered on the radar, referred to as local coordinate system. Geocentric aircraft reports were this way transformed to local Cartesian coordinates (ξ_k^GPS , η_k^GPS) and local polar coordinates (ρ_k^GPS , θ_k^GPS). For the sake of notational simplicity, the superscript GPS will be suppressed in the following.

IMMPDAF PERFORMANCE WITH REAL AND SIMULATED DATA

To characterize the position error of the IMMPDAF algorithm at time k, differences in each coordinate were initially computed as:

where ξ and η at time k. and are, respectively, the IMMPDAF estimates for Cartesian coordinates

Estimates of the ensemble averages of the errors at time k for each Cartesian coordinate have been calculated using simulated data as:

and

where Єⁱ_ξ,k and Єⁱ_η,k are the errors at time k in coordinates ξ and η, for the i^th realization, and N_ENS is the total number of realizations.

Figure 4 shows the results obtained for _ξ,k and _η,k with N_ENS = 4,000 independent simulations. It can be observed in this figure that the mean of errors in both coordinates varies significantly with time. It is worth noticing that the most significant variations may be associated to intense maneuvers, since it has been recognized that Kalman filter innovations are not zero-mean during these maneuvers (Bar-Shalom et al. 2001).

Figure 4
Estimated averages of IMMPDAF errors from simulation.

To gain access to other properties of the IMMPDAF errors with simulated measurements, their standard deviation estimates at time k were computed as:

and

The results obtained after averaging over 4,000 independent simulation runs are shown in Fig. 5, where it can be observed that the errors in coordinate η are more intense than those in ξ. This is explained by the fact that the flight was carried out mostly eastwards from the radar, where conversion of measurement noise from polar to Cartesian coordinates yielded higher variance in the η axis (Bar-Shalom et al. 2011). Besides, significant variations of the standard deviation in both coordinates are exhibited in this figure.

Figure 5
Estimated standard deviation of IMMPDAF errors from simulation.

In short, the estimates shown in Figs. 4 and 5 indicate that the IMMPDAF errors are highly non-stationary in both firstand second-order statistical moments. This is an important fact to be taken into account in the analysis of errors obtained with real data (radar measurements).

It is important to remark that a natural difficulty in comparing performances using real and simulated measurements arises from the fact that real data is usually rare, so a limited number of samples are available and it is not possible to obtain reliable estimates of statistical moments such as those shown in Figs. 4 and 5.

The strategy here adopted to circumvent this difficulty was to verify, through a statistical analysis, if the samples of errors obtained in real conditions of radar operation can be considered as having the same order of magnitude of those obtained with simulated data.

To carry out such verification we first calculated the normalized zero-mean sample of errors obtained with simulated data as follows:

and combined them to form a single normalized error measure q_k defined as:

For the sake of probabilistic modeling it was assumed that q_ξ,k and q_η,k are independent zero-mean Gaussian random variables with unit variance. Under this assumption, the normalized error amplitude q_k has a Rayleigh probability density and mean √π/2. This is adopted here as a reference probability density for the error amplitude, denoted by p_r(.).

So, given a tail probability ∏ arbitrarily set, we are able to evaluate an acceptance interval [0, λ] such that:

Now, with the aim of comparing the IMMPDAF performance subjected to the two sources of measurements, we further define the variable q^R_k, which is the normalized error measure obtained with real data. Accordingly, this variable is evaluated using Eq. 13 and error values obtained with real measurements in a similar way of q_ξ,k and q_η,k in Eqs. 11 and 12, respectively.

The data obtained in 4,000 independent simulation runs and two values for tail probability ∏ (0.01 and 0.001, resulting in λ = 3.03 and 3.71, respectively) have been considered to plot Fig. 6, which shows the sample function of q^R_k × k and the thresholds (λ) associated to the acceptance intervals. A large number of values of q^R_k above these thresholds, i.e. outside the acceptance interval, may be observed.

Figure 6
Normalized amplitude of IMMPDAF errors obtained with real measurements.

On the other hand, a very different behavior is observed in Fig. 7, where samples of normalized errors obtained from four simulation runs are shown. The same thresholds included in the previous figures are used as references. It can be observed that in these cases the thresholds are rarely surpassed, which is an indication of the compliance with the assumed statistical framework.

Figure 7
Samples of normalized error amplitude of IMMPDAF obtained with simulated measurements.

A quantitative indication of the differences observed in the IMMPDAF performance under the two sources of measurements is given in Table 1, in which the percentages of instantaneous values of q^R_k observed above several threshold (λ) values are presented, including the two thresholds shown in Fig. 6.

Table 1
Quantitative evaluation of samples outside acceptance intervals.

Tail probability (H)	Percentage of qRk values above the threshold (%)
0.001	3.33
0.01	7.47
0.05	14.30
0.1	18.76
0.15	23.21

It is shown both in Fig. 6 and Table 1 that the number of instantaneous samples of q^R_k obtained with radar measurement data that surpass the chosen thresholds is considerably larger than what should be expected if the IMMPDAF errors with real radar data followed the reference probability distribution.

For further investigation and comparison of the IMMPDAF errors obtained from simulated and real radar data, the RMS position error along with a given (say the i ^th) simulation run has been calculated as follows:

where N is the number of points in the trajectory. In a similar way, the RMS error along with the single sample function obtained with real data has been evaluated. It is denoted by Є^RMS_real

The average and standard deviation of the RMS error samples obtained after 4,000 simulation runs have been calculated and are denoted here as Є^RMS_avg and σ^RMS_avg, respectively. A comparison with Є^RMS_real is shown in Table 2.

Table 2
IMMPDAF RMS error summary.

Measurement type	RMS error (m)
Real	ЄRMSreal = 617.20
Simulated	ЄRMSavg= 472.78, σRMSavg= 22.01

It can be seen that Є^RMS_real is 6.56 standard deviations greater than Є^RMS_avg, a significant difference of performance. To statistically quantify this difference, we regard the RMS error value as a random variable with first and second moments Є^RMS_avg and σ^RMS_avg, respectively. By using the Chebyshev’s inequality, we find that the probability of observing values of RMS error above Є^RMS_real is lower than 0.0232.

Therefore, this comparison of RMS errors leads us to observe once again that the IMMPDAF performance indexes obtained using real radar data seem not to be statistically consistent with those obtained from simulations.

NOISE ANALYSIS

Radar measurement errors in distance and azimuth were obtained according to:

where z_ρ,k and z_θ,k are radar measurements at time k in distance and azimuth, respectively.

Considering that the target trajectory was the same for the evaluation of both types of measurements, i.e. simulated and real, we may infer that the difference in IMMPDAF performance using real and simulated data cannot be caused by the target dynamics, but rather resides in the difference of noise characteristics in the two scenarios.

Since bias was compensated, the real measurement noise is assumed to be zero-mean. To further investigate its characteristics, Fig. 8 shows estimates of normalized autocorrelation and crosscorrelation functions for radar errors in distance (Є_ρ,k) and azimuth (Є_θ,k), obtained from a single realization of simulation and from real measurements.

Figure 8
(a) Normalized autocorrelation of simulated radar distance noise; (b) Normalized autocorrelation of real radar distance noise; (c) Normalized autocorrelation of simulated radar azimuth noise; (d) Normalized autocorrelation of real radar azimuth noise; (e) Normalized cross-correlation of simulated radar noise; (f) Normalized cross-correlation of real radar noise.

Figure 8 shows that the estimates of autocorrelation functions generated from simulated measurements present typical white noise behavior (Figs. 8a and 8c), as expected. It also confirms that simulated measurements in distance (range) and azimuth are uncorrelated (Fig. 8e), as it is usually assumed (Bar-Shalom et al. 2011).

On the other hand, estimates of autocorrelation functions obtained from real measurements (Figs. 8b and 8d) clearly show that the measurement error components at different time instants exhibit correlation. So these results suggest that the noise present in real environments would be better modeled as coloured stochastic processes. In Fig. 8f, it is shown that measurement components in polar coordinates present some cross-correlation, in contrast to what is normally assumed in radar tracking algorithms.

CONCLUSION

This paper presented an evaluation of the difference in the IMMPDAF algorithm performance using real and simulated measurements for the same target trajectory. The real measurements were obtained in a flight inspection performed by the Brazilian Air Force to evaluate a ground-based long-range surveillance radar. These radar measurements and GPS position data onboard the aircraft were employed to obtain performance indexes of the IMMPDAF algorithm in a real scenario and compare to those obtained by simulating radar measurements. It was shown that the IMMPDAF performance with real data was worse than that predicted by simulation, and this difference seems to be rooted on differences between statistical properties of simulated and real measurements. In particular, it was observed that the noise embedded in real radar measurements is not well modeled as perfectly white, contrary to the usual assumption made in simulations. These results point to the need of more realistic noise modeling in simulation-based performance evaluation of the IMMPDAF algorithm and, as one might infer, in the evaluation of tracking algorithms in general.

ACKNOWLEDGEMENTS

The authors would like to thank the Departamento de Controle do Espaço Aéreo for supporting this research.

REFERENCES

• Bar-Shalom Y, Daum F, Huang J (2009) The probabilistic data association filter. IEEE Control Syst 29(6):82-100. doi: 10.1109/ MCS.2009.934469
  » https://doi.org/10.1109/MCS.2009.934469
• Bar-Shalom Y, Li X, Kirubarajan T (2001) Estimation with applications to tracking and navigation: theory, algorithms and software. Hoboken: Wiley.
• Bar-Shalom Y, Willet P, Tian X (2011) Tracking and data fusion: a handbook of algorithms. Storrs: YBS Publishing.
• Blackman S, Popoli R (1999) Design and analysis of modern tracking systems. Norwood: Artech House.
• Blom HAP, Bar-Shalom Y (1988) The interacting multiple model algorithm for systems with Markovian switching coefficients. IEEE Trans Autom Control 33(8):780-783. doi: 10.1109/9.1299
  » https://doi.org/10.1109/9.1299
• Departamento de Controle do Espaço Aéreo (2014) Manual Brasileiro de Inspeção em Voo. Brasília: Ministério da Defesa; Comando da Aeronáutica.
• Departamento de Controle do Espaço Aéreo (2015) GEIV — Grupo Especial de Inspeção em Voo; [accessed 2015 Dec 23]. http://www.decea.gov.br/?page id=108
  » http://www.decea.gov.br/?page id=108
• Engel A (2005) Coordinate transformation algorithms for the handover of targets between POEMS interrogators. Technical report. Brussels: EUROCONTROL.
• Heidger R, Mathias A (2008) Multiradar tracking in PHOENIX and its extension to fusion with ADS-B and multilateration. Proceedings of the 5th European Radar Conference; Amsterdam, The Netherlands.
• Hess M, Heidger R, Bredemeyer J (2014) Tracker quality monitoring by nondedicated calibration flights. IEEE Aerosp Electron Syst Mag 29(8):10-16. doi: 10.1109/MAES.2014.120028
  » https://doi.org/10.1109/MAES.2014.120028
• Kirubarajan T, Bar-Shalom Y, Blair W, Watson G (1998) IMMPDAF for radar management and tracking benchmark with ECM. IEEE Trans Aerosp Electron Syst 34(4):1115-1134. doi: 10.1109/7.722696
  » https://doi.org/10.1109/7.722696
• Renes JJ, vd Kraan P, Eymann C (1985) Flightpath reconstruction and systematic radar error estimation from multi-radar range-azimuth measurements. Proceedings of the 24th IEEE Conference on Decision and Control. doi: 10.1109/CDC.1985.268714
  » https://doi.org/10.1109/CDC.1985.268714
• Rong Li X, Jilkov VP (2005) Survey of maneuvering target tracking. Part V. Multiple-model methods. IEEE Trans Aerosp Electron Syst 41(4):1255-1321. doi: 10.1109/TAES.2005.1561886
  » https://doi.org/10.1109/TAES.2005.1561886
• Sinha A, Kirubarajan T, Bar-Shalom Y (2006) Tracker and signal processing for the benchmark problem with unresolved targets. IEEE Trans Aerosp Electron Syst 42(1):279-300. doi: 10.1109/TAES.2006.1603423
  » https://doi.org/10.1109/TAES.2006.1603423
• Zeebroek Y (2010) SASS-C V6.7 user manual. SASS-C-UM-MAN-30, ed. 1.90. Brussels: EUROCONTROL.

Publication Dates

• Publication in this collection
  Apr-Jun 2016

History

• Received
  25 Dec 2015
• Accepted
  18 Apr 2016