The PEGASUS Method

6. Process Guidelines + Metrics for HAD Assessment

In general, an automated vehicle is a risk for different focus groups. The first two groups are the users of the HAD vehicle and the potentially involved accident partner, who can be any individual traffic participant (non-users). As previously discussed, the reason for the different views on accepted risk of the two groups results from the benefits the groups get. The third group is the society. Different from the first two groups, the fate of an individual is not relevant for society but the total accident number is. Hereinafter, we only discuss the occurrence of fatalities (index d), so instead of risk, we are allowed to give quantitative requirements for the occurrence rate or frequency of fatal accidents.

Quantitative requirements for different types of usage are given in Figure 1. It is concluded that the accepted frequency for a person’s death per year ƒ_inv is 10^-6 k_d ⁄ a for involuntary exposure, 10^-5 k_d ⁄ a for professional exposure (ƒ_prof), and a theoretically unlimited risk for voluntary exposure with typical acceptance rates ƒ_vol of up to 10^-2 k_d ⁄ a. In general, the accepted risk varies with the benefit for the user or focus group. Most of these considerations are from the 70’s and 80’s regarding the discussions on safety of nuclear power plants. However, the assumptions are still valid in general, but should be adapted to today’s level of safety. We suggest a factor of one forth, similar to the development of MEM from the 70’s to today. (Reduction by a change in accident rate would be another approach with a similar outcome.) In the following, the lower risk today compared to the numbers above is indicated by its index with year and country of the underlying statistic.

Users
The fatal risk for users is assumed equivalent to the risk of a fatal accident of a HAD vehicle (neglecting a higher damage with more than one user at the same time). As depicted in Figure 1, the type of exposition is relevant for accepting risks. In most use cases, HAD-F are used voluntarily; they must be actively bought and activated. Professional use is also plausible but the use for the job is not expected during the first introduction phase. Involuntary use is excluded in typical use cases. This consideration suggests following the risk acceptance rate of fprof, 2016, GER equal to 1.4∙10^-9 k_d ⁄ km . Similar rates are also present in the US (2.5∙10^-9 k_d ⁄ km) [FATALITY RATE PER 100 MILLION ANNUAL VMT - 2013, U.S. Department of Transportation Federal Highway Administration. https://www.fhwa.dot.gov/policyinformation/statistics/2013/pdf/fi30.pdf. Accessed July 17, 2018]. For other countries, data about the mileage on different road types is not always available. The accident rate on all roads’ combined mileage is in a similar order of magnitude for most developed countries.[Oguchi, T. Achieving safe road traffic—the experience in Japan. IATSS research, Vol. 39, No. 2, 2016, pp. 110–116] [Comparison of 2013 VMT Fatality Rates in U.S. States and in High-Income Countries, U.S. Department of Transportation NHTSA, October 2016. https://crashstats.nhtsa.dot.gov/Api/Public/Publication/812340. Accessed July 17, 2018].

However, the substitution of conventional driving also suggests comparing the risk of today’s driving with the suggested rate of ƒ_prof. In the following, both considerations will be examined and compared. For today’s driving risk, driving on Autobahn in Germany will be taken as a reference. This has several advantages. First, driving on controlled-access highways is one of the safest, if not the safest way of travelling in a car, especially when taking the accident rate per mileage as reference. Second, it is likely that the first HAV will drive on a controlled-access highway. Third, accident data on highways are well documented. Even minor accidents often result in the involvement of police because of the traffic disturbance and the measured traffic density estimates the travelled distance. Assuming an average travel distance d, the time-based frequency (index t) can be transmitted to a distance-based frequency (index s) and vice versa. In this example, 4000 km/a are assumed as an average travel distance according to VDA (see footnote 4) and an average velocity of 100 km/h.

In the following equation (1), the GAMAB principle and the MEM principle are combined. We consider this the upper limit for tolerable frequency because a new technology is introduced that comes with new risk. Additional risk is acceptable because the user experiences a benefit from that new technology. Note that we only consider the risk for the user in this section. This is not applicable to non-users or society at all, what will be discussed in the following sections.

Interestingly, the order of magnitude according to equation (1) corresponds to the accepted frequency for professional exposure. This strengthens the hypothesis that both estimations result in acceptable values for users of automated vehicles. However, higher risk could be accepted by the user (similar to motorbikes or extreme sport) but the user should be aware of this potentially increased risk.

Non-users
For all other traffic participants, HAD has no personal benefit (besides the decreased total risk for all traffic participants assuming that HAD is generally safer than the average driver). However, non-users could have a lower risk acceptance threshold because they are skeptical about the new technology or might even have (subjective) disadvantages e.g. due to slow vehicles on the road. Extraordinarily critical is the risk of new types of accidents because non-users would blame HAD for those accidents despite a potential reduction of the total number.

New risks could be caused for example by performance limitation (according ISO/PAS 2144), systematic software failures or cyber-attacks. The total new risk of the technology for an individual non-user should be below ƒ_inv,_2016,GER equal to 2.5∙10^-7 k_d ⁄ a  (= ¼ of the value for involuntary exposure). So how can the individual risk for a non-user be calculated? As long as there are not many HAD vehicles on the market, the exposure is very low and the probability that the individual traffic participant is involved in a HAD accident is low. So, the risk is multiplied with the field share µ. The risk for non-users is diluted by the exposure to vehicles equipped with HAD.

According to equation (2), the accepted risk for a single HAD vehicle is lower with increasing number of HAD vehicles. This is intuitively obvious because the exposure multiplies with the number of potential single threats. Comparing the risk level with equation (1) results in a number of 1.625·10⁶ HAD vehicles in Germany, until the risk acceptance of the other traffic participants becomes dominant. This deliberately neglects that the non-user also has benefits if the system is safer than the human driver it replaces. This will only be acknowledged by non-users if there is an undeniable difference in accident statistic. Otherwise, the (subjective) disadvantage of the new technology stays dominant.

Society
For society, the fate of individuals is of lesser importance. Benefits and costs of HAD are measured by the total number of accidents and if they are reduced over time. In general, a decreasing trend of accident rate throughout the years can be observed in Germany [Statistisches Bundesamt (Destatis). Verkehrsunfälle - Fachserie 8 Reihe 7 - 2015, 2015] and the US [Fatality Analysis Reporting System, U.S. Department of Transportation NHTSA, 2017. https://www-fars.nhtsa.dot.gov/Main/index.aspx. Accessed July 18, 2018]. However, we can observe that this trend has been diminishing over the last 5 years for accidents with injuries and even had a slight (but insignificant) increase during these years. For fatal accidents, this trend of a more slowly decreasing rate is observable as well, but less significant. One could suspect that there is a natural limitation with current road network, traffic density, and state-of-the-art vehicles.

When introducing HAD, there will still be a non-zero risk of severe accidents and therefore it is likely that HAD will be involved in those severe or even fatal accidents. So, what are the requirements by society if individual accidents do not influence the total number significantly? What is the upper total accident rate limit accepted by society?

The overall target is to reduce the amount of accidents over time with the introduction of new technology. If we follow the argumentation of Wachenfeld [Wachenfeld, W. H. K., Dissertation, Technische Universität Darmstadt, 2017] and Kalra [Kalra, N., and D. G. Groves. The Enemy of Good. Estimating the Cost of Waiting for Nearly Perfect Automated Vehicles, 2017], we should allow a certain risk to bring HAD to the market and to gain further knowledge. At the same time, it is not acceptable for the whole society that the total risk is increased in a noticeable way. However, there is no way to check how accident numbers would have evolved without the technology as soon as it has entered the market. However, in the last decade, the decrease of fatal accidents and accidents with injuries diminished. At the same time, the annual travel distance increased. Hence, it seems justified to use recent numbers as reference. When using the accident rate for fatal accidents ƒ_s,d , an exponential regression is a better fit than a linear regression. Interestingly, this is also the case for accidents in aviation (comp. footnote 5). The standard deviation for the exponential regression for all years since 2010 results in:

Multiplying the standard deviation with the average annual mileage in 2016 results in 23 fatal accidents per year. It must be pointed out that the type of regression and the number of years influence the result. It is also possible to use the double or triple standard deviation as a measure. However, the results will be in a similar order of magnitude. In the following, the result from equation (3) will be used.

The requirements by society should be that the risk from HAD is significantly lower than the described exponential trend observed in the latest data, so HAD should be at least one standard deviation σ_7y,exp better than the predicted performance of conventional driving. However, society should give HAD time to reach this high safety reference. Similar to air traffic, it is necessary to monitor the performance to allow improvement in functions, infrastructure, and user experience. In the following formula, it is suggested to allow additional risk of one standard deviation at the beginning of introduction and demand, that risk shall be three standard deviations lower than the extrapolation, when full market share is reached. Therefore, the acceptable risk not only depends on the development of the risk in conventional traffic over the years, but also on the market share of µ of HAD.

In the following, a field share µ is assumed that develops similar to the field share of other driving functions such as electronic stability control (comp. (5)). Full field share is assumed to be reached after 30 years and described by a sine function (1+sin⁡π∙t/T) ⁄ 2; 0≤t≤T, in Figure 1 but also other parameters are time dependent because the actual safety on the roads is expected to change over time, also without HAV.

Summary of Safety Requirements
In the previous sections, safety requirements for three different stakeholders were deduced. For society, the required risk depends on the market share of HAD. The authors suggest to allow an increase in total risk by one standard deviation of the predicted accident rate, so HAD can be introduced when the knowledge about its safety level is not yet complete. Additionally to society’s requirements, non-users (as part of society) have increased requirements for new risks that come with automation. For users, we suggest constant risk requirements although they might increase with the current traffic safety over the years. However, the user’s requirements are only dominant in the early introduction phase (comp. Figure 1) when the market share is relatively low. From a market share of about 10% on, the requirements of society (and non-users) are dominant. However, if the market share reaches 100%, there are no non-users remaining.

In Table 1, the requirements are summed up. Note that we currently only give values for Germany, because statistics about mileage on controlled-access highways is available. Since data for the accident rate on the whole road network is in the same order of magnitude for developed countries (see above), we do not expect significant changes in safety requirements.

Proof of Safety and Probation in the Filed
Although we have found individual and social risk acceptance criteria for HAD it is neither possible

• to derive safety requirements or test criteria for certain vehicles from risk acceptance criteria nor
• to integrate a set of scenarios to approve that individual and social risks of HAD are accepted.

What we need is for 1^st bullet point a Proof of Safety and a Probation in the Field for 2^nd.

Proof of Safety
Social consensus regarding acceptable risk is regulated by liability laws, e.g. German ProdSG §5(2), according to which a product that conforms to standards or other relevant technical specifications is presumed to comply with product safety requirements, as far as they are covered by the standards or other relevant technical specifications. Development according to ISO 26262 ensures “absence of unreasonable risk” (i.e. “risk, judged to be unacceptable in a certain context according to valid societal moral concepts”). Other relevant technical specification is e.g. ISO/PAS 21448 for SOTIF, represented by automation risks in PEGASUS. But conformity with standards or other relevant technical specifications is only a necessary condition.

A Sufficient condition is given by the rules of the Ethics Committee [Ethik-Kommission Automatisiertes und Vernetztes Fahren, BMVI, Juni 2017], which says:

• HAD is reasonable if it promises to reduce damage in the sense of a positive balance of risk compared to human performance.
• If there is a fundamentally positive balance of risk, technically unavoidable residual risks do not preclude an introduction.

A fundamentally positive balance of risk compared to human performance can be argued following expert’s expectations of a generally benefit of vehicle automation for traffic safety by

• avoidance of accidents and
• reduction of consequences [Rechtsfolgen zunehmender Fahrzeugautomatisierung, Berichte der Bundesanstalt für Straßenwesen, Heft F 83, 2012].

Moreover, the test concept developed in PEGASUS ensures exemplarily, that the systems achieve at least human driving performance.
Beside the general benefit for traffic safety, HAD will provide automation risks in individual cases because of performance limitations and inadequate interaction with drivers and other traffic participants. Those automation risks are identified and - as far as possible - quantified in PEGASUS to ensure that HAD systems cause only technically unavoidable residual risks.
However, product development according to standards and specifications does not replace product tests in frame of verification and validation. An evaluation of different test strategies is given by Junietz et al [Junietz, P., W. Wachenfeld, K. Klonecki, and H. Winner. Evaluation of Different Approaches to Address Safety Validation of Automated Driving. Accepted Paper. In 2018 Intelligent Transportation Systems Conference (ITSC), 2018].
A crucial limitation of all a-priori considerations consists in the fact, that a valid statistical proof, that the systems actually meet the expectations, cannot be provided before they are launched on the market. This results in requirements for Probation in the Field as well as in measures to be derived therefrom for the continuous improvement of the systems.

Limited Introduction and Probation in the Field
Despite all consideration, normative specification and product tests, there will be an uncertainty about the future performance of HAD at the time of the initial introduction to the market. This uncertainty either can be accepted if all stakeholders agree that the residual risk is sufficiently small or controlled by reducing the number of sold vehicles in the introduction phase (see footnote 15). The market share µ is controlled and the exposure for the society and non-user reduced. Only the user has to accept the uncertainty if he wants to use HAD. However, the performance of HAD should be observed and statistics publicly discussed to build trust in customers and the society, but also to identify critical situations and improve the system with updates.

Recent measures of field observation have limitations regarding consideration of special aspects of HAD, detection of rare events in a timely manner and consistency during complete product lifecycle. Potential improvements are for example

• Further development of accident models
• Definition of indicators for deficiencies of HAD systems
• Increase density of accident analyses
• Field studies with HAD systems across manufacturers
• Surveys of customers who own vehicles with HAD systems
• Effective testing of HAD systems during Periodical Technical Inspection.

It is necessary to implement those improvements before market introduction and to install an independent organization for execution of proof of probation in the field. For efficient product monitoring and continuously optimization of HAD, event data recording (EDR) is necessary. Therefore, standards have to be developed and implemented as well as procedures to deal with data.

The legislature has just created the framework for the introduction of HAD with the amendment of the German Road Traffic Act (StVG) in 2017. A proof of Probation in the Field is therefore also an essential prerequisite for the desired and required further development of the legal framework for the introduction of higher levels of automation.

7. Data in PEGASUS-format

For the PEGASUS project, a feasible way to gather data from different sources and different partners needed to be developed. As pointed out in step 2 the sources differ among others from simulated data over FOT and NDS data up to data from development vehicles. In order to get this data into a single database, a common data format needed to be established. The main goal is to ensure a consistent handling of data throughout the different processing steps. To achieve this, different requirements are checked, and a definition based on this information is generated.

First a structure for all the different data needs to be developed. Since the processing of data is done in Matlab, a Matlab structure is chosen. However, Matlab is not available to every partner in the project. Therefore, also hdf5-files can be used, but they need to have the same internal structure as the matlab-files.

The main structure inside the data-files is designed to have a channel for every signal that is saved in it. Inside this channel, a Time as well as a Value vector corresponding to each other need to be provided for every signal. The time vector should have a (mostly) fixed frequency throughout the entire measurement, but the frequency can vary between different signals. Values which do not change within an entire measurement, can be stored only once with a timestamp of 0. With this structure, all required data can be stored. All mandatory, but also further useful signals are defined. Additional signals can be enclosed, so that special data, which may be available in different test vehicles, can be integrated into the format.
Different requirements were assessed during this process step. The main requirements were given by the different data sources as well as from the further processing of the data. The main goal was identified as to be able to represent every possible situation which may occur to the highway chauffeur function in the project. Since the highway chauffeur is bound to highway scenarios, the data format only needs to be capable of representing these scenarios. Since most of the data is captured from vehicles, an ego-perspective is used in the developed format.

Generally two things need to be represented by the data format:

• Objects in the current scenario
• Environment of the current scenario

Used coordinate systems
To ensure a consistent representation, two coordinate systems are used in the format. A global coordinate system (North, East, Altitude) to describe the overall movement of the ego vehicle and a local coordinate system (x,y,z) bound to the rear axle of the ego vehicle. The directions in as well as the connections between these two are shown in Figure 1.

Most of the data is located in the vehicle coordinate system, since the main source for data are recordings made in different vehicles. Nevertheless a global coordinate system is necessary to be able to represent the overall trajectories of the ego and all surrounding objects.

Object representation
The object representation has a maximum of 65 object slots. This includes the ego vehicle and up to 64 surrounding vehicles. To distinguish the data between these object slots, every signal is replicated for every object slot. This is done by including the current slot number into the name of every signal. Like this the signal names become idX_SignalName with X reaching from 0 to 64. If there are less objects, not all of these slots need to be used. For every object, signals for typical translation and rotation values as well as other properties can be stored. All these values are stored relative to the ego vehicle, since this is the common format for measurements taken from a test vehicle.

To be capable of storing more than 65 vehicles with this format, a global object id is stored as one signal for every object slot. With this global object ID, every slot is able to store different objects over time, since single objects are likely to get out of range after some time. So every object within the measurement data gets a unique global object ID. For the further data processing steps, these global object IDs, and thus the current object, need to stay within a single slot of the data format. Since the number of used slots needs to stay constant within a single measurement, currently not used object slots need to be indicated by a global object ID of -1.

For the ego vehicle additional signals are necessary, to give a full representation of the current scenario. To be able to recalculate the entire trajectories, values for Northing and Easting of the ego vehicle as well as its current heading angle need to be provided. These values are not needed for the surrounding objects, the complete trajectories of these can be derived from the ego vehicles position.

Environment representation
As already mentioned, the environment in the PEGASUS project is restricted to highway scenarios. Therefore, an easy approach to describe the current situation for the data format can be used. The environment is described by the lane markings and the roadside infrastructure. All these are described by a third degree polynomial using four signals each (distance, heading, curvature and curvature derivative). With the help of these values, the position along the road of each feature can be calculated. Figure 2 and Figure 3 show the general concept of the lane markings and also the definition for heading and distance. Just like for distance and heading in Figure 3, the curvature and curvature derivative also need to be given at the rear axle of the ego vehicle

The Equation 1 to Equation 3 show how to compute the position, heading and curvature in the ego vehicles coordinate system:

Additional data, like lane marking color, lane marking type or type of roadside infrastructure (barrier, guard rail, grass, …) can be given with an additional signal.

Necessary recommended and optional signals
Based on the requirements of the database process some signals need to be available in every measurement. These signals are shown in Figure 4. For a basic representation of the environment, at least the left and right lane marking need to be available. These are bound to the ego vehicle. Further the positions and speeds, as well as the global object ID of all objects, are mandatory for every measurement. For the recalculation of the global trajectories of all objects, the absolute position of the ego vehicle is also needed in every measurement.

If this data is available in a measurement, all further process steps can be carried out. As already mentioned, a lot more signals are defined within the format. These can be used for further extensions of the database processes or other specific calculations. The remaining signals are divided into recommended and optional signals. The recommended signals should be provided if possible, the optional ones are possibly only necessary for special applications.

In the following a list of mandatory PEGASUS JSON-signals is shown. These signals must be available in test data. The convention is that the index for the EGO is 0 and is set between 1 and 64 for surrounding objects.

8. Application of Metrics + Mapping to Logical Scenarios

Mapping to Logical Scenarios

For the mapping from measurement data to logical scenarios, different signals and data are needed. To this purpose, a processing framework is set up. It takes the recorded data as input and outputs the mapped scenarios together with key performance indicators (KPI) that describe the logical scenario.
In a first step, the data is checked for errors and, if possible, these errors are handled and corrected automatically. If not, the data conversion step must be repeated. Since there are many different KPIs that may be desired for the analysis of the different logical scenarios, the processing framework is flexible concerning the computed measures. In order to achieve this, it analyzes the given scripts and functions needed to derive the measures builds a computation tree from the requirements of the distinct scripts. It analyzes input and output of each script and orders them in a way that always, all needed signals have already been calculated.

The functions and scripts are put in four distinct groups: derived signals, scenario probabilities, scenario extraction and indicators. Derived signals are the first step of computation. Here signals such as the time-to-collision, the time headway or also the assignment of objects to particular lanes are calculated. Scenario probabilities return a probability for each time step of the recording, indicating whether that time step is part of a scenario and with which probability. Functions for scenario extraction then take these probabilities and extract scenarios, thereby not only taking into account the single probabilities but also accounting for sensory deficiencies and processing hiccups. The last group of function, the indicators, extract KPIs for each of the extracted scenarios. Depending on the type of scenario, these indicators can be more or less extensive.
After all these functions have been processed, the data is aggregated for further use. In the current state of the processing framework, it is passed on to the database for further aggregation across trips and recordings. To this purpose, the extent of a scenario and its KPIs are transferred to a JavaScript Object Notation (JSON) file. This can then be read in by the data base.

All of this framework is executed for one sole purpose: the extraction of scenarios and the KPIs describing those scenarios. Due to its modular nature, the framework can always be extended to extract more scenarios or calculate more KPIs. If the naming conventions of the used signals is kept and the needed signals are available, the framework will easily integrate any new function.

The before mentioned scenarios are recognized in the functions of the scenario probability group. From the data, the probabilities are derived using a four-stage process. For this purpose, in the first stage, comfort zones are set up around the ego vehicle. These zones are shown in Figure 1 The zone shown by the green square is limited by the ego lane markings to the left and the right. Limitation to the front is handled dynamically by the time headway

(THW). For the second comfort zone (red in Figure 1), the dimensions are set statically. A common selection here is a few meters before and behind the car and 1 m beside the car. Bear in mind, that other objects are handled by their center, e.g. if the distance from the side of the ego vehicle to the object is 1 m it is already very close.

As noted above, the framework has a modular approach. Therefore, the assignment of objects to the lanes is given as derived signal. This means, that when the mapping to the logical scenarios is performed, the function already knows if objects are in the ego lane at a distinct point in time or not. The same goes for the infringement of the proximity (red) zone. Therefore, the second stage to the recognition is computing all entrances into the ego lane across the complete trip. This is then complemented with the information of the exact position of the infringing object, thereby taking into account the THW thresholds used to define the relevance of the logical scenarios. Entrances into the (green) comfort zone are only considered relevant, if they occur below a certain threshold.

Once a candidate for entering the comfort zone is found, the third stage is the mapping to the actual logical scenario. To this purpose, the time before and after the object entering the comfort zone is investigated. Using the objects dynamic information, the framework can deduce which logical scenario this entrance belongs to.

The fourth and last stage is the reduction of parameters needed for the description of the scenario. For brevity in the description of the scenarios and thereby also for memory efficiency, not the complete trajectory is stored. Moreover, only parameters allowing the precise reconstruction of the trajectory are computed. For entrances from the front and rear, this is rather simple, since a straight movement relative to the road can be assumed. For entrances that include a lane change, a fitting of a simpler function describing the lane change is done. This way, even more complex scenarios can be described using only a small subset of the parameters in the raw recording data.

After a scenario has been fully recognized and its parameters have been extracted, the probability of the scenario is stored for further usage in the extraction process and for the computation of the KPIs.

Application of Metrics

In the following the purpose and conception of the criticality metrics are described as one example of an application of metrics. More detail can be found in the publication by Junietz et al. [1] When deducing scenarios from real world data, a filter to find relevant scenarios is required. Most driving especially on motorways is monotone and does not require a special test case [2]. The metric should detect all scenarios with increased driving requirements in order to derive test cases that are challenging. It can be applied on data from NDS/FOT and during HAD test drives to detect relevant scenarios. When the metric is applied online during driving, real-time computation is necessary. If this requirement is not met, a metric to filter all definitely irrelevant scenarios could be used in a first assessment step in order to reduce the number of scenarios that need further computational effort. The Worst-Time-to-Collision [2] is a metric designed to work as a first filter to efficiently find those scenarios and could be used here.

Requirements for a Criticality Metric
Besides the absence of accidents, metrics that describe the criticality of the test scenario could be used in order to assess the safety (compare section 4.3.2). Criticality is hereby defined as the driving requirements that are present in the scenario. The driving requirements are highly dependent on the scenario, the HAD-F is exposed to. Hence, a general requirement cannot be deduced. The occurrence of scenarios with high requirements can only be compared statistically in real world driving. If two different HAD-F perform in a different way in the same artificial test (simulation/test field), there is not necessarily a difference in safety, as a more conservative driving strategy does necessarily mean increased safety, if both HAD-F drive accident-free. An exception would be if the driving capabilities of the HAD-F are known (e.g. a reaction time until a full brake due to computation time and brake system delay). In this case it is a requirement that the driving capabilities are sufficient to perform accident-free and the driving strategy has to be adapted accordingly.

As all kinds of accidents shall be prevented, the accident severity is not considers here, so it is not a risk metric. Ideally, the metric should describe the probability of an accident in a specific situation for AD or a human driver. As different drivers and different AD also have different driving performance and therefore different collision probability, the metric shall describe the driving requirements in a situation that is independent of the available driving performance [3]. The driving requirements consist of the following entities:

1. Necessary acceleration in lateral and longitudinal direction
2. Margin for corrections of course angle (side distance)
3. Margin for correction of speed (front distance)

These entities depend on each other. A higher margin for corrections can be bought with a higher deceleration early on. Hence, the three entities should be normalized and combined into a joint value.

Comparison with State of the Art
Various metrics exist for the identification of scenarios, trajectory planning, and the assessment of criticality in general. They can be classified into a posteriori metrics, which are often used to analyze NDS [4, 5], and metrics with deterministic [6–9] and probabilistic trajectory prediction [9–13]. An overview and a classification of metrics with trajectory prediction can be found in [10] and [14]. However, those metrics do not combine all of the mentioned entities. In [15, 16] monte-carlo sampling is used to address combined maneuvers. In [17], the reachable and available free space is assessed to derive criticality. However, these approaches do not address the difficulty of driving on those trajectories. In a combination of different criteria is often done in trajectory planning and optimization [18, 19]. Defining an optimization function to be minimized offers the opportunity to combine different entities and weighting factors. Typically, the resulting trajectory is of interest together with the required control values in case model predictive control (MPC) is used. In PEGASUS, we use a similar approach with an optimization function designed to describe criticality as the driving requirements mentioned above. The minimized value of the optimization function is the criticality in the situation. It is important to note the difference between this approach and most use-cases for trajectory optimization. The metric has no direct feedthrough to the trajectory control. Its only purpose is to optimize possible future trajectories in order to calculate the criticality. This could be either done online during test drive, or a posteriori from recorded data.

Computation
To obtain the desired criticality metric, we define the state-space model with state vector x=[x_w y_w v_v ψ_c] and input vector u=[a_x a_y]. The problem depends on the current ego velocity in natural coordinates v_v, the course angle ψ_c, and the acceleration a_x,a_y in natural coordinates. Additionally, the position in world coordinates is required because the position of the objects is not updated into vehicle coordinates once the prediction horizon is initialized. For most MPC applications, a linear single-track-model is used. This has the advantage that the actuator inputs can be optimized directly. However, in the application of the criticality metric, the accelerations are of interest. As we do not use the model for actual vehicle control, additional deviations from a simplified model (e.g. because the neglecting of tire cornering stiffness) can be neglected. Instead, a point mass model is used that is extended by a course angle that is required to transform the vehicle motion into world coordinates. A non-linear vehicle model is described by:

Assuming small course angle changes and small changes in velocity allows linearization of the model. Initializing the model at course angle zero and using a constant velocity v_old, which is updated only at the beginning of the prediction horizon (comp. [18]), the equations simplify to:

Resulting in the system matrices:

The goal of the MPC is to optimize the trajectory subject to the criticality within a given prediction horizon N. The first two influence factors are the accelerations that are minimized. The margin for corrections of angle is dependent on the lateral distance and the velocity. A higher distance gives more space to correct the course angle. A lower velocity has two influence factors: A disturbance (e.g. side wind) has a smaller influence towards a deviation in the position (because the velocity is integrated) and the course angle can be corrected with less effort in a_y. As we want to minimize a term that includes maximization of distances, we use the position with the maximum lateral distance to all obstacles as reference r_y and a maximum deviation d_y resulting in the following term:

The margin in longitudinal direction cannot be used in such way, because only front distance is relevant. As reference r_x we use the recommended following distance in Germany (0.5∙v/(km/h)). We use the piecewise linear function:

With the minimization of an objective function alone, the collision with other objects is not excluded. Hence, the outputs are bound by constraints that are updated at the start of the prediction horizon assuming constant velocity and course angle. Again, we neglect objects coming from behind and represent the left, right, and front boundaries to be constrained by c_l,c_r, and c_f respectively. Additionally, the inputs must be bound because the dynamics of a vehicle are limited by the available friction coefficient. According to Kamm’s circle:

However, to allow efficient solving of the optimization problem, Kamm’s circle is approximated with 12 linear constraints using , and  taken from [18]. Modeling the objective function with the sum of the defined elements with constant weighting factors  gives us following optimization problem:

The weighting factors are chosen in a way that allows equal weight of the four elements of the optimization function. However, R_x typically results in values 10 to 20 times higher than the other elements. Because the ego vehicle is approaching, the elements of the objective function are summed up for subsequent steps. Hence, we choose a weighting factor of 1/10 here, while all other factors remain 1. The final criticality is the maximum of the acceleration during prediction divided by the maximal acceleration possible.

Developed Workflow
The computational effort of the proposed metric is high compared to state of the art metrics. Hence it is suggested to preselect scenarios that might be relevant or in other words filter scenarios that are definitely not relevant. We suggest to use the metric Worst-Time-to-Collision (WTTC) [2] as it addresses all kinds of scenarios. It calculates the time until a collision if both vehicles perform the maneuver that leads to the fastest collision possible (Figure 2).

As threshold, a WTTC value of at least one second is recommended based on calibration with real world scenarios [3] (Figure 3). Other metrics would also be possible in scenarios were they can be applied (e.g. front-to-rear collisions for TTC, TTB or required acceleration). The identified threshold are similar to those found in the SHRP2 database [20].

If sufficient data is available, machine learning could be used to classify scenarios for a first evaluation. During training, false negatives should be prevented using a cost-matrix punishing those events. For classification an ordinal scale is suggested to classify the scenario into uncritical, slightly critical and critical. For a cut-in scenario, the metrics WTTC, Time-Headway, Time-to-Steer and Collision Index (CI=v²/TTC) were favored using methods of feature selection. With more data available, a trained classification model could replace the suggested filter by WTTC only.

References
[1]    P. Junietz, F. Bonakdar, B. Klamann, and H. Winner, “Criticality Metric for the Safety Validation of Automated Driving using Model Predictive Trajectory Optimization,” in 2018 21st International Conference on Intelligent Transportation Systems (ITSC), 2018, pp. 60–65.
[2]    W. Wachenfeld, P. Junietz, R. Wenzel, and H. Winner, “The worst-time-to-collision metric for situation identification,” in 2016 IEEE Intelligent Vehicles Symposium (IV), 2016, pp. 729–734.
[3]    P. Junietz, J. Schneider, and H. Winner, “Metrik zur Bewertung der Kritikalität von Verkehrssituationen und -szenarien,” in 11. Workshop Fahrerassistenzsysteme, Walting, 2017.
[4]    M. Benmimoun, Automatisierte Klassifikation von Fahrsituationen: fka Forschungsgesellschaft Kraftfahrwesen mbH, 2015.
[5]    T. A. Dingus, R. J. Hanowski, and S. G. Klauer, “Estimating crash risk,” Ergonomics in Design: The Quarterly of Human Factors Applications, vol. 19, no. 4, pp. 8–12, 2011.
[6]    C. Rodemerk, S. Habenicht, A. Weitzel, H. Winner, and T. Schmitt, “Development of a general criticality criterion for the risk estimation of driving situations and its application to a maneuver-based lane change assistance system,” in Intelligent Vehicles Symposium (IV), 2012 IEEE, 2012, pp. 264–269.
[7]    H. Winner, S. Geyer, and M. Sefati, “Maße für den Sicherheitsgewinn von Fahrerassistenzsystemen,” in Maßstäbe des sicheren Fahrens. 6. Darmstädter Kolloquium Mensch + Fahrzeug, 2013.
[8]    R. K. Satzoda and M. M. Trivedi, “Safe maneuverability zones & metrics for data reduction in naturalistic driving studies,” in Intelligent Vehicles Symposium (IV), 2016 IEEE, 2016, pp. 1015–1021.
[9]    F. Damerow and J. Eggert, Eds., Predictive risk maps. Intelligent Transportation Systems (ITSC), 2014 IEEE 17th International Conference on, 2014.
[10]    M. Schreier, Bayesian environment representation, prediction, and criticality assessment for driver assistance systems. Dissertation, Technische Universität Darmstadt, 2016. Düsseldorf: VDI Verlag GmbH, 2016.
[11]    J. Eggert and T. Puphal, “Continuous Risk Measures for ADAS and AD,” in FAST-zero '17, 2017.
[12]    J. Eggert, “Risk estimation for driving support and behavior planning in intelligent vehicles,” at-Automatisierungstechnik, vol. 66, no. 2, pp. 119–131, 2018.
[13]    D. Althoff, J. Kuffner, D. Wollherr, and M. Buss, “Safety assessment of robot trajectories for navigation in uncertain and dynamic environments,” Auton Robot, vol. 32, no. 3, pp. 285–302, http://dx.doi.org/10.1007/s10514-011-9257-9, 2012.
[14]    S. Lefèvre, D. Vasquez, and C. Laugier, “A survey on motion prediction and risk assessment for intelligent vehicles,” (English), Robomech J, vol. 1, no. 1, pp. 1–14, http://dx.doi.org/10.1186/s40648-014-0001-z, 2014.
[15]    A. Broadhurst, S. Baker, and T. Kanade, “Monte carlo road safety reasoning,” in Intelligent Vehicles Symposium, 2005. Proceedings. IEEE, 2005, pp. 319–324.
[16]    A. Eidehall and L. Petersson, “Statistical Threat Assessment for General Road Scenes Using Monte Carlo Sampling,” Intelligent Transportation Systems, IEEE Transactions on, vol. 9, no. 1, pp. 137–147, 2008.
[17]    C. Schmidt, Fahrstrategien zur Unfallvermeidung im Straßenverkehr für Einzel- und Mehrobjektszenarien. KIT, Diss.--Karlsruher Institut für Technologie, 2013. Karlsruhe, Baden: KIT Scientific Publishing, 2014.

9. Logical Scenarios & Parameter Space

Logical scenarios that were previously defined are stored in this data container together with their variants extracted from measurement data. Logical scenarios are an abstract description of a traffic scenario. They are characterized by the fact that not all scenarios, such as distances or speeds, have a concrete value. Instead, some values are left open and only parameter ranges are specified. Which combinations of parameters occur how often can be extracted from measurement data. Several components are required to be able to save these logical scenarios together with the variants found from measurement data (concrete scenarios). First, parameters are derived from the modeling of the logical scenarios that uniquely describe the concrete characteristics of the scenarios. These parameters are then determined automatically in the extraction step for the scenarios found. In order to be able to use the modelled scenarios in simulations or on the test track, templates are created in OpenSCENARIO and OpenDRIVE. These templates are created manually for each scenario and stored in the database.

After scenarios in the previous process steps have been found and extracted from measurement data, e.g. from test runs, they are stored in this data container. However, scenarios must not only be extracted, but it must also be possible to play them back in the simulation and on the test track. On the one hand, it is important that a scenario found in measurement data, including the static environment, can be algorithmically correctly extracted and modelled. On the other hand, it is at least as important that these scenarios can be described technically so that the events can be reproduced precisely and that the description format used can be used in the simulation and the test track.

Within this project, the two description formats OpenDRIVE and OpenSCENARIO are used to fulfill this task. Both formats are open standards and are based on the XML description language. They solve the problem that an open standard is needed to describe the static and dynamic environment. Thus, within the database, a single combination of standards can be used to aggregate data from different sources such as FOTs, NDSs or simulations and simultaneously use them on the test track or simulations from different vendors.

OpenDRIVE was first released in 2005 and allows to describe a static environment with a focus on streets. Thus, individual road elements such as straight lines and curves can be parameterized and combined with other elements to create entire routes. OpenSCENARIO was first introduced in 2014 and further developed during the PEGASUS project. Unlike OpenDRIVE, OpenSCENARIO is used to describe the temporally dynamic elements of a scenario. For example, the position of a traffic light is defined in OpenDRIVE, while the switching times are defined in OpenSCENARIO. However, the focus is on the vehicles themselves. For example, vehicles are defined in catalogs and positioned on a route that is referenced in OpenDRIVE. The behavior of the vehicles in the scenario can be described by a story. Vehicle actions can be triggered when conditions occur. The actions include, for example, a lateral movement to change lanes on different levels of abstraction. A vehicle can be given only a new destination lane, the transverse movement can be described by geometrical primitives or precisely given by a trajectory in the form of a list of positions.

Although OpenDRIVE and OpenSCENARIO are technically powerful enough to define scenarios, there is still room for improvement. For example, the definition of routes in OpenDRIVE is often time-consuming, especially for more complex routes. For this reason, DLR introduced the SimpleRoad description language. This makes it easier to generate routes by providing a shorter and easier to understand description. Files in SimpleRoad format can be translated into OpenDRIVE by a provided converter.

Even when describing the dynamic parts of a scenario in OpenSCENARIO, compromises must be made in the current version with regard to parameterization. This becomes clear with the example of the description of a lane change trajectory. Within the scenario concept, lane changes are described by fourth degree polynomials. However, since the current version of the OpenSCENARIO standard does not support these as geometric primitives, such a trajectory must be defined as a list of positions. This makes it possible to use the polynomial for lane changes, but this trajectory cannot be described by a few parameters in OpenSCENARIO itself. On the other hand, supported geometric shapes such as straight lines can be easily modified by parameters, which is an important requirement for stochastic variation. The solution to this problem is to allow the introduction of non-standard compliant elements into OpenSCENARIO. For these, however, an appropriate script must be provided, which is executed by an introduced Transpiler and translates the new definition into standards-compliant OpenSCENARIO. Thus, new maneuvers and other elements can be easily integrated and distributed to partners for evaluation before ultimately eventually being integrated into the standard.

These data formats (OpenDRIVE and OpenSCENARIO) and extensions (SimpleRoad and Transpiler) are used to define the logical scenarios. As already mentioned, not all parameters are assigned a concrete value. Since elements such as distances and speeds are not fixed in a logical scenario, placeholder variables are specified in the files at these points. Before the scenarios can be used for a simulation, for example, these variables must be replaced by concrete values so that a concrete scenario is created from the logical scenario.

Method to define acceptance criteria

As an example, the single reeving shall be considered from the right, which is shown in Figure 30. For the dynamic elements of this logical scenario, a total of 8 parameters were required, which are listed in Table 1. If a value is defined for each parameter, the constellation of the vehicles shown in this scenario can be uniquely described. The space of all possible parameter combinations describes the parameter space. It contains all possible variations of this one scenario. After, for example, a right-hand cut-in has been detected in the data, the values of all parameters are determined from the trajectories and saved. After this has been carried out for a large number of scenarios, analyses can be carried out in the next process steps, for example on the frequency of certain characteristics, in order to use the findings for the simulation.

coming soon

12. Application of Test Concept incl. Variation Method

Test Concept

The PEGASUS Testing Method is located in the upper right side of the general V-model (see figure 1). It is important to note what is in respectively not in the scope of the Testing Method.

In the scope of the PEGAUS Testing Method is:

• Safeguarding the AD-Function regarding risk of collision
• Sensor performance as an input for system performance

Not in the scope: Other HIGHLY Automated Driving Functions relevant topics covered by OEM, Suppliers tests or regulations: (examples)

• Testing according ISO26262
• Direct safeguarding of the sensor performance
• Safeguarding
  
  • interaction with driver (HMI, …)
  • compliance with traffic rules

The Goal of the test concept is the development of a method to generate an evidence regarding the safeguarding of a level 3 system (Autobahn-Chauffeur, max. 130km/h). The test instances used are shown in figure 2.

Input for the test concept are a set of logical test cases with corresponding parameter space incl. Exposure of parameters and Pass-/Fail criteria as well as special concrete scenarios (for example from certification purpose, accident data, automation risk analysis if testable in the described context (see above)).

The expected output are evaluated Scenarios and the probability for collision scenarios (with crash severity). Starting from space of logical test cases (= logical scenarios + parameter space + Pass/Fail-Criteria + exposure of parameter) test cases get allocated to test validation levels.

Starting from the space of test cases, the test cases are assigned to the PEGASUS test instances.
Hereby, “all” logical scenarios get tested in the simulation to create a high efficiency. Based on the automatized/stochastic variation of the logical scenarios parameter, concrete test cases are created. These concrete test cases get evaluated after the Pass-/Fail-Criteria.

Interesting or critical cases (i.e. not fulfilled or close fulfilled Pass-criteria) get tested or validated in real cars on a proving ground. In addition, manual selected concrete test cases can be evaluated on the test ground (i.e. test as per ECE R79 or rating tests).

Within field tests it is not possible to test specific test cases of the space of logical test cases. Instead, the behavior of drive features get tested in real traffic, whereby guidelines of route, time or weather, challenges for the HAF- function can be created. The major target is to find “surprises” (e.g. new scenarios, new parameters) and create new measured data for further analyzes in Replay2Sim.

How does the test case allocation take place and how does the test procedure works?

Test case allocation:

• Simulation:
  
  • All logical test cases regarding scenario based testing with a high number of scenarios but a low relevance regarding real sensor performance
    
• Proving ground:
  • Pre-selected tests, e.g. certification tests
  • Test with a high relevance regarding drive dynamics and real sensor performance
  • Rare events which can hardly be seen in field tests
    
• Field tests: Tests with a high relevance regarding real system performance under a high variation of surrounding conditions

Deviation of test cases for simulation is done by stochastic parameter variation and test automatization respectively, for the test ground based on manual selection or identification of relevant scenarios within the simulation.

Test result:
Based on Pass/Fail criteria evaluated concrete scenarios for simulation, test ground and field test Probability for accident scenarios

Test end criteria regarding simulation (suggestion):

• Create transfer function between scenario parameters (input) and test result (output, e.g.accident yes/no, distance between ego-vehicle and relevant target):
  Target value for quality of transfer function (e.g. R_Qd-value) ≥ 80%*
• Calculate standard deviation σ of computed probability for accident scenarios:
  Target value for σ ≤ 20%* (*according state of the art)

Starting from space of logical test case the test cases and test level get assigned. Hereby “all” logical scenarios within the space of logical test cases get tested in the simulation. Based on the automatized/stochastic variation of the logical scenarios parameter, concrete test cases are created. These test cases get evaluated after the Pass-/Fail-Criteria. Examples of these criteria are shown in figure 5. Critical cases (i.e. not fulfilled or close fulfilled Pass-criteria) get evaluated in real cars on a proving ground. In addition, manually selected concrete test cases can be evaluated on the test ground (i.e. accident scenarios, rating or certification tests). Within field tests it is not possible to test specific test cases of the space of logical test cases. Instead, the behavior of drive features get tested in real traffic.

The major target is to find “surprises” (i.e. new scenarios, new parameters). These surprises may be enforced by different guidelines in route (i.e. tunnel) or time (i.e. low sun) and can be further analyzed in Replay2Sim.

Test of the HAD-function in real world traffic (long term testing)

• identification of specific individual situations within the framework of event-based simulation
• Identification of faults/impairments of the system as a result of environmental influences cannot currently be simulated directly via models, as no suitable physical models are yet available.
• Perform special “pass” assessments (e.g. risk in passing or following other vehicles)

Variation Method

The processing step of the stochastic variation plays a central role within the concept of a scenario-based assessment of an automated driving function. Given a scenario data model with all relevant parameter spaces as well as additional a-priori knowledge about these spaces from a central database, the variation method has the main task to find all relevant critical parameter subsets within each logical scenario space. According to the PEGASUS test concept this variation is mainly applied within the simulation since it is the only test instance capable of handling a large number of test runs at relative low cost. The stochastic variation of parameters within all relevant logical scenario spaces offers the most complete exploration of the extremely large scenario space representing the reality as faced by the automated driving function. The variation and simulation should deliver a description of all relevant critical scenario subspaces. This description or so-called characterization of the identified critical subspaces including knowledge about their occurrence probability is then passed to the risk assessment subsequently accumulating these results to a total risk measure. This risk measure is the basis of the PEGASUS safety argument.

Given a logical test case (e.g. OpenSCENARIO, OpenDRIVE, parameter space description, and metrics for safety assessment of simulation results) the goals of the parameter exploration can be defined as one of the following:

1. Find one or more worst cases of the safety metrics in the defined parameter space. This can be defined as an optimization problem.
2. Characterize the subspace of parameter values where a metric reports a safety violation
3. Quantify (e.g. find an approximation of) the probability of safety violation in the defined parameter space, if possible, also with a confidence measure in the probability value.

A combination of algorithms following the goals of (1) and (2) can also be used to provide evidence that inside the most probable subspace of parameter values no safety violations occur – in case, the search inside this subspace does not deliver any scenario violating the safety limits.

More formally, the input of a parameter exploration algorithm can be defined as follows:

• N parameters {P1, .., Pn} with mixed discrete and continuous value domains and with defined min-max bounds.
• One or more safety metrics {M1, M2, … }, such as time-to-collision (TTC), distance-to-collision (DTC), and others.
• Optionally the parameter description can include constraints among parameter values, such as p1 < p2.
• The parameter description can also include prior probability distributions and correlations among the parameter values – a required input for all exploration methods that have the goal to assess the integrated probability of safety violations.

In order for the results of the simulation to be useful for a safety argumentation the following properties of the simulation models, the input data and the logical scenarios must be ensured:

• The simulation model must be realistic, i.e. it must deliver results close to reality.
• The parameters influencing the function/safety must be complete (e.g. no important parameter/effect must be missing), moreover, the min-max limits must be correct.
• The metrics must deliver a realistic safety assessment.
• In case the prior probability distributions are used in the argumentation, the distributions must be derived from data having sufficient statistical relevance.

Test coverage

Coverage measures attempt to define completeness criteria for a test. For the assessment of automated driving functions at system level no accepted completeness measures exist to date. Due to the fact that, due to its complexity, a complete characterization of the parameter space is impossible, is it reasonable to assume that in future, even if one or more coverage measures will be defined and recommended for automated driving vehicle safety assessment, they will be used as indicators of incompleteness, rather than as indicators of completeness.

Metrics used in the conducted parameter exploration experiments

Within PEGASUS, we considered a set of simple metrics to identify the criticality of scenario.  Such Metrics are all representing a sort of distance measure, e.g. the distance itself, time headway, time-to-collision or time-to-react. Two example metrics, which are experimented and are used in illustrations in the subsequent sections, are:

• time-to-collision (TTC) – no assessment of the severity level
• time-to-collision-collision-speed (TTC_VCOL) – uses the relative collision velocity as severity assessment in case a collision occurs. The values are defined to be equal to:
  
  • TTC, when TTC > 0
  • – (abs(relative collision velocity)), when TTC = 0.

The Figure 1 below compares the response surfaces of the two metrics in an experiment.

As the above figure illustrates, a metric that can distinguish among severity levels provides more information and structure for the search space and probably makes more sense when searching for worst-cases, for instance.
Definition and computational challenges related to the metrics:

• How to define metrics that give a realistic measure of the safety violations, including the severity of the incidents? Moreover, the computation of the metric models should not be computationally too expensive (for instance, a FEM crash simulation would probably not be acceptable considering the large number of scenario simulations needed).
• How to define smooth metrics which allow more efficient optimization algorithms?
• How to define metrics that do not expose several spurious local minima or large “plateau areas”?
• How to scale several metrics in order to get to comparable values, in case several metrics are to be used in a logical scenario?
•  One should also be aware that sampling effects and numerical errors in the simulators can make the identification of a zero-crossing (for instance for TTC) more difficult.

In case that more than one safety metric has to be used in the analysis of a logical scenario, the metrics’ output must ideally be comparable among the metrics. Since all metrics refer to the safety of a scenario, scaling the metric outputs, for instance between [-1, 1], could be possible solution (for instance, as a combination of a measure of the distance to an incident and a measure of the incident’s severity – consider that, in a Failure Mode and Effects Analysis (FMECA) the severity of an event is assessed using 10 severity levels).

Finding worst cases / optimization algorithms

There exist a large number of optimization algorithms that can be used for searching safety violations using the metrics, respectively worst-cases of the scenario severity. The output of an optimization algorithm usually cannot be used to characterize the defined parameter space – unless no safety violation is found and an argument about the coverage of the searched space can be built. The following properties of the response function, i.e. here the safety metric(s), can pose challenges to different classes of optimization algorithms (Multiple restarts with random starting points can often compensate in a certain degree the mentioned weaknesses of the optimization algorithms.). Therefore, these properties should be inspected and their potential effect should be assessed in the evaluation results:

• Plateau subspaces: these are subspaces where the metrics have a constant value and do not deliver any orientation towards the safety violations or the worst cases
• Multiple (sub-optimal) local minima: for instance, if the TTC is used as a metric, it is possible that, depending on its implementation, each time the ego vehicle passes another car/traffic object, a local minimum is reached – even when the cars are driving in parallel lanes and there is no high risk of collision. The search around local minima can consume a lot of resources.
• Abrupt changes of the safety metrics or singularities: such “islands with high severity” might be hard to detect.

Several optimization algorithms have been implemented and analyzed. Two optimization algorithms are:

• Local Search (LS) and
• KD-OPT gradient optimization (KD-OPT).

Local Search

This is a variant of the “simulated annealing” classical optimization algorithm. Starting form a given seed, the algorithm searches neighboring points and follows the current best/worst case found so far. Mixed discrete and continuous parameters as well as (simple) constraints among the parameter values can be supported.

The algorithm is specified by the starting point (starting seed), the relative distance within which neighboring pints are generated and the total number of points (concrete scenarios) that are to be generated. The generated points are attracted by local or global minimum points. The choice of the configuration parameters can influence the outcome of the algorithm (which local/global minimum point is found). The higher the dimensionality of the problem (number of parameters) and the higher the distance between the starting seed and a local/global minimum is, the more simulation points are needed for the convergence to a solution point.

KD-Opt gradient optimization on sub-spaces

This algorithm searches for the worse cases along individual axes iteratively until a convergence point is found (In general, the worst cases are searched iteratively until a convergence point is found, on axes, planes, and sub-spaces up to a maximal dimensionality K.). This often leads to a more direct search for a local or global minimum, even for larger numbers of parameters. Mixed discrete and continuous parameters as well as (simple) constraints among the parameter values can be supported.
Even in cases when several local minimum points exist the algorithm can still “jump over” the local minimum points and find a global minimum - although, depending on the shape of the response function, finding a global minimum cannot be guaranteed. Plateau regions of the response function can be traversed often without dramatic problems. When the worst case is part of a plateau region (for instance a big subspace where TTC = 0) then this algorithm delivers one (arbitrary) point from the region as result. The choice of the starting point (seed) can influence the found worst case.

Characterization algorithms

Goal of the characterization algorithms is to deliver information about the response of the severity metric(s) in the whole parameter space associated with a logical scenario. In general, this is more valuable than just finding one worst case point, as this is closer to a safety assessment in the complete parameter space. At the same time, solving characterization problems is computationally more expensive than solving optimization problems. Therefore, when the dimensionality increases, compromises with respect to the coverage of the parameter space have to be made. For this, the following characterization algorithms can be used:

• Fixed sampling with full or with partial cross products (FS)
• Adaptive sampling with full or partial cross products (AS)
• Random generation (RND)
• Multi-stage mix of strategies

Fixed sampling with full cross products

This is a classical cross product among selected parameter values. Mixed discrete and continuous parameters as well as (simple) constraints among the parameter values can be supported.

The complexity of the algorithm is exponential in the number of parameters. More precisely, assuming N parameters with S sampling points per parameter the total number of parameter combinations in the cross product is SN. The exponential increase of the number of value combinations limits the usability of the algorithm to only small N (and S). The table above depicts the time required to explore by full cross products scenarios that require 10s of simulation for various values of N and S.

The algorithm is easy to implement and the results are easy to comprehend, e.g. can be depicted with a variety of visualization diagrams, including projections on intuitive 3D plots, as depicted in the Figure 4 below.

While the algorithm provides in intuitive characterization of the space for small N (and a sufficiently large S), we should mention that the worst case (e.g. as found by optimization algorithms) is usually not included in the sampled grid. In order to compensate this, one can of course, use the algorithm in combination with optimization algorithms executed before or after the fixed sampling.

Fixed sampling with partial cross products (size K)

This is a variant of the cross-product algorithm where the size of the cross products is limited to at most K parameters. Mixed discrete and continuous parameters as well as (simple) constraints among the parameter values can be supported.

For N parameters there exist C(N,K) combinations of K parameters, for each of which a size K cross product can be built. The number of generated value combinations is only exponential in K, not in N, e.g. for S sampling points per parameter this is C(N,K)*S^K.

For small K one can increase N. But is it “safe” to reduce the cross-products size from N to K? Apart of some cases, when subsets of parameters do not interact with each other, the reduction of the cross-product size is not guaranteed to be “safe”. The figure below shows the reduction of the sampled space when K is reduced from 3 to 2 dimensions: most of the volume is not sampled. If the difference between N and K is larger the tested subspace literally vanishes inside the parameter space. You can compare the figure below with the diagrams generated for the full cross product in the previous section - note that here N-K=1.
On the other hand, we should mention the following positive properties of the FS-KD algorithms:

• can be used even for larger N values
• coverage measures for K parameter combinations can be demonstrated
• if some parameters have a higher influence on the severity metrics then they can be placed in a cross-product set, while the rest of parameters can be analyzed serially around selected points from the more significant subspace (e.g. around worst points or border points resulted from the analysis of more significant subspace)
• intuitive 3D surface plots of the response function can be easily built. This helps to easier understand the nature of the 2D/3D parameter interactions in order to develop a parameter exploration strategy in an iterative manner.

The influence of the seed point around which the 2D/3D/KD projections are built is worth discussing. The figure below displays the results of the FS-KD (with K=2) when the seed is (a) in the middle of each parameter domain, versus (b) in a point with a bad severity value resulted from a previous search with an optimization algorithm. As can be seen, the projections of the worst-case severity are closer to the ones obtained with full cross products when the projections are around a worst-case point. Therefore, if the goal is to focus on the characterization on severity violations, it appears reasonable to first search for a worst case and then to apply the FS-KD search using the worst case as a seed.

Adaptive sampling with full/partial cross products

Adaptive sampling has the goal to identify better the local minima (local worst cases) and the borders between accepted and not accepted severity metric values. The algorithm starts with a fixed sampling (full or partial cross products) and then increases the sample rate near one or both of the (a) severity border crosses or (b) local minima / worst cases. Mixed discrete and continuous parameters as well as (simple) constraints among the parameter values can be supported.

The figure below depicts a response surface with 4 local min/max points (left surface plot), the severity response with a relatively high fixed sampling rate (13 points / axis, middle heatmap) and the result of adaptive sampling with a rather small fixed sampling rate (5 points / axis), super-sampled around severity borders and worst cases.

The complexity of the algorithm is higher than the one of the fixed sampling FS/FS-KD, e.g. above C(N,K)*S^K. Because the borders of the regions with severity violations can be very long if explored with a high resolution in a multi-dimensional space, the algorithm is only useful when applied to rather small 2D or 3D parameter subspaces, for instance, most useful when applied around a previously found interesting point, such as a worst case found using optimization algorithms.

Random generation

Random generation is a simple basic sampling method that can often provide a good complement to other, more goal oriented, methods. Although it is simple, it often can compensate some weaknesses that more sophisticated heuristic algorithms can exhibit. Mixed discrete and continuous parameters as well as (simple) constraints among the parameter values can be supported.

Random generation provides a reasonable degree of coverage (kind of, uniformly distributed sampling points), if the number of generated value combinations is not too small. For instance, for the cases when FS is not feasible and FS-KD does not provide enough coverage (due to a high difference between N and K), it is reasonable to switch to RND, or at least to add also a number of random generated scenarios to the sampled set.

The figures below depict the outcome of random generation on a parameter space with 5 dimensions. You might want to compare the results with the ones depicted in the previous sections for FS and FS-KD.

Multi-stage mix of strategies

Without prior knowledge about the response surfaces of the severity metrics for a specific logical scenario, it is not easy to specify an algorithm adequate for characterizing the parameter space. The best general-purpose solution that we have identified after experimenting with a few scenarios is to be able to specify a mix of all strategies: optimization with several random restarts, random generation and fixed sampling with partial cross products built around interesting points, such as, around previously found worst cases.

The above figure depicts the mix of (1) random generation, (2) optimization with local search, followed by (3) fixed sampling with 3D partial cross products. Last stage was added also in order to be able to plot the severity response surfaces around the worst case found after the first two stages. You might note that here we used TTC_VCOL as severity metric, a metric that uses not only the estimation of the time left till a collision occurs, but also the relative collision speed in case a collision occurred.

Characterization with quantification of probability of safety violations

Assessing the probability of safety violations for a logical scenario space with a given confidence measure is a challenging goal of the stochastic parameter exploration. In order to approach such a goal, several preconditions about the input data must be ensured:

• All relevant parameters that influence the experiment are known, including their prior probabilities, correlations, constraints and other dependencies.
• The simulation model (ego car, street, traffic, environment) delivers values that are close to reality.
• The severity metrics deliver realistic evaluations of the severity of a scenario.

The assessment of the probability remains challenging for more than just a few parameters, due to the high number of samples required for a probability estimation with a reasonable confidence, even when all of the above assumptions are met.

Theoretically a full cross product parameter exploration, with a sufficiently fine sampling resolution, followed by an integration of the probabilities around the severity violations can lead to a probability estimate. The method is, however, impractical for more than 3-4 parameters.

In principle, Monte-Carlo simulations (or derivates that optimize the number of samples, such as Importance Sampling) can be used to estimate the probability of severity violations. The large number of simulations required for obtaining a probability estimate with a reasonable confidence is very large, so that these methods can only be applied for scenarios with a very small number of parameters.

A particular fortunate case, when an estimate of the probability can be made, or at least, an argument for the safety inside a parameter space can be given, is when, after using a combination of optimization and characterization methods (with a arguably sufficient coverage), no concrete scenario violating the safety conditions can be found. This is not a very likely case for the complete parameter space of a logical scenario, but the method can be used to argument that, inside the most probable subspace of the parameter space, no safety violations occur.

Other approaches, that have the potential to lead to reasonable estimates with a (significantly) smaller number of simulations, for instance using surrogate models, machine-learning and DoE methods, should be investigated in future.

Recommendations for a simulation-supported safety argumentation of logical scenarios

Generally accepted and sufficient coverage goals for the safety assessment of logical scenarios do not exist to date. Increasing the coverage of the analyzed parameter space towards the feasibility limits is an option that should be taken into consideration.

Even if a completeness guarantee cannot be given, simulation results can contribute to a safety argumentation. Two pragmatic analysis approaches that appear reasonable to us to date are:

1. Absence of safety violations in the most probable parameter subspace:

• Absence of safety violations in the most probable subspace is an argumentation step that appears to be within the reach of current technology. This goal could be approached using a combination of optimization and characterization algorithms.
• In the absence of a more complete and reliable assessment of the cumulated probability of the safety violations, at least the next goal should be attempted as well.

2. Parameter space characterization using a combination of algorithms:

• Random generation – for a “fair” coverage of the parameter space.
• Optimization algorithms with multiple random restarts – for a more goal-directed identification of safety violations and worst cases in differing regions of the parameter space.
• Fixed sampling with K-size cross products – for K-dimension coverage of value combinations around a previously chosen seed, as well as for more intuitive visualizations of the response function with 2D and 3D projections, as these might be essential for the assessment conducted in the next step (they could help identify areas with unexpected parameter interactions, for instance).
• Manual assessment of the results by experts. An intuitive and interactive visualization and analysis of the results on sub-spaces is likely to be necessary for conducting this activity.

13. Test Cases

Test Cases in the PEGASUS context are concrete scenarios with PASS-/FAIL-criteria, which can be executed either in Simulation or on proving grounds. The test automation generates one new concrete scenario in each (testing) step. This concrete scenario is available to the Simulation or Proving Ground module for further processing purposes. The data format is OpenDrive: „.xodr“ and OpenScenarion: „osc“.

14. Test HAD-F: Simulation, Proving Ground, Real World Drive

Simulation

The simulation is a test platform used for the assessment of the functional implementation of the HAD-function. It plays a central role within the PEGASUS method, because it has the capability to cover a large number of test scenarios and enables to vary freely the relevant scenario parameters. In contrast to the proving ground or real world drives, simulation models are used to interact with the implemented HAD-function. In each simulation run, the HAD-function is assessed for a defined concrete scenario and concrete road network by means of given pass / fail criteria (metrics).

One central test platform used to assess the functional implementation of a HAD-Function in a large number of scenarios and variety of scenario parameters is the simulation. The essential idea behind this test platform is that input signals of the HAD-Function are provided by means of simulation models and output signals of the HAD-Function are then used to dynamically update the simulation.

In contrast to simulation, in other test platforms such as proving ground and real world drive a real vehicle is used to test the HAD-function. There, the function typically runs on a real ECU integrated in the vehicle. All input signals of the HAD-function are provided by the vehicle itself and not by simulation models. Typical input signals are, e.g. vehicle dynamic data (velocity, steering angle, etc.) or environmental data (e.g. detected objects, lane information, etc.). The latter ones are provided by means of environmental sensors based, e.g. on radar, camera, or LIDAR technology, but may also be provided by some downstream data processing like a sensor data fusion. Output signals of the HAD-function, e.g. steering commands, may then be used to control the real vehicle. Therefore, testing a HAD-function using a real vehicle has the advantage to be closer to reality than simulation test instances. The drawback of real driving tests is the impossibility to cover a large test space such as all possible scenarios on highway.

The simulation is able to cover more efficiently the large scenario test space and the necessary variation of scenario parameters. Simulation models are used to provide input signals for the HAD-Function. Furthermore, the used simulation models may be dynamically updated by signals provided by the HAD-Function. In general, the closeness to reality of the simulation is determined by the accuracy of the used models for each situation. That means, the required closeness to reality should be considered when choosing the simulation models. However, the verification as well as the validation of the simulation, e.g. the demonstration that the simulation meets the required accuracy, is an important and challenging task. Without a certain validation the simulation results offer only a limited knowledge.

In the following we take a look at the interaction between the simulation models and the HAD-function more closely. Figure 1 gives a schematic overview of a typical setup of a simulation environment considered in the project.

A common approach is to use a simulation tool in which all models, such as the vehicle model, the environment simulation, etc. run.

To configure the simulation it is necessary to define the road network and the scenario. Among other things, the desired behavior of the traffic surroundings as well as the subject vehicle (velocity profile, lane association, etc.) are defined. An important decision within the project was to define the road network and the scenario by the vendor independent and open file formats OpenDRIVE and OpenSCENARIO, respectively. Its prototypical support by the simulation tools of the project partners IPG, Vires, and dSpace was implemented or improved within the scope of the project with the goal of establishing an internationally accepted open source standard.

Based on the simulated environment, which in the first place is the simulated ground truth, the simulated sensor signals for the environment sensors can be derived. This is done by means of sensor models for integrated sensors like, e.g. radar, lidar, and camera. The main intention of sensor models is to simulate the behavior of real sensors, like e.g. the field of view or the error characteristics. Topics related to sensor models, including used standards like Functional-Mockup-Interface (FMI) or Open-Simulation-Interface (OSI), are explained later in detail.

Based on all input signals the HAD-function then computes the internal or output signals, e.g. signals for vehicle control, used to dynamically update the simulation including the vehicle model. This so-called closed-loop simulation may be performed on different platforms, such as MiL, SiL, or HiL platforms.

Figure 2 shows an overview of how the closed-loop simulation described above is integrated in the overall simulation environment.

Inputs to the simulation environment are a concrete road network and a concrete scenario. Concrete means that its parameterization is already set beforehand by the stochastic variation / the test automation tool to a set of concrete values (see P12 for details about the stochastic variation).

Within the PEGASUS method, the concrete scenario defined in the OpenSCENARIO file format may contain not yet standard conform extensions of the format, e.g. a not yet standard conform representation of the trajectory of a cutting in vehicle. This PEGASUS specific OpenSCENARIO file is then transferred into an actual standard conform OpenSCENARIO by means of the so-called Transpiler and an appropriate translation script. The non-standard content of the PEGASUS specific OpenSCENARIO is actually addressed by the Association for Standardization of Automation and Measuring Systems (ASAM) within its standardization process of the next generation OpenSCENARIO. This allows to evaluate format extensions without modifications of the used simulation tool. It may also be required that the standard conform OpenSCENARIO file contains specific, practically non parameterizable, elements. That are elements for which it is complicated or practically infeasible to define parameters, e.g. a trajectory definition consisting of a time series containing pairs of time and position. The consequence is that these elements may not be variable in the stochastic variation. Nevertheless, it may be required that the concrete OpenSCENARIO contains these elements, e.g. to allow testing the concrete scenario in another test platform like proving ground.

As an example, let us consider the trajectory of a cutting in vehicle. For the stochastic variation the trajectory must be practically parameterizable via, ideally human interpretable, parameters like the duration of the cut-in maneuver or parameters of a mathematical representation of the cut-in trajectory. In contrast, for testing the same concrete scenario on the proving ground it may be best if the cut-in trajectory is represented in a standard conform OpenSCENARIO via a time series containing pairs of time and position of the cutting in vehicle. The Transpiler can then be used to transfer the practically parameterizable OpenSCENARIO into an OpenSCENARIO where the same cut-in trajectory is represented via pairs of time and position. The latter one can then be used for both, the closed-loop simulation and the proving ground.
Beside the scenario description, the parameters of the road network, such as curve radius or lane width, may also be varied by the stochastic variation. This results in a concrete road network, e.g. the road network with a fixed parameter set, constituting an input to the simulation environment. However, a variation of a road network defined in the file format OpenDRIVE is rather challenging. Therefore, the file format SimplifiedRoad is used instead. The road network defined in a concrete Simplified Road is then converted in an OpenDrive using the OpenDrive-Generator.

The generated concrete standard conform OpenSCENARIO and the generated concrete OpenDRIVE are then used to perform the simulation with the HAD-function. While simulating, data of certain signals, such as velocity of the subject vehicle and the challenger, are recorded using the PEGASUS format as described in data container D9. Based on the recorded data the test is then evaluated using defined pass / fail criteria (metrics). This may include the evaluation of certain metrics. For the stochastic variation, typically a certain (criticality) metric, e.g. distance, time-to-collision (TTC) or time-to-react (TTR), will be evaluated to obtain a measure for the criticality. The development and definition of evaluation criteria and metrics are not covered within this chapter (see P8 and D6). They are input to this processing step. The result of the evaluated criteria then forms an output of this processing step which is then used for further assessment.

Figure 3 schematically illustrates how the simulation environment described above is integrated in the overall software architecture including relevant project partners. Note, that some components of the architecture are not considered within this chapter, but in other processing steps. Furthermore, relevant project partners for the simulation environment are shown.

Transpiler

OpenSCENARIO and OpenDRIVE are two file format specifications that help to accurately define environments and scenarios relevant for the safety validation of automated driving. However, both data formats in its current version have several limitations, which make it cumbersome to model complex scenarios. Also, changes and extensions of the dataset format have to go through a lengthy standardization process and have to be implemented in the (simulation) tools, which both inhibit a quick development of the standards. Therefore, an OpenSCENARIO Transpiler is introduced to tackle both problems. The main idea behind the Transpiler is to allow the addition custom elements to the OpenSCENARIO formats, which are not part of the standards, but can be translated to standardized OpenSCENARIO format elements. The translation is done by translation scripts, which are executed by the Transpiler. By combining the translation scripts with a documentation, new ideas and extensions can easily be shared with project partners and thus can easily be evaluated before integrating them into the standard. It is executed after creating a concrete scenario from a logical scenario by specifying the parameters and before handing it to e.g. the simulation environment or any other tool, which needs standard conform OpenDRIVE and OpenSCENARIO files.

Both, the OpenDRIVE and the OpenSCENARIO file format standards solve multiple problems regarding the definition static environments and scenarios during the development and safety validation of automated driving. The OpenSCENARIO format, for example, is a technical description of scenarios. It is based on the XML description language and is developed by a consortium of more than 25 partners parallel to and in coordination with the PEGASUS project. In this description format, the elements of a scenario, such as the start positions or the maneuvers of road users, are clearly defined by predefined parameters. On the other hand, this format allows the elements to be linked to a story by conditions, triggers and sequences.
The development of this standard has many advantages and solves essential problems at the same time. The definition of the format makes it easier for tool manufacturers to adapt their software to the requirements of the safety validation of automated vehicles. At the same time, interoperability between tools on the same or different levels as the simulation and the test track is ensured. Only a standardized file format makes it possible for a scenario to be used in several tools once it has been specified.

However, OpenSCENARIO is still under development and so it became clear in the PEGASUS project that changes and enhancements are necessary to create scenarios for the safety validation of automated vehicles. Since it makes sense to share and discuss these enhancements with the project partners before they become part of the standard, the Transpiler was created.  With the help of this tool, OpenSCENARIO can be extensively extended without tool manufacturers having to adapt their software to these changes. The trick is to provide an appropriate script for each extension, which translates the extension into standards-compliant OpenSCENARIO. An example of this is the introduction of new maneuvers. To display lane changes, the lateral movement should be described as a fifth degree polynomial, which OpenSCENARIO in its current version does not support as a geometric form. Any trajectories can be described as a series of positions in an OpenSCENARIO file. However, this representation cannot be parameterized, which is a necessary feature for the stochastic variation in the testing process. Therefore, the standard description format was extended by a parameterizable element for lane changes described by a fifth degree polynomial. A suitable script was implemented, which translates the new element into a series of positions. This allows the new maneuver to be used for validation before the element is included in the standard and implemented in the corresponding tools.

The Transpiler can also be used to easily add complex but at the same time parameterizable elements to a scenario, which cause changes in several places of the file. One example are action restrictions, which e.g. describe vehicles blocking a lane (see Figure 4). An action restriction can consist of one or more vehicles, which have to be added to the scenario and equipped with a start position and a behavior. In the current version of OpenSCENARIO, these restrictions would not be parameterizable, but at most cumbersome to implement manually. On the other hand, a new custom element with a number of parameters could be added, which is converted fully automatically into the corresponding standardized elements by the Transpiler. The Transpiler is an open source and free software provided by the PEGASUS project.

OpenDrive-Generator

OpenDRIVE is an open, vendor-independent mature XML-based file format, which contains all features to model real road networks and is supported by commonly used simulation software. Therefore, OpenDRIVE is a natural choice to model virtual tracks for test cases. In combination with OpenSCENARIO for describing dynamic behavior of objects and traffic participants (an open and XML-based file format as well), these two formats build a solid foundation to model environments for virtual test cases.

Direct modelling of different test track variants in OpenDRIVE is rather challenging. A simple change in a track to create variants usually leads to several other changes in the same OpenDRIVE file caused by the dependencies of the OpenDRIVE data model. Therefore, a domain specific language (DSL) is used to efficiently model various variants of highways to create adequately realistic tracks to describe test cases. This DSL is called Simplified Road and is specially designed to simplify the prototyping of road tracks.

The OpenDRIVE-Generator processes Simplified Road files and its parametrization to generate OpenDRIVE files. These are finally used for the simulation and testing activities.

Interfaces

The Simplified Road format is specified by an XML schema. Figure 5 visualizes an extract of that schema file. Some node details are collapsed, to limit the shown information to provide just an example. Especially, the course’s children are the same for lane, elevation and lateral profile. Furthermore, children of center are the same for left and right.

The root element of each Simplified Road document is the “track”-element. It can contain a “metaInformation” element, which can have attributes to describe the document with information of author, version and textual description. The track element further contains an unbounded sequence of section elements, which represent parts / sections of the described street. Each section can have children of the type “course”, “road”, “elevation”, “lateralProfile”, “signals” or “objects” to specify details of the specific section. The course element specifies the curvature of the street segment by using one of the mathematical functions “constant”, “linear” function, “arc”, “clothoid” or “cubic” function.

In the element “road” it is defined how many lanes the road section has including their width and other attributes for the “lane” and “road marking”. These lanes and road markings are specified separately for the “left” side, the “center” and the “right” side of the road. Similar to the “course”, the “elevation” and the “lateral profile” are defined with mathematical functions for road sections. Further details of these elements are hidden in Figure 5.

If the element “signals” is present, it contains street signs to be located along the road. The element “objects” contains more generic “vegetation” like trees and bushes or “infrastructure” along the road. Infrastructure can contain elements of the type guide post, railing, noise barrier and guard rail.

With these elements, it is currently possible to define one long street without junctions. With the focus on test scenarios, this limitation is perfectly admissible for the PEGASUS project scope. In the future, it should be possible to define highway entries and exits based on guidelines and a possibility of modification, too. From the perspective of the test scenario, it is just needed to provide the road currently under test.

There are several simplifications compared to OpenDRIVE, as less XML elements are needed to specify equal content. Furthermore, attributes have meaningful default values as often as possible. Besides of guide posts there are currently further supported infrastructure elements, such as noise barrier, railing and guard rail. In some tests, simplified road needed only between 7% and 21% of the lines required in OpenDRIVE to describe a specific road [Noyer, Ulf; Richter, Andreas and Scholz, Michael (2018) Generation of Highway Sections for Automated Test Case Creation. In: Proceedings of the DSC 2018 Europe VR, pages 171-174. Driving Simulation Conference Europe 2018 VR, 05.-07 Sep. 2018, Antibes, France. ISBN 978-2-85782-734-4 ISSN 0769-0266].
Input: Parameter file

Parameter files define variables, which are substituted in simplified road definitions. Based on these variables different variants of road tracks can be managed easily. Using a parameter file with parameters is optional.

Output: OpenDRIVE
The output file generated by the OpenDRIVE generator is the transformation of the simplified road input file with applied parameters of the parameter file. It contains the defined road track in the OpenDRIVE format. As OpenDRIVE is a commonly used industry standard, it can be further processed by all major simulation tools and environments.

Sensor models

Scope

Goals & Objectives
Here, the activities were focused on developing suitable Radar, Lidar- and camera sensor models, for the integration and usage within existing commercial (e.g. IPG CarMaker, Vires VTD, dSPACE ASM) and proprietary OEM simulation tool chains for driving function assessment. To minimize the development efforts for the sensor models and to maximize the reusability of development and validation efforts for the different sensor models, a standardized interface and packaging method is defined as another main goal.

Work Plan
With respect to sensor modeling, the main activities basically followed these steps:

• Requirements Elicitation
• General Simulation Architecture Definition
• Interface Definition
• Model Classification and Selection
• Model Implementation
• Model Validation
• Model Tool Integration (commercial and OEM proprietary environments)
  
  

Requirements Elicitation
Based on this available expert knowledge, the primary sensor phenomena of radar, lidar and camera were identified and prioritized with respect to the expected driving function to be supplied with simulated sensor data. Furthermore, interface requirements were derived from the abovementioned effect lists and at this early stage first aligned with commercial and proprietary simulation tool chains. That and the already stated overall demand of standardization led to certain change request on the tool vendor side, which yielded to the Open-Simulation-Interface (OSI) standard. OSI is an open source project hosted on GitHub and maintained by the PEGASUS members. The definitions are aligned with the upcoming ISO23150 standard describing a generic sensor output interface containing object data and detections. By requesting generally target platform independence, no or only limited system constraints to the target environments should limit model usage across different platforms.

Design / Architecture
One of the major design decisions to be made, was the selection of the appropriate type of sensor simulation model. Basically, two kinds of models were considered:

• Phenomenological models
• (Quasi-) physical models

The first one is based on statistical behavior on object data as in- and output in most cases and therefore performs on a higher level of abstraction. The latter approach targets a more physically related implementation, which usually requests a so-called raw-data (e.g. radar detections, lidar point clouds, raw images) interface and therefore has more specific demands in terms of the data provided by the simulation environment. Each kind has its strengths and weaknesses in terms of e.g. performance, accuracy, and implementation effort. For the project scope, the phenomenological type was selected as the most appropriate choice regarding the desired functions for automated highway pilots.

A phenomenological model receives ideal objects from environment simulation and modifies them taking into account sensor specific phenomena. The output of the sensor model contains detected objects (including false positives (ghosts)) which have sensor specific properties (e.g. deviations in dimension, position, velocity and individual existence probabilities).
Another important decision to cover standardization and exchangeability of sensor models was made on the one hand by defining the Open-Simulation-Interface (OSI), which utilizes Google-Protocol-Buffer (aka ‘Protobuf’) protocol and on the other hand by selecting the Functional-Mockup-Interface (FMI) standard, which led in conjunction to the Open-Simulation-Model-Packaging (OSMP) approach.

The selection of simulation environments, which are dedicated to act as the master simulation controller, was basically done during project setup by choosing the following project partners and their commercial tool environments:

• IPG (CarMaker)
• Vires (VTD - Virtual Test Drive)
• dSPACE  (ASM) [Associated Project Partner]

Besides these OEM project partner proprietary environments were considered for compatibility with the abovementioned simulation model design. In summary, the design architecture is shown in Figure 7.

Implementation
Sensor model implementation was realized according the following workshare:

• Bosch:
• Radar Model
• Camera Model
• TU Darmstadt:
• Lidar Model
• A.D.C.:
• Camera Model

Parameterization of those models were mainly derived from real sensors (e.g. range, field of view, dynamic and statistical behavior attributes). By providing FMI model packaging templates based on OSMP, implementation efforts could be lowered and reaching compliance according the selected standards was simplified.

Validation
In order to answer the question, how the sensor model validation in terms of ‘Is the correct model implemented?’ shall be designed, different strategies were discussed. In general, the validation process must consider a comparison with real sensor data – e.g. reference data. Of course, gathering reference data for the sake of comprehensive sensor model validation shall not lead to higher efforts compared to real world testing. On the other hand, comparison of real world and synthetic data will definitely suffer from non-perfect reference data taken during real measurements and used for generation of synthetic data for validation via replay-to-sim. Neither a data acquisition tool for real sensors nor any simulation environment will capture or provide all environmental attributes representing the reality. Since deviation between both is implicit, the question will be what deviation is tolerable and what kind of attributes are relevant and therefore impacting the model quality?
The overall goal of sensor model validation in the PEGASUS context is the ability to use a validated sensor model provided by a supplier in the validation of the automated driving functions at the OEM. Since there is no guaranteed commonality in the simulation environments between supplier and OEM, the validation needs to remain valid if the sensor model is used in a different simulation environment than the one it was validated against. To enable this, we used a two-stage validation approach:

• Validation test suite for ground truth source as sensor model input:
  Based on several representative sample scenarios, test cases are generated that compare the actual ground truth presented by the environment simulation to reference values to ensure that the ground truth lies within acceptable limits (based on the actual requirements of the sensor models in question).
• Validation test suite for sensor models:
  Based on representative KPIs and the representative sample scenarios, test cases are generated that compare the sensor output to reference bounds, ensuring that the sensor model gives results, which are characteristic and lie within the boundaries of the assured KPI ranges.

Based on the two-step approach, the sensor model can be considered valid, if it passes the sensor model validation test suite, and if it is connected to an environment simulation that passes the ground truth validation test suite.
It is important to state, that the validity of the sensor model is confined to the validation scenarios. For all other scenarios no prediction on the sensor model’s accuracy is permissible.

A concept and first implementation of the validation test suite for the sensor models, which covers the abovementioned, was developed and applied to sensor models (s. Figure 8).

Initiated and Ongoing Tasks
Building up the PEGASUS tool chain was supported by having some pre-existing artefacts on the one hand like the OpenDRIVE and OpenSCENARIO standards and on the other hand the experience from tool and sensor supplier side. At the end, the abovementioned OpenX standards plus the Open-Simulation-Interface were published by transferring ownership to ASAM e.V. There, further development by its members will take place and broaden usage on an international level. The ASAM is investigating a potential liaison with the ISO/TC 22/SC 33/WG 9 which desires to reference and refine the OpenX Standards bringing them to an official international level.

Abbreviations/References

• ADC - Automotive Distance Control Systems GmbH: subsidiary of Continental AG;
• FZD - Fachgebiet Fahrzeugtechnik TU Darmstadt www.fzd.tu-darmstadt.de
• FMI - Functional Mockup Interface; fmi-standard.org
• KPI – Key Performance Indicator;
• OSI - Open Simulation Interface; github.com/OpenSimulationInterface
• OSMP - Open Simulation Model Packaging; github.com/OpenSimulationInterface/osi-sensor-model-packaging
• PROTOBUF - Google Protocol Buffer; developers.google.com/protocol-buffers

Proving Ground Tests

Scenario-based testing on proving grounds is one essential component of assessing highly automated driving functions. It serves to investigate the real behavior of these functions under defined and thus manageable real environment conditions. It generates supporting points used to verify simulation results. In order to carry out reproducible tests it is necessary to use state of the art tools. The demands on the measurement technology and on the communication between the test participants are very high. In the PEGASUS project a complete tool chain has been developed to meet the requirements for proving grounds.

The tool chain essentially consists of traffic simulation vehicles that automatically (e.g. with the help of driving robots) execute concrete scenarios in order to test the performance of the vehicle under test. In order to observe and coordinate the test sequence, a control station is part of the tool chain. To minimize dangers and achieve a high level of controllability, safety drivers are included in this tool chain. The safety for all directly and indirectly involved test participants plays an important role in this description.

The following figures and paragraphs will outline the technical implementation of the PEGASUS test tool chain for proving grounds. The tool chain consists of a static part containing the control station and the stationary RTK correction data transmitter as well as a dynamic part consisting of the vehicle under test (VUT), traffic simulation vehicle (TSV) and/or movable soft crash target (mSCT) involved in the experiment (see Figure 9). Non-critical scenarios are usually performed with TSVs. If a scenario is classified as critical, traversable elements can be used; the movable soft crash targets (mSCTs). These can be used if the hazard and risk assessment allows it and if there are no people at the test site or on the test track. At any time during the experiment, all dynamic entities (VUT, TSVs, mSCTs) must know the exact positions of all other participants. For this reason a robust and state of the art radio communication (usually WiFi mesh) is required. In order to achieve the highest possible positioning accuracy, all location-variant test participants must be equipped with an Inertial Measurement Unit (IMU), which receives RTK-correction data from a differential Global Positioning System (D-GPS) base station via radio transmission.

The communication of the static and dynamic components can be described in three levels.

• Green: A time-critical communication level between the vehicles involved in the test that is dependent on short latency times.
• Blue: The monitoring level between the vehicles involved in the test and the control station.
• Red: The RTK correction data level, the correction data for the inertial measurement technology used in the dynamic elements of the tool chain.

Traffic Simulation Vehicle

The traffic simulation vehicles (TSV) (see Figure 10) play an important role in the scenario-based testing approach. Due to the exact control and positioning of the TSV in the scenario, the VUT can always be tested in the same, reproducible situation. Only if the test is precise and repeatable, the pass and fail criteria can be safely evaluated for a reliable assessment. In addition to the IMU and WiFi units, the TSVs are equipped with a programmable longitudinal and lateral controller. This is necessary to ensure that repeated test cases yield comparable results. The vehicle control system used to control the internal actuators for longitudinal and lateral guidance can be configured by means of an interface. As implemented in PEGASUS, the vehicle guidance can be implemented by controlling the internal actuators or conventionally by means of driving robots for steering, brake and accelerator pedal.

The safety concept requires that a safety driver is physically present in the vehicle to intervene in case of a fault. He constantly activates a dead man's switch located directly on the steering wheel during the test case execution. When the driver releases the switch, the automated vehicle guidance is switched off immediately.

A tablet or laptop is used to constantly inform the safety driver about the current test status. To obtain the current position of the TSV, a high-precision IMU (location technology) or a comparable technology must be installed in the vehicle. The IMU determines the position of the TSV at any time using GPS data and D-GPS correction data. An external WiFi (-Mesh) antenna is used for communication with the other test participants and the control station. As an alternative to the D-GPS correction, correction data may also be received using. In order to ensure a permanent power supply of the measuring technology and the real-time computer, an independent energy provision must be ensured, which is separated from the electrical system. Furthermore, all vehicles involved in the test and the control station must be equipped with powerful synchronization software and hardware (real-time computer).

Vehicle Under Test

The vehicle under test (VUT) (see Figure 11), occupies a special position as a dynamic, location-variant object that can also apply a non-deterministic driving strategy. Like the TSV, the VUT must be equipped with software and hardware for synchronization, since in certain scenarios it can reach indeterminable states that in turn triggers defined TSV maneuvers.
Since the VUT is only observed, the interface for controlling the internal actuators is omitted here. In order to carry out various scenarios, however, it is necessary that the exact position of the VUT is available to the TSVs. This is done analog to the TSVs by means of an IMU which in turn determines the position of the VUT at any time by means of D-GPS and dual GPS antennas. In order to transmit the information to the other test participants, a WiFi(-Mesh) is used. In order to synchronize all test vehicles with each other with low latency, the setup requires a powerful computer with corresponding software. The VUT must also be powered by a reliable power supply. A tablet/laptop is used by the safety driver to obtain information. The safety driver can always interrupt the test in critical situations and dissolve the test network by using the emergency stop.

Master Control Station

The master control station (MCS) is the central unit for monitoring the performance of proving ground tests (see Figure 12). It is used by the test manager to coordinate all test participants and to check the correct test execution. For this purpose, the MSC is connected to the dynamic parts of the tool chain via the supervision communication level.
The control station can be setup stationary in a building or, as implemented in the PEGASUS tool chain, mobile in a vehicle. The location of the control station on the test site must be selected by the test supervisor in such a way that a good overview of the scenario execution area is ensured and access as well as exit routes to this area can be supervised. The supervisor obtains all information from the vehicles in real time by the monitoring software. Furthermore, the software offers the functionality to interrupt the test by means of a software emergency stop. If the location of the control station does not allow the test manager to have the required overview of the test site, video surveillance can be used instead.
In addition to the monitoring activity, the control station is also able to evaluate data and carry out simulations making use of a high-performance computer.

Methodology Pre-Tests and Main-Test

After receiving the test case, a high-level description of the scenario to be driven is created with the help of expert knowledge. Using suitable software, this description is then converted into an automatically executable scenario. The safety drivers, who are part of the safety concept, carry out the first preliminary tests at reduced speed. After a few iterations and adjustments of automation parameters, the main test is performed several times. The generated measurement data are used to evaluate the VUT with regard to pass/fail criteria and the measured data are made available to the PEGASUS database. The first part of the tool chain, which is shown in Figure 13, shows the test preparation up to the main test with the generated data.

The expert knowledge includes know-how about test execution, test parameters, setting details for the measurement technology and, above all, the safety concept.

The following rules must be strictly adhered in order to guarantee safety during trial operation:

• In all tests where longitudinal and/or lateral guidance of the test vehicles is performed by driving robots or controllers, safety drivers are present in all vehicles as the fallback level.
• Drivers must be specially trained and have a recognized test driving license. On the day of the tests, all drivers must be informed separately about the risks of the respective scenarios and agree on reaction strategies for the case the scenario is terminated. The risk assessment carried out for each scenario serves as an aid.
• If the risk assessment indicates an increased risk for safety drivers and/or other involved and uninvolved persons, the experimenter must reject the tests. The test supervisor's approval will also be refused or withdrawn if, during the implementation on the test site, further primary concerns (feedback from the safety drivers) or secondary concerns (unsuitable weather, coefficients of friction, co-users on the test site, suitability for the test, etc...) are raised.
•  Before each test day, the test leader has to check whether all safety drivers are in possession of a valid driving license of the required vehicle class and can present a corresponding proof (driving license). If a test driver cannot present a driving license due to a driving ban, but can prove possession of the driving license by means of other recognized documents, this can be accepted for the tests on the test site (in agreement with the test site operator).
• At all times during the test, audio communication between the safety drivers in the TSVs, the VUT and the test supervisor in the control station must be ensured.
• All scenarios are simulated in advance of the physical test case execution in order to allow a better assessment of potential collision risks. All maneuvers are also tested in advance at low speeds (< 30 km/h). The speed is increased, where the speed increments for each scenario are determined by the safety drivers in a joint dialogue at their discretion.
•  Even if a scenario has already been tested previously, at the beginning of a new test day a run is performed at low speed again. This is also at the discretion of the safety drivers.
• Before a new scenario is tested, the data entered in the scenario description must be checked. The corresponding user interfaces of the control station environment or of the vehicles have to provide the possibility to perform the required checks. In particular, attention has to be paid to physical limits or all vehicles (required vs. possible longitudinal and lateral accelerations etc.). If there exist no implausible values, the scenario can be performed at low speeds.
• Before starting the test, the hardware and software must be checked for function on each test day. Only if all components have been tested, secure data communication between the vehicles has been established, and the safety drivers have checked the communication between each other the tests can be started.
• All vehicles incl. control station must be equipped with a first aid kit according to DIN 13157.
• All drivers involved in the test, or at least 3 safety drivers and one control station operator, must attend a first aid course every two years (2 days or 16 training units).
• The checklists must be completed on each test day (Table 1, Table 2, Table 3, and Table 4).

• The safety drivers have further hardware and software functions at their disposal to increase safety, which are described in more detail in the next paragraph:
  
  • Virtual fences
  • Collision warning
  • Previous Simulation of the Scenario
  • Approaching target speeds
  • Risk assessment of the scenario
  • Evaluation of the test site incl. cooperative testing

Robot-controlled vehicles have both, hardware and software safety features. The primary responsibility for the safety of the vehicle and the surrounding vehicles/pedestrians lies with the safety driver. The built-in tools for additional safety are:

• Hardware (VUT and TSV):
  
  • Dead man's switch - releasing the activation switch positioned close to the steering wheel gives the safety drivers in the TSV full control and maximum safety. In the VUT, which is merely observed, an emergency stop is always within reach to abort the attempt and to disengage the entire system.
  • Emergency stop button, also accessible from the front passenger seat.
  • Spring-loaded safety brake, which is held by an electromechanical clutch, only for driverless testing (not used in the PEGASUS tool chain).
• Software (VUT and TSV):
  • Steering angle, speed and acceleration can be restricted for path tracking.
  • Test abort if the control deviation of the individual axes (throttle, brake, steering) exceeds a set value.
  • The accuracy of the IMU is monitored. If it falls below a threshold or if the connection is lost, the test is aborted.
  • Path sequence error limits can be set. If the error reaches the set value, the test is aborted.
  • Error limits for speed control can be set. If the error reaches the set value, the test is aborted.
  • Virtual guide rails can be used to change the error limits for web guiding and speed control within defined segments of the path.
  • bort parameters ensure that a predefined reaction occurs when a test is aborted (e.g. brakes, information to all vehicles).
  • Critical sections can be configured to allow defined control of steering, brake and accelerator pedal during a test abort.

• Communication and monitoring software (VUT, TSV and control station):
  • Tests are performed with fully synchronized vehicles. They adjust their current speed and path based on the error of the reference vehicle (VUT or TSV).
  • Tests can be aborted from any vehicle by pressing the emergency stop button.
  • Each vehicle knows the position of the other vehicles.
  • Relative parameters to other vehicles can be used to trigger a test abort.
  • The setup data for synchronized vehicles are checked to ensure that they all use the same coordinate system and the same clock (GPS time).
  • With each test protocol, a code can be defined to ensure that all vehicles are running the same scenario.
  • The positions of all vehicles on the test site are displayed to the service user.
  • The control station performs a collision check, which can stop vehicles if necessary.
  • The control station can stop all vehicles.
  • The control station sends a real-time watchdog to each vehicle. If sending is stopped or the signal is lost, then the vehicle stops.

Once the hazards of the concrete scenario have been assessed and the risk has been minimized, the actual main experiment is performed using the planned driving scenario with correct boundary conditions according to specification (OpenSCENARIO). In addition to applying the safety concept (see above) and performing the scenario training for safety drivers, some further technical aspects have to be taken into account by the test performers. The actual main test must be checked for validity immediately after the end of the test. At least three valid repetitions are required for the main test to be considered completed. The test is valid if the required tolerances from the OpenSCENARIO file are met. If they are not described precisely enough or cannot be implemented technically and physically (due to measurement uncertainties and error propagation), the following reference values can be used (see Table 5). These reference values are empirically determined in practical use of the test tool chain. Robust operation of the tool chain is possible if these values are adhered to.

In order to evaluate the validity, it is necessary to record the corresponding quantities. In addition, channels for later evaluation of the VUT and other variables must be recorded as informative channels (Table 6). It is recommended that all vehicle data of the participating vehicles are recorded on the same time axis (e.g. GPS time).

After the test has been carried out, the data are processed further. The data generated by the data recorder is converted to the PEGASUS file format and evaluated with regard to pass/fail criteria. Figure 14 provides an overview of the process.

After the necessary activities have been carried out, the results are manually transferred to the PEGASUS database.

Real World Test Drive

Field test is a test platform to test the HAF-function in a real traffic environment with different environment conditions, but without having concrete test cases.

Test of the HAD-function in real world traffic (long term testing)

• identification of specific individual situations within the framework of event-based simulation (Unknown behavior, New scenarios and/or new parameters, Pass-Criteria violated)
• Identification of faults/impairments of the system because of environmental influences that cannot currently be simulated directly via models, as no suitable physical models are yet available.
• Perform special “pass” assessments (e.g. risk in passing or following other vehicles)

The field test takes place on public streets using existing infrastructure (e.g. test field A9, geodetic surveyed routes within the Pegasus project). Where necessary, guidelines for route and time to create challenging situations (e.g. lots of traffic, tunnel or weather) have to be considered.

Input is: global guidance of conditions (e.g. o a route, weather conditions, time of day, week or month, special environment e.g. tunnels, special curvatures …), pass criteria, original vehicle as system under test.

The output is: evaluated real world test drive, measurement data as input for database (return flow to database). This means: measurement data in OEM-Format, measurement data in PEGASUS-format as input for the database, analyzing the data will be performed on database estimated results are for example: evaluated test drive data, new scenarios and/or new parameters for existing scenarios, input for Replay2Sim.