Mobile augmented reality in support of building damage and safety assessment

Rapid and accurate assessment of the state of buildings in the aftermath of a disaster event is critical for an effective and timely response. For rapid damage assessment of buildings, the utility of remote sensing (RS) technology has been widely researched, with focus on a range of platforms and sensors. However, RS-based approaches still have limitations to assess structural integrity and the specific damage status of individual buildings. Structural integrity refers to the ability of a building to hold the entire structure. Consequently, ground-based assessment conducted by structural engineers and first responders is still required. This paper demonstrates the concept of mobile augmented reality (mAR) to improve performance of building damage and safety assessment in situ. Mobile AR provides a means to superimpose various types of reference or pre-disaster information (virtual data) on actual post-disaster building data (real buildings). To adopt mobile AR, this study defines a conceptual framework based on the level of complexity (LOC). The framework consists of four LOCs, and for each of these, the data types, required processing steps, AR implementation and use for damage assessment are described. Based on this conceptualization we demonstrate prototypes of mAR for both indoor and outdoor purposes. Finally, we conduct a user evaluation of the prototypes to validate the mAR approach for building damage and safety assessment.


Introduction
Natural disasters bring about economic damage and victims, and despite signs of a decline in the number of events in recent years, between about 300 and 500 events are recorded worldwide annually, with total damage frequently exceeding USD 100 billion (Kerle and Alkema, 2012).After the occurrence of a disaster event, in particular in urban areas, rapid and accurate building damage and safety evaluation is critical.The former only provides information on the physical level of impairment of a structure, while the latter assesses to what extent buildings can be safely entered, which also takes the state of adjacent buildings into account.Rapid building assessment plays a major role for initiating effective emergency response actions (Brunner et al., 2010), and also constitutes the basis for post-disaster needs assessment (PDNA).Building safety inspection is related to building usage after disaster events; hence time delays before buildings can be declared safe again also translate into economic losses (Dai et al., 2011).Thus, a comprehensive and rapid assessment of building stock following disaster events is a prerequisite for effective rehabilitation and reconstruction.
Remote sensing (RS) technology has been widely used for rapid building assessment, with many sensor/platform combinations having been tested and with principal focus on seismic damage (e.g.Dell'Acqua and Gamba, 2012).Although RS technology has improved the speed and accuracy of damage mapping at a macro level, e.g. at a city block scale, challenges persist at more detailed, per-building levels.For instance, while RS-based approaches can detect the extreme damage categories (none or complete) quite accurately, detection of intermediate damage states continues to be marked by great uncertainty (Kerle, 2010;Fernandez Galarreta et al., 2015).
Given the problems in assessing damage and safety and structural integrity of individual buildings with RS data (Kamat and El-Tawil, 2007), ground-based appraisals remain necessary.This is usually conducted by reconnaissance teams comprising certified inspectors or structural en-gineers, and using assessment guidelines such as the ATC-20 field manual (ATC, 2005) or the European Macroseismic Scale (EMS-98).For example, in the aftermath of the 2010 Haiti earthquake, some 300 engineers mapped approximately 400 000 buildings using an adapted version of ATC-20, focusing on determining the residual vertical capacity of each structure, which even with airborne multi-perspective oblique RS images is challenging to determine (Gerke and Kerle, 2011;Kerle and Hoffman, 2013;Dell'Acqua and Gamba, 2012).ATC-20 classifies the building state into safe, unsafe and limited use, while EMS-98 uses damage scores ranging from 1 (no damage) to 5 (heavily damaged).However, ground surveys are inefficient in terms of cost and time of evaluation.In addition, the ground surveys rely on human skills and knowledge, so that human-induced errors can affect the quality of data during the mapping or subsequent transcription and analysis process (Kerle, 2010).
Ground-based surveys are mainly hindered by (i) limited site access, (ii) problems with orientation in the field with landmarks and street signs also having been affected and (iii) the lack of efficient tools to provide pre-disaster reference data to facilitate a pre-and post-disaster data comparison, as is done in RS-based change detection.Therefore, this study proposes an approach based on mobile augmented reality (mAR) that can efficiently deliver pre-disaster reference data of various types and levels of complexity, provide location and navigation information, but also support additional situational awareness, such as by warning of the presence of nearby hazardous facilities or substances.Augmented reality (AR) superimposes computer-generated graphics or contents on real-world imagery, for example as captured by a camera of a mobile device.The main purpose of AR is to combine real and virtual world information and provide realtime interaction to users (Azuma, 1997).While a conventional data overlay (e.g. as done in geographic information systems (GISs)) replaces reality with virtual data, AR supplements reality (Azuma, 1997).The roots of augmented reality go back to the 1990s (Valentini et al., 2010), and the first prototype of mAR was developed by Feiner et al. (1997) to provide tour information for buildings based on 3-D graphical information.
The main objective of this study is to demonstrate an mAR concept that can increase the efficiency and safety of groundbased building damage and safety assessment.We first identify the main obstacles of ground-based building assessment that are directly connected with the user requirements.Based on this analysis a conceptual framework is defined, which defines specific methods and processes to apply mAR for building damage and safety assessment.In order to validate the efficiency of mAR, a prototype is designed and implemented, showing how information ranging from simple labels to complex 3-D building models can be provided to a user.Based on an online survey that demonstrates the prototype, we analyse to what extent current limitations of ground-based damage and safety assessment can be assisted with mAR, qualitatively and quantitatively.

Overview of previous work
Building damage and safety assessment was the focus of a number of earlier studies.Much research has focused on remote-sensing-based damage inventorization (for a recent overview see Dell'Acqua and Gamba, 2012), while work focusing on assessing building safety is comparatively scarce.Altan et al. (2001) used ground-based photogrammetry to determine critical damage and deformation that may compromise structural integrity, though in an approach that required entering a potentially unstable structure.Schweier and Markus (2006) analysed 3-D CAD models derived from laser scanning data to identify geometric changes between preand post-disaster buildings, including volume or height reduction, as well as orientation changes, which were linked to stability.The main challenge of their method is to obtain preand post-disaster 3-D data at sufficient accuracy to support change quantification.Curtis and Mills (2012) developed a video-based field reconnaissance system that allows tornado damage to be mapped, though no pre-event reference information is provided.Adams et al. (2009) introduced VIEWS (Visualizing Impacts of Earthquake With Satellites).This is a video-based scene assessment system that not only uses satellite imagery, but supports it with video data and photos that are taken in the field after a disaster.The captured video data and photos are georeferenced, hence the user can combine and compare the reference satellite data with video footage and photos for a comprehensive damage assessment.
Although VIEWS provides an interface that can readily visualize all data on one screen, it requires post-processing of the video data and photos.Moreover, the actual damage assessment task is not meant to be done in the field.
Only few studies have investigated the utility of AR for post-disaster situations.Leebmann (2004) designed software and hardware of an AR system to overlay disaster-related information onto the real-world view through a digital video camera.Leebmann's research demonstrated that AR is accurate enough to register virtual information onto real objects in the context of disaster response.At that time focus was mostly on how to improve matching accuracy between real and virtual objects, and how to integrate hardware components such as video camera, GPS and laptop.Tsai et al. (2012) proposed mAR to support evacuation of nuclear power plants following events such as earthquakes.The system displays direction and distance to shelters on a live camera image taken with a smartphone, using Google Maps and the electronic compass function.Boddhu et al. (2013) introduced context-aware event detection with mAR for first responders.This system gathers and analyses data that are distributed by social networks such as Twitter and Facebook.Because SNs can deliver important information, such as the user's location, in real time, they can play a role as a sharing tool during or after a disaster.In gathering and analysing these spatiotemporal data, the system tried to support first responders in managing the disaster situation (Boddhu et al., 2013).To display analysed data for first responder in the field, the system used AR that superimposes virtual data regarding historic or live events occurring around a user's location.Wani et al. (2013) proposed a workflow for collaborative AR in a fire emergency situation.In the workflow a wearable computer was adopted for the collaboration between experts in an office and the operator in the field.For instance, a medical doctor in the office can, through the AR device, interactively indicate parts of a patient's body which are in need of an initial treatment.
While the above systems simply focus on the visualization of exiting information, Kamat and El-Tawil (2007) tried to generate new information from AR to support seismic building damage assessment.Their approach focused on comparing baseline information from pre-disaster 3-D building data with the actual post-event structure to identify differences, detecting deformation through the interstory drift ratio (IDR).For the IDR measurements, the pixel offset between the image of the real building and the 3-D model baseline was counted, using both sensor-based and vision (marker)-based AR.The former uses the sensors included in mobile devices, such as GPS, compass and gyro sensors, to obtain the user's location and orientation, while vision-based AR analyses images taken by the camera identify unique features of real objects, using computer-vision technology (Rabbi and Ullah, 2013).While Kamat and El-Tawil (2007) measured displacement between the pre-disaster 3-D model and a real building, Dong et al. (2013) determined IDR without a pre-disaster 3-D model, but instead also using vision-based AR.They used edge detection and corner detection methods (e.g. a line segment detector) to extract building baselines from reference images of an affected building.Subsequently, the generated baseline model was superimposed on the real (post-event) object to calculate the IDR.Although these approaches provide efficient methods for damage assessment, they have traditionally required an array of separate pieces of equipment and sensors (e.g.laptop, camera, GPS).In addition, their approaches only provide one indicator, IDR, which is too narrow to improve the overall assessment procedure of building damage and safety in the field.For instance, standard evaluation forms used in rapid damage assessment ask assessors to provide comprehensive information, such as building address, type and height, which existing systems cannot provide.Additionally, response to large disasters is marked nowadays by a large variety of stakeholders, lack of cooperation and coordination and limited situational awareness (Kerle, 2013).mAR has the potential to improve this situation by not only allowing orientation or providing a certain type of reference information.Instead, it can provide pre-event reference data on a number of scales of complexity (from simple labels of where buildings used to be or the function of a given building, to complex geometry fitted over building remains), in addition to data on environmental parameters that directly affect the safety of the responder, or information on other response teams in the area or on prior assessment work.Given the trend in miniaturization in mobile devices, some of the mAR features discussed in this work will likely eventually be available for AR platforms such as Google Glass, which has great potential to provide critical, customized information to disaster responders, with the additional benefit of hands-free operation.

Conceptualization of mAR for building damage and safety assessment
First we define the overall concept of mAR for building damage and safety assessment.It shows how mAR can support ground-based building assessment, and which methods and data sets are required by different users, who we also review and categorize.The stakeholders we envision for this system comprise structural or geotechnical engineers, architects, search and rescue teams, or professional groups who carry out building damage and safety assessment in the field (see for example Ghosh et al., 2011).Based on the defined concept, prototypes of indoor (marker-based) AR and outdoor (sensor-based) AR are developed.Subsequently the concepts and prototypes are evaluated through a user evaluation.

Overall concept of mAR for building damage and safety assessment
To apply mAR for building damage and safety assessment in the field, we conceptualize the relationship between AR and the different levels of assessment as shown in Table 1.AR superimposes various reference data sets onto real buildings, which include simple location information (e.g.building coordinates), attributes (e.g.building information), linked attributes (e.g.street-level imagery) and geometry (e.g.3-D building frame).
To conceptualize the utility of mAR in a post-disaster situation for different complexity levels, we use a term, level of complexity (LOC), which ranges from LOC1 (low complexity) to LOC4 (high complexity; Table 1).The complexity levels relate to increasing number of data dimensions, levels of accuracy, etc. LOC1 simply indicates the existence of a building.This is useful in case of a completely collapsed building, or for seismic or tornado damage, where vast debris fields challenge the identification of individual buildings that existed before.After recognizing the existence of a building, LOC2 provides information on the type of structure and basic attributes (e.g.building classification, materials, height and story).Subsequently, LOC3 provides pre-event reference imagery (e.g.photos of the building, or Google Street View imagery, GSV), while LOC4 includes pre-event 3-D information.This may include imagery of the building exterior, but also data on building frames and internal structures.Each LOC level is defined in detail in terms of procedure, flow, accuracy and uncertainty in the next section.

Simple location-based (LOC1) and attribute-based (LOC2) visualization
The purpose of LOC1 is to provide simple but essential information on the (prior) existence of a structure.Meanwhile, LOC2 information helps to reduce the building assessment duration, since the most time-consuming task is building type classification (Flesch, 2007).Both concepts, including the required data processing steps, are illustrated in Fig. 1.

Procedure and flow
Building information can be extracted from several sources, such as cadastral databases maintained by local governments, private building management companies and emergency response organizations.Following the 2010 Haiti earthquake, organizations such as the Remote Sensing Laboratories at the University of Zurich and Swisstopo provided extensive building data (Corbane et al., 2011).A common challenge is that those data sets come in different formats, so that data need to be preprocessed and converted in order to be processed through AR.In our LOC1 concept, the data type is mainly text format that is very irregular with respect to data providers.Therefore, it is important to refine raw data that can be interpreted by AR.In terms of data clutter on the screen, AR needs to adjust the display range (radius), i.e.AR only displays information that is within a given radius from the user location.Lastly, LOC1 and LOC2 are interactively connected, as shown in Fig. 1.Initially, only the building name is displayed on a target building to let the user rec-ognize the building existence (LOC1).If the user needs additional building attributes, LOC2 information is provided on a touch basis.

Accuracy
As mentioned earlier, text format data are usually irregular, necessitating data refinement, including georeferencing.Both processes may cause data changes or loss.However, text information is a form of point data, and as long as it is displayed within the building boundary, it is still interpretable.Hence, the georeferencing only needs to fall anywhere within the building footprint.

Uncertainty
Uncertainty refers to uncontrollable and unpredictable parameters that can affect a given procedure.Irregularity of data sources and formats can affect data availability and usability and thus all subsequent LOC processes.Information sharing itself can be restricted because of political and privacy issues.GPS signal availability may be compromised in densely built-up areas, undermining the ability to associate database information with a given structure in the field, also affecting all LOCs.

Linked attribute-based (LOC3) and geometry-based (LOC4) visualization
The concept of LOC3 is to provide multimedia data, such as photos, video clips and sounds, that are linked (related) to a specific building.The data sets needed for LOC3 can be obtained through internet services such as YouTube, Flickr or GSV.The concept of LOC4 is to provide building geometry that includes 3-D information of the structure's exterior and possibly the interior.The 3-D building exterior shows the building's original shape and frame structure, while 3-D building interior displays features within the building, such as stairs, elevators, rooms and so on, facilitating better damage interpretation, or evacuation planning.The detailed data sources and processing steps are shown in Fig. 2.

Procedure and flow
LOC3 mainly depends on external APIs to extract data from service providers.These APIs require query parameters such as coordinates and keywords, meaning the results will vary depending on a specific search string used.After locating and extracting a data set, those data need to be converted to a data format understood by the AR platform.For instance, Metaio API used in this study adopts MPEG4 and 3G2 format to display video clips while others use AVI format.Lastly, AR imports the converted data.Meanwhile, GSV employs its own registration process -each panorama is georeferenced; hence GSV data can be expected to be directly integrable.Therefore, GSV can be visualized in a separate window without an AR process.With LOC3, a user can call up pictures of buildings, as well as of the surrounding area.
LOC4 uses 3-D data sets generated by or extracted from various sources that can be grouped into primary and sec-ondary data sets.The former contain 3-D buildings, used directly without any data generation processes, for instance, 3-D buildings extracted from Google Earth or from a building management company.Conversely, secondary data sets require a process to generate 3-D building information, for example from 2-D building footprints, or lidar data.The secondary data sets are typically processed on a PC, and subsequently converted to the 3-D data format required by the AR platform.In order to visualize these 3-D buildings through AR, the following parameters should be defined: coordinates, scale, rotation of building, exchange data format and level of detail.

Accuracy
Accuracy of data sources for LOC3 depends on the query condition to extract meaningful data from APIs (e.g.keyword).Additionally, the locational accuracy of geotagged photos and video clips is not guaranteed.Although GSV does not rely on the registration process of AR, it also uses its own registration process, meaning that data accuracy can be affected by accuracy of the mAR device sensors.Multimedia files are basically point data that can be meaningfully associated with a building, as long as they are displayed within the building boundary.
The accuracy of the original 3-D data and the data conversion process can affect both data source and data process accuracy for LOC4.The accuracy of the primary data set is a direct function of the accuracy of the original data generation, while the preprocessing or conversion steps listed above determine the accuracy of the secondary data sets.For the AR process of building exterior and interior, as long as a user can identify which 3-D building belongs to which real building, the accuracy of data registration is acceptable (see for example the concept image for LOC1 in Fig. 1).Data meant to facilitate a more detailed damage analysis (e.g.inclination; Dai et al., 2011), such as building frames, need to be more accurately registered.

Uncertainty
In the concept of LOC3, the main uncertainty stems from the network connection and service coverage, since the data sets are extracted by external APIs through a live internet connection.Another uncertainty results from data size, since storage on a mobile device is limited.Although in a disaster situation network connection might be limited, several alternatives for internet access are already in practical use.For instance, in the case of the Great East Japan Earthquake, 2011, portable satellite internet Earth stations and satellite phones were installed to use satellite-based internet connection (Sakurai et al., 2014).
In the concept of LOC4, file size of the 3-D data can affect the performance of both data process and AR process.In addition, the number of 3-D buildings that can be displayed simultaneously can be limited depending on the performance of the mobile device.The higher the 3-D data precision and accuracy, the more performance and storage of the mobile device is required.

Prototype implementation
For the user evaluation of the LOCs developed, we designed and implemented an mAR prototype.It was developed based on the following APIs: Android API, AR API (http://www.metaio.com/products/sdk/)and Google Maps Android API (https://developers.google.com/maps/documentation/android/). Android API is a framework to develop applications, involving core modules of the operating system.In this research, Android API 4.3 (code name: Jellybean) was used.The AR API is a core part of mAR, and includes registration and visualization (superimposition) functions.In this research an AR API called Metaio was used, which supports common 3-D data formats (e.g.obj format) without any conversion, and also supports video clip display.In order to display base maps in separate windows, we adopted Google Maps Android API V2.The prototype comprised two systems, an indoor and an outdoor mAR. Figure 3 shows the user interface of each system.The indoor mAR uses markers such as image and fiducial marker to recognize the plane of AR visualization.With predefined markers, the required rotation, scale and translation of the 3-D object can be defined.The indoor AR is proposed for stakeholders who work inside, e.g. of an emergency headquarters or organization.The prototype of indoor mAR simply shows 3-D buildings and a video clip on a satellite image that covers the city of Enschede, the Netherlands.It also shows specific building information when the user touches a virtual building.
The outdoor mAR uses the GPS to obtain the user's current location (coordinates), and the gyro sensor to determine the orientation and movements of the mobile device.The outdoor mAR visualizes the data for LOC1 to LOC4 with geographic coordinates within the city centre of Enschede, the Netherlands.The outdoor mAR has the following functions: a radar and a map display, warning for hazard zones, display of current address and 3-D model controls (e.g.rotation, zoom-in, zoom-out by touch control).
The 3-D models of building exterior, interior and frame were created with SketchUp (www.sketchup.com)for demonstration purposes.To visualize a 3-D building exterior that has an accurate scale, some buildings were extracted from the Google Earth warehouse.For visualization of the multimedia data set, the prototype used video clips that were downloaded from YouTube.
A demonstration of the specific functions and operations of indoor and outdoor mAR can be accessed through the following links and QR codes (Fig. 4): indoor mAR (http: //youtu.be/fL1yaPnBoI4)and outdoor mAR (http://youtu.be/0tCzXaGMx4Y).

Result of user evaluation
Based on the prototypes, an online survey was conducted to evaluate how much mAR can improve building damage and safety assessment in a disaster situation.The online questionnaire form was distributed to several associations of structural engineer (e.g.National Council of Structural Engineers Associations in the United States) and online communities (such as Facebook, LinkedIn) of structural engineers and first responders.The questionnaire consisted of six parts: (1) demonstration of indoor and outdoor mAR for each concept (online video clips were used), (2) questions about experience of building damage (safety) assessment, (3) efficiency of mobile AR, (4) locational awareness, (5) usability and functionality and (6) essential information.To evaluate the efficiency of mobile AR, after playing video clips of AR demonstration, we asked them how much the mobile AR can reduce assessment time of building damage, and how much AR can improve assessment accuracy with five scales between 10 % and more than 50 %.To evaluate locational (and situational) accuracy, we showed them two illustrations.The first one displayed a map of the current location with its address.The other one uses the so-called radar function to show a user's current location and relative distance and direction of hazardous facilities (e.g. a chemical plant) and essential facilities (e.g.hospital, power station).Then we asked how much it can improve the locational awareness if they can identify the current street name or address in a disaster area using such a system.In addition, we referred to the improvement of situational awareness, if they know the location of a target facility in a disaster situation.For the usability evaluation of AR based on video clips that we showed in the first stage, we explained to them how the user can efficiently view the 3-D model or map using the touch screen of a mobile device.We then asked them how much this kind of usability could improve operational performance.
In total 34 responses were collected from users with experience in building damage assessment.Of those, 24 responses (70 %) came from structural engineers with more than 5 years experience in building damage and safety assessment.Most of the responders were structural engineers (91 %), with the remaining responses (9 %) coming from post-graduate students and building managers.Eight (24 %) responses indicated prior experience with mAR.Table 2 shows the results of the online survey.
To understand the level of importance for each type of reference data, the survey showed simple illustrations of each concept to the users, covering building information, streetlevel imagery, 3-D building exterior, 3-D building interior, 3-D building frame and automatic change detection, increasing in order of complexity.The users were asked to rate them from 1 (not necessary) to 5 (very important).Table 3 shows the result of the survey about what is considered essential information.
The results show that users consider the 3-D building frame (4.2), followed by building information (4.1), to be the most important reference data types.They also indicate that there is a strong need for street-level imagery (4.1).On the other hand, multimedia data (3.1) are considered to be less useful.However, the score of these data still shows they are necessary data for building damage assessment.

Discussion
A comprehensive and rapid assessment of building stock should be conducted quickly after disaster events.In this paper, we developed an mAR concept for post-disaster field reconnaissance, conceptualizing and testing how various reference data, at increasing levels of complexity, can be delivered to field staff to increase efficiency and safety of groundbased building damage and safety assessment.For this, we defined a conceptual framework in terms of LOCs.Subsequently, based on the framework, prototypes were developed and evaluated through the online survey taken by experienced users.
The results of the survey showed that mAR can improve the assessment accuracy (objectiveness) and time.Although the results indicate the potential of mAR to facilitate fieldlevel damage assessment, the following factors need to be considered carefully.First, there is no standard range for accuracy and speed of the building assessment.The procedure tends to be ad hoc, and strongly shaped by specific circumstances of the event (location of the disaster, accessibility, extent of the damage, presence of skilled responders and required infrastructure, etc.).This means that no benchmark in terms of time needed or accuracy achieved exists against which to measure any improvement afforded by the concept defined in this study objectively.Secondly, indoor AR did not get the positive feedback we expected (only 3.07 out of 5), likely because the work of responders typically takes place outside, where site assessment challenges are greatest.However, indoor AR might still be a useful solution for someone managing the overall disaster situation and response.
The survey about locational (situational) awareness showed interesting results that indicate that the user wants geographic information that identifies the location of both the user and the building.It implies that mAR gains in value when combined with a map system (GIS).The utility of map information is linked to the need for locational awareness that decreases especially in a severely damaged area.Furthermore, disaster responders are typically dispatched to unfamiliar places at short notice, resulting in limited locational and situational awareness.To deliver various maps to mobile devices, service-oriented architecture (SOA) and Open Geospatial Consortium (OGC) web services can be a good approach.Both SOA and OGC web services can distribute or share data between different systems working on different OS and hardware environments.
In addition, users rated the usability of mAR based on touch gestures (e.g.zoom-in/out or rotation of 3-D building with two fingers) highly that let users manipulate contents on the screen interactively.The results showed that the interaction can improve the operational performance in a disaster situation.In addition to touch handling, current mobile devices are increasingly adopting voice recognition functions, allowing yet further means to manipulate the data and functions.Although the performance of voice recognition in current mobile devices is still being improved, this is strongly driven by developments of next-generation consumer devices such as Google Glass.Thus, voice recognition is also expected to increase in usability in a disaster situation.
In terms of essential information, the results showed that the most important data type is building frame, followed by building attribute and street-level imagery.In particular, the strong need for large data sets such as the latter can pose a challenge in real disaster situations, where suitable mo-  With sequential images, mAR can extract 3-D geometry using structure from motion (SfM) methods that are widely used in several application fields (Quan and Wu, 2013;Verstockt et al., in press).Using SfM in combination with dense image matching, 3-D models of post-disaster buildings can be reconstructed.Since mobile AR uses the camera of the mobile device, a user can directly take photos of the target building in the disaster area.If a network connection is available, photos can be uploaded directly to a server to generate 3-D building models (for example, using freely available smartphone apps such as Autodesk's 123D Catch; www.123dapp.com/catch).Instead of using sequential images, tools such as "Spike" (http://www.ikegps.com/spike/)showed the possibility of scanning 3-D objects directly to generate 3-D data from an active scanner mounted onto the mobile device.Spike combines imagery with a laser range finder and position sensors to generate real-time 3-D data of building objects.Google have also been testing similar functions in its "Project Tango" (http://www.google.com/atap/projecttango/#project) to extract 3-D geometry directly from a smartphone.Tango can give accurate information (e.g.building shape, height, inclination) of real buildings without relying on GPS or gyro sensors as it scans in real time.In LOC5 and LOC6, users can straightforwardly determine the extent of building damage, and which parts of the building have changed during a given period based on automatically calculated information.
Since LOC5 and LOC6 only emerged as possibilities from the user evaluation of our prototypes, in this paper they were only conceptualized but not yet implemented and tested.However, with progress in 3-D reconstruction technologies based on smart devices, it is thus reasonable to expect that mAR in various forms will soon become a common information generation and communication method, including in post-disaster settings.

Conclusions
AR is being increasingly adopted in various fields.Especially the rapidly growing use of mobile devices means that mAR can be adopted at a lower cost but with higher usability than traditional AR.With the development of not only new technologies such as wearable devices and 3-D mobile scanners, but also improvements in the mobile devices themselves, it is expected that mAR will play a major role in comprehensive and rapid building assessment.
In this paper we developed a number of concepts of increasing complexity to provide stakeholders operating in a disaster area with pertinent reference information to facilitate their orientation, the assessment of damage and safety, and to improve their own safety in a potentially hazardous environment.The potential of mAR in a disaster situation was confirmed by experienced disaster responders.Nevertheless, a number of limitations persist.First, our utility assessment was based on an online survey, meaning that the concept, not the actual practical usability, was assessed.Secondly, because mAR delivers and processes 3-D data, it is important to control performance given available resources.The experts emphasized the need for fast and responsive mAR.Current consumer devices, which can readily be used as mAR platforms, struggle when confronted with the variety and size of the data streams and processing steps required for more advanced LOC performance (in particular photogrammetric processing).Lastly, obtaining and sharing of data acquired in a disaster situation raises privacy issues that need to be considered carefully.
In this study, AR-based change detection (LOC5/LOC6) was only conceptualized.A comprehensive implementation will require components in vision recognition, photogrammetry, 3-D data reconstruction and multi-data integration, vision-based AR, etc.However, rapid developments in mobile 3-D reconstruction (e.g.Spike or Google's Project Tango) will provide ready solutions for some of the above, and also reduce the need for server-based processing that requires large bandwidth.In addition, we also need to expand our mAR concepts in terms of not only delivering reference data, but also allowing further augmentation with information directly fed into a mobile device in the disaster area.Even though satellite imagery obtained following a disaster (e.g. through the Disaster Charter) is currently not made available to response organizations, in principle, such a data stream can also be meaningfully incorporated into an mAR framework.
Edited by: T. Glade Reviewed by: two anonymous referees

Figure 1 .
Figure 1.The concept of simple location-based (LOC1) and attribute-based visualization (LOC2), data processing workflows, and aspects that determine accuracy and uncertainty of the superimposed information.

Figure 3 .
Figure 3. User interface of the (a) outdoor mAR and (a) indoor mAR.

Figure 4 .
Figure 4. QR codes for video clips of AR demonstration.(a) Indoor mAR, (b) outdoor mAR.

Table 1 .
Level of complexity (LOC) in augmented reality.

Table 3 .
Results of the survey on the value of different LOC concepts and data types: mean (M), median (Md), standard deviation (SD).
bile internet access is quickly compromised.GSV recently introduced a new function that shows multi-temporal im-agery.With this function, a user can directly compare the current building status with street-level imagery taken in dif-