cholesteric liquid crystals; autonomous robots; digital twin; fiducial markers; localization; construction; building information modeling
Abstract :
[en] The ability to label and track physical objects that are assets in digital representations of the world is foundational to many complex systems. Simple, yet powerful methods such as bar- and QR-codes have been highly successful, e.g. in the retail space, but the lack of security, limited information content and impossibility of seamless integration with the environment have prevented a large-scale linking of physical objects to their digital twins. This paper proposes to link digital assets created through building information modeling (BIM) with their physical counterparts using fiducial markers with patterns defined by cholesteric spherical reflectors (CSRs), selective retroreflectors produced using liquid crystal self-assembly. The markers leverage the ability of CSRs to encode information that is easily detected and read with computer vision while remaining practically invisible to the human eye. We analyze the potential of a CSR-based infrastructure from the perspective of BIM, critically reviewing the outstanding challenges in applying this new class of functional materials, and we discuss extended opportunities arising in assisting autonomous mobile robots to reliably navigate human-populated environments, as well as in augmented reality.
Precision for document type :
Critical notes/Edition
Disciplines :
Mathematics
Author, co-author :
Schwartz, Mathew
Geng, Yong ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Physics and Materials Science (DPHYMS)
Agha, Hakam ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Physics and Materials Science (DPHYMS)
Kizhakidathazhath, Rijeesh ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Physics and Materials Science (DPHYMS)
Liu, Danqing
Lenzini, Gabriele ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > IRiSC
Lagerwall, Jan ; University of Luxembourg > Faculty of Science, Technology and Medicine (FSTM) > Department of Physics and Materials Science (DPHYMS)
External co-authors :
yes
Language :
English
Title :
Linking Physical Objects to Their Digital Twins via Fiducial Markers Designed for Invisibility to Humans
Publication date :
04 June 2021
Journal title :
Multifunctional Materials
Special issue title :
Special issue on Morphological Computing
Volume :
4
Issue :
2
Pages :
022002
Peer reviewed :
Peer reviewed
Focus Area :
Physics and Materials Science
FnR Project :
FNR11688643 > Jan Peter Felix Lagerwall > SSh > Security In The Shell > 01/05/2018 > 30/04/2021 > 2017
Funders :
CE - Commission Européenne FNR - Fonds National de la Recherche ONR - Office of Naval Research Global
Pfeifer R 2009 Morphological Computation-Connecting Brain, Body, and Environment Creating Brain-Like Intelligence: From Basic Principles to Complex Intelligent Systems Sendhoff B, Körner E, Sporns O, Ritter H and Doya K (Berlin: Springer ) 66-83.
Li Y, Zheng Y, Morys B, Pan S, Wang J and Li K 2020 Threat Assessment Techniques in Intelligent Vehicles: A Comparative Survey IEEE Intell. Transp. Syst. Mag. 1.
Zhang L, Xiao W, Zhang Z and Meng D 2020 Surrounding Vehicles Motion Prediction for Risk Assessment and Motion Planning of Autonomous Vehicle in Highway Scenarios IEEE Access 8 209356-76.
Klomp M, Jonasson M, Laine L, Henderson L, Regolin E and Schumi S 2019 Trends in vehicle motion control for automated driving on public roads Veh. Syst. Dyn. 57 1028-61.
Hamid U Z A, Saito Y, Zamzuri H, Rahman M A A and Raksincharoensak P 2018 A review on threat assessment, path planning and path tracking strategies for collision avoidance systems of autonomous vehicles Int. J. Veh. Auton. Syst. 14 134-69.
Kim J, Schwartz M and Zarzycki A 2016 The Wave of Autonomous Mobility: Architecture Facilitating IndoorAutonomous Navigation eCAADe 2016: Complexity & Simplicity 34th eCAADe 2016 August 24-26, 2016 Oulu, Finland vol 1 Herneoja A, Österlund T and Markkanen P pp 53-8.
Iglínski H and Babiak M 2017 Analysis of the Potential of Autonomous Vehicles in Reducing the Emissions of Greenhouse Gases in Road Transport Proc. Eng. 192 353-8.
Alfeo A L et al 2019 Urban swarms: A new approach for autonomous waste management Int. Conf. on Robotics and Automation (ICRA) 20-24 May 2019 Montreal, QC, Canada vol 2019 4233-240 (IEEE).
Jones S, Milner E, Sooriyabandara M and Hauert S 2020 Distributed Situational Awareness in Robot Swarms Adv. Intell. Syst. 2 2000110.
Eastman E 2017 Tokyo, Japan (available at: Https://unsplash.com/photos/4HG5hlhmZg8).
Castelvecchi D 2020 Is facial recognition too biased to be let loose Nature 587 347.
Sacks R, Eastman C M, Lee G and Teicholz P 2018 BIM Handbook: A Guide to Building Information Modeling for Owners, Managers, Designers, Engineers and Contractors 3r (New York: Wiley).
Grieves M 2005 Product Lifecycle Management: Driving the Next Generation of Lean Thinking (New York: McGraw-Hill).
Garrido-Jurado S, Munõz-Salinas R, Madrid-Cuevas F and Marín-Jiménez M 2014 Automatic generation and detection of highly reliable fiducial markers under occlusion Pattern Recognit. 47 2280-92.
Zhang H, Zhang C, Yang W and Chen C Y 2015 Localization and navigation using QR code for mobile robot in indoor environment 2015 IEEE Int. Conf. on Robotics and Biomimetics (ROBIO) 6-9 Dec. 2015 Zhuhai, China (IEEE) pp 2501-6.
Sagitov A, Shabalina K, Sabirova L, Li H and Magid E 2017 Artag, Apriltag and Caltag Fiducial Marker Systems: Comparison in a Presence of Partial Marker Occlusion and Rotation 2017 Madrid, Spain 14th International Conference on Informatics in Control, Automation and Robotics (ICINCO 2017) 182-91.
Krogius M, Haggenmiller A and Olson E 2019 Flexible layouts for fiducial tags 2019 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS) 3-8 Nov. 2019 Macau, China (IEEE) pp 1898-903.
Schwartz M, Lenzini G, Geng Y, Rønne P B, Ryan P Y and Lagerwall J P 2018 Cholesteric Liquid Crystal Shells as Enabling Material for Information-Rich Design and Architecture. Adv. Mater. 30 1707382.
Geng Y, Kizhakidathazhath R and Lagerwall J P F 2021 Encoding Hidden Information onto Surfaces Using Polymerized Cholesteric Spherical Reflectors Adv. Funct. Mater. 31 2100399.
Koski M H and Ashman T L 2014 Dissecting pollinator responses to a ubiquitous ultraviolet floral pattern in the wild Funct. Ecol. 28 868-77.
Klomberg Y, Dywou Kouede R, Bartos M, Mertens J, Tropek R, Fokam E and Janěcek S 2019 The role of ultraviolet reflectance and pattern in the pollination system of Hypoxis camerooniana (Hypoxidaceae). AoB Plants 11 plz057.
Humar M and Musevic I 2010 3D Microlasers From Self-assembled Cholesteric Liquid-crystal Microdroplets Opt. Express 18 26995-7003.
Noh J, Liang H L, Drevensek-Olenik I and Lagerwall J P 2014 Tuneable Multicoloured Patterns From Photonic Cross Communication Between Cholesteric Liquid Crystal Droplets J. Mater. Chem. C 2 806-10.
Asshoff S, Sukas S, Yamaguchi T, Hommersom C, Le Gac S and Katsonis N 2015 Superstructures of chiral nematic microspheres as all-optical switchable distributors of light. Sci. Rep. 5 14183.
Fan J, Li Y, Bisoyi H, Zola R, Yang D, Bunning T, Weitz D and Li Q 2015 Light-directing omnidirectional circularly polarized reflection from liquid-crystal droplets. Angew. Chem., Int. Ed. Engl. 54 2160-4.
Lee S S, Seo H J, Kim Y H and Kim S H 2017 Structural Color Palettes of Core-Shell Photonic Ink Capsules Containing Cholesteric Liquid Crystals. Adv. Mater. 29 1606894.
Wang L et al 2017 Optically reconfigurable chiral microspheres of self-organized helical superstructures with handedness inversion Mater. Horiz. 4 1190-5.
Uchida Y, Takanishi Y and Yamamoto J 2013 Controlled Fabrication and Photonic Structure of Cholesteric Liquid Crystalline Shells Adv. Mater. 25 3234-7.
Geng Y, Noh J, Drevensek-Olenik I, Rupp R, Lenzini G and Lagerwall J P 2016 High-fidelity spherical cholesteric liquid crystal Bragg reflectors generating unclonable patterns for secure authentication Sci. Rep. 6 26840.
Geng Y, Noh J, Drevensek-Olenik I, Rupp R and Lagerwall J 2017 Elucidating the fine details of cholesteric liquid crystal shell reflection patterns Liq. Cryst. 44 1948-59.
Myung D and Park S 2019 Optical Properties and Applications of Photonic Shells. ACS Appl. Mater. Interfaces 11 20350-9.
Iwai Y, Iijima R, Yamamoto K, Akita T, Uchida Y and Nishiyama N 2020 Shrinkage of Cholesteric Liquid Crystalline Microcapsule as Omnidirectional Cavity to Suppress Optical Loss Adv. Opt. Mater. 8 1901363.
Park S, Lee S and Kim S 2020 Photonic Multishells Composed of Cholesteric Liquid Crystals Designed by Controlled Phase Separation in Emulsion Drops. Adv. Mater. 32 e2002166.
Bisoyi H, Bunning T and Li Q 2018 Stimuli-Driven Control of the Helical Axis of Self-Organized Soft Helical Superstructures. Adv. Mater. 30 e1706512.
Kitzerow H and Bahr C 2000 Chirality in Liquid Crystals (Partially Ordered Systems) (Berlin: Springer).
Geng Y, Jang J H, Noh K G, Noh J, Lagerwall J P and Park S Y 2018 Through the Spherical Looking-Glass: Asymmetry Enables Multicolored Internal Reflection in Cholesteric Liquid Crystal Shells Adv. Opt. Mater. 6 1700923.
Lenzini G, Ouchani S, Roenne P, Ryan P Y, Geng Y, Lagerwall J and Noh J 2017 Security in the shell: An optical physical unclonable function made of shells of cholesteric liquid crystals 2017 IEEE Workshop on Information Forensics and Security (WIFS) (IEEE) pp 1-6.
Pappu R, Recht B, Taylor J and Gershenfeld N 2002 Physical one-way functions Science 297 2026-30.
McGrath T, Bagci I E, Wang Z M, Roedig U and Young R J 2019 A PUF taxonomy Appl. Phys. Rev. 6 011303.
Barbosa F 2017 Reinventing Construction through a Productivity Revolution (available at: Www.researchandmarkets.com/reports/4846343/global-construction-outlook-to-2023-q3-2019) (Accessed 30 May 2021).
Agarwal R, Chandrasekaran S and Sridhar M 2016 Imagining construction's digital future (McKinsey & Company).
Evers A 2020 How autonomous robots are changing construction (available at: Www.cnbc.com/2020/11/30/how-autonomousrobots-by-are-changing-construction.html) (Accessed 31 December 2020).
Birgisdottir H and Rasmussen F N 2016 Introduction to LCA of buildings http://www.trafikstyrelsen.dk/~/media/Dokumenter/15%20byggeri/Baredygtigt%20byggeri/TBST-2016-02-Introduction-LCA-english.ashx (Danish Transport and Construction Agency).
Bayer C, Gamble M, Gentry R and Joshi S 2010 AIA guide to building life cycle assessment in practice https://www.aia.org/resources/7961-building-life-cycle-assessment-in-practice (The American Institute of Architects).
Mathews M, Robles D and Bowe B 2017 BIM+ Blockchain: A solution to the trust problem in collaboration? CITA BIM Gathering 2017 November 23rd-24th 2017 (https://doi.org/10.21427/D73N5K).
Li J, Greenwood D and Kassem M 2019 Blockchain in the built environment and construction industry: A systematic review, conceptual models and practical use cases Autom. Constr. 102 288-307.
Haas P 2018 Unobtrusive tagging of the environment to aid robotic or computer vision classification using infrared reflective computer vision tags US Patent App. 15/879, 067.
Lim C, Kim C, Park J I and Park H 2016 Mobile augmented reality based on invisible marker 2016 IEEE Int. Symp. on Mixed and Augmented Reality (ISMAR-Adjunct) (IEEE) pp 78-81.
Park H and Park J I 2006 Invisible marker based augmented reality system Visual Communications and Image Processing 2005 vol 5960 (Int. Society for Optics and Photonics) p 59601I.
Chan A P and Owusu E K 2017 Corruption forms in the construction industry: Literature review J. Constr. Eng. Manage. 143 04017057.
Engebø A, Kjesbu N, Lædre O and Lohne J 2017 Perceived Consequences of Counterfeit, Fraudulent and Sub-standard Construction Materials Proc. Eng. 196 343-50.
Pishdad-Bozorgi P, Yoon J H and Dass N 2020 Blockchain-based Information Sharing: A New Opportunity for Construction Supply Chains EPiC Series in Built Environment vol 1 pp 274-82.
Carley L R, Colak G, Chomas L, Pileggi L and Mai K 2016 Technologies for secure RFID authentication of medicinal pills and capsules 2016 IEEE Int. Conf. on RFID Technology and Applications (RFID-TA) (IEEE) pp 10-15.
Nam H, Song K, Ha D and Kim T 2016 Inkjet Printing Based Mono-layered Photonic Crystal Patterning for Anti-counterfeiting Structural Colors Sci. Rep. 6 30885.
Chen Y, Mihcąk M K and Kirovski D 2005 Certifying authenticity via fiber-infused paper ACM SIGecom Exch. 5 29-37.
Kangari R and Halpin D W 1989 Potential Robotics Utilization in Construction J. Constr. Eng. Manage. 115 126-43.
Paulson B C Jr 1985 Automation and Robotics for Construction J. Constr. Eng. Manage. 111 190-207.
Feng C, Xiao Y, Willette A, McGee W and Kamat V R 2015 Vision guided autonomous robotic assembly and as-built scanning on unstructured construction sites Autom. Constr. 59 128-38.
Richard R B 2005 Industrialised building systems: Reproduction before automation and robotics Autom. Constr. 14 442-51.
Schwartz M 2016 Use of a low-cost humanoid for tiling as a study in on-site fabrication 36th Conf. Association for Computer Aided Design in Architecture-Posthuman Frontiers: Data, Designers and Cognitive Machines (ACADIA) pp 214-23.
Dörfler K, Sandy T, Giftthaler M, Gramazio F, Kohler M and Buchli J 2016 Mobile robotic brickwork Robotic Fabrication in Architecture, Art and Design 2016 (Berlin: Springer) pp 204-17.
Lundeen K M, Kamat V R, Menassa C C and McGee W 2019 Autonomous motion planning and task execution in geometrically adaptive robotized construction work Autom. Constr. 100 24-45.
Kapadia M, Ninomiya K, Shoulson A, Garcia F and Badler N 2013 Constraint-aware navigation in dynamic environments Proc. Motion on Games MIG '13 (New York: Association for Computing Machinery) pp 111-20.
Augugliaro F et al 2014 The Flight Assembled Architecture installation: Cooperative construction with flying machines IEEE Control Syst. Mag. 34 46-64.
Occupational Safety and Health Administration 2019 Commonly used statistics: Worker fatalities (Accessed 30 May 2021) (https://www.osha.gov/data/commonstats).
Neelamkavil J 2009 Automation in the prefab and modular construction industry 26th Symposium on Construction Robotics ISARC.
Fuchs K, Grundmann T and Fleisch E 2019 Towards identification of packaged products via computer vision: Convolutional neural networks for object detection and image classification in retail environments Proc. 9th Int. Conf. on the Internet of Things pp 1-8.
Santra B and Mukherjee D P 2019 A comprehensive survey on computer vision based approaches for automatic identification of products in retail store Image Vis. Comput. 86 45-63.
Varadarajan S, Kant S and Srivastava M M 2020 Benchmark for generic product detection: A low data baseline for dense object detection Int. Conf. on Image Analysis and Recognition (Springer) pp 30-41.
Jeon D, Kim C K and Kim J 2015 Dynamic object tracking system 2015 12th Int. Conf. on Ubiquitous Robots and Ambient Intelligence (URAI) (IEEE) pp 512-15.
Wei J, Snider J M, Kim J, Dolan J M, Rajkumar R and Litkouhi B 2013 Towards a viable autonomous driving research platform 2013 IEEE Intelligent Symp. pp 763-70.
Santa J, Gómez-Skarmet A and Sánchez-Artigasz M 2008 Architecture and evaluation of a unified V2V and V2I communication system based on cellular networks Comput. Commun. 31 2850-61.
United States Department of Transportation 2015 U.S Department of Transportation releases vehicle to pedestrian technical scan summary (available at: Www.its.dot.gov/press/2015/v2p-tech.htm) (Accessed 25 January 2021).
Ahmadyan A, Zhang L, Wei J, Ablavatski A and Grundmann M 2020 ( Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotations 2012.09988 [cs.CV].
Stanley B M and Cornick M T 2018 Method and system for localization of a vehicle using surface penetrating radar US Patent App. 15/830, 398.
Fagnant D J and Kockelman K 2015 Preparing a nation for autonomous vehicles: Opportunities, barriers and policy recommendations Transp. Res. A 77 167-81.
Litman T 2017 Autonomous Vehicle Implementation Predictions (Victoria: Victoria Transport Policy Institute).
Schneier M, Schneier M and Bostelman R 2015 Literature Review of Mobile Robots for Manufacturing (Gaithersburg, MD: US Department of Commerce, National Institute of Standards and Technology) (https://doi.org/10.6028/NIST.IR.8022).
United States Department of Transportation Vehicle-to-infrastructure (V2I) resources (available at: Www.its.dot.gov/v2i/index.htm) (Accessed 25 January 2021).
Kucuk S D, Gerengi H and Guner Y 2018 The effect of Tinuvin derivatives as an ultraviolet (UV) stabilizer on EPDM rubber Period. Eng. Nat. Sci. 6 52-62.
Kuila D, Kvakovszky G, Murphy M A, Vicari R, Rood M H, Fritch K A, Fritch J R, Wellinghoff S T and Timmons S F 1999 Tris(hydroxyphenyl)ethane Benzotriazole: A Copolymerizable UV Light Stabilizer Chem. Mater. 11 109-16.
Zayat M, Garcia-Parejo P and Levy D 2007 Preventing UV-light damage of light sensitive materials using a highly protective UV-absorbing coating Chem. Soc. Rev. 36 1270-81.
Hwang D, Moon J, Shul Y, Jung K, Kim D and Lee D 2003 Scratch Resistant and Transparent UV-Protective Coating on Polycarbonate J. Sol-Gel Sci. Technol. 26 783-7.
Sethi S K and Manik G 2018 Recent Progress in Super Hydrophobic/Hydrophilic Self-Cleaning Surfaces for Various Industrial Applications: A Review Polym.-Plast. Technol. Eng. 57 1932-52.
Feng W, Broer D and Liu D 2018 Oscillating Chiral-Nematic Fingerprints Wipe Away Dust Adv. Mater. 30 1704970.
Mantha B and Garcia de Soto B 2019 Designing a reliable fiducial marker network for autonomous indoor robot navigation Proc. 36th Int. Symp. on Automation and Robotics in Construction (ISARC) ed M Al-Hussein (Banff: Int. Association for Automation and Robotics in Construction (IAARC)) pp 74-81.
Yong K S, Chiew K L and Tan C L 2019 A survey of the QR code phishing: The current attacks and countermeasures 2019 7th Int. Conf. on Smart Computing and Communications, ICSCC 2019 pp 3-7.
Wahsheh H A M and Luccio F L 2020 Security and Privacy of QR Code Applications: A Comprehensive Study, General Guidelines and Solutions Information 11 217.
