Prompt engineering for zero‐shot and few‐shot defect detection and classification using a visual‐language pretrained model

Yong, Gunwoo; Jeon, Kahyun; Gil, Daeyoung; Lee, Ghang

doi:10.1111/mice.12954

Cited by 32 publications

(10 citation statements)

References 109 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Additionally, as deep learning and some new methods rapidly advance, numerous models have emerged in various related engineering fields, such as engineering defect detection (Chen et al., 2023; Pan & Yang, 2023; Yong et al., 2023), engineering object detection (Carranza‐García et al., 2022; Foresti et al., 2022; Guo et al., 2023), small object detection (Chaverot et al., 2023), engineering object tracking (Pan et al., 2023; Urdiales et al., 2023), and structural health monitoring system (Park, Park, et al., 2015). Meanwhile, some new and sophisticated machine learning methods have also been developed for engineering applications, such as neural dynamic classification (Rafiei & Adeli, 2017), dynamic ensemble Learning (Alam et al., 2020), finite element machine for fast learning (Pereira et al., 2020), and self‐supervised learning (Rafiei et al., 2022).…”

Section: Discussionmentioning

confidence: 99%

Monocular 3D object detection for construction scene analysis

Shen,

Jiao,

Zhang

et al. 2023

Computer aided Civil Eng

View full text Add to dashboard Cite

Three‐dimensional (3D) object detection, that is, localizing and classifying all critical objects in a 3D space, is essential for downstream construction scene analysis tasks. However, accurate instance segmentation, few 2D object segmentation and 3D object detection data sets, high‐quality feature representations for depth estimation, and limited 3D cues from a single red‐green‐blue (RGB) image pose significant challenges to 3D object detection and severely hinder its practical applications. In response to these challenges, an improved cascade‐based network with a transformer backbone and a boundary‐patch‐refinement method is proposed to build hierarchical features and refine object boundaries, resulting in better results in 2D object detection and instance segmentation. Furthermore, a novel self‐supervised monocular depth learning method is proposed to extract better feature representations for depth estimation from construction site video data with unknown camera parameters. Additionally, a pseudo‐LiDAR point cloud method and a 3D object detection method with a density‐based clustering algorithm are proposed to detect 3D objects in a construction scene without help from 3D labels, which will serve as a good foundation for other downstream 3D tasks. Finally, the proposed model is evaluated for object instance segmentation and depth estimation on the moving objects in construction sites (MOCS) and construction scene data sets. It brings a 9.16% gain in terms of mean average precision (mAP) for object detection and a 4.92% gain in mask mAP for object instance segmentation. The average order accuracy and relative mean error for depth estimation are improved by 0.94% and 60.56%, respectively. This study aims to overcome the challenges and limitations of 3D object detection and facilitate practical applications in construction scene analysis.

show abstract

Section: Discussionmentioning

confidence: 99%

Monocular 3D object detection for construction scene analysis

Shen,

Jiao,

Zhang

et al. 2023

Computer aided Civil Eng

View full text Add to dashboard Cite

show abstract

“…It should be noted that CLIP has been utilized in the field of civil engineering, specifically for classifying and detecting building defects (Yong et al., 2022). In this study, the focus was on identifying effective prompts (text inputs) for CLIP to optimize its performance by utilizing its zero‐shot and few‐shot capabilities.…”

Section: Related Workmentioning

confidence: 99%

Improving visual question answering for bridge inspection by pre‐training with external data of image–text pairs

Kunlamai,

Yamane,

Suganuma

et al. 2023

Computer aided Civil Eng

View full text Add to dashboard Cite

This paper explores the application of visual question answering (VQA) in bridge inspection using recent advancements in multimodal artificial intelligence (AI) systems. VQA involves an AI model providing natural language answers to questions about the content of an input image. However, applying VQA to bridge inspection poses challenges due to the high cost of creating training data that requires expert knowledge. To address this, we propose leveraging existing bridge inspection reports, which already include image–text pairs, as external knowledge to enhance VQA performance. Our approach involves training the model on a large collection of image–text pairs, followed by fine‐tuning it on a limited amount of training data specifically designed for the VQA task. The results demonstrate a significant improvement in VQA accuracy using this approach. These findings highlight the potential of AI models for VQA as valuable tools for assessing the condition of bridges.

show abstract

“…Despite these promising developments, RPE is not without challenges. Recent discussions by different author for downstream task of Text Classi cation [9][10][11][12][13][14][15] underscore the data diversity, scalability, and interpretability issues associated with PE. This thought-provoking work encourages the community to address these challenges and set new research directions.…”

Section: Challenges and Future Directionsmentioning

confidence: 99%

From Static to Recursive: Transforming Prompts for Enhanced Language Models

Tripathi

2023

Preprint

View full text Add to dashboard Cite

In the dynamic field of Natural Language Processing (NLP), a revolutionary paradigm shift known as Large Language models has emerged. This research article embarks on exploration of Prompt Engineering, unveiling its innovative techniques, confronting its challenges, and highlighting its transformative impact on NLP applications. The proposed prompt engineering which is Recursive Prompt Engineering (RPE) redefines conventional prompt engineering, providing NLP models with the ability to iteratively refine responses. Through carefully designed experiments and real-world applications, we showcase RPE’s ability to enhance performance in language generation, question answering, and sentiment analysis. However, this journey into uncharted territory uncovers formidable challenges, including issues related to data diversity, scalability, and model interpretability. These challenges, while illuminating, also serve as stepping stones toward further innovation. Traditional evaluation methods prove inadequate, prompting us to introduce novel evaluation metrics that capture the essence of recursive adaptability. Our work sets the stage for redefining the criteria for measuring RPE’s effectiveness. In presenting this work, we envision a future where RPE reshapes the NLP landscape. As Recursive Prompt Engineering leads us to uncharted frontiers in NLP, opening doors to unprecedented possibilities and innovation. This article serves as a guiding beacon in this new era of NLP exploration.

show abstract

Prompt engineering for zero‐shot and few‐shot defect detection and classification using a visual‐language pretrained model

Cited by 32 publications

References 109 publications

Monocular 3D object detection for construction scene analysis

Monocular 3D object detection for construction scene analysis

Improving visual question answering for bridge inspection by pre‐training with external data of image–text pairs

From Static to Recursive: Transforming Prompts for Enhanced Language Models

Contact Info

Product

Resources

About