The Impact of Dataset Splits on Classification Performance in Medical Videos

Fox, Markus; Schoeffmann, Klaus

doi:10.1145/3512527.3531424

Cited by 5 publications

(4 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Fox et. al [21] used both the KFCV and normal split techniques to divide the ITEC LapGyn4 Gynecologic Laparoscopy Image Dataset [22] and employed them to classify the images using a convolutional neural network (CNN) model and scale-invariant feature transform (SIFT) classification. Their results show better performance for the KFCV using a CNN.…”

Section: Related Workmentioning

confidence: 99%

Automatic Optimization of Deep Learning Training through Feature-Aware-Based Dataset Splitting

Shahrabadi,

Adão,

Peres

et al. 2024

Algorithms

View full text Add to dashboard Cite

The proliferation of classification-capable artificial intelligence (AI) across a wide range of domains (e.g., agriculture, construction, etc.) has been allowed to optimize and complement several tasks, typically operationalized by humans. The computational training that allows providing such support is frequently hindered by various challenges related to datasets, including the scarcity of examples and imbalanced class distributions, which have detrimental effects on the production of accurate models. For a proper approach to these challenges, strategies smarter than the traditional brute force-based K-fold cross-validation or the naivety of hold-out are required, with the following main goals in mind: (1) carrying out one-shot, close-to-optimal data arrangements, accelerating conventional training optimization; and (2) aiming at maximizing the capacity of inference models to its fullest extent while relieving computational burden. To that end, in this paper, two image-based feature-aware dataset splitting approaches are proposed, hypothesizing a contribution towards attaining classification models that are closer to their full inference potential. Both rely on strategic image harvesting: while one of them hinges on weighted random selection out of a feature-based clusters set, the other involves a balanced picking process from a sorted list that stores data features’ distances to the centroid of a whole feature space. Comparative tests on datasets related to grapevine leaves phenotyping and bridge defects showcase promising results, highlighting a viable alternative to K-fold cross-validation and hold-out methods.

show abstract

Section: Related Workmentioning

confidence: 99%

Automatic Optimization of Deep Learning Training through Feature-Aware-Based Dataset Splitting

Shahrabadi,

Adão,

Peres

et al. 2024

Algorithms

View full text Add to dashboard Cite

show abstract

“…To the best of our knowledge, only two works addressing the analysis of dataset splits for surgical phase or instrument recognition have been published so far. Fox and Schoeffmann [29] show that random sampling of video frames without considering patient split might result in training and test sets containing video frames that are visually similar. This significantly distorts the evaluation results on the test set and yields overly optimistic results.…”

Section: Related Workmentioning

confidence: 99%

“…In the 40/-/40 split, which is used in the studies [7,36], all surgical phases are represented in both sets. However, a closer inspection of phase transitions unveils a group of nine surgeries (10,13,19,22,23,29,32,33,38) that deviate from the standard workflow by skipping the first phase and initiating the surgery directly in the second phase (see Fig. 4A).…”

Section: /-/40 Splitmentioning

confidence: 99%

Surgical phase and instrument recognition: how to identify appropriate dataset splits

Kostiuchik,

Sharan,

Mayer

et al. 2024

Int J CARS

View full text Add to dashboard Cite

Purpose Machine learning approaches can only be reliably evaluated if training, validation, and test data splits are representative and not affected by the absence of classes. Surgical workflow and instrument recognition are two tasks that are complicated in this manner, because of heavy data imbalances resulting from different length of phases and their potential erratic occurrences. Furthermore, sub-properties like instrument (co-)occurrence are usually not particularly considered when defining the split. Methods We present a publicly available data visualization tool that enables interactive exploration of dataset partitions for surgical phase and instrument recognition. The application focuses on the visualization of the occurrence of phases, phase transitions, instruments, and instrument combinations across sets. Particularly, it facilitates assessment of dataset splits, especially regarding identification of sub-optimal dataset splits. Results We performed analysis of the datasets Cholec80, CATARACTS, CaDIS, M2CAI-workflow, and M2CAI-tool using the proposed application. We were able to uncover phase transitions, individual instruments, and combinations of surgical instruments that were not represented in one of the sets. Addressing these issues, we identify possible improvements in the splits using our tool. A user study with ten participants demonstrated that the participants were able to successfully solve a selection of data exploration tasks. Conclusion In highly unbalanced class distributions, special care should be taken with respect to the selection of an appropriate dataset split because it can greatly influence the assessments of machine learning approaches. Our interactive tool allows for determination of better splits to improve current practices in the field. The live application is available at https://cardio-ai.github.io/endovis-ml/.

show abstract

“…Um bom extrator de características visuais deve garantir repetibilidade e precisão, sendo invariante a mudanças geométricas e fotométricas, ao mesmo tempo em que deve ser capaz de distinguir objetos diferentes (ABDEL-HAKIM;FARAG, 2006;GRAUMAN;LEIBE, 2011). Como exemplo desse tipo de descritor, podemos citar o Scale-Invariant Feature Transform (SIFT) (LOWE, 2004), usado em tarefas de análise multimídia (PATEL;DABHI;PRAJAPATI, 2020;MAHMOODZADEH, 2021;FOX;SCHOEFFMANN, 2022).…”

Section: Características Visuaisunclassified

Operadores de fusão prévia para segmentação temporal de vídeo em cenas

Beserra¹

View full text Add to dashboard Cite

Técnicas de fusão prévia têm sido propostas em tarefas de análise multimídia como uma maneira de melhorar a eficácia ao gerar representações de dados mais compactas, expressivas e capazes de preservar a semântica presente nos dados. Os trabalhos recentes no domínio de vídeo digital empregam multimodalidade fazendo jus à natureza multimodal de um vídeo. Esse espaço heterogêneo, somado à dificuldade de se obter uma etapa de fusão prévia desacoplada e separável do restante do processamento, limita possíveis melhorias que poderiam ser alcançadas nas etapas isoladamente. Além disso, técnicas foram projetadas para problemas específicos, não podendo ser generalizadas, o que também as tornam inseparáveis da tarefa de análise de vídeo em questão. Motivado por esse cenário, este trabalho de mestrado propõe a aplicação dos operadores de fusão prévia, Soma, Máximo e Concatenação, que atuem no médio nível semântico, desacoplando o operador de qualquer tarefa específica e, ao mesmo tempo, com um custo computacional mais simples. Os operadores foram aplicados em duas bases de dados publicamente disponíveis da tarefa de Segmentação Temporal de Vídeo em Cenas. Os resultados atingidos competem com os do estado da arte com a vantagem de simplicidade computacional.

show abstract

The Impact of Dataset Splits on Classification Performance in Medical Videos

Cited by 5 publications

References 4 publications

Automatic Optimization of Deep Learning Training through Feature-Aware-Based Dataset Splitting

Automatic Optimization of Deep Learning Training through Feature-Aware-Based Dataset Splitting

Surgical phase and instrument recognition: how to identify appropriate dataset splits

Operadores de fusão prévia para segmentação temporal de vídeo em cenas

Contact Info

Product

Resources

About