RobustSPAM for Inference from Noisy Longitudinal Data and Preservation of Privacy

Palczewska, Anna; Palczewski, Jan; Aivaliotis, Georgios; Kowalik, Łukasz

doi:10.1109/icmla.2017.0-137

Cited by 2 publications

(11 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A simplified approach is to convert any interval-based event e j taking place within an interval [t } , where the superscripts s and e denote synthetic events corresponding to the beginning and end of the interval for event e j . In this way we should solve a the problem of pointwise time series events, see examples in [34][35][36][37]. A required sequential or time series pattern mining algorithm can be applied afterwards.…”

Section: Temporal Datamentioning

confidence: 99%

“…Parameter β j is allowed to be zero when we know precisely when an event took place. Examples of databases with uncertainties can be found in [29,37,47].…”

Section: Uncertainty In Datamentioning

confidence: 99%

“…The database consists of approximately 100,000 adult social care records over a period of 15 years (also used in [37]). The records cannot be processed outside the secure facilities of Leeds Institute for Data Analytics (LIDA) at the University of Leeds.…”

Section: Adult Social Care Databasementioning

confidence: 99%

“…This approach has the advantages of using less storage space and being easier to implement, e.g. [34][35][36][37]. Another approach to mining through timestamped events is relevance weight pattern mining [38].…”

Section: Introductionmentioning

confidence: 99%

“…This will lead to incorrect estimates of how frequent the patterns AB and BA are in the dataset. For example, the algorithm RobustSpam [37] takes into account the possibility of inaccuracies in the way timestamps are recorded. This approach is focused on using time points instead of intervals and fitting probabilistic models for the errors in the time stamps around these time points.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Fast implementation of pattern mining algorithms with time stamp uncertainties and temporal constraints

Titarenko

Aivaliotis

et al. 2019

J Big Data

Self Cite

View full text Add to dashboard Cite

Pattern mining is a powerful tool for analysing big datasets. Temporal datasets include time as an additional parameter. This leads to complexity in algorithmic formulation, and it can be challenging to process such data quickly and efficiently. In addition, errors or uncertainty can exist in the timestamps of data, for example in manually recorded health data. Sometimes we wish to find patterns only within a certain temporal range. In some cases real-time processing and decision-making may be desirable. All these issues increase algorithmic complexity, processing times and storage requirements. In addition, it may not be possible to store or process confidential data on public clusters or the cloud that can be accessed by many people. Hence it is desirable to optimise algorithms for standalone systems. In this paper we present an integrated approach which can be used to write efficient codes for pattern mining problems. The approach includes: (1) cleaning datasets with removal of infrequent events, (2) presenting a new scheme for time-series data storage, (3) exploiting the presence of prior information about a dataset when available, (4) utilising vectorisation and multicore parallelisation. We present two new algorithms, FARPAM (FAst Robust PAttern Mining) and FARPAMp (FARPAM with prior information about prior uncertainty, allowing faster searching). The algorithms are applicable to a wide range of temporal datasets. They implement a new formulation of the pattern searching function which reproduces and extends existing algorithms (such as SPAM and RobustSPAM), and allows for significantly faster calculation. The algorithms also include an option of temporal restrictions in patterns, which is available neither in SPAM nor in RobustSPAM. The searching algorithm is designed to be flexible for further possible extensions. The algorithms are coded in C++, and are highly optimised and parallelised for a modern standalone multicore workstation, thus avoiding security issues connected with transfers of confidential data onto clusters. FARPAM has been successfully tested on a publicly available weather dataset and on a confidential adult social care dataset, reproducing results obtained by previous algorithms in both cases. It has been profiled against the widely used SPAM algorithm (for sequential pattern mining) and RobustSPAM (developed for datasets with errors in time points). The algorithm outperforms SPAM by up to 20 times and RobustSPAM by up to 6000 times. In both cases the new algorithm has better scalability.

show abstract

Section: Temporal Datamentioning

confidence: 99%

“…Parameter β j is allowed to be zero when we know precisely when an event took place. Examples of databases with uncertainties can be found in [29,37,47].…”

Section: Uncertainty In Datamentioning

confidence: 99%

Section: Adult Social Care Databasementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Fast implementation of pattern mining algorithms with time stamp uncertainties and temporal constraints

Titarenko

Aivaliotis

et al. 2019

J Big Data

Self Cite

View full text Add to dashboard Cite

show abstract

Machine learning, materiality and governance: A health and social care case study

Keen

Ruddle

Palczewski

et al. 2021

Self Cite

View full text Add to dashboard Cite

There is a widespread belief that machine learning tools can be used to improve decision-making in health and social care. At the same time, there are concerns that they pose threats to privacy and confidentiality. Policy makers therefore need to develop governance arrangements that balance benefits and risks associated with the new tools. This article traces the history of developments of information infrastructures for secondary uses of personal datasets, including routine reporting of activity and service planning, in health and social care. The developments provide broad context for a study of the governance implications of new tools for the analysis of health and social care datasets. We find that machine learning tools can increase the capacity to make inferences about the people represented in datasets, although the potential is limited by the poor quality of routine data, and the methods and results are difficult to explain to other stakeholders. We argue that current local governance arrangements are piecemeal, but at the same time reinforce centralisation of the capacity to make inferences about individuals and populations. They do not provide adequate oversight, or accountability to the patients and clients represented in datasets.

show abstract

RobustSPAM for Inference from Noisy Longitudinal Data and Preservation of Privacy

Cited by 2 publications

References 17 publications

Fast implementation of pattern mining algorithms with time stamp uncertainties and temporal constraints

Fast implementation of pattern mining algorithms with time stamp uncertainties and temporal constraints

Machine learning, materiality and governance: A health and social care case study

Contact Info

Product

Resources

About