Interactive visual machine learning in spreadsheets

Sarkar, Advait; Jamnik, Mateja; Blackwell, Alan F.; Spott, Martin

doi:10.1109/vlhcc.2015.7357211

Cited by 15 publications

(12 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These pragmatic considerations have led to the development of interactive machine learning systems, such as Microsoft Excel FlashFill, BrainCel, Gneiss, etc. (Chang & Myers, 2014;Gulwani, 2011;Sarkar et al, 2015), in which labeling is carried out "online", with feedback from the partially trained classifier being used to inform the user about current performance and potential weaknesses of the statistical model so far, and users being provided with tools to correct the model (for an early paradigm of interactive model construction in the HCI literature, see Fails & Olsen, 2003). This process of interactive labeling can be considered as a variety of programming for example, where the user causes the system to work as desired by demonstrating how it ought to behave (Lieberman, 2000;Menon et al, 2013).…”

Section: Conversational Labelingmentioning

confidence: 99%

“…From this perspective, the process of observing system behavior and providing new labels to correct erroneous behavior is a kind of debugging (Kulesza et al, 2015). Through direct interaction with data, it is likely that future semi-automated classification and inference systems will routinely demonstrate mixed-initiative interaction characteristics (Sarkar et al, 2015).…”

Section: Conversational Labelingmentioning

confidence: 99%

See 1 more Smart Citation

Perception of rhythmic agency for conversational labeling

Blackwell

Cross

2021

Human–Computer Interaction

Self Cite

View full text Add to dashboard Cite

Machine learning algorithms often need to be trained with sample datasets, prepared by humans who label the content manually. However, data labeling is tedious and repetitive, with humans often replicating trivial decisions that could have been automated, and not getting sufficient time to devote to cases where their judgment would have added more value. Much recent research has therefore focused on approaches to labeling that optimize information gain by assisting the user with trivial cases, or directing them toward more complex ones, so that the user can devote time and attention to subtle cues and ambiguous cases, potential bias (e.g., Binns et al., 2017), or inventing new label categories (e.g., Kulesza et al., 2014). The consequence of this increasingly common strategy is that labeling interfaces become more conversational. The system is providing implicit feedback to the user about the model under development, while the user's decisions on more difficult cases are implicitly challenging the system to update the model it holds. Such a perspective shifts from the relatively predictable view of machine learning, in which facts are simply repeated until the machine retains them, to one that more closely imitates teaching and learning situations between two humans, where the knowledge being acquired is dialogically shared, probed and questioned as in a conversation. We report investigations into a particular characteristic of human conversation that has not previously been explored for its relevance to interactive machine learning-rhythmic timing in conversation. This can be contrasted on one hand to earlier models of "dialog" in HCI where the user takes the initiative, issuing commands while the system responds with information as soon as it can in order to be seen as "smooth" (Miller, 1968; Nielsen, 1993), or on the other hand to task optimization models where the system takes the initiative, prompting for information that is supplied by the user as quickly and efficiently as possible, with minimum latency (Bernstein et al., 2011). In mixed initiative interaction (Horvitz, 1999), neither of these existing design models is appropriate, and we suggest that attention to rhythm and timing becomes far more important. In design approaches where the system has the potential to complete the user's actions, or even to take the initiative and make decisions, there will be a back-and-forth flow of initiative between the user and the system, resembling participatory turn-taking in human conversation. In human conversation, 1 poorly timed participation is associated with negative or inappropriate social effects (see, e.g., Benus et al., 2011; Richardson et al., 2006). We ask: during mixed-initiative interaction such as interactive labeling, how do the rhythmic timing characteristics of the interaction influence users' experience, and how should these be designed?

show abstract

Section: Conversational Labelingmentioning

confidence: 99%

Section: Conversational Labelingmentioning

confidence: 99%

Perception of rhythmic agency for conversational labeling

Blackwell

Cross

2021

Human–Computer Interaction

Self Cite

View full text Add to dashboard Cite

show abstract

“…Multiple representations have previously been applied in spreadsheets in the interactive machine learning domain [14], but not as simultaneous editing experiences. Programming languages theory has a concept of 'lenses' [15] which is a form of infrastructure enabling multiple representations.…”

Section: Related Work a Multiple Representations And Spreadsheementioning

confidence: 99%

Calculation View: multiple-representation editing in spreadsheets

Sarkar

Gordon

Jones

et al. 2018

2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)

Self Cite

View full text Add to dashboard Cite

Spreadsheet errors are ubiquitous and costly, an unfortunate combination that is well-reported. A large class of these errors can be attributed to the inability to clearly see the underlying computational structure, as well as poor support for abstraction (encapsulation, re-use, etc). In this paper we propose a novel solution: a multiple-representation spreadsheet containing additional representations that allow abstract operations, without altering the conventional grid representation or its formula syntax. Through a user study, we demonstrate that the use of multiple representations can significantly improve user performance when performing spreadsheet authoring and debugging tasks. We close with a discussion of design implications and outline future directions for this line of inquiry.

show abstract

“…In contrast to other automation approaches, like AutoML [9,3], in VisualSynth the task to be performed (e.g., classification) is not assumed to be fixed or known. While interacting with spreadsheet users has been studied on a simple machine learning task [8], VisualSynth considers a wide range of data science tasks using a unified interaction system.…”

Section: Introductionmentioning

confidence: 99%

VisualSynth: Democratizing Data Science in Spreadsheets

Gautrais

Dauxais

Kolb

et al. 2021

Machine Learning and Knowledge Discovery in Databases. Applied Data Science and Demo Track

View full text Add to dashboard Cite

We introduce VisualSynth, a framework that wants to democratize data science by enabling naive end-users to specify the data science tasks that match their needs. In VisualSynth, the user and the spreadsheet application interact by highlighting parts of the data using colors. The colors define a partial specification of a data science task (such as data wrangling or clustering), which is then completed and solved automatically using artificial intelligence techniques. The user can interactively refine the specification until she is satisfied with the result.

show abstract

Interactive visual machine learning in spreadsheets

Cited by 15 publications

References 18 publications

Perception of rhythmic agency for conversational labeling

Perception of rhythmic agency for conversational labeling

Calculation View: multiple-representation editing in spreadsheets

VisualSynth: Democratizing Data Science in Spreadsheets

Contact Info

Product

Resources

About