Leveraging Community-Generated Videos and Command Logs to Classify and Recommend Software Workflows

Wang, Xu; Lafreniere, Benjamin; Grossman, Tovi

doi:10.1145/3173574.3173859

Cited by 21 publications

(16 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this paper, we developed a hybrid crowd-computer vision system to enable access to dynamic touchscreens in-the-wild. One unique contribution of this work is that we demonstrated the possibility of extracting state diagrams from existing pointof-view videos instead of screenshots or screencast videos [2,26,37]. For existing physical devices whose underlying hardware or software cannot be modified, point-of-view videos are more prevalent and easier to acquire compared to screencast videos, which makes our approach generalizable to a large variety of devices and scenarios.…”

Section: Generalizabilitymentioning

confidence: 99%

StateLens

Guo

Kong

Rivera

et al. 2019

Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology

View full text Add to dashboard Cite

Blind people frequently encounter inaccessible dynamic touchscreens in their everyday lives that are difficult, frustrating, and often impossible to use independently. Touchscreens are often the only way to control everything from coffee machines and payment terminals, to subway ticket machines and in-flight entertainment systems. Interacting with dynamic touchscreens is difficult non-visually because the visual user interfaces change, interactions often occur over multiple different screens, and it is easy to accidentally trigger interface actions while exploring the screen. To solve these problems, we introduce StateLens -a three-part reverse engineering solution that makes existing dynamic touchscreens accessible. First, StateLens reverse engineers the underlying state diagrams of existing interfaces using point-of-view videos found online or taken by users using a hybrid crowd-computer vision pipeline. Second, using the state diagrams, StateLens automatically generates conversational agents to guide blind users through specifying the tasks that the interface can perform, allowing the StateLens iOS application to provide interactive guidance and feedback so that blind users can access the interface. Finally, a set of 3Dprinted accessories enable blind people to explore capacitive touchscreens without the risk of triggering accidental touches on the interface. Our technical evaluation shows that State-Lens can accurately reconstruct interfaces from stationary, hand-held, and web videos; and, a user study of the complete system demonstrates that StateLens successfully enables blind users to access otherwise inaccessible dynamic touchscreens.

show abstract

Section: Generalizabilitymentioning

confidence: 99%

StateLens

Guo

Kong

Rivera

et al. 2019

Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology

View full text Add to dashboard Cite

show abstract

“…Recently, Workflow Recommenders (WRs) [14] in PAISs have proposed recommendation services that aim at suggesting frequent combinations of workflow tasks for reuse. Some of these recommenders apply data mining techniques such as similarity measurement to help users find items to improve their workflow designs by prediction [14,15]. For instance Wang et al [15] has proposed and evaluated a design space of four different WR algorithms based on data mining, which can be used to recommend new workflows and their associated videos to software users.…”

Section: Related Workmentioning

confidence: 99%

“…Some of these recommenders apply data mining techniques such as similarity measurement to help users find items to improve their workflow designs by prediction [14,15]. For instance Wang et al [15] has proposed and evaluated a design space of four different WR algorithms based on data mining, which can be used to recommend new workflows and their associated videos to software users. One of the limitations of the algorithms proposed in [15] is that they used heuristics.…”

Section: Related Workmentioning

confidence: 99%

“…For instance Wang et al [15] has proposed and evaluated a design space of four different WR algorithms based on data mining, which can be used to recommend new workflows and their associated videos to software users. One of the limitations of the algorithms proposed in [15] is that they used heuristics. Besides, the authors in [15] did not evaluate their proposed algorithms with different datasets of different domains.…”

Section: Related Workmentioning

confidence: 99%

“…One of the limitations of the algorithms proposed in [15] is that they used heuristics. Besides, the authors in [15] did not evaluate their proposed algorithms with different datasets of different domains. Tosta et al [14], as another example proposed a WR method that works potentially in different domains.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations