Jarrett Interaction Design

Smarties: An Input System for Wall Display Development by Oliver Chapuis

• wall display input issues: mouse/keyboard – stuck to desk; touch – too close; laser pointer, air gesture, hard to track; prefer mobile tablet
• input programming is complex and expensive; provide a system for easy and fast prototyping; mobile device(s) -> protocol -> library -> wall
• interface: multiple pucks representing “cursors” at multiple locations on the wall; each puck has functions of a mouse with multi-touch and other functions like text; collaborative; you have your own pucks and share control; store and retrieve pucks
• protocol and libraries: multi-client event based; synchronization for shared pucks; C++, Java, JavaScript; similar to mouse management but applied to each puck
• applications: attach pucks to wall lenses for exploration of map
• software available

Conductor: Enabling and Understanding Cross-Device Interaction by Peter Hamilton

• vision: the move from physical desk to mobile device has lost some of the utility of large physical desk space; interactions across multiple devices by one user; “symphony” of devices
• ***** Conductor: targeted transmissions to other devices; cue broadcasting; minimally invasive; contextual actions; persistent connections -> duet functional bonding;'duet management; cross-application views; peripheral device sharing (eg share a Bluetooth keyboard); cross-device task manager
• user study: large search and integration task given a whiteboard, paper and pen, 5 Nexus 10 and 5 Nexus 7 devices; all used the devices; using spacial memory to store specific pieces of data on individuals devices

Panelrama: Enabling Easy Specification of Cross-Device Web Applications by Jishuo Yang

• by 2017 the average household will have 4 internet enabled devices; how to take advantage of those
• automatically reassign UI elements to the available devices based on best device for certain functions (example uses laptop/projector = slide, phone = remote control, pebble watch = presentation time, google glass = presenter notes)
• panel: UI building block; group of UI components with a shared purpose
• Panelrama: attributes – scorn size, proximity to user, keyboard, touchscreen; developers score each characteristic for each panel – extensible attributes; Panelerama models attributes of devices; optimize layout across multiple devies for each panel; minimize code changes, just define panel tag and panel definition
• developer study: 8 developers converted single device apps to multi-device in 40 minutes or less
• code available soon

Interactive Development of Cross-Device User Interfaces by Michael Nebeling

• meeting room scenario: parts of UI distributed to speaker tablet, projector, and audience phones; device-centric with different roles
• classroom scenario: teacher's device and 3 student groups; role centric ie student + group or teacher
• design time: device types including unknown devices; user roles; adapt UI elements: adapt UI across devices; design and testing; reuse
• run-time requirements: dynamic integration of devices, update the distribution, matching and adaptation
• XDStudio: GUI builder for multi-device UI; “DUI” = distributed user interface; on device and simulated authoring modes; user study validated that different modes are preferred for different situations; define distribution profiles for devices, device classes, and roles; client-server architecture
• evaluation: would people use authoring modes? used the scenarios with mix of mode availability; device-centric scenario benefits more from both modes, where classroom was fine with simulation

Easy Does It: More Usable CAPTCHAs by Angelique Moscicki

• CAPTCHA: block low grade, automated abuse on low risk tasks; many variations in specific features
• usability measures: accuracy, solving time, satisfaction
• automatic variations in features and parameters; 97,000 Mechanical Turk participants on 750,000 tests; 5,000 satisfaction surveys
• findings: users sensitive to font choice, prefer simpler character sets, eg numeric; not sensitive to screen resolution, length; many feature interactions, 20% had nonlinear relationships! user testing required; preference for positive words, digits, and common words; random strings least preferred
• tested and deployed new algorithm: numeric digits, removed confusion between 1 and 7 and o (oh) and 0 (zero); +6.7% accuracy, -55% reloads, -10% failed

Using Personal Examples to Improve Risk Communication for Security and Privacy Decisions by Marian Herbach

• 67 million apps downloaded per day on Google Play in 2013; users entrust personal data to devices
• many people do not understand permissions and get habituation so ignore and just grant them
• use concrete and personal examples to demonstrate risk; eg show photos that could be deleted or describe explicit risks like viruses or show example contacts
• study: mockup app with pilot and Mechanical Turk; pick 2-6 apps to install and present permission screen;
• findings: 14-23% of the time participants chose less-requesting apps or none even after app selected; didn't prevent users from choosing to install at least one app (in most cases); brand and high ratings didn't change decisions; showing personal information created negative affect including paying more attention to real permission screens

Experiences in Account Hijacking by Iulia Ion and Richard Shay

• account compromise: example Mat Honan; lots of effort for a small goal (twitter handle) on a normal person with devastating impact to that person
• goal: how to encourage people to use good security practices; experiences and attitudes
• study: 294 Mechanical Turk participants; 15-30% said they had an account compromised and received different survey
• findings: accounts are often valuable and used often; attackers unknown and known (effect relationship); harm is concrete and emotional; accept some responsibility for security; incomplete security understanding; 50% notified by others, 30% noticed content, 30% notified by service, 17% locked; 33% had email sent from account; 20% said no concrete harm; most felt negative emotions; 2/3 said it improved their security behavior; most say user and service provider are responsible; often said responsibility related to passwords; services should prevent and inform user of compromises
• implications: use stories with emotional appeal to drive people to better security behavior; emphasize that there is more to security beyond passwords; services should have good notification mechanisms (alternative channels)

Experimenting at Scale with Google Chrome's SSL Warning by Adrienne Felt

• active network attack: intercepting traffic between user and server; SSL supposed to protect; if something is wrong with SSL, warning is shown
• 68% of the time people ignore warning; often annoyed by false warning; but warning could be improved, eg FireFox only has 33% clock through level on their warning; want to stop annoying people and get informed consent
• study: 17,000 impressions per condition over a week; with FireFox warning in chrome, lower level but still higher than in FireFox; images of people didn't impact, despite expectations from psychology; styling changes had no effect; number of extra clicks has no effect
• other factors? better headlines and calls to action; separate action buttons physically and make less similar

Betrayed by Updates: How Negative Experiences Effect Future Security by Rick Wash

• eg police warning at Michigan State about IE security vulnerability
• most attacks target known vulnerabilities where patch is available; why do people not patch?
• interviewed 37 non-expert Windows users, mostly grad students (high risk if computer compromised, low cash to replace)
• findings: don't want unexpected changes to user interface; unused and unrecognized software, like Java; current version already works, why bother, like Adobe Reader
• ref: Microsoft Security Intelligence Report v13, 2013; browser, Java, adobe account for large proportion of attack vector

Mobile Attachment – Causes and Consequences for Emotional Bonding with Mobile Phones by Alexander Meschtscherjakov

• 7 billion population, 6.8 billion mobile subscriptions
• psychology: attachment theory, extended self; consumer research and design: brand attachment, product attachment
• mobile attachment: cognitive and emotional target-specific bond connecting a person's self and a mobile device that varies in strength
• model: causes, influences, consequences
• causes: device-self linkage routes, design space determinants; empowerment -> utility; self (past + private + public +collective) enrichment -> memory + self image + affiliation + world view; self-gratification -> pleasure
• influences: user (personality, brand history, ownership, etc), environment (ads, narratives, other devices), device (design, functions, quality, etc)
• consequences: investment of resources and self-image resources, behavioral and emotional responses
• conclusions: attachment exists, causes and consequences are not mutually exclusive, helpful to investigate theories in different disciplines
• “I Just Want to Be Your Telephone” song

Hooked On Smartphones: Overuse Among College Students by Uichin Lee

• smartphone overuse: disrupt social interactions, mental health, sleep patterns; technological addiction – behavioral not chemical
• goal: identify detailed usage behaviors related to problematic usage
• study: 95 students for a semester; 36 in risk group; android detailed usage logging tool (unlock, app use, lock, app notifications)
• addiction scale: interference, virtual world orientation, withdrawal, tolerance
• findings: risk groups spend more time, more frequently, and longer sessions; top 1 or 2 apps dominate usage, at risk groups show even more skewed; risk groups use more always but especially morning and evening; risk groups use web apps more and possibly communication apps; instant messaging dominant; external triggers (notifications) 450+ per day in risk group; 3 times more web page visits in group
• problematic usage: feel more compelled to check devices (more anxious), less conscious and structured in usage (less self-regulation)

Influence of Personality on Satisfaction with Mobile Phone Services by Nuria Oliver

• relationship between personality and satisfaction with devices
• satisfaction drives sustained consumption, is a focus of marketing, and an important measure in usability; used Big 5 personality dimensions
• model: relate personality, customer satisfaction, perceived usability, device usage
• study: 603 participants, young, gender balanced, rural and urban, use phone at least 6 months; call data records, so basic feature phone usage; structural equation modeling (ESM)
• findings: biggest factor is perceived usability and satisfaction (.48), mostly perceived efficiency; usage negatively correlates with satisfaction (mostly calls and duration); extroversion influences usage; extroversion and conscientiousness effect perceived usability; conscientiousness has positive influence on satisfaction, intellect negative
• implications: personality-based service personalization, minimize disruptions, be aware of user saturation points and usage/time budgets

Broken Display = Broken Interface? by Florian Schaub

• 37% of mobile phones damages in 1st 3 months; 23% of iPhones have damaged screens (disclaimer: study by insurance company)
• 95 photos of damaged screens from Mechanical Turk; image analysis; annotate and code damage; statistical damage analysis; damage topology
• damage categories: minor, medium, severe; compare to self-reported touch damage, correlates with extent of screen damage but minor screen damage perceive more touch damage
• 98% continued to use after damage for an average of 5 months, 8.4% more than a year; 70% did not plan to repair; still usable, damage insignificant, financial considerations
• viewing issues: location, extent, opacity, typing impacted by reading impact, depends on orientation; input issues: tactile sensations, source of injury, UI elements unreachable
• coping strategies: preventive, viewing, touch and input, calling, interaction; eg ignore or get used to, move content around, be more careful, alternative interaction paths, move to another app
• design considerations: support scrolling and device rotation, layout and theme customization (eg dark background makes damage less noticeable), alternative interaction paths, adaptive representations when sensing damage

Structuring the Space by Nathalie Henry Riche

• people often refer to information visualizations as maps; many visualizations use spatial metaphors; picked up on contour lines from topographical maps for data
• mental model: spatial structure + landmarks; do they help or hinder readability and understanding?
• user study: hinder ability to find common neighbors? help perform comparison of similar graphs? help revisit nodes? 3 conditions – no structure, grid, contour lines
• findings: no changes in readability shown, contour lines better than grids for comparison but no difference between grids and control, ensure data sets have salient features like clusters, contour better than grid or control for revisitation even though people thought the grid helped

Highlighting Interventions and User Differences by Giuseppe Carenini

• investigate user-adaptive visualizations; what to adapt to? when to adapt? how to adapt?
• evaluate 4 types of highlighting interventions: bold, connected arrows, de-emphasis, reference lines
• highlighting can layer in relevant information in a complex visualization
• user study: 62 participants, bar graphs, tasks – retrieve value + compute derived value; look at impact of user characteristics; varied timing of intervention
• findings: deemphasis is best but bold and arrow worked too, dynamic timing erased edge of deemphasis, more complex task also took deemphasis to parity with bold and arrow, all interventions rated useful, visual working memory related to perceived usefulness of reference line

Evaluating a Tool for Improving Chart and Graph Accessibility by Gita Lindgaard

• how do blind people form a mental representation of a graph?
• descriptions must be consistent, order of descriptions follow order of questions, star from oldest data, need to interrogate the graph, vocabulary – x/y axis + up/down
• iGraph: extracts semantics from excel charts and generates natural language output plus supports interaction commands
• usability study 1: complex graphs took longer, blind people used twice as many commands as sighted and found it easier to use, all used more command than necessary, skipmwasmconfusing, didn't use where am I command
• usability study 2 (improved system): graph complexity had no effect, blind users used many more commands still (double check understanding), blind users navigated left more often, sighted start over more often
• field study: system could handle most of the questions the user had about user's chosen graphs; order of information: title, type, then other info; presentation of graphs were often missing a great deal of critical metadata about the graphs
• test expert to novice vocabulary usage: iGraph vocabulary mentioned by all participants

Understand Users' Comprehension and Preferences for Composing Information Visualizations by Huahai Yang

• develop a system to automatically compose a visualization from multiple charts and pick best representation; choice depends on insight you are looking for, eg side by side bar good at extrema identification, lines good at correlation comparison
• study: describe composite visualizations to discover vocabulary and concepts; mechanical Turk led to 1,500 useful descriptions, then coded them; 4 basic insights – read value, extrema identification, characterize distribution, correlation; all can be used for comparison as well; prioritize insights for different types of charts; value comparison, extrema, and correlation swamp other insights (Zipf function)
• most preferred: crossed-bar (side by side) except for correlation comparison which prefers crossed-line

Odin: Contextual Document Opinions on the Go by Joshua Hailpern

• Odin: mobile solution to get through hundreds and thousands of docs quickly; UrLs, Google News, upload zip, streams like RSS; finds most relevant, most aligned, most divergent; does an executive summary based on sentence scoring; can go to statement in context; can get summary on any document
• algorithm: topic modeling (rank order of topics) -> sentiment detection (sentence diagramming) -> aggregation (weighted distribution on ranked keywords)
• user studies: pilot Odin vs Google News – preferred Odin all tasks, core process really solved the problem, extend for domains; comparative study Odin vs RevMiner vs Google News – choose own doc set then summarize, Odin and Google rated high on SUS, Odin had high value added to work, all participants said Odin was the best, summary is powerful

Monadic Exploration: Seeing the Whole Through the Parts by Marian Dork

• when working with networks we can see micro (one node) or macro (the network as a whole); visual exploration between part and whole
• monads: point of view on all entities taken severally and not as a totality; neither whole nor part, but a single element's perspective
• principles: having (relational aspects), difference (distinct position), movement (navigate overlapping perspectives); could lead to many approaches; treat elements as vantage and navigation points + elastic layout, show difference! integrate search
• current visualization puts monad detail at center, other elements in brief in ordered circle around at a distance based on relevancy and transparency for more distant relationships possibly as just a dot
• case study: people found it valuable to draw themselves into the content of the network (based on highly cross-linked book on activation)

Photographing Information Needs by Zhen Yue

• role of photos in data collection; ESM (experience sampling method) – use of photos for jogging memory to be less disruptive in actual moment in data collection
• collect qualitative data periodically and optionally add a photo; end of each day, send a survey to ask for elaboration including photo to trigger memory
• findings: 1/3 used at least one photo, women more likely to share photo, older people more likely to share photo; fewer photos shared after first day; photos led to more complete surveys; photos led to higher quality responses, but interrupted work for longer responses but didn't interfere with ease of use ratings; 1/3 of photos were useful and relevent to researchers, especially for clarification and disambiguation

Design Insights for the Next Wave Ontology Authoring Tools by Markel Vigo

• ontology: logical actions that represent a field of interest; very complex and authoring is complex; very large; semantics, reasoning, inference; applied in critical domains like health; tools with poor utility
• need to improve tools because ontologies are being used more widely including by amateurs
• interviewed 15 ontology authors in different fields
• recommend: provide overviews of hierarchy and complexity, provide filtering, increase situational awareness, bulk entry of large numbers of elements, retrieve from external ontologies, intelligent reasoning, evaluation features

The Role of Interactive Byclusters in Sensemaking by Maoyuan Sun

• how to find relationships between elements in a large body of documents; visual analytics is useful
• bicluster: cluster by two attributes simultaneously
• Bixplorer: tool to help with interactive biclusters
• user study: task – identify possible terrorist plots; 15 participants with no prior experience
• findings: most started with biclusters, most found relevant documents and abandoned irrelevant docs; 1/2 of interactions was to find relevant docs; indicate potentially important entities; 1/2 of users created custom layouts using biclusters to label data

Communiplay: A Field Study of Public Display Mediaspace by Jorg Muller

• multiple interactive displays linked across varied public spaces; how will people interact with displays in public media spaces?
• key metric: conversion rate – percentage of people that start interacting; also explore the honey-pot effect
• Communiplay: 6 locations in public parts of buildings; six conditions including fake users
• observations: 1,234 interactions out of 30,888 passers; honey pot effect exists, local and remote, and more participants leads to more; local is much stronger; fake users didn't show differences; interaction duration increases with more users; play together and with objects, waving, punching/kicking, mimicking; ghost effect – remote passer by local turns around; landing effect – pass then return and do a small interaction

P-LAYERS – A Layered Framework for Public Displays by Nemenja Memarovic

• many more interactive displays today, and growing; people like them especially when well executed; what are the design attributes for a successful one?
• 3 different installations with different attributes
• hardware layer: use same hardware in lab and production, communicate affordances on screen, hardware failures and support issues
• system architecture: consider scalability, keep up with 3rd party APIs
• content: user-generated and auto-generated perform the same, keep content fresh and relevant
• system interaction: skipped for time
• community interaction design: communicate value prop to the user, avoid effects of competition, guilt, and other negative impacts
• interplay between layers, self-reflection on individual awareness and interests, tabulating issues, understand effort required at each layer

Posting for Community and Culture: Design of Interactive Digital Bulletin Boards by Claude Forth

• what types of content are on non-digital boards, how do they impact the community? classify postings
• 59 bulletin boards, 1,297 postings
• findings: geographic relevance, contextual relevance, aesthetic aspects; postings were highly local; postings highly related to the purpose of the place where board is located such as cultural or entertainment; more personal ads on outdoor and commercial boards; affordances are important to cause action; tangibility and texture; lighting and contrast; controlled boards were much more neatly arranged than uncontrolled; empty boards stay empty, messy ones change often; spill beyond board surface; modern spaces don't have boards, so no community or ownership; retail shops have boards with their own identity
• conclusion: physical and local instead of virtual and global

I Can Wait a Minute: Optimal Delay Time for Public Display Content by Miriam Greis

• many boards don't yet have user-generated content yet; often only in university settings
• does it need moderation? risk of innapropriate content; what delay would be tolerated and what effects does it have?
• expectations: 83% of participants think moderation is needed, but expect to appear instantly; without moderation, willing to wait 1 minute, with informed of moderation 60% would wait longer
• research app: display 12 recent tweets to handle; no notice of moderation; 0, 30, or 90 second delays; 519 messages from 95 users
• findings: longer delays led to less posting, but didn't affect likelihood of additional postings

Visualizing Dynamic Networks with Matrix Cubes by Benjamin Bach

• dynamic networks: networks that change over time, eg social networks, regional brain signals, migration flows, messages between people and systems
• how can we help people understand changes over time? much research over time with different representations supporting different tasks; would like to integrate these views while keeping it simple and powerful
• matrix cubes: node-link diagrams + matrices; matrix per timeslice stacked by time; 2 dimensions of nodes and 1 of time; more of a mental model for overview and pivot control, not a 3D visualization because of problems like occlusion; project 2D reps using cube to manipulate and explore; filtering, slicing, etc
• evaluation: antenna network domain + brain signals domain; quickly understood, liked consistency, animated transitions helpful, like linking and filtering, views for particular task, first legible visualization

A Table! Improving Temporal Navigation in Soccer Ranking Tables by Charles Perin

• soccer has lots of spatial-temporal events, results, and ranking tables
• ranking tables change each week; how to improve ranking navigation?
• analyzed 44 tables; 51 result articles; 33 temporal tasks; weighted tasks by importance and popularity
• drag-cell: drag in cell to change time and update data
• viz-rank: select cell to expand into line charts showing multiple metrics over time
• evaluation: basic tasks effective and faster for many tasks; more accurate but slower for complex tasks; can mix simple and advanced interaction techniques in same table, discoverability and learnability remain problematic, empower legacy visualization techniques

Kinetica: Naturalistic Multi-touch Data Visualization by Jeff Rzeszotarski

• multi-variate data interaction with touch on mobile devices; post-WIMP/naturalistic interfaces; exploit innate human capabilities
• ****** visual sedimentation for log data
• Kinetica: naturalistic metaphors to explore data; fun and intuitive; data are pseudo-tangible objects, physics, fluidity
• interrogation vs manipulation tools; force-based plotting; sifting; size and highlighting; merge points into groups
• benefits: rich mental models of information; intuitive awareness of amount and distribution; outliers stand out; physical traces of filtered points, minimize training for new users
• compare performance in excel vs Kinetica: car buying task + open ended task based on titanic data; newcomers could work with even 5 dimensions or more; excel users had point and statistic findings but kinetics had comparative findings

Traffigram: Distortion for Clarification via Isochronal Cartography by Sunghoo Hong

• spatial distance is not equal to spatial accessibility; hard to estimate time from distance, especially on a map; add temporal dimension to map via isochronal cartography; also show change over time; can this benefit users in a usable way?
• Traffigram: side by side physical map + distorted for travel time and ability to select time to display on maps; user in center to show time to other places
• evaluation: 12 local students and 13 distant to control for familiarity; 4 tasks; projection sparseness leads to trouble guessing contorted positions, but too complex can confuse, so must find right balance; faster and more accurate than using google map with traffic

• work with carriers and advertisers to enable micro payments for services and data collection via mobile devices, especially in emerging markets
• mobile phones are huge enablers in developing countries
• $24 for a low-cost android phone: 3G WiFi GPS touchscreen
• 1.35 billion android users vs 1.3 billion windows users worldwide
• more than 50 of internet users are in developing countries; Facebook on track to earn majority of revenue in emerging markets in next few years
• mobile operator data: CDR; track normal patterns in deidentified data; can detect unusual events, like earthquakes and disease outbreaks, based on deviations from normal; drive policy decisions and disaster response
• small data: predict cholera outbreaks in Rwanda a week prior; not detecting cholera but roads washing out; need more than CDR for correlation to reality
• issues of consent
• prepaid airtime is 8-12% of daily wage; looking to offset costs by having people participate in data and research and content; marketing spend can offset costs; redirect $200 billion away from 1950's ads like billboards to mobile device interaction and help people make money; good for carriers, advertisers, and individuals

Automatic Generation of Semantic Icon Encodings for Visualizations by Vidya Setlur

• examples of using icons for data points on visualizations
• why use icons? Represent semantics, easier to parse spatially and visually, better aesthetics, but takes time to find or create icons; can we automate it?
• context establishment->query construction and image retrieval->global feature clustering; discover icons with semantic meaning to the data then cluster by similarity of visual representation
• used Mechanical Turk to evaluate automated icon set versus hand-picked, the automated set were preferred for general media concepts but less for domain specific sets or in a narrow area where auto generated icons were too realistic and hard to tell apart; Turkers like round icons
• Q: surprised that word net algorithm made such good choices for semantics; how? 'symbol' is very well represented in WordNet and Google has good search semantics to instrument
• Q: in other domains would it be this good? send the data; it works without manual intervention unless domain too narrow or hard to represent
• Q: plan to look at user-guided vs. automated or user-generated? good idea

Task-Driven Evaluation of Aggregation in Time Series Visualization by Danielle Albers

• certain visualizations are better for specific visualization tasks, but not strong guidelines available
• visual aggregation tasks: summary info in display regions; point vs summary comparisons; avoid visual clutter
• design variables: visual, mapping, computational
• compared tasks using Mechanical Turk; monthly sales data set with controlled variability; did find correlations; eg position better at point, color at summary, and box plots better than line charts for minima (surprised the researchers); users can find insights in complex data sets using visualizations
• Q: can we overlay things and get one chart for all tasks? likely not, but insights in the better choices to make in design; multiple views supports the many micro tasks

Dive In! Enabling Progressive Loading for Real-Time Navigation of Data Visualizations by Michael Glueck

• interaction transaction: user provides input and system responds; stepped vs continuous interactions
• current data exploration uses mostly stepped interactions because of large sets
• Splash Framework: explore, continuous, and instantaneous; progressively load data; well known but little used; progressive loading is hard to do right; most research scientists are not expert programmers; abstract levels of detail; improves performance in use; works with existing visualizations
• Splash Aggregator->Data Transport->Splash Cache
• compared progressive vs non-progressive representations on coarse, global, and fine features; progressive is roughly equivalent in all cases and better in most low bandwidth scenarios (except fine features); helps maintain “real-time” interaction (<200ms)
• detail can be tuned by data curator, and researchers were successful at choosing a good tuning level
• Q: what are the limitations? simplified level of detail settings, no semantic hierarchies, no dynamic elements, however supports many data types

Sample-Oriented Task-Driven Visualizations: Allowing Users to Make Better, More Confident Decisions by Niven Ferreira

• uncertainty is often ignored in visualizations; adds complexity to tasks and experts have trouble interpreting it; how can we improve understanding of uncertainty?
• important tasks: retrieve data, find extrema, find range; incorporating uncertainty: how likely? sorting and ranking tasks as well
• design guidelines: easy to interpret, consistency across tasks, stability across sample sizes, minimize visual noise; simple, light-weight, annotations, smooth transitions, natural representations
• eg use color to overlay the uncertainties when selecting a bar; use pie chart to represent likelihood of extrema; probability of ranking in list
• tested how these encodings worked in real-world analysis; 7 participants, 75 questions, measure answer + timing + confidence; all were comfortable, didn't have to think much, rank/sort was complex, sample size greatly influenced confidence, not significantly slower when using them, we're not more accurate however with these simple questions (easy to guess), correctness and confidence correlated

Mining Online Software Tutorials: Challenges and Open Problems by Adam Fourney

• tutorials are a rich resource for procedural informatio
• want to extract structured semantic data to identify timing, tasks, etc
• can automatically detect commands and interface elements, but nowhere near automated
• at least a dozen problems: coreference resolution, spatial reasoning, purpose clauses, parameters and values, etc
• parameters and values: underspecified, constrained
• anti-patterns: some instructions are what not to do
• titles and topic phrases: narrative content beyond steps, not operations
• just enumerating the problems advances the work: common vocabulary, sub-problems for tailored approaches (using machine learning or crowd-sourcing)
• Q: why is automation interesting? not a goal in itself, but leads to enough understanding to use in other ways
• Q: anti-patterns, negated can't be detected? an example where identifying it may help address the problem

HCI Over Multiple Screens by Caroline Jay

• collaboration with U Manchester and BBC
• people watch TV with more than one device; broadcasters would like to develop content and experiences for companion devices
• most research has focused on social context, but not cognitive details of attention
• lean forward: web and social media; lean backward: newspaper, film, TV; multiple screens allows combinations
• observation and uncontrained interaction with eye tracking
• challenges: track movement over two screens, ecological realistic/valid; found good tracking calibration, eye tracking and video analysis matched well but hands may occlude eye tracker, good match with and without eye tracking
• split attention across two screens: tablet fixation when topic changes, updates and action (to tablet “where's the dolphin? to TV “there, there, there, there!”)
• additional challenges: eye tracking only usable in lab, investigating logging on device; many other factors like interaction, content, environment; even if we can do this, what about privacy?
• Q: is this primarily for filling in gaps in attention? individual interests influence the attention but that's not the primary goal

A Funny Thing Happened on the Way to the Website: Telling About Browsing by John Fass

• how do we derive meaning from our browsing history?
• people average 30s per page; tolerate 450ms for a web site to load, same as human blink; many tabs; multiple devices
• browser history is a list like many other elements of web content; easy to implement but life is more of an interrupted journey; many attempts at visualizing the journey over the last 20 years but no real change in browsers
• “machine” (browser and cloud services) is learning about your behavior, much more than you; don't really know where it goes, and it's algorithmic
• human meaning making: narrativity, continuous and episodic time cross-over, framing effects
• narrative: story (events over time), versions (texts), fabula (affect of the characters)
• people think of their browser history experientially
• people understand the richness of multiple aspects; need to be able to forget things
• visual comic-like layouts, curation, and collections
• also discovered conductive ink; maybe we could interact physically with our browser history
• Q: what motivated you? do the research in public with diverse participation; learned that it's very hard to get a computer to do a human story
• Q: did you see people imagining the future when reflecting on their past? internet is so fast it's hard to look far back in time to effectively imagine future
• Q: curation may delete unpredictable serendipity? yes, serendipity is another area we may want to facilitate
• Q: how might visual communication inform our work? this conference is very technical, visual design comes from a very different place; hybrid systems are very interesting

Translation from Text to Touch: Touching a “Japanese Old Tale” by Susuki et al

• tactile sense is non-verbal communication: signal (single tap on shoulder vs double), language (massag)'
• text has tactile impression: how can we extract tactile senses from text? text/words + emotion = tactile impression; eg strong/angry words are short and clipped, weak leads to many words
• create tactile score with notation like musical score
• translate long story: used natural language processing to break into chunks and coun syllables; created tactile score from counts; English translation was very different; used tactile score to translate into dance, haptic device, music and graphics; did performance based on all those translations; have also created tactile books
• “The Massage is the Medium.”
• Q: does translating to the tactile score leave out some of the richness of touch? massage therapists have a 46 element language that is used for the tactile score, so it is quite rich

Discussion: How does your research mesh with each other?

• Semantic analysis with a machine is difficult, but humans understand the narrative easily.
• HCI is traditionally so focused on tasks, and we're now starting to deal with so many new types of interactions, and we don't know what it may lead to.
• Maybe this is about bringing meaning back into our interactions with computers and algorithms, searching for ways to include and retain meaning.