Design and Implementation of Online Braille Math Support in the Accessible Equation Editor Sam Dooley Pearson Assessment Dan Brown Pearson Assessment Susan Osterhaus Texas School for the Blind and Visually Impaired Braille math is hard! Blind students need high-quality braille math Advanced preparation is costly and takes time Math teachers are often unfamiliar with braille

2 TVIs are often unfamiliar with math notation Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Braille math software is harder! 3 Software translation can be problematic Forward translation takes many steps Real-time back translation is non-existent

No support for online or classroom use Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Braille math needs to be online! Blind students need: 4 A level playing field for STEM instruction To read and write online braille math To interact with sighted instructors and peers

To participate in online activities Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Braille math should be math! Math concepts are independent of notation Braille math codes capture all math notation Math software can be independent of notation Blind students only have full access to math if their math is treated the same as printed math.

5 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Classroom challenges for blind math students 6 They cannot read math the teacher can write They cannot write math the teacher can read Speech solutions incur high cognitive load

They need to read and write for themselves Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Braille math translation Visual math software is not truly accessible MS Word, MathType, Scientific Notebook Braille math translation software is not instantaneous is not totally accurate

LaTeX, Duxbury, HIMS, HumanWare 7 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Online Braille Math with the Accessible Equation Editor 8 Sighted user can create math for a blind user Blind user can create math for a sighted user Real-time, two-way braille math translation

Instantaneous interactions with math content Discoverable braille math encoding rules Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Nemeth Braille (1952) 9 Abraham Nemeth (AFB, APH, BANA) Tactile encoding for print math

Math for technical publications Presentational math structures Consistent with content markup Design and Implementation of the Accessible Equation Editor 10 Apr 2016 TeX/LaTeX (1978/1985) Donald Knuth, Leslie Lamport Typesetting language for print math

Math for technical publications Presentational math structures Fully programmable macro language 10 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 W3C MathML (1998) World Wide Web Consortium (W3C) XML element/attribute vocabulary

Typical K-12 and higher ed. math Presentation and content markup 11 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Presentation v. Content Presentation encodes signs/symbols Content encodes functional structure "x^2+1" v. "1346, 45, 23, 5, 346, 2"

"x^2+1" v. (plus (power x 2) 1) 12 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 QWERTY keyboard input Math symbols on the keyboard Other symbols on the buttons Keyboard cursor navigation

Implicit and explicit selection Backspace, delete, clear 13 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Braille terminal output Math output on the screen Braille output on the screen Braille output on the terminal

Screen reader support 14 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Braille terminal input Each braille key is treated like a keyboard key Sequences of braille keys create the math Button tooltips with text and braille Input cursor position and routing

15 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 QWERTY terminal output Braille input creates math content Math content becomes math output Math output becomes braille output The math markup is exactly the same 16 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Braille math is math! Blind students can read the same math Blind students can create the same math The math can be shared the same way The math can be scored the same way Blind students can now have full access to math since their math is the same as printed math. 17 Design and Implementation of the Accessible Equation Editor 10 Apr 2016

Equation Editor Architecture Syntax expression trees Expression tree transformations Configuration and customization Universal design decision points 18 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Expression trees

An operator with its child arguments The operator as the initial child argument Parse trees, syntax expressions, XML elements Often annotated with semantic attributes Functional programming languages (Lisp) 19 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Content expressions

The functional structure of a math expression Does not itself define any semantic behavior The operators carry the semantic behaviors Operator, child elements, child attributes Direct translations to and from XML markup Boxed reference to presentation expression

20 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Presentation expressions The visual notation for a math expression Does not itself define functional structure The operators carry the notational schema Operator, child elements, child attributes Direct translations to and from XML markup

Boxed reference to content expression 21 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Expression references Math document user interface elements Content node matched with a presentation Pointers to child expression references

References to the browser display nodes Resolves from DOM to presentation to content 22 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Browser display nodes Browser document user interface elements Direct references into the browser DOM tree Attached pointers into content, presentation

Listeners for math content change events Wrap nodes not owned by the equation editor 23 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Expression trees are universal! Generic encodings for math formulas Separating notation from function Separating interface from function

Enabling automated regression testing Universal design for math behaviors 24 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Transformation architecture Input transformations create expressions Output transformations create XML markup Layouts transform content to presentation

Templates create new content from old Generic rewrite rules on expression trees Enabling data-driven math behaviors 25 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Configuration architecture External data defines conventional behaviors

Allows highly customizable math interactions Fine control for solving math usability issues Allows reconfiguration for new populations Layouts, templates, buttons, palettes, keys 26 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Layout transformations Transform structure into notation

Structure encoded as Content MathML Notation encoded as ... pick your target! Layout rules encoded as custom XML Source pattern as Content MathML Output pattern as layout schemata 27 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Content to Presentation

Math operators determine layout rules Minimal contextual dependencies Presentation links back to content (plus x 1) v. "x + 1" 28 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Content to Braille

Math operators determine braille symbols Additional contextual dependencies Flattening to linear output format (plus x 1) v. "1346, 346, 2" ⠬ 29 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Template transformations

Create new structure from old structure Template rules encoded as custom XML Template names and key event names Source pattern as Content MathML Output pattern as Content MathML Template variables control rewriting

30 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Keyboard to Content Keys and button names select template rules Minimal sequential dependencies Presentation updated on each key "x + 1" v. (plus x 1)

31 Content MathML Markup from Nemeth Braille Input 23 Mar 2016 Braille to Content Braille cells indirectly select template rules Non-trivial sequential dependencies Presentation updated on each key "1346, 346, 2" v. (plus x 1)
32 Content MathML Markup from Nemeth Braille Input 23 Mar 2016 Braille key transformations Transform braille cells to template names Braille cells are received as ASCII braille Each braille cell assigned a symbolic name Symbolic names for braille cell sequences

Braille input mode transforms to key names Previous template rules operate unchanged 33 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Braille cells to keys Single cells that appear as keys Single cells that cause actions Sequences that cause actions

Sequences that extend others 34 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Braille finite state machine Symbolic names for braille prefixes Symbolic names identify input states

Braille cells identify state transitions Accepting input states invoke templates Subsequent states may replace inputs 35 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Universal design for online math Expression trees provide universal data Data-driven universal tree transformations

Flexible configuration customization rules Universal presentation output notation Universal input event transformations 36 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Braille Input Examples Numeric indicator Baseline indicator

Type form indicators Shape indicators Negated operators Composed relations 37 Content MathML Markup from Nemeth Braille Input 23 Mar 2016 Braille Input Testing Web browser input test page

JavaScript equation editor Screen reader device drivers Web accessibility APIs Braille terminal input 38 Content MathML Markup from Nemeth Braille Input 23 Mar 2016 Research Studies Two research studies in Fall 2015

(KY/AZ - Sep, TX - Oct) Goal - to collect feedback from multiple populations on EE functionality Populations - blind, low vision, regular print readers, learning disabled Criteria - high school students who had completed Algebra I 39 Content MathML Markup from Nemeth Braille Input 23 Mar 2016 Research Outcomes

Students had limited knowledge of Nemeth Erasing math content was unpredictable Working with grouping symbols was difficult Entering and closing fractions was unexpected Ending trigonometric expressions was unusual 40 Content MathML Markup from Nemeth Braille Input 23 Mar 2016 Advanced preparation

Create math content in advance Share with students by email Include math content with slides Students use the equation editor 41 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Extemporaneous content Teacher uses the equation editor

Projects the screen to the class Blind student uses braille terminal Everyone can read the same content 42 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Shared presentations WebEx, Google Hangouts, etc.

Shared screens for visual users Individual work shown to the class Blind students can show their work Blind students viewing others work 43 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Online assessments Common Core, PARCC, TestNav

Spring 2015 admin (15m responses) Machine scoring for constructed math Blind students can use online forms Blind students create online responses 44 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Nemeth Braille curriculum Interactive Nemeth braille exercises

Immediate practice for lessons learned Experimentation with braille concepts Discoverability for unfamiliar encodings 45 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Homework assignments Blind student uses the equation editor Email homework files to sighted teacher

Teacher reads using the equation editor Teacher marks up the math content Returns marks to the blind student 46 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 TVI support The role of the TVI remains essential

The technical work load can be reduced The software can help assist the process Discoverability for print math notations 47 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Math teachers The role of the math teacher is vital The technical work load can be reduced

Engaging directly with blind students Discoverability for braille math encodings 48 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Group interactions Effective group work with blind students Each student has an opportunity to contribute Each can learn from the work of others

The software can help assist the process 49 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Online Accessible Math! Blind students can have: A level playing field for STEM instruction Tools to read and write online braille math Interaction with sighted instructors and peers

Active participation in online activities 50 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Expression trees are universal! Generic encodings for math formulas Separating notation from function Separating interface from function Enabling automated regression testing

Universal design for math behaviors 51 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Braille math is math! Blind students can read the same math Blind students can create the same math The math can be shared the same way The math can be scored the same way

Blind students can now have full access to math since their math is the same as printed math. 52 Design and Implementation of the Accessible Equation Editor 10 Apr 2016 Accessible Equation Editor Try it out, give us feedback http://accessibility.pearson.com/mathex-app/ Please let us know how you would use it [email protected] [email protected] [email protected] 53 Design and Implementation of the Accessible Equation Editor 10 Apr 2016