Curriculum Vitae

Mary Mark Ockerbloom
(Mary A. Mark, M.Sc.)

130 W. Nippon St.
Philadelphia, P.A.
U.S.A. 19119
Phone: (215) 248-1248
Email: celebration.women@gmail.com


"The knowledge engineer is part student, part teacher, part consultant, part model builder, and part programmer."
Harmon, Maus & Morrissey, Expert Systems Tools and Applications, 1988.


Experience:

PROJECT LEADER, CELEBRATION OF WOMEN WRITERS

Mary Mark (Ockerbloom)
1994 - present (Part-time)
URL: http://digital.library.upenn.edu/women/
Blog: http://merrigold.livejournal.com

I maintain "A Celebration of Women Writers" website, listing 19,000+ women authors, 13,000+ author information links, and 10,000+ freely readable on-line books. The inclusion of Library of Congress authorized and variant names, birth and death dates, countries where authors lived, and ethnicity in the author database supports complex searches targeting particular areas of interest, such as Jewish women writers in Germany in the 1940s. I have also republished over 370 books by women, working independently and with volunteers. Particular areas of interest include women travelers, utopian fiction, and Newbery award winners.

Operating Systems: Mac OS X, Windows ; Languages/Tools: HTML5, CSS3, python, javascript, Photoshop, various software products for scanning and OCR

HEAD CORPUS LIBRARIAN, WORDNIK.COM

Supervisor: Erin McKean, Wordnik Founder (erin@wordnik.com)
Telecommuting to New York, NY and San Mateo, CA
2008 - 2011 (Full-time)
URL: http://wordnik.com

At Wordnik.com my goal was to build a broad-ranging corpus of sentences illustrating English language use for the Wordnik dictionary. I was responsible for outlining the company's initial metadata specifications, finding and harvesting online data sources for Wordnik , and extracting metadata. Data sources included online books, government and legal records, oral history archives, movie scripts, song lyrics, poetry, and blogs. I was also involved in testing and providing feedback for Wordnik.

Operating Systems: Mac OS X, Windows; Languages/Tools: HTML, Perl, wget, Mantis

RESEARCH PROGRAMMER, INTELLIGENT TUTORING SYSTEMS

Supervisor (1992-1995): Dr. John Anderson (ja+@cmu.edu)
Department of Psychology
Supervisor (1995-1998): Dr. Ken Koedinger (koedinger@cmu.edu)
Computer Science Department
Carnegie-Mellon University, Pittsburgh, PA
1992 - 1997 (Full-time)
1997 - 1998 (Part-time)

In John Anderson's lab, I was the primary research programmer for the Grade Nine Algebra Word Problem Tutor. I worked with Bill Hadley, a curriculum designer and teacher in the Pittsburgh Public School system, to develop a tutor to fulfill the objectives of his PUMP Algebra Curriculum (Pittsburgh Urban Mathematics Project). I worked on implementation, content development, testing, debugging, and evaluation. The project involved working with other programmers and designers in the lab, and with teachers and students in the Pittsburgh public schools. I developed tools for evaluation of student protocol data, and carried out evaluations of tutor effectiveness. The Algebra Word Problem Tutor became a core product for the startup Carnegie Learning Partners, which was recently purchased by the Apollo Group.

Languages/Tools: Lisp, ACT Tutor Development Kit (ACT production system language), HTML


Student Work Experience:

[show]

COMPUTER PROGRAMMER

Faculty of Education
Simon Fraser University
Burnaby, British Columbia
Supervisors: Dr. Phil Winne (604) 291 - 4858 (winne@whistler.sfu.ca)
Nov. 1, 1991 - March 1, 1992

I was hired by Dr. Winne to do further work on the STUDY project. This included developing a demonstration module in the domain of education, as well as testing and documenting the STUDY system and developing user support information.
Operating Systems: Macintosh IIx; Languages/Tools: Nexpert Object, Hypercard, Superpaint

RESEARCH ASSISTANT

Learning and Collaboration Group
Alberta Research Council
3rd Floor, Digital Building
6815 - 8 Street North East
Calgary, Alberta
Supervisors: Dr. Marlene Jones (403) 297 - 2600 (marlene@arcsun.arc.ab.ca)
Janet McCracken (403) 297 - 7584 (janet@arcsun.arc.ab.ca)
June 1, 1991 - Sept 15, 1991

As a senior summer student at the Alberta Research Council, I was involved in several projects, requiring a variety of skills. As a major project, I developed an instructional module for WHMIS (Workplace Hazardous Materials Information System) using the STUDY system. STUDY is a prototype authoring and delivery environment based on Nexpert Object and Hypercard. I also obtained and documented demonstration versions of AI systems for instructional authoring and delivery from developers in North America and Europe.
Operating System: Macintosh IIfx; Languages/Tools: Nexpert Object, Hypercard, Superpaint

COMPUTER PROGRAMMER

Sci-Tec Instruments, Inc.
1526 Fletcher Road
Saskatoon, Saskatchewan
Supervisors: Lee James (306) 934 - 0101
May 1, 1990 - Aug. 31, 1990

I was one of several programmers developing an expert system for the installation and monitoring of fluid containment tanks. The expert system was written in Prolog and integrated with a tank maintenance system written in C.
Operating System: Unix; Computer Languages: Prolog, C

RESEARCH ASSISTANT (part-time)

Education Department
University of Saskatchewan
Saskatoon, Saskatchewan
Supervisors: Dr. John McLeod (306) 966-5245
Dr. Marlene Jones (403) 297 - 2600 (marlene@noah.arc.ab.ca )
Nov. 1, 1988 - Sept. 30, 1989

I replaced another programmer in the third year of a three year project . The project's goal was to develop an expert system for diagnosis of learning problems in children. I synthesized the existing work, extended it, and wrote final research reports and documentation for the project.
Operating Systems: MSDOS (PC); Computer Languages: Prolog

RESEARCH ASSISTANT

ARIES Laboratory
Computer Science Department
University of Saskatchewan
Saskatoon, Saskatchewan
Supervisors: Dr. Gord McCalla (306) 966 - 4886 (mccalla@cs.usask.ca)
Dr. Jim E. Greer (306) 966 - 8655 (greer@cs.usask.ca)
Feb. 1, 1988 - Oct. 14, 1988

I helped develop a diagnostic system for identifying strategies used by students when writing Lisp programs. I implemented a prototype system using Lisp and the ART Inference Engine for rule-based inference. I was also responsible for maintenance of the Symbolics Lisp machines used in the project, and for supervision of undergraduate and masters students who were learning to use them.
Operating Systems: Symbolics Lisp; Computer Languages: Symbolics Lisp

COMPUTER PROGRAMMER

NSERC Undergraduate Summer Research Award
Computer Science Department
University of Saskatchewan
Saskatoon, Saskatchewan
Supervisor: Dr. Gord McCalla (306) 966 - 4886
May 1, 1987 - Aug. 31, 1987

As a summer student, I reviewed the computer science research literature on natural language generation, and developed a program for generating a restricted subset of natural language.
Operating System: Suntools (Sun); Computer Languages: Franz Lisp.

COMPUTER PROGRAMMER

Softart Microsystems
421 King Street
Waterloo, Ontario
Supervisor: Richard McMurray (519) 746 - 6750
May 12, 1986 - Sept. 12, 1986

I converted a package of computer development utilities to C, and worked on the development of a communication protocol for computer networks.
Operating System: Unix; Computer Languages: C.


Degrees:

Master of Science in Computational Science 1991
University of Saskatchewan, Saskatoon, Saskatchewan, CANADA

Advanced Certificate in Science in Computational Science 1988
University of Saskatchewan, Saskatoon, Saskatchewan, CANADA

Bachelor of Arts High Honours in Psychology 1984
University of Saskatchewan, Saskatoon, Saskatchewan, CANADA


Scholarships and Awards:

[show]
University of Saskatchewan
Saskatoon, Saskatchewan

Theses:

Mark, M.A. The VCR Tutor: Design and evaluation of an intelligent tutoring system.
Master's Thesis, University of Saskatchewan, Saskatoon, Saskatchewan, 1991.
Advisor: Prof. J. E. Greer, Department of Computational Science. [Abstract]
This thesis describes the design, implementation and evaluation of a tutoring system for instruction in device operation. While design and implementation are important components of this research, a major focus is evaluation. Evaluation is used to compare versions of the system and examine psychological and educational theories underlying system development.

The purpose of the tutoring system is to teach an adult how to program a video cassette recorder (VCR) to automatically record television programs. Actions involved in operating a VCR are procedural, but their execution may depend upon cognitive knowledge of the device, its behaviour, and the relationships between device features and device actions. Questions about the relationship of knowledge to learning in the VCR domain include: "Does the domain involve only procedural knowledge, or factual and referential knowledge as well?" "Will knowledgeable feedback in a computer-based instructional system support learning more effectively than other types of feedback?" "How should instructional feedback be designed to best facilitate learning of procedures in this domain?"

To examine these questions, prompting, sequence-based, device-based, and knowledgeable versions of a VCR tutoring system have been implemented. They incorporate different tutorial approaches and types of knowledge. An experimental study has compared the effectiveness of these versions. Subjects who used the knowledgeable tutor learned to program a VCR using fewer steps and with fewer errors than those who used the prompting version. Students with knowledgeable tutoring showed fewer types of errors than prompting students and completed the testing task more quickly. Subjects who used the device-based version of the tutor executed the task with fewer types of errors than prompting students, but took significantly longer than prompting students to learn the task.

These results have implications for the design and development of instructional systems for the VCR domain. They support the view that the VCR programming domain involves factual and referential as well as procedural knowledge. They demonstrate that knowledgeable feedback supports learning in the VCR domain more effectively than other types of feedback, and they suggest that the presentation of knowledgeable feedback is a desirable design for a tutoring system in this type of domain.

Mark, M.A. Effects of insulin on food intake in the rat and the thirteen-lined ground squirrel: A comparative study.
Bachelor of Arts Honours Thesis, University of Saskatchewan, Saskatoon, Saskatchewan, 1984.
Advisor: E.K. Walls, Department of Psychology.


Invited Talks:

Mark Ockerbloom, Mary. A Celebration of Women Writers. Presentation for the Pennsylvania Library Association Centennial Conference, "Digital Projects Across the Commonwealth", 2001. [Slides]

Publications:

Mark Ockerbloom, Mary & Mark Ockerbloom, John. "Women Writers and On-Line Books." In Still, Julie (Ed.) Creating Web Accessible Databases: Case Studies for Libraries, Museums and Other Non-Profits, pp. 40-52. Medford, N.J. : Information Today, c2001.

Mark, M. A., & Koedinger, K. R. "Strategic Support of Algebraic Expression-Writing." Proceedings of the Seventh International Conference on User Modeling. Banff, Alberta, Canada, June 20-24, 1999. [Paper] [Abstract]
In completing a problem with the PAT Algebra I Tutor, students solve a number of algebraically related tasks. We have observed a characteristic pattern of students' success rates at these tasks. We have also observed that students' success on specific skills may differ depending on whether students previously carried out algebraically related activities in the same problem. Finally, the sequences in which tasks are completed may indicate overall problem-solving strategies of the student. These observations have important implications for the future design of the PAT Tutor and its user model.

Mark, M. A., Koedinger, K. R., & Hadley, W. H. "Elaborating Models of Algebraic Expression-Writing." In Goettl, B. P., Halff, H. M., Redfield, C. L., & Shute, V. J. (Eds. ) Intelligent Tutoring Systems, Proceedings of the Fourth International Conference (San Antonio, Texas, August 16-19, 1998). Lecture Notes in Computer Science, 1998, Volume 1452/1998, 524-533. Berlin: Springer-Verlag. [Paper] [Abstract]
This paper discusses the refinement of the algebraic expression-writing rules for the PAT Algebra I tutor, between 1992 and 1997. Direct observation of students in class, step-by-step PAT tutor protocols tracing student behavior, and statistical analysis of protocol data, have all informed our understanding of students' skills, and our refinement of this part of PAT's production model.

Koedinger, K.R., Anderson, J. R., Hadley, W. H. & Mark, M.A. "Intelligent tutoring goes to school in the big city." International Journal of Artificial Intelligence in Education (Special Issue of AI-ED 95 best papers), 1997, 8, 30-43. [Paper] [Abstract]
This paper reports on a large-scale experiment introducing and evaluating intelligent tutoring in an urban High School setting. Critical to the success of this project has been a client-centered design approach that has matched our client's expertise in curricular objectives and classroom teaching with our expertise in artificial intelligence and cognitive psychology. The Pittsburgh Urban Mathematics Project (PUMP) has produced an algebra curriculum that is centrally focused on mathematical analysis of real world situations and the use of computational tools. We have built an intelligent tutor, called PAT, that supports this curriculum and has been made a regular part of 9th grade Algebra in 3 Pittsburgh schools. In the 1993-1994 school year, we evaluated the effect of the PUMP curriculum and and PAT tutor use. On average, the 470 students in experimental classes outperformed students in comparison classes by 15% on standardized tests and 100% on tests targeting the PUMP objectives. This study provides further evidence that laboratory tutoring systems can be scaled up and made to work, both technically and pedagogically, in real and unforgiving settings like urban high schools.

Koedinger, K.R., Anderson, J. R., Hadley, W. H. & Mark, M.A. "Intelligent tutoring goes to school in the big city." Proceedings of the Seventh World Conference on Artificial Intelligence in Education. Washington, D.C., August 16-19, 1995. (Awarded 1 of 2 Best Paper Awards.) [Abstract]
This paper reports on a large-scale experiment introducing and evaluating intelligent tutoring in an urban High School setting. Critical to the success of this project has been a client-centered design approach that has matched our client's expertise in curricular objectives and classroom teaching with our expertise in artificial intelligence and cognitive psychology. The Pittsburgh Urban Mathematics Project (PUMP) has produced an algebra curriculum that is centrally focused on mathematical analysis of real world situations and the use of computational tools. We have built an intelligent tutor, called PAT, that supports this curriculum and has been made a regular part of 9th grade Algebra in 3 Pittsburgh schools. In the 1993-1994 school year, we evaluated the effect of the PUMP curriculum and and PAT tutor use. On average, the 470 students in experimental classes outperformed students in comparison classes by 15% on standardized tests and 100% on tests targeting the PUMP objectives. This study provides further evidence that laboratory tutoring systems can be scaled up and made to work, both technically and pedagogically, in real and unforgiving settings like urban high schools.

Mark, M.A. & Greer, J.E. "The VCR tutor: Effective instruction for device operation." Journal of the Learning Sciences, 1995, 4(2), 209-246. [Paper] [Abstract]
A critical issue for intelligent tutoring systems is whether knowledgeable diagnosis and feedback offers instructional benefits compared with less knowledge-intensive instructional methods. The effectiveness of different instructional methods is examined in a device operation task: setting a VCR to record a program. Four tutoring simulations use different instructional approaches emphasizing specific types of knowledge. The possibility of acquiring this knowledge is manipulated by increasing opportunities to interact with the simulation and by explicitly providing knowledgeable diagnosis and feedback. Eighty novice adults were randomly assigned to the four tutors for training. Planned comparisons of learner achievement indicate greater learning with knowledgeable diagnosis and feedback, compared to correctness feedback. Analysis of variance to detect predicted linear trends shows increased instructional effectiveness across versions. Results are not simply a function of differences in training time. They support the contention that knowledgeable diagnosis and feedback can offer benefits over other instructional methods.

Mark, M.A. & Greer, J.E. "Evaluation methodologies for intelligent tutoring systems." Journal of Artificial Intelligence in Education (Special Issue on Evaluation), 1993, 4 (2/3), 129-153. [one of the top 20 most referenced papers of the International Journal of Artificial Intelligence and Education] [Paper] [Abstract]
As intelligent tutoring system (ITS) issues are investigated and intelligent tutoring systems are developed, evaluation methodology becomes important. Basic researchers, system developers, and educators working with ITS all have motives for becoming involved in ITS evaluation. In formative evaluation, researchers examine a system under development, to identify problems and guide modifications. By contrast, summative evaluation is carried out to support formal claims about the construction, behaviour of, or outcomes associated with a completed system. Different methodologies are suitable for different types of evaluation, some focusing on internal considerations, such as architecture and behaviour, others on external considerations, such as educational impact.

This paper draws upon the areas of intelligent tutoring systems research, expert systems design, computer-based instruction, education, and psychology to identify techniques for the formative and summative evaluation of ITS. Evaluation techniques are discussed in terms of the motivations for using them, situations for which they are particularly suitable, and their strengths and weaknesses. Techniques are illustrated with reference to actual studies whenever possible.

Winne, P.H., Butler, D.L., McGinn, M., Sugarman, J.H., Jones, M., Mark, M., & Field, D. "STUDY: A tool for authoring adaptive learning environments and for advancing instructional research." ITS '92: System Demonstrations (Second International Conference on Intelligent Tutoring Systems). Montreal, June 10-12, 1992. [Abstract]
STUDY is a tool for authoring Adaptive Learning Environments. We refer to the system as state-of-the-art because it is based fundamentally on knowledge of computing and instructional psychology that is available now. Rather than offering a revolutionary re-definition of education or implying radically new views of learning and the learner, STUDY has been designed and implemented to step modestly beyond high quality instruction that can be provided by a superlative textbook and its accessories. To achieve this goal, STUDY can be used to create tutorials in which students develop more than just knowledge of the subject matter provided in the text. STUDY also serves as a valuable research tool for bootstrapping research regarding the nature of effective instruction in Adaptive Learning Environments.

Mark, M.A. & Greer, J.E. "Methods for intelligent tutoring system evaluation." Proceedings of the East West Conference on Emerging Computer Technologies in Education, 6-9 April 1992, Moscow, Russia. ICSTI: Moscow, Russia, 1992. Awarded 1 of 5 Best Paper Awards, for ITS-related papers presented at the conference. [Abstract]
As intelligent tutoring system (ITS) issues are investigated and intelligent tutoring systems are developed, the issue of evaluation begins to receive attention. In formative evaluation, researchers examine a system under development, to identify problems and guide modifications. Summative evaluation is carried out to support formal claims about the construction, behaviour of, or outcomes associated with a completed system. Different methodologies are suitable for different types of evaluation, some focusing on internal considerations such as architecture and behaviour; others on external considerations such as educational impact. This paper draws upon the areas of ITS research, expert systems design, computer-based instruction, education and psychology to identify techniques for the formative and summative evaluation of ITS. Evaluation techniques are discussed in terms of motivations for using them, situations for which they are particularly suitable, and strengths and weaknesses. Techniques are illustrated with reference to actual studies whenever possible.

Jones, M., Mark, M., Field, D., & Winne, P.H. The STUDY system: A learning environment to facilitate reading comprehension and studying. Paper presented at Survival Skills in the Nineties: A conference about media and information literacies, Edmonton, Alberta, November 1 - 2, 1991.

Mark, M.A. & Greer, J.E. "The VCR Tutor: Evaluating instructional effectiveness." Proceedings of the 13th Annual Conference of the Cognitive Science Society, August 7-10, 1991. COGSCI: Chicago, Illinois, 1991. [Abstract]
People use a wide variety of devices. Operation of a device can usually be described in terms of knowledge of specific procedural sequences. However, execution of procedures may also depend upon knowledge of the device, its behaviour, and the relationships between device features and device actions. A video cassette recorder (VCR) is one commonly used device. Programming a VCR to automatically record a chosen television program is an example of a device manipulation task. In designing a device tutor, it is relevant to ask how instruction about device operation should be designed, and to ask whether knowledge engineering for a device tutor should focus on procedural knowledge or involve factual and referential knowledge as well. Four versions of a tutoring system for the VCR device and programming task have been implemented, incorporating different tutorial approaches using different types of knowledge. The effectiveness of these versions has been examined experimentally. Subjects who used the knowledgeable tutoring version learned to program a VCR simulation using fewer steps and with fewer errors and error types than subjects who used a prompting version of the tutor.

Mark, M.A. & Greer, J.E. "Tutoring systems for device operation." Proceedings of the International Conference for Cognitive Science for the Development of Organizations (ICO '91), Montreal, May 2-4, 1991. [Abstract]
People use a wide variety of devices. Operation of a device can usually be described in terms of knowledge of specific procedural sequences. However, execution of such procedures may also depend upon knowledge of the device, its behaviour, and the relationships between device features and device actions. It is relevant to ask how instruction about device operation should be designed, and whether knowledge engineering for a device tutor should involve only procedural knowledge, or factual and referential knowledge as well.

A video cassette recorder (VCR) is one example of a commonly used device. Programming a VCR to automatically record a chosen television program is an example of a device manipulation task. Four versions of a tutoring system for this task and device have been implemented, incorporating different tutorial approaches and types of knowledge. The effectiveness of these versions has been examined in a one-way ANOVA with four experimental groups. Students who used the knowledgeable tutoring version were found to be most successful at carrying out the task. They used less steps to program the tutor, and programmed the tutor in less time, than any of the other groups.

Mark, M.A. "Evaluation of intelligent tutoring systems." Proceedings of the Second Annual Graduate Symposium on Computational Science, April 5, 1990, Saskatoon, Saskatchewan. Saskatoon, Saskatchewan: Department of Computational Science, 1990. [Abstract]
As intelligent tutoring system (ITS) issues are investigated and intelligent tutoring systems are developed, the issue of evaluation begins to receive attention. Basic researchers, system developers, and educators working with ITS all have motives for becoming involved in their evaluation. At different times in the life cycle of a project, researchers may be concerned with different types of evaluation. In formative evaluation, researchers examine a system under development, to identify problems and guide modifications. In contrast, summative evaluation is carried out to support formal claims about the construction, behaviour of, or outcomes associated with a completed system.

This paper draws upon the areas of ITS research, expert systems design, computer-based instruction, education and psychology to identify techniques for the formative and summative evaluation of ITS. Evaluation techniques are discussed in terms of the motivations for using them, situations for which they are particularly suitable, and their strengths and weaknesses. Techniques are illustrated with reference to actual studies whenever possible.

Greer, J. E., Mark, M.A., & McCalla, G.I. "Incorporating granularity-based recognition into SCENT." Proceedings of the Fourth International Conference on Artificial Intelligence and Education, Amsterdam. May, 1989, 107-115. [Abstract]
Understanding the strategies that underlie students' problem-solving behaviour in complex domains requires significant expertise and large amounts of domain specific knowledge. In this paper, a computational model for granularity-based recognition of strategies in LISP programming is described in accordance with theoretical principles governing granularity in knowledge representation. The implemented system recognizes students' programming strategies at various levels of abstraction and aggregation and provides an overall diagnosis. The design of the system guarantees that full recognition will occur at some grain size and that partial recognition at finer grain sizes is probable. The combination of these two types of recognition provides a more robust understanding of students' strategies. Granularity-based strategy recognition seems to be an appropriate model for diagnosis in the SCENT advisor.

Coulman, R., Bhuiyan, S., Mark, M., Prasad, B., Schweigardt, M., Greer, J., & McCalla, G. "The Circuits Intelligent Tutoring System." Proceedings of the Sixth Canadian Symposium on Instructional Technology, Halifax, Canada, May 1989.

McCalla, G., Greer, J., & Members of the SCENT Research Team. "Intelligent advising in problem-solving domains: The SCENT-3 architecture." Proceedings of Intelligent Tutoring Systems, ITS-88. Montreal, June 1-3, 1988. [Abstract]
The SCENT project has focussed on developing an intelligent advising environment for students learning to program in Lisp. This paper presents a status report on SCENT, a summary of recent research contributions, and the design of the new SCENT-3 system. SCENT-3 is an architecture for a full scale student advising system. Work on SCENT-3 is progressing concurrently on many fronts with new research contributions in dynamic planning and blackboard control, student modelling, strategy judging and diagnosis, and program analysis. The SCENT-3 architecture has been designed to achieve the goal of creating an intelligent advising system for higher order problem-solving activities with minimal domain dependence. Much more research remains to be done to fully prove out SCENT-3, but the investigations carried out so far are promising.

Walls, E.K., Mark, M.A., & Wishart, T.B. "Suppression of haloperidol and apomorphine effects in streptozotocin-induced diabetic rats." Proceedings of the Canadian Federation of Biological Studies, 1984, 27, 154.